starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-09-09 00:04:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into glm/onboarding-cloud

Browse Source 
# Conflicts:
#	AGENTS.md
This commit is contained in:
Guilhem Lemouel
2026-09-08 10:46:38 +02:00
parent 38762db6cb 33f9828c3e
commit 1745573ec6
197 changed files with 12446 additions and 1441 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/local-review-codex/SKILL.md
+4 -3
View File
@@ -1,6 +1,6 @@
---
name: local-review-codex
description: Run the CI Codex PR review locally against this branch's unpushed work (committed + uncommitted) before pushing. Same policy, model, and reasoning effort as the codex-pr-review GitHub action.
description: Run the CI Codex PR review locally against this branch's unpushed work (committed + uncommitted) before pushing. Same policy and reasoning effort as the codex-pr-review GitHub action, on a newer model.
---
# Local Codex Review (pre-push)
@@ -11,17 +11,18 @@ before the PR exists. Use this before `git push` on a non-trivial change.
**Correspondence with CI** — identical:
- Policy: `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test coverage).
- Model: `gpt-5.6-sol`, `model_reasoning_effort="xhigh"`.
- Reasoning effort: `model_reasoning_effort="xhigh"`.
- Output: markdown starting with `## Codex Review`, findings tagged P0 / P1 / P2 with file:line.
**Differences from CI** — local-only:
- Model is `gpt-6-astra`; CI stays on `gpt-5.6-sol`. Not an oversight to reconcile: `gpt-6-astra` is confirmed on the ChatGPT auth `codex login` uses locally, while CI authenticates with `OPENAI_API_KEY` (`codex-pr-review.yml` prefers it over `CODEX_AUTH_JSON`) and that tier is unverified for the model. Move CI once API access is confirmed, or once CI switches to `CODEX_AUTH_JSON`.
- Scope is the current branch vs `main` at the merge-base, **including uncommitted changes** (CI reviews a pushed PR diff).
- Sandbox is `read-only` (CI uses `danger-full-access` on an ephemeral runner). Codex reads the diff and files but cannot modify your working tree.
- Fresh context is inherent: `codex exec` is a separate cold process, so it does not anchor on the current chat session — the same reason `local-review` insists on a subagent.
## Prerequisites
- `codex` CLI **>= 0.144.1** installed and authed (`codex login` or `OPENAI_API_KEY`). Older CLIs reject `gpt-5.6-sol` with "requires a newer version of Codex". Upgrade with `npm install --global @openai/codex@0.144.1` (may need `sudo` for a global install). Keep this in sync with the pin in `.github/workflows/codex-pr-review.yml`.
- `codex` CLI **>= 0.153.4** installed and authed via `codex login` (an `OPENAI_API_KEY` in the environment takes priority and may not reach `gpt-6-astra` — see the model note above). Older CLIs reject the model with "requires a newer version of Codex"; `run.sh` checks the version up front. Upgrade with `npm install --global @openai/codex@0.153.4` (may need `sudo` for a global install). This matches the pin in `.github/workflows/codex-pr-review.yml` — the CLI version is the same on both sides, only the model differs.
- `git fetch` the base ref if it's stale, so the merge-base is accurate.
## Run
.agents/skills/local-review-codex/run.sh
+27 -4
View File
@@ -1,21 +1,44 @@
#!/usr/bin/env bash
# Local Codex review — mirrors the .github/workflows/codex-pr-review.yml CI job,
# but scoped to this branch's unpushed work (committed + uncommitted) so you can
# review before pushing. Same policy (REVIEW.md), same model (gpt-5.6-sol) and
# reasoning effort (xhigh) as CI. Runs read-only: Codex cannot modify your tree.
# review before pushing. Same policy (REVIEW.md) and reasoning effort (xhigh) as CI.
#
# The model deliberately differs from CI: gpt-6-astra is confirmed available on the
# ChatGPT auth `codex login` uses here, but CI authenticates with OPENAI_API_KEY and
# that tier is unverified for it, so codex-pr-review.yml stays on gpt-5.6-sol.
#
# Usage: run.sh [BASE_REF] (BASE_REF defaults to "main")
set -euo pipefail
MODEL="gpt-6-astra"
CODEX_MIN="0.153.4"
BASE_REF="${1:-main}"
REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT"
if ! command -v codex >/dev/null 2>&1; then
echo "codex CLI not found. Install with: npm install --global @openai/codex@0.144.1" >&2
echo "codex CLI not found. Install with: npm install --global @openai/codex@$CODEX_MIN" >&2
exit 1
fi
# Older CLIs reject the model with an error that never names the CLI version as the
# cause, so check it up front rather than letting the exec fail opaquely. The `|| true`
# keeps an unrecognised --version format from aborting under `set -e`: an unparseable
# version means "cannot tell", which must fall through to the exec, not kill the review.
CODEX_VER="$(codex --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)"
if [ -n "$CODEX_VER" ] && [ "$(printf '%s\n%s\n' "$CODEX_MIN" "$CODEX_VER" | sort -V | head -1)" != "$CODEX_MIN" ]; then
echo "codex $CODEX_VER is too old for $MODEL (need >= $CODEX_MIN). Upgrade with: npm install --global @openai/codex@$CODEX_MIN" >&2
exit 1
fi
# codex prefers OPENAI_API_KEY over the ChatGPT credentials `codex login` stores, and
# that tier is not confirmed for $MODEL — the resulting failure names the model, not the
# auth that selected it.
if [ -n "${OPENAI_API_KEY:-}" ]; then
echo "warning: OPENAI_API_KEY is set and takes priority over 'codex login' credentials; $MODEL may be unavailable on that tier." >&2
fi
# Resolve the base to a concrete commit, preferring a local ref but falling back to
# the remote-tracking ref — checkouts (CI, single-branch clones) often have only
# origin/main, not a local main.
@@ -80,7 +103,7 @@ EOF
codex exec \
-C "$REPO_ROOT" \
-m gpt-5.6-sol \
-m "$MODEL" \
-c 'model_reasoning_effort="xhigh"' \
-s read-only \
-o "$OUT" \
.github/workflows/codex-pr-review.yml
+1 -1
View File
@@ -219,7 +219,7 @@ jobs:
- name: Install Codex CLI
if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
run: npm install --global @openai/codex@0.144.1
run: npm install --global @openai/codex@0.153.4
- name: Configure Codex auth
if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
.release-please-manifest.json
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "1.804.0"
".": "1.805.0"
}
AGENTS.md
+1 -1
View File
@@ -36,7 +36,7 @@ Open-source platform for internal tools, workflows, API integrations, background
- **Backend patterns**: use the `rust-backend` skill when writing Rust code
- **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
- **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi. For a Codex-driven pass that mirrors the `codex-pr-review` GitHub action against your unpushed work (committed + uncommitted) before you push, use `/local-review-codex` (`.agents/skills/local-review-codex/`) — same `REVIEW.md` policy, `gpt-5.6-sol`, `xhigh` reasoning; requires the `codex` CLI >= 0.144.1.
- **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi. For a Codex-driven pass that mirrors the `codex-pr-review` GitHub action against your unpushed work (committed + uncommitted) before you push, use `/local-review-codex` (`.agents/skills/local-review-codex/`) — same `REVIEW.md` policy and `xhigh` reasoning, on `gpt-6-astra` rather than the action's `gpt-5.6-sol`; requires the `codex` CLI >= 0.153.4.
- **Domain guides**: `.claude/skills/native-trigger/`
- **Brand/UI guidelines**: `frontend/brand-guidelines.md`
- **Domain vocabulary**: `CONTEXT.md` — the words this codebase uses for its own concepts (step, step setting, trigger step, …). Name things the way it does.
CHANGELOG.md
+25
View File
@@ -1,5 +1,30 @@
# Changelog
## [1.805.0](https://github.com/windmill-labs/windmill/compare/v1.804.0...v1.805.0) (2026-09-07)
### Features
* **git-sync:** sync extra_perms for variables ([#11004](https://github.com/windmill-labs/windmill/issues/11004)) ([ee9e550](https://github.com/windmill-labs/windmill/commit/ee9e550a484fda286eeab43b7db5f314b8b2d0d9))
* go to referenced row from foreign-keyed cells in the database manager ([#10998](https://github.com/windmill-labs/windmill/issues/10998)) ([e2b63d1](https://github.com/windmill-labs/windmill/commit/e2b63d177ae4e5c980cb5da34154540c90771b63))
* let `// materialize` declare a `dbt://` warehouse-relation write ([#10978](https://github.com/windmill-labs/windmill/issues/10978)) ([c6e0302](https://github.com/windmill-labs/windmill/commit/c6e0302d7c1c60147f19d55a3923be8b1aa99c9c))
* report resource type picks to the hub and rank pickers by popularity ([#10982](https://github.com/windmill-labs/windmill/issues/10982)) ([48a5615](https://github.com/windmill-labs/windmill/commit/48a56158c135c3b13a02f73b7b8438bc691f85b4))
* run a linked AI agent's draft when testing a flow, and offer to deploy it ([#10993](https://github.com/windmill-labs/windmill/issues/10993)) ([7feaf61](https://github.com/windmill-labs/windmill/commit/7feaf619cf0ec021d66be14ef535cf2149bec58a))
* show the new-tab icon on a chat path pill while the modifier is held ([#10976](https://github.com/windmill-labs/windmill/issues/10976)) ([5da4ea4](https://github.com/windmill-labs/windmill/commit/5da4ea43fbd01e43aa14e75dc597d7ce5d8797ab))
### Bug Fixes
* **cli:** keep permissioned_as on single-item push, as sync push does ([#11000](https://github.com/windmill-labs/windmill/issues/11000)) ([5f3f99b](https://github.com/windmill-labs/windmill/commit/5f3f99ba6915b7c5df663a30b35f4cd02050e728))
* **cli:** say which workspace id is targeted, and when wmill.yaml is bypassed ([#11006](https://github.com/windmill-labs/windmill/issues/11006)) ([7643e9b](https://github.com/windmill-labs/windmill/commit/7643e9bd77c56f72596b8dca50801baf58984198))
* **frontend:** no phantom draft when opening a CLI-pushed script ([#10997](https://github.com/windmill-labs/windmill/issues/10997)) ([1be390a](https://github.com/windmill-labs/windmill/commit/1be390aa878e15a58f530f3a878e8f9caeb89c43))
* **frontend:** stop hover flicker on asset nodes shared with an overflow popover ([#10996](https://github.com/windmill-labs/windmill/issues/10996)) ([519a5c8](https://github.com/windmill-labs/windmill/commit/519a5c8bc70b44a7417e83c26c7b9c58b2c4fb9c))
* let a draft-only schedule, trigger or resource be deleted ([#11010](https://github.com/windmill-labs/windmill/issues/11010)) ([8d0f475](https://github.com/windmill-labs/windmill/commit/8d0f4754e4e0c78696ee0c97ff2de3016ece3bac))
* point the app viewer's edit button at the editor for the app's kind ([#11009](https://github.com/windmill-labs/windmill/issues/11009)) ([8f553ea](https://github.com/windmill-labs/windmill/commit/8f553eab353103fd8a28a00532e1766f133590de))
* seed runs page filter defaults through the url so they survive sync ([#11005](https://github.com/windmill-labs/windmill/issues/11005)) ([f381acd](https://github.com/windmill-labs/windmill/commit/f381acdb37f66f5e272bc37938e69f734987d53f))
* stop an untouched item's form from saving a draft nobody wrote ([#10964](https://github.com/windmill-labs/windmill/issues/10964)) ([c3f7f8a](https://github.com/windmill-labs/windmill/commit/c3f7f8a45830fb548aa628ebf6e2b6c95c6de67f))
* write and read python job files as utf-8, not the platform locale ([#10994](https://github.com/windmill-labs/windmill/issues/10994)) ([670404f](https://github.com/windmill-labs/windmill/commit/670404ffe27fedc3858b46b0c6b3312fbe175e13))
## [1.804.0](https://github.com/windmill-labs/windmill/compare/v1.803.0...v1.804.0) (2026-09-05)
backend/.sqlx/query-06c1a79bfc24b17acbd79411295c718161d95f7ad65a26525c5f43241267608d.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, tags)\n VALUES ($1, $2, $3, $4, 'model.p.stock', 'model', 'stock',\n 'u/a/wh/analytics/stock', '{}'),\n ($1, $2, $3, $4, 'model.p.stock_daily', 'model', 'stock_daily',\n 'u/a/wh/analytics/stock_daily', '{}')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Uuid"
]
},
"nullable": []
},
"hash": "06c1a79bfc24b17acbd79411295c718161d95f7ad65a26525c5f43241267608d"
}
backend/.sqlx/query-1008ed150f30b56baf17b3c6e6bfc8ee144b14d564a3e589a60678b3e68effbf.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, parent_column, child_unique_id,\n child_column, lineage_kind)\n VALUES ($1, $2, $3, $4, 'model.p.stock', 'sku', 'model.p.stock_daily', 'sku', 'copy')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Uuid"
]
},
"nullable": []
},
"hash": "1008ed150f30b56baf17b3c6e6bfc8ee144b14d564a3e589a60678b3e68effbf"
}
backend/.sqlx/query-178cb9cd8dfda66e878cd924f64a58ec742d7079703d417c886c8cf0a0e767a6.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM dbt_column_edge\n WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": []
},
"hash": "178cb9cd8dfda66e878cd924f64a58ec742d7079703d417c886c8cf0a0e767a6"
}
backend/.sqlx/query-1d8effff5dd1e4a177efed7366e84411c125cbfada861992e3d6032de635bad6.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings\n SET dbt_warehouses = '{\"main\": {\"resource_path\": \"u/test-user/wh\"}}'::jsonb\n WHERE workspace_id = 'test-workspace'",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "1d8effff5dd1e4a177efed7366e84411c125cbfada861992e3d6032de635bad6"
}
backend/.sqlx/query-1f7608defb5748db687145750520cb1059f63c1ea51db3471c213018b2983704.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind)\n VALUES ('test-workspace', 'main/analytics/marts', 'dbt', 'w', 'u/test-user/project',\n 'script')",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "1f7608defb5748db687145750520cb1059f63c1ea51db3471c213018b2983704"
}
backend/.sqlx/query-210441eb7bee09afd27a927e0e44ff3bc1143dab285e51a9b54b2c2f240b9a8c.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, parent_column, child_unique_id,\n child_column, lineage_kind)\n VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000',\n 'model.p.raw_orders', 'id', 'model.p.orders', 'order_id', 'copy'),\n ($1, $2, $3, '00000000-0000-0000-0000-000000000000',\n 'model.p.raw_orders', 'status', 'model.p.orders', 'order_id', 'scan')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8"
]
},
"nullable": []
},
"hash": "210441eb7bee09afd27a927e0e44ff3bc1143dab285e51a9b54b2c2f240b9a8c"
}
backend/.sqlx/query-23151591ed03ea6d2b017e66f54115b2241215d660d25df7615c8b12d704e92c.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, tags)\n VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'source.q.' || $4,\n 'source', $4, $5, '{}'),\n ($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'model.q.' || $6,\n 'model', $6, $7, '{}')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Text",
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "23151591ed03ea6d2b017e66f54115b2241215d660d25df7615c8b12d704e92c"
}
backend/.sqlx/query-2e562fb35cfa71702e5fdbc505d2fa6f3114a151b66b0a116b1d98dd84115188.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, materialized, materialize_strategy, unique_key,\n tags, description, test_kind, test_column, test_args, severity, attached_node,\n columns, column_schema, freshness, raw_code, original_file_path, ingested_at)\n SELECT $2, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, materialized, materialize_strategy, unique_key,\n tags, description, test_kind, test_column, test_args, severity, attached_node,\n columns, column_schema, freshness, raw_code, original_file_path, ingested_at\n FROM dbt_node\n WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "2e562fb35cfa71702e5fdbc505d2fa6f3114a151b66b0a116b1d98dd84115188"
}
backend/.sqlx/query-33aa15957f56281947963cd696f7f518433741305d891d99059184eaf39fa2db.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT path FROM asset WHERE workspace_id = 'test-workspace' AND kind = 'dbt' AND usage_path = 'u/test-user/ingest' AND usage_access_type = 'w'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "33aa15957f56281947963cd696f7f518433741305d891d99059184eaf39fa2db"
}
backend/.sqlx/query-3b7858e47e4e3f31f861e114c7ff26b467185edcfde151f22c6e903d15722ad4.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, parent_column, child_unique_id, child_column, lineage_kind,\n ingested_at)\n SELECT $2, script_path, script_hash, job_id, parent_unique_id, parent_column,\n child_unique_id, child_column, lineage_kind, ingested_at\n FROM dbt_column_edge\n WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "3b7858e47e4e3f31f861e114c7ff26b467185edcfde151f22c6e903d15722ad4"
}
backend/.sqlx/query-3fc12582cbae2ebc77ecfcaeed4bce43c749a44cbc4e7b8f719bf2a598ce57f3.json
Generated
+21
View File
@@ -0,0 +1,21 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_environment_state (workspace_id, script_path, environment, job_id,\n manifest, manifest_key, run_results,\n run_results_key, updated_at)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now())\n ON CONFLICT (workspace_id, script_path, environment) DO UPDATE SET\n job_id = EXCLUDED.job_id, manifest = EXCLUDED.manifest,\n manifest_key = EXCLUDED.manifest_key, run_results = EXCLUDED.run_results,\n run_results_key = EXCLUDED.run_results_key, updated_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Text",
"Uuid",
"Text",
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "3fc12582cbae2ebc77ecfcaeed4bce43c749a44cbc4e7b8f719bf2a598ce57f3"
}
backend/.sqlx/query-446909206f289fc3f9201a6ad025cc73eef1e3d01825031ba263881b7cfd5ed4.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM script WHERE workspace_id = $1 AND hash = 1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "446909206f289fc3f9201a6ad025cc73eef1e3d01825031ba263881b7cfd5ed4"
}
backend/.sqlx/query-49d905cae6ba42a3df62bad9385e613381cb4dcf7e851197150a3771ab43c99e.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "WITH producer AS (\n SELECT 'dbt://' || a.path AS trigger_ref, a.usage_path, s.language\n FROM asset a\n JOIN script s ON s.workspace_id = a.workspace_id AND s.path = a.usage_path\n AND s.archived = false AND s.deleted = false\n WHERE a.workspace_id = $1 AND a.kind = 'dbt'\n AND a.usage_kind = 'script' AND a.usage_access_type IN ('w', 'rw')\n AND 'dbt://' || a.path = ANY($2)\n )\n SELECT DISTINCT st.trigger_ref || ' → ' || st.runnable_path AS \"edge!\"\n FROM script_trigger st\n WHERE st.workspace_id = $1 AND st.trigger_kind = 'asset'\n AND st.trigger_ref = ANY($2)\n AND EXISTS (SELECT 1 FROM producer p\n WHERE p.trigger_ref = st.trigger_ref\n AND p.usage_path <> st.runnable_path\n AND p.language = 'dbt')\n AND NOT EXISTS (SELECT 1 FROM producer p\n WHERE p.trigger_ref = st.trigger_ref\n AND p.usage_path <> st.runnable_path\n AND p.language <> 'dbt')\n ORDER BY 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "edge!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
null
]
},
"hash": "49d905cae6ba42a3df62bad9385e613381cb4dcf7e851197150a3771ab43c99e"
}
backend/.sqlx/query-4ce90ff818e5058a7e31da24abd938eb135d6eb9714ca115359221d7c02861f0.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_environment_state (workspace_id, script_path, environment, job_id,\n manifest)\n SELECT $1::varchar, $2::varchar, 'main||analytics|wh'::text, $3::uuid, '{}'::text\n WHERE EXISTS (SELECT 1 FROM script\n WHERE workspace_id = $1 AND path = $2\n AND deleted = false AND archived = false\n AND language = 'dbt'\n AND (hash = $4 OR $4 = ANY(parent_hashes)))\n ON CONFLICT (workspace_id, script_path, environment) DO UPDATE SET job_id = EXCLUDED.job_id",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Uuid",
"Int8"
]
},
"nullable": []
},
"hash": "4ce90ff818e5058a7e31da24abd938eb135d6eb9714ca115359221d7c02861f0"
}
backend/.sqlx/query-9309262d8a37827e5ab0b3534d1595cb6d94118432fd1770f1f89f7cb52a4f12.json → backend/.sqlx/query-4efca4ff8de0bd297de9eaf0fb7f86f320e4ff58aac8794248925c084707f9b5.json
Generated
+11 -5
View File
File diff suppressed because one or more lines are too long
backend/.sqlx/query-55af8c19888ddc222a0ef2db04fac8ee7e664e64a51972327875b1138f80db5d.json
Generated
+50
View File
@@ -0,0 +1,50 @@
{
"db_name": "PostgreSQL",
"query": "SELECT p.asset_path AS \"from_path!\", e.parent_column AS \"from_column!\",\n c.asset_path AS \"to_path!\", e.child_column AS \"to_column!\",\n e.lineage_kind AS \"kind!\"\n FROM unnest($2::text[], $3::bigint[], $4::uuid[])\n AS o(script_path, script_hash, job_id)\n JOIN dbt_column_edge e ON e.workspace_id = $1\n AND e.script_path = o.script_path\n AND e.job_id = o.job_id\n -- `=` still, with the NULL-to-NULL case\n -- spelled out and gated on the pin: a\n -- version-less row's hash is NULL on both\n -- sides, which `=` never matches, but\n -- `IS NOT DISTINCT FROM` would cost the\n -- equality its index bound everywhere else.\n AND (e.script_hash = o.script_hash\n OR ($5::text IS NOT NULL\n AND o.script_hash IS NULL\n AND e.script_hash IS NULL))\n JOIN dbt_node p ON p.workspace_id = e.workspace_id\n AND p.script_path = e.script_path\n AND p.script_hash IS NOT DISTINCT FROM e.script_hash\n AND p.job_id = e.job_id\n AND p.unique_id = e.parent_unique_id\n JOIN dbt_node c ON c.workspace_id = e.workspace_id\n AND c.script_path = e.script_path\n AND c.script_hash IS NOT DISTINCT FROM e.script_hash\n AND c.job_id = e.job_id\n AND c.unique_id = e.child_unique_id\n WHERE e.lineage_kind IN ('copy', 'mod')\n AND p.asset_path IS NOT NULL AND c.asset_path IS NOT NULL",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "from_path!",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "from_column!",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "to_path!",
"type_info": "Text"
},
{
"ordinal": 3,
"name": "to_column!",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "kind!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"TextArray",
"Int8Array",
"UuidArray",
"Text"
]
},
"nullable": [
true,
false,
true,
false,
false
]
},
"hash": "55af8c19888ddc222a0ef2db04fac8ee7e664e64a51972327875b1138f80db5d"
}
backend/.sqlx/query-58231bdfb04fe73a8a41601fcc3c77cdae5377120441229490e468d47112819d.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms)\n VALUES ($1, $2, $2, '{}', '{}')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar"
]
},
"nullable": []
},
"hash": "58231bdfb04fe73a8a41601fcc3c77cdae5377120441229490e468d47112819d"
}
backend/.sqlx/query-58ec340c78af046a40296b15543beeedc63b4a941e5b17ab7bb5d3c259f05147.json
Generated
+24
View File
@@ -0,0 +1,24 @@
{
"db_name": "PostgreSQL",
"query": "SELECT 1 FROM script\n WHERE workspace_id = $1 AND path = $2\n AND deleted = false AND archived = false AND language = 'dbt'\n AND (hash = $3 OR $3 = ANY(parent_hashes))\n FOR SHARE",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
null
]
},
"hash": "58ec340c78af046a40296b15543beeedc63b4a941e5b17ab7bb5d3c259f05147"
}
backend/.sqlx/query-5c7260785ebcece2ddb04dc85d09b6a36c50d70e20c5a10310f75a39930db521.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_environment_state (workspace_id, script_path, environment, job_id,\n manifest)\n VALUES ($1, $2, 'main||analytics|wh', $3, '{}')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Uuid"
]
},
"nullable": []
},
"hash": "5c7260785ebcece2ddb04dc85d09b6a36c50d70e20c5a10310f75a39930db521"
}
backend/.sqlx/query-5cbff3d68b684f794bf43ea981716bd03da56192754610d9b34d69e567bd1f10.json
Generated
+18
View File
@@ -0,0 +1,18 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, parent_column, child_unique_id,\n child_column, lineage_kind)\n VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000',\n 'source.q.' || $4, 'k', 'model.q.' || $5, 'k', 'copy')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "5cbff3d68b684f794bf43ea981716bd03da56192754610d9b34d69e567bd1f10"
}
backend/.sqlx/query-6890e2be43ff8c653ce62f8cde1b9877190091923a2615d7d94bdf89a12e43c7.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO script (workspace_id, hash, path, summary, description, content, created_by,\n language)\n VALUES ('test-workspace', 1, 'u/test-user/project', '', '', '', 'test-user', 'dbt')",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "6890e2be43ff8c653ce62f8cde1b9877190091923a2615d7d94bdf89a12e43c7"
}
backend/.sqlx/query-69c9ac0e5761c00a9e9a7ff96431e5ebfce96786448fb5273d49f13978df65c0.json
Generated
+28
View File
@@ -0,0 +1,28 @@
{
"db_name": "PostgreSQL",
"query": "SELECT resource_type, count(*) as \"count!\" FROM resource WHERE workspace_id = $1 GROUP BY resource_type",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "resource_type",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "count!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "69c9ac0e5761c00a9e9a7ff96431e5ebfce96786448fb5273d49f13978df65c0"
}
backend/.sqlx/query-7334351af91382d5619ff437a253c2d1acc3cd857e089e8c6972fcd276094837.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, parent_column, child_unique_id,\n child_column, lineage_kind)\n SELECT $1, $2, $3, '00000000-0000-0000-0000-000000000000',\n 'model.p.raw_orders', 'c' || i, 'model.p.orders', 'c' || i, 'copy'\n FROM generate_series(1, 6000) i",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8"
]
},
"nullable": []
},
"hash": "7334351af91382d5619ff437a253c2d1acc3cd857e089e8c6972fcd276094837"
}
backend/.sqlx/query-7f421bdf3dc4d47b36457af703ce69ef1e0784e9f25ada27e1b0f5cc0944e9ba.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE dbt_environment_state SET script_path = $3\n WHERE workspace_id = $1 AND script_path = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "7f421bdf3dc4d47b36457af703ce69ef1e0784e9f25ada27e1b0f5cc0944e9ba"
}
backend/.sqlx/query-8165a447f458d62f7bafb9827d87d192f5e926b932c3dd0e1bef4822cf722630.json
Generated
+48
View File
@@ -0,0 +1,48 @@
{
"db_name": "PostgreSQL",
"query": "SELECT job_id, manifest, manifest_key, run_results, run_results_key\n FROM dbt_environment_state\n WHERE workspace_id = $1 AND script_path = $2 AND environment = $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "job_id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "manifest",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "manifest_key",
"type_info": "Text"
},
{
"ordinal": 3,
"name": "run_results",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "run_results_key",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": [
false,
true,
true,
true,
true
]
},
"hash": "8165a447f458d62f7bafb9827d87d192f5e926b932c3dd0e1bef4822cf722630"
}
backend/.sqlx/query-895a4feb0b3cc01ad711ce0eff8eef4f68cc3ecaf680c55a94175f6dfc2e2912.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, parent_column, child_unique_id,\n child_column, lineage_kind)\n VALUES ($1, $2, NULL, $3, 'model.p.draft_src', 'raw', 'model.p.draft', 'clean', 'mod')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Uuid"
]
},
"nullable": []
},
"hash": "895a4feb0b3cc01ad711ce0eff8eef4f68cc3ecaf680c55a94175f6dfc2e2912"
}
backend/.sqlx/query-94c0aae349172b9295b81b9e61e2f8f0ca36920ab66e8922815157a0c147bc47.json
Generated
-15
View File
@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, materialized, materialize_strategy, unique_key,\n tags, description, test_kind, test_column, test_args, severity, attached_node,\n columns, freshness, raw_code, original_file_path, ingested_at)\n SELECT $2, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, materialized, materialize_strategy, unique_key,\n tags, description, test_kind, test_column, test_args, severity, attached_node,\n columns, freshness, raw_code, original_file_path, ingested_at\n FROM dbt_node\n WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "94c0aae349172b9295b81b9e61e2f8f0ca36920ab66e8922815157a0c147bc47"
}
backend/.sqlx/query-97fd4f9939d96176f4cfc4a07e69466279aad62f33c57f392835e8265ed40f26.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT count(*) FROM dbt_column_edge WHERE workspace_id = $1 AND script_hash = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Int8"
]
},
"nullable": [
null
]
},
"hash": "97fd4f9939d96176f4cfc4a07e69466279aad62f33c57f392835e8265ed40f26"
}
backend/.sqlx/query-9f0979110f86dffc7452d80ea0410ba39025d9c9787a71a9456241cf8a9b9d82.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, tags)\n VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'model.p.raw_orders',\n 'model', 'raw_orders', 'u/a/wh/analytics/raw_orders', '{}'),\n ($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'model.p.orders',\n 'model', 'orders', 'u/a/wh/analytics/orders', '{}')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8"
]
},
"nullable": []
},
"hash": "9f0979110f86dffc7452d80ea0410ba39025d9c9787a71a9456241cf8a9b9d82"
}
backend/.sqlx/query-a06e1d9f6f95e4c4c2b98310ebddcc9d963cc033582bf2e945e8bf3a301b4247.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT pg_advisory_xact_lock($1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "pg_advisory_xact_lock",
"type_info": "Void"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
null
]
},
"hash": "a06e1d9f6f95e4c4c2b98310ebddcc9d963cc033582bf2e945e8bf3a301b4247"
}
backend/.sqlx/query-ae4c0e8aeeef90d08a9b9c919b881b0a03e44beea80da732a4ad7166cf4c06cf.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT trigger_ref FROM script_trigger WHERE workspace_id = 'test-workspace' AND runnable_path = 'u/test-user/consumer' AND trigger_kind = 'asset'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "trigger_ref",
"type_info": "Text"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "ae4c0e8aeeef90d08a9b9c919b881b0a03e44beea80da732a4ad7166cf4c06cf"
}
backend/.sqlx/query-b8161f6481460bed1c985bedc7c638a42f03599ce5eb872cf601f1da2e76b946.json
Generated
+63
View File
@@ -0,0 +1,63 @@
{
"db_name": "PostgreSQL",
"query": "SELECT s.path AS \"path!\", s.language AS \"language!: ScriptLang\"\n FROM asset a\n JOIN script s ON s.workspace_id = a.workspace_id AND s.path = a.usage_path\n AND s.archived = false AND s.deleted = false\n WHERE a.workspace_id = $1 AND a.kind = 'dbt' AND a.path = $2\n AND a.usage_kind = 'script' AND a.usage_access_type IN ('w', 'rw')\n AND a.usage_path <> ALL($3)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "language!: ScriptLang",
"type_info": {
"Custom": {
"name": "script_lang",
"kind": {
"Enum": [
"python3",
"deno",
"go",
"bash",
"postgresql",
"nativets",
"bun",
"mysql",
"bigquery",
"snowflake",
"graphql",
"powershell",
"mssql",
"php",
"bunnative",
"rust",
"ansible",
"csharp",
"oracledb",
"nu",
"java",
"duckdb",
"ruby",
"rlang",
"dbt"
]
}
}
}
}
],
"parameters": {
"Left": [
"Text",
"Text",
"TextArray"
]
},
"nullable": [
false,
false
]
},
"hash": "b8161f6481460bed1c985bedc7c638a42f03599ce5eb872cf601f1da2e76b946"
}
backend/.sqlx/query-b820bf7b0a93476fb7762e8ff2819b2c2b7df2788c9eb0115abc60ea7b733d64.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM dbt_environment_state WHERE workspace_id = $1 AND script_path = $2\n AND NOT EXISTS (SELECT 1 FROM script\n WHERE workspace_id = $1 AND path = $2\n AND deleted = false AND archived = false)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "b820bf7b0a93476fb7762e8ff2819b2c2b7df2788c9eb0115abc60ea7b733d64"
}
backend/.sqlx/query-b9c200365ea426ebe01b2e67c378ca41990e4c5d49b5baaee41acfe409bb1c24.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, tags)\n VALUES ($1, $2, $3, $4, 'model.p.raw_orders', 'model', 'raw_orders',\n 'u/a/wh/analytics/raw_orders', '{}')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Uuid"
]
},
"nullable": []
},
"hash": "b9c200365ea426ebe01b2e67c378ca41990e4c5d49b5baaee41acfe409bb1c24"
}
backend/.sqlx/query-c793b147014cb1b6e138aed1c2eb8b93b4a8383d8cf835a7d350056565669e7f.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, raw_code, tags)\n VALUES ($1, $2, NULL, $3, 'model.p.draft', 'model', 'draft',\n 'u/a/wh/analytics/draft', 'select 3', '{}'),\n ($1, $2, NULL, $3, 'model.p.draft_src', 'model', 'draft_src',\n 'u/a/wh/analytics/draft_src', 'select 4', '{}')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Uuid"
]
},
"nullable": []
},
"hash": "c793b147014cb1b6e138aed1c2eb8b93b4a8383d8cf835a7d350056565669e7f"
}
backend/.sqlx/query-d2c02f1c7f4555fa849f5cd86169eb8e43d384084b68dc1bbb3b961206ff4b6c.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM dbt_environment_state WHERE workspace_id = $1 AND script_path = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "d2c02f1c7f4555fa849f5cd86169eb8e43d384084b68dc1bbb3b961206ff4b6c"
}
backend/.sqlx/query-d8b7241518ce0822288fb123b48205427cc50af031a84d334f65f870d86302ac.json
Generated
+48
View File
@@ -0,0 +1,48 @@
{
"db_name": "PostgreSQL",
"query": "SELECT job_id, manifest, manifest_key, run_results, run_results_key\n FROM dbt_environment_state\n WHERE workspace_id = $1 AND script_path = $2 AND environment = $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "job_id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "manifest",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "manifest_key",
"type_info": "Text"
},
{
"ordinal": 3,
"name": "run_results",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "run_results_key",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": [
false,
true,
true,
true,
true
]
},
"hash": "d8b7241518ce0822288fb123b48205427cc50af031a84d334f65f870d86302ac"
}
backend/.sqlx/query-e37b74f77cfa8769aee4d95155a3b3b6856b44d49c0712f08e307f950f210570.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind)\n VALUES ('test-workspace', 'main/analytics/orders', 'dbt', 'w', 'u/test-user/project',\n 'script')",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "e37b74f77cfa8769aee4d95155a3b3b6856b44d49c0712f08e307f950f210570"
}
backend/.sqlx/query-e5417b36319dafd5fcb2e42d812f778b2d3cf8b0f13cf4620192604cc3824a77.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM dbt_column_edge\n WHERE job_id <> '00000000-0000-0000-0000-000000000000'\n AND ingested_at < now() - make_interval(days => $1)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int4"
]
},
"nullable": []
},
"hash": "e5417b36319dafd5fcb2e42d812f778b2d3cf8b0f13cf4620192604cc3824a77"
}
backend/.sqlx/query-eb0df8f3f1d66dd7dfc9aedc7945989fae585115a24f19946a4d58257c93bbe2.json
Generated
+41
View File
@@ -0,0 +1,41 @@
{
"db_name": "PostgreSQL",
"query": "SELECT DISTINCT n.script_path AS \"script_path!\", n.script_hash, n.job_id AS \"job_id!\"\n FROM dbt_node n\n WHERE n.workspace_id = $1 AND n.asset_path = ANY($2)\n -- The run's snapshot, or the deployed graph when that job stored\n -- none -- a build pins only if it wrote one.\n AND n.job_id = CASE WHEN $5::uuid IS NOT NULL AND EXISTS (\n SELECT 1 FROM dbt_graph_snapshot g\n WHERE g.workspace_id = $1 AND g.job_id = $5)\n THEN $5::uuid\n ELSE '00000000-0000-0000-0000-000000000000'::uuid END\n -- The gate, re-decided for every project the walk reaches. That\n -- is what resolving owners in a loop is for: being entitled to\n -- one project is not being entitled to the one that declares a\n -- relation it hands over.\n AND ( $6\n OR n.script_path = ANY($7)\n OR EXISTS ( SELECT 1 FROM unnest($8::text[]) AS pfx\n WHERE n.script_path = pfx\n OR left(n.script_path, length(pfx) + 1) = pfx || '/' ) )\n AND CASE\n -- Pinned: which version comes from a job this caller was\n -- already granted, so `script` does not decide THAT -- but\n -- it still decides whether the project may be read, the\n -- same second gate `script_visible` is on the graph. Being\n -- entitled to a run is not being entitled to the SQL\n -- behind it, and column lineage is that SQL's shape. A\n -- version-less row is exempt because it is an editor\n -- buffer, which has no `script` row to ask and reaches\n -- this only through the parse job that wrote it.\n --\n -- One project answers, so a pinned trace never crosses\n -- into another: neither does the graph it annotates.\n WHEN $4::text IS NOT NULL\n THEN n.script_path = $4 AND n.script_hash IS NOT DISTINCT FROM $3::bigint\n AND ($3::bigint IS NULL OR EXISTS (\n SELECT 1 FROM script sc\n WHERE sc.workspace_id = $1 AND sc.path = n.script_path\n AND sc.hash = $3))\n -- A named version: the deployed one an editor is drawing.\n -- `script` is read under RLS, so this is the visibility\n -- check as well as the existence one. A hash names one\n -- script row, so this arm answers for one project too —\n -- and deliberately: a pin says which stored graph is on\n -- screen, and another project's live graph is not it.\n WHEN $3::bigint IS NOT NULL\n THEN n.script_hash = $3 AND EXISTS (\n SELECT 1 FROM script sc\n WHERE sc.workspace_id = $1 AND sc.path = n.script_path\n AND sc.hash = $3)\n -- Otherwise the version deployed now: an older one's rows\n -- outlive it in `dbt_node` until the sweep, and describe a\n -- project that is no longer what runs. `language` narrows\n -- it the way the graph's own resolution does, so a path\n -- that has since become a script of another kind draws and\n -- explains the same version rather than disagreeing. Read\n -- under RLS, so a project the caller cannot see resolves\n -- to NULL and matches nothing.\n ELSE n.script_hash = (\n SELECT sc.hash FROM script sc\n WHERE sc.workspace_id = $1 AND sc.path = n.script_path\n AND sc.language = 'dbt'\n AND sc.deleted = false AND sc.archived = false\n ORDER BY sc.created_at DESC LIMIT 1)\n END",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "script_path!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "script_hash",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "job_id!",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Text",
"TextArray",
"Int8",
"Text",
"Uuid",
"Bool",
"TextArray",
"TextArray"
]
},
"nullable": [
false,
true,
false
]
},
"hash": "eb0df8f3f1d66dd7dfc9aedc7945989fae585115a24f19946a4d58257c93bbe2"
}
backend/.sqlx/query-f49cd9b5ea1e94d7cd10dcd1706cad62b5c1eff191f06905a3a78eb600e3bb4b.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM dbt_column_edge WHERE workspace_id = $1 AND script_path = $2\n AND script_hash = $3 AND job_id = $4",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Int8",
"Uuid"
]
},
"nullable": []
},
"hash": "f49cd9b5ea1e94d7cd10dcd1706cad62b5c1eff191f06905a3a78eb600e3bb4b"
}
backend/.sqlx/query-f5e061eed38d483980ee3691aea821dde2c093bb926b77ec3f99a1c0d9283a14.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT count(*) FROM dbt_environment_state WHERE workspace_id = $1 AND script_path = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "f5e061eed38d483980ee3691aea821dde2c093bb926b77ec3f99a1c0d9283a14"
}
backend/.sqlx/query-f6de1512fa3e46883b32d56fd19ffe8aaf6dfe664c330ac8cea51a157a4fe52e.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, parent_column, child_unique_id,\n child_column, lineage_kind)\n VALUES ($1, $2, $3, $4, 'model.p.raw_orders', 'id', 'model.p.orders', 'order_id',\n 'copy')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Uuid"
]
},
"nullable": []
},
"hash": "f6de1512fa3e46883b32d56fd19ffe8aaf6dfe664c330ac8cea51a157a4fe52e"
}
backend/.sqlx/query-f963ea31d1744ff0d76ff86140f01cca0bcbc832f01eb3340050504b8b3875a2.json
Generated
+30
View File
@@ -0,0 +1,30 @@
{
"db_name": "PostgreSQL",
"query": "SELECT manifest_key, run_results_key FROM dbt_environment_state\n WHERE workspace_id = $1 AND script_path = $2 AND environment = $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "manifest_key",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "run_results_key",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": [
true,
true
]
},
"hash": "f963ea31d1744ff0d76ff86140f01cca0bcbc832f01eb3340050504b8b3875a2"
}
backend/.sqlx/query-ffde6e45841090440bc99e22788d313db5945faa0b93c127833e44d38665a392.json
Generated
+30
View File
@@ -0,0 +1,30 @@
{
"db_name": "PostgreSQL",
"query": "SELECT manifest_key, run_results_key FROM dbt_environment_state\n WHERE workspace_id = $1 AND script_path = $2 AND environment = $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "manifest_key",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "run_results_key",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": [
true,
true
]
},
"hash": "ffde6e45841090440bc99e22788d313db5945faa0b93c127833e44d38665a392"
}
backend/Cargo.lock
Generated
+102 -101
View File
@@ -2763,18 +2763,18 @@ dependencies = [
[[package]]
name = "crossbeam-channel"
version = "0.5.16"
version = "0.5.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d85363c37faeca707aef026efa9f3b34d077bce547e48f770770625c6013679e"
checksum = "98b0cc327b5bc766e7fda9c9260cc0fa81b43a8e240440422dff70788e3f9ef1"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-deque"
version = "0.8.7"
version = "0.8.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb"
checksum = "622f3fc73690be383c7214310406f28a90e6edeadc3cea882f9d71e495b9711a"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
@@ -2782,27 +2782,27 @@ dependencies = [
[[package]]
name = "crossbeam-epoch"
version = "0.9.20"
version = "0.9.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f"
checksum = "dc74980687109a3b14c72fd458107bf0baa1da1a1a805e178d15501ba9b86d9d"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-queue"
version = "0.3.13"
version = "0.3.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "803d13fb3b09d88be9f4dbc29062c66b19bf7170867ceb746d2a8689bf6c7a26"
checksum = "03e8bd762f7479489c70ed6c768ddca99d7296857de437a68dcb2a94365b3fae"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.22"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17"
checksum = "a31eee39dddec8330830986fcd7625edb5a24ec90ea038215273bbc3adb08ac6"
[[package]]
name = "crossterm_winapi"
@@ -6640,9 +6640,9 @@ dependencies = [
[[package]]
name = "io-uring"
version = "0.7.14"
version = "0.7.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d64d8ca234d152948ceaede1f419b6a83983a5ecccaac05fb337a809c96d3aa6"
checksum = "ed3bd0ecfbb87805f538bb7b32e5239ca0763890c623e349860ecba69469f2bb"
dependencies = [
"bitflags 2.13.1",
"cfg-if",
@@ -6664,9 +6664,9 @@ dependencies = [
[[package]]
name = "ipnet"
version = "2.12.1"
version = "2.12.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78"
checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]]
name = "ipnetwork"
@@ -9077,9 +9077,9 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "pest"
version = "2.9.0"
version = "2.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a07a60cc7a4d00c91f95c685609d1d2f79050e6804b70ebedd7650f0b839bcf"
checksum = "6d45aeb61b4bf818e12d4205f2466f8c4748f85f4fce0146d1c03d69d753f0ad"
dependencies = [
"memchr",
"ucd-trie",
@@ -9087,9 +9087,9 @@ dependencies = [
[[package]]
name = "pest_derive"
version = "2.9.0"
version = "2.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b3a83744a5c8455b8b3e0dc5031362780a347c878bdd11584d1a8984228cc88d"
checksum = "89cc5a242e25ed4e7704d0be240f2cfbe20a8c27e7e252d94835be93d92dc39f"
dependencies = [
"pest",
"pest_generator",
@@ -9097,9 +9097,9 @@ dependencies = [
[[package]]
name = "pest_generator"
version = "2.9.0"
version = "2.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0cd3451aa3de60d4b9a1e736885e4dea6b31617598026f12256ad566d63304a"
checksum = "7abf21475cc3820fe4b2ca2dc2142902f67a02189f3b5b3a229f4febc01a43e5"
dependencies = [
"pest",
"pest_meta",
@@ -9110,9 +9110,9 @@ dependencies = [
[[package]]
name = "pest_meta"
version = "2.9.0"
version = "2.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e04d3a0849e241d7dfce834c83b1c5edc8622009e8dd51a12ba1927c32f05496"
checksum = "adba4db388f687393c18c51348d44a41d870ca9df71a2c98172ea3035dc6936e"
dependencies = [
"pest",
]
@@ -14747,7 +14747,7 @@ dependencies = [
[[package]]
name = "windmill"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-nats",
@@ -14835,7 +14835,7 @@ dependencies = [
[[package]]
name = "windmill-ai"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"async-stream",
"async-trait",
@@ -14868,7 +14868,7 @@ dependencies = [
[[package]]
name = "windmill-alerting"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14881,7 +14881,7 @@ dependencies = [
[[package]]
name = "windmill-api"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"argon2",
@@ -15021,7 +15021,7 @@ dependencies = [
[[package]]
name = "windmill-api-agent-workers"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15044,7 +15044,7 @@ dependencies = [
[[package]]
name = "windmill-api-assets"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15061,7 +15061,7 @@ dependencies = [
[[package]]
name = "windmill-api-auth"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15087,7 +15087,7 @@ dependencies = [
[[package]]
name = "windmill-api-client"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"reqwest 0.12.28",
"serde",
@@ -15097,7 +15097,7 @@ dependencies = [
[[package]]
name = "windmill-api-configs"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15114,7 +15114,7 @@ dependencies = [
[[package]]
name = "windmill-api-debug"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"base64 0.22.1",
@@ -15136,7 +15136,7 @@ dependencies = [
[[package]]
name = "windmill-api-embeddings"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15159,7 +15159,7 @@ dependencies = [
[[package]]
name = "windmill-api-flow-conversations"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15175,7 +15175,7 @@ dependencies = [
[[package]]
name = "windmill-api-flows"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15197,7 +15197,7 @@ dependencies = [
[[package]]
name = "windmill-api-groups"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15218,7 +15218,7 @@ dependencies = [
[[package]]
name = "windmill-api-inputs"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15232,7 +15232,7 @@ dependencies = [
[[package]]
name = "windmill-api-integration-tests"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-nats",
@@ -15267,7 +15267,7 @@ dependencies = [
[[package]]
name = "windmill-api-jobs"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15292,7 +15292,7 @@ dependencies = [
[[package]]
name = "windmill-api-npm-proxy"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15320,7 +15320,7 @@ dependencies = [
[[package]]
name = "windmill-api-openapi"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15342,7 +15342,7 @@ dependencies = [
[[package]]
name = "windmill-api-schedule"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15362,7 +15362,7 @@ dependencies = [
[[package]]
name = "windmill-api-scripts"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15400,7 +15400,7 @@ dependencies = [
[[package]]
name = "windmill-api-settings"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15428,7 +15428,7 @@ dependencies = [
[[package]]
name = "windmill-api-sse"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"lazy_static",
"serde",
@@ -15440,7 +15440,7 @@ dependencies = [
[[package]]
name = "windmill-api-users"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"argon2",
"axum 0.8.9",
@@ -15464,7 +15464,7 @@ dependencies = [
[[package]]
name = "windmill-api-workers"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15478,7 +15478,7 @@ dependencies = [
[[package]]
name = "windmill-api-workspaces"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15513,7 +15513,7 @@ dependencies = [
[[package]]
name = "windmill-audit"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"chrono",
"lazy_static",
@@ -15527,7 +15527,7 @@ dependencies = [
[[package]]
name = "windmill-autoscaling"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15546,7 +15546,7 @@ dependencies = [
[[package]]
name = "windmill-common"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"aes-gcm",
"aho-corasick",
@@ -15652,7 +15652,7 @@ dependencies = [
[[package]]
name = "windmill-dep-map"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"chrono",
"futures",
@@ -15672,7 +15672,7 @@ dependencies = [
[[package]]
name = "windmill-git-sync"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"regex",
"serde",
@@ -15687,7 +15687,7 @@ dependencies = [
[[package]]
name = "windmill-indexer"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -15714,7 +15714,7 @@ dependencies = [
[[package]]
name = "windmill-jseval"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"futures",
@@ -15731,7 +15731,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"itertools 0.14.0",
"lazy_static",
@@ -15747,7 +15747,7 @@ dependencies = [
[[package]]
name = "windmill-mcp"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15768,7 +15768,7 @@ dependencies = [
[[package]]
name = "windmill-native-triggers"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15799,7 +15799,7 @@ dependencies = [
[[package]]
name = "windmill-oauth"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"arc-swap",
@@ -15824,7 +15824,7 @@ dependencies = [
[[package]]
name = "windmill-object-store"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-stream",
@@ -15858,7 +15858,7 @@ dependencies = [
[[package]]
name = "windmill-operator"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"futures",
@@ -15876,7 +15876,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"convert_case 0.6.0",
"serde",
@@ -15885,7 +15885,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15897,7 +15897,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde_json",
@@ -15909,7 +15909,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"gosyn",
@@ -15921,7 +15921,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15933,7 +15933,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde_json",
@@ -15945,7 +15945,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -15956,7 +15956,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -15967,7 +15967,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -15979,7 +15979,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -15990,7 +15990,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -16012,7 +16012,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde_json",
@@ -16024,7 +16024,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16038,7 +16038,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"convert_case 0.6.0",
@@ -16055,7 +16055,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16068,7 +16068,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde",
@@ -16080,7 +16080,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16098,7 +16098,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -16114,7 +16114,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -16130,7 +16130,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16144,7 +16144,7 @@ dependencies = [
[[package]]
name = "windmill-queue"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -16183,7 +16183,7 @@ dependencies = [
[[package]]
name = "windmill-runtime-nativets"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"const_format",
@@ -16223,7 +16223,7 @@ dependencies = [
[[package]]
name = "windmill-sql-datatype-parser-wasm"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"getrandom 0.3.4",
"wasm-bindgen",
@@ -16234,7 +16234,7 @@ dependencies = [
[[package]]
name = "windmill-store"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -16269,7 +16269,7 @@ dependencies = [
[[package]]
name = "windmill-test-utils"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16293,7 +16293,7 @@ dependencies = [
[[package]]
name = "windmill-trigger"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16326,7 +16326,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-amqp"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16353,7 +16353,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-azure"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16386,7 +16386,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-email"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16406,7 +16406,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-gcp"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16440,7 +16440,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-http"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16476,7 +16476,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-kafka"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16499,7 +16499,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-mqtt"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16523,7 +16523,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-nats"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-nats",
@@ -16547,7 +16547,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-postgres"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16582,7 +16582,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-sqs"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16610,7 +16610,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-websocket"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16635,7 +16635,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"bitflags 2.13.1",
@@ -16654,7 +16654,7 @@ dependencies = [
[[package]]
name = "windmill-worker"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-once-cell",
@@ -16699,6 +16699,7 @@ dependencies = [
"opentelemetry 0.30.0",
"opentelemetry-proto 0.30.0",
"oracle",
"parquet",
"pem 3.0.6",
"pep440_rs",
"postgres-native-tls 0.5.3",
@@ -16771,7 +16772,7 @@ dependencies = [
[[package]]
name = "windmill-worker-volumes"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"bytes",
"futures",
backend/Cargo.toml
+8 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.804.0"
version = "1.805.0"
authors.workspace = true
edition.workspace = true
@@ -88,7 +88,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.804.0"
version = "1.805.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -665,6 +665,12 @@ process-wrap = { version = "8.2.1", features = ["tokio1"] }
systemstat = "0.2.4"
datafusion = "47.0.0"
# The row API only: a dbt engine's parquet index is six string columns, so this
# needs no arrow and no writer. `parquet` is already in the tree with `arrow` for
# every shipped edition (`oss_core`), and cargo unifies the features there; this
# set is what a build WITHOUT object storage compiles. ZSTD is what the engine
# writes today, snap what parquet writers most often default to.
parquet = { version = "55.2.0", default-features = false, features = ["snap", "zstd"] }
object_store = { git = "https://github.com/apache/arrow-rs-object-store", rev = "36752c975d4f29e20b57c91f81a10872dcd48ae7", features = ["aws", "azure", "gcp"] }
openidconnect = { version = "4.0.0-rc.1" }
aws-config = "^1"
backend/migrations/20260904135713_dbt_environment_state.down.sql
+1
View File
@@ -0,0 +1 @@
DROP TABLE IF EXISTS dbt_environment_state;
backend/migrations/20260904135713_dbt_environment_state.up.sql
+52
View File
@@ -0,0 +1,52 @@
-- The dbt state one project last built into one environment: the `manifest.json`
-- (and the `run_results.json` beside it) that `dbt --defer --state <dir>` resolves
-- an unbuilt `ref()` through.
--
-- Separate from `dbt_run_state`, which answers a different question. That one is
-- keyed by the executing principal and holds the LAST run whatever its outcome,
-- so `dbt retry` can resume its failures; this one is keyed by environment and
-- holds the last SUCCESSFUL run, because a relation a later run defers to has to
-- exist. Merging them would make a retry resume a run that is not the last one,
-- or a deferral point at relations a failed run never wrote.
CREATE TABLE IF NOT EXISTS dbt_environment_state (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE,
script_path VARCHAR(255) NOT NULL,
-- The workspace warehouse, the dbt target, and the database and schema that
-- target resolves to. All four, because deferring is resolving a relation
-- NAME: a repointed warehouse or a moved schema makes the stored manifest
-- describe relations that are not where this run would look for them, and the
-- run has no other way to notice. A move therefore reads as an environment
-- with no state yet rather than as state that silently no longer fits.
--
-- TEXT rather than VARCHAR(255): a project bringing its own `profiles.yml`
-- spells its own schema and database, so the length is the project's.
environment TEXT NOT NULL,
-- The run that published it, so a deferring run can say what it deferred to.
job_id UUID NOT NULL,
-- Exactly one home each. A manifest grows with the project and passes a few
-- hundred KB on a handful of models, so a large one goes to the INSTANCE's
-- object storage and this row keeps the key; a small one stays here, where it
-- costs no round trip and works on an instance that has configured no storage
-- at all. The instance's and not the workspace's, because a member can write
-- the workspace bucket under a key of their choosing, and a manifest is what a
-- later run resolves every unbuilt `ref()` through. `run_results.json` is a
-- tenth of the size and takes the same two homes rather than a rule of its own.
manifest TEXT,
manifest_key TEXT,
run_results TEXT,
run_results_key TEXT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (workspace_id, script_path, environment),
CONSTRAINT dbt_environment_state_manifest_one_home
CHECK (num_nonnulls(manifest, manifest_key) = 1),
CONSTRAINT dbt_environment_state_run_results_one_home
CHECK (num_nonnulls(run_results, run_results_key) <= 1)
);
-- No age sweep, unlike the per-run graph rows next door: this table holds one
-- row per script per environment and replaces it in place, so it does not grow
-- with runs, and its reader is every later run of that script — a project that
-- runs monthly must still find last month's state. It goes with the script
-- instead, alongside `dbt_run_state`.
GRANT ALL ON dbt_environment_state TO windmill_user;
GRANT ALL ON dbt_environment_state TO windmill_admin;
backend/migrations/20260904143633_dbt_column_lineage.down.sql
+2
View File
@@ -0,0 +1,2 @@
DROP TABLE IF EXISTS dbt_column_edge;
ALTER TABLE dbt_node DROP COLUMN IF EXISTS column_schema;
backend/migrations/20260904143633_dbt_column_lineage.up.sql
+68
View File
@@ -0,0 +1,68 @@
-- Column-level lineage, from the engine's own static analysis.
--
-- `manifest.json` carries none, which is why decision 14 recorded the feature as
-- unavailable. The edges exist in a different artifact: an engine that does
-- static analysis writes `target/index/dbt.column_lineage.parquet` under
-- `dbt compile --static-analysis strict --write-index`. That pass is opt-in per
-- project (`column_lineage: true`), because strict analysis rejects SQL the
-- default accepts and must never become a silent requirement of running a build.
-- One column-to-column edge, keyed exactly like `dbt_edge`: a version's graph
-- dies with its version through the composite foreign key, a run's snapshot is
-- keyed by `job_id` with the zero UUID meaning "the version's own graph", and an
-- editor buffer's parse carries a NULL `script_hash` keyed to its preview job.
CREATE TABLE IF NOT EXISTS dbt_column_edge (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE,
script_path VARCHAR(255) NOT NULL,
script_hash BIGINT,
job_id UUID NOT NULL DEFAULT '00000000-0000-0000-0000-000000000000',
parent_unique_id TEXT NOT NULL,
parent_column TEXT NOT NULL,
child_unique_id TEXT NOT NULL,
child_column TEXT NOT NULL,
-- dbt's own word for how the value travelled: `copy` (passthrough), `mod`
-- (transformed), `scan` (the column was read to produce the ROW rather than
-- the value -- a join key, a `where` predicate, a `group by`). TEXT rather
-- than an enum because the engine treats the set as open: its own reader maps
-- those three and returns anything else verbatim.
lineage_kind TEXT NOT NULL,
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- Two partial unique indexes rather than a primary key, for the reason
-- 20260801121717 gives: a versioned graph is keyed by its version, a buffer
-- parse by its job alone. `lineage_kind` is part of the key because it is part
-- of the fact: a column that is both projected and used as a predicate for the
-- same output column has a `copy` edge AND a `scan` one, and the digest counts
-- both. Leaving it out let `ON CONFLICT DO NOTHING` drop the second while the
-- digest still claimed it was stored.
CONSTRAINT dbt_column_edge_script_fkey FOREIGN KEY (workspace_id, script_hash)
REFERENCES script (workspace_id, hash) ON DELETE CASCADE ON UPDATE CASCADE
);
CREATE UNIQUE INDEX IF NOT EXISTS dbt_column_edge_versioned_key
ON dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id, child_column,
lineage_kind)
WHERE script_hash IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS dbt_column_edge_editor_key
ON dbt_column_edge (workspace_id, job_id,
parent_unique_id, parent_column, child_unique_id, child_column,
lineage_kind)
WHERE script_hash IS NULL;
-- Same age sweep as the other per-run rows, and the same reason there is no
-- foreign key to `v2_job`.
CREATE INDEX IF NOT EXISTS idx_dbt_column_edge_run_age ON dbt_column_edge (ingested_at)
WHERE job_id <> '00000000-0000-0000-0000-000000000000';
-- The real column schema of a node, which only static analysis knows: an
-- ordered `[{"name": …, "type": …}]`, from `dbt.node_columns.parquet`.
--
-- Beside `columns` rather than folded into it. `columns` is the DECLARED
-- metadata `manifest.json` carries -- the names an author wrote in `schema.yml`
-- and the prose against them -- and stays exactly that, so a project that
-- documents two of forty columns keeps saying so. This is the other forty,
-- typed, in the order the model produces them.
ALTER TABLE dbt_node ADD COLUMN IF NOT EXISTS column_schema JSONB;
GRANT ALL ON dbt_column_edge TO windmill_user;
GRANT ALL ON dbt_column_edge TO windmill_admin;
backend/parsers/windmill-parser-wasm/Cargo.lock
Generated
+24 -24
View File
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windmill-common"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"aho-corasick",
"anyhow",
@@ -6274,7 +6274,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"proc-macro2",
"quote",
@@ -6286,7 +6286,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"convert_case",
"serde",
@@ -6295,7 +6295,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6307,7 +6307,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6319,7 +6319,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"gosyn",
@@ -6331,7 +6331,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6343,7 +6343,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6355,7 +6355,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -6366,7 +6366,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6377,7 +6377,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6389,7 +6389,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6400,7 +6400,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -6422,7 +6422,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6434,7 +6434,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6448,7 +6448,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"convert_case",
@@ -6465,7 +6465,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6478,7 +6478,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde",
@@ -6490,7 +6490,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6508,7 +6508,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -6524,7 +6524,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6540,7 +6540,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wasm"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"getrandom 0.2.17",
@@ -6572,7 +6572,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6586,7 +6586,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.804.0"
version = "1.805.0"
dependencies = [
"anyhow",
"bitflags",
backend/parsers/windmill-parser-wasm/Cargo.toml
+1 -1
View File
@@ -12,7 +12,7 @@ resolver = "2"
members = ["."]
[workspace.package]
version = "1.804.0"
version = "1.805.0"
edition = "2021"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
backend/parsers/windmill-parser-yaml/src/dbt.rs
+84 -11
View File
@@ -58,6 +58,31 @@ impl DbtEngine {
pub fn emits_node_events(&self) -> bool {
matches!(self, DbtEngine::DbtCore1x)
}
/// Whether the engine's CLI has `--write-index`, the flag that writes the
/// parquet index column lineage lives in. False for 1.x, whose Python CLI
/// has no such option.
///
/// True is not a promise that the artifact appears: `dbt-core` 2.0.0-alpha.5
/// accepts the flag, declares the views over `dbt.column_lineage` in its own
/// `views.sql`, and writes neither that parquet nor `dbt.node_columns`. Only
/// Fusion does today. Attempting the pass on both is what lets a later 2.x
/// release pick the feature up with no change here.
pub fn writes_column_index(&self) -> bool {
!matches!(self, DbtEngine::DbtCore1x)
}
/// Whether the engine has `--defer-state`, the deferral-only half of
/// `--state`.
///
/// It matters on one command. `dbt retry` reads the run it resumes from
/// `--state`, so an engine with only that flag cannot be told to defer and
/// to resume from the job's own results at once: handed the deferral's
/// directory, it resumes the all-green run stored there and rebuilds
/// nothing. Only dbt-core 1.x separates the two.
pub fn has_defer_state_flag(&self) -> bool {
matches!(self, DbtEngine::DbtCore1x)
}
}
/// How the warehouse connection is supplied. Both paths are supported
@@ -127,6 +152,18 @@ pub struct DbtDescriptor {
pub selector: Option<String>,
#[serde(default)]
pub test_behavior: DbtTestBehavior,
/// Ingest column-to-column lineage and the real column schemas, from the
/// engine's static analysis.
///
/// Opt-in, and it has to be: the artifact only appears under
/// `--static-analysis strict`, which rejects SQL the default accepts (an
/// unresolvable identifier is an error there and compiles fine otherwise).
/// Turning it on for everyone would make a stricter dialect the price of
/// deploying a dbt project. It is a separate `dbt compile` pass, so nothing
/// it decides can change what a build does; a project it cannot analyze
/// keeps the graph it has today.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub column_lineage: bool,
/// `--vars`. dbt vars are typed — numbers, booleans, lists and objects are
/// all normal — so values keep their YAML type; only string leaves carry
/// `{{ arg }}` placeholders the worker substitutes from job args. Coercing
@@ -137,6 +174,16 @@ pub struct DbtDescriptor {
pub threads: Option<u32>,
#[serde(default)]
pub full_refresh: bool,
/// Resolve a `ref()` a run does not build through the state the last
/// successful run of this environment published, rather than through the
/// schema that run writes into.
///
/// Only the default for the `build` block's own `defer`, since the choice is
/// per run: the run that publishes an environment's state and the run that
/// defers to it are two invocations of ONE script (decision 6), so a project
/// that could only defer by descriptor could never populate what it reads.
#[serde(default)]
pub defer: bool,
/// Automatic in-job retry of the nodes a build failed on.
///
/// dbt already confines a failure to its own subtree, and `dbt retry`
@@ -258,6 +305,7 @@ pub const RESERVED_ARG_NAMES: &[&str] = &[
"exclude",
"vars",
"full_refresh",
"defer",
"dbt_command",
"dbt_retry_job",
"model",
@@ -345,15 +393,26 @@ fn command_variants(d: &DbtDescriptor) -> Vec<(&'static str, Vec<Arg>)> {
"build",
selection()
.into_iter()
.chain([Arg {
name: "full_refresh".to_string(),
otyp: None,
typ: Typ::Bool,
has_default: true,
default: Some(serde_json::json!(d.full_refresh)),
oidx: None,
otyp_inferred: false,
}])
.chain([
Arg {
name: "full_refresh".to_string(),
otyp: None,
typ: Typ::Bool,
has_default: true,
default: Some(serde_json::json!(d.full_refresh)),
oidx: None,
otyp_inferred: false,
},
Arg {
name: "defer".to_string(),
otyp: None,
typ: Typ::Bool,
has_default: true,
default: Some(serde_json::json!(d.defer)),
oidx: None,
otyp_inferred: false,
},
])
.collect(),
),
(
@@ -546,7 +605,9 @@ fn property_of(arg: &Arg) -> serde_json::Value {
),
"select" => Some(
"dbt selection syntax, e.g. `tag:nightly`, `stg_orders+`, \
`config.materialized:incremental`. Empty runs the descriptor's own selection.",
`config.materialized:incremental`. `state:modified+` and `result:error+` \
compare against the state a previous run published, so they need `defer` on. \
Empty runs the descriptor's own selection.",
),
"exclude" => Some("Nodes to leave out of the selection above, same syntax."),
"vars" => Some(
@@ -554,6 +615,11 @@ fn property_of(arg: &Arg) -> serde_json::Value {
exist makes this run store its own graph rather than the deployed one.",
),
"full_refresh" => Some("Rebuild incremental models from scratch instead of appending."),
"defer" => Some(
"Resolve a `ref()` this run does not build to the relation the last successful \
run of this warehouse and target published, instead of to the schema this run \
writes into.",
),
"model" => Some(
"The model to preview, by name — `stg_orders`, or `my_package.stg_orders` when \
two packages share a name. Any dbt selector resolving to ONE node works.",
@@ -687,8 +753,15 @@ full_refresh: true
};
let (build, build_args) = of("build");
assert_eq!(build_args, ["exclude", "full_refresh", "select", "vars"]);
assert_eq!(
build_args,
["defer", "exclude", "full_refresh", "select", "vars"]
);
assert_eq!(build["properties"]["full_refresh"]["type"], "boolean");
// `defer` is a per-run toggle rather than a descriptor-only setting: the
// run that publishes an environment's state and the run that defers to
// it are two invocations of ONE script.
assert_eq!(build["properties"]["defer"]["type"], "boolean");
// Defaults come from the descriptor, so an untouched run reproduces it.
assert_eq!(
build["properties"]["select"]["default"],
backend/parsers/windmill-parser/src/asset_parser.rs
+44 -25
View File
@@ -29,10 +29,10 @@ pub enum AssetKind {
Ducklake,
DataTable,
Volume,
/// A warehouse relation a dbt project builds or reads,
/// `dbt://<warehouse>/<schema>/<name>`, the warehouse named as the
/// workspace configures it. The scheme names the producer, the path stays
/// the relation — see `windmill_types::AssetKind::Dbt`.
/// A warehouse relation, `dbt://<warehouse>/<schema>/<name>`, the warehouse
/// named as the workspace configures it. The scheme names the namespace dbt
/// made — a script in any language but dbt's own can declare a write to one —
/// and the path stays the relation. See `windmill_types::AssetKind::Dbt`.
Dbt,
}
@@ -288,11 +288,13 @@ pub struct RetrySpec {
}
// `// materialize [manual] <asset> [append] [key=<col>] [history] [track=<c1,c2>]`
// — declares that this script produces a *managed* materialization of `<asset>`
// (a `ducklake://` table). By default the runtime generates the write DDL around
// — declares that this script produces `<asset>`. A `ducklake://` table is
// materialized *managed* by default: the runtime generates the write DDL around
// the script's single trailing `SELECT` and owns idempotency, partition-state
// and snapshot capture. `manual` is the escape hatch: the script writes its own
// DDL and the runtime only records state (track-only). The reconciliation
// DDL and the runtime only records state (track-only) — and it is the only mode a
// `dbt://` warehouse relation has, since nothing generates warehouse DDL (deploy
// enforces that; see docs/dbt-runtime.md). The reconciliation
// strategy options apply to managed mode: none → DELETE-by-partition + INSERT
// (replace); `key=<col>` → MERGE (dedup within slice, SCD type 1); `append` →
// INSERT-only. `append` wins if both are given (deploy-time warning).
@@ -804,6 +806,18 @@ pub fn canonicalize_table_asset_path(path: &str) -> String {
)
}
/// Whether a `dbt://` path names a whole relation, `<warehouse>/<schema>/<name>`.
///
/// Every producer spells one that way — the manifest ingest derives it from
/// `relation_name`, a `// materialize` target is checked against it — so anything
/// else can be produced by nothing and read by nothing. Both sides of the deploy
/// ask here rather than counting segments themselves: a subscription and a write
/// that disagreed on the shape would refuse and accept the same string.
pub fn is_full_relation_path(path: &str) -> bool {
let mut segments = path.split('/');
segments.clone().count() == 3 && !segments.any(str::is_empty)
}
/// A doubled delimiter inside a quoted identifier is that delimiter, literally —
/// the same rule the worker's `split_relation` applies to `relation_name`. Both
/// have to decode it or one spelling of a table becomes two graph nodes: the dbt
@@ -1740,10 +1754,7 @@ mod pipeline_annotation_tests {
// just stop being the same node and the cross-boundary cascade never fires.
#[test]
fn table_paths_from_every_spelling_canonicalize_to_one_key() {
let canonical = Some((
AssetKind::Dbt,
Cow::Owned("main/analytics/orders".into()),
));
let canonical = Some((AssetKind::Dbt, Cow::Owned("main/analytics/orders".into())));
for spelling in [
// Hand-written annotation.
"dbt://main/analytics/orders",
@@ -1765,6 +1776,17 @@ mod pipeline_annotation_tests {
}
}
/// The shape both halves of the deploy check against: a subscription and a
/// write that disagreed on it would refuse and accept the same string.
#[test]
fn a_whole_relation_is_three_non_empty_segments() {
assert!(is_full_relation_path("main/analytics/orders"));
assert!(is_full_relation_path("main/archive.sales/orders"));
for partial in ["main", "main/analytics", "main/analytics/orders/x", "", "main//orders"] {
assert!(!is_full_relation_path(partial), "{partial} is not a relation");
}
}
// A relation that overrode its database carries `<database>.<schema>` in
// one segment, and each half can be quoted independently. Stripping only
// the outer pair leaves a key the manifest ingest never produces, so the
@@ -1791,10 +1813,7 @@ mod pipeline_annotation_tests {
// database qualifier.
assert_eq!(
parse_asset_syntax("dbt://main/\"sales.v2\"/orders", false),
Some((
AssetKind::Dbt,
Cow::Owned("main/sales.v2/orders".into())
))
Some((AssetKind::Dbt, Cow::Owned("main/sales.v2/orders".into())))
);
}
@@ -1813,14 +1832,8 @@ mod pipeline_annotation_tests {
"dbt://main/analytics/\"order\"\"s\"",
"main/analytics/order\"s",
),
(
"dbt://main/`da``ta`/`orders`",
"main/da`ta/orders",
),
(
"dbt://main/[my]]schema]/[orders]",
"main/my]schema/orders",
),
("dbt://main/`da``ta`/`orders`", "main/da`ta/orders"),
("dbt://main/[my]]schema]/[orders]", "main/my]schema/orders"),
// And in one half of a database-qualified segment.
(
"dbt://main/\"arch\"\"ive\".\"sales\"/orders",
@@ -1839,8 +1852,14 @@ mod pipeline_annotation_tests {
// apart. A lone delimiter treated as opening a quote would be dropped —
// `sa"les` filed as `sales` — and the two derivations would split.
for (decoded, spelled) in [
("dbt://main/sa\"les/orders", "dbt://main/\"sa\"\"les\"/orders"),
("dbt://main/analytics/order\"s", "dbt://main/analytics/\"order\"\"s\""),
(
"dbt://main/sa\"les/orders",
"dbt://main/\"sa\"\"les\"/orders",
),
(
"dbt://main/analytics/order\"s",
"dbt://main/analytics/\"order\"\"s\"",
),
(
"dbt://main/arch\"ive.sales/orders",
"dbt://main/\"arch\"\"ive\".\"sales\"/orders",
backend/summarized_schema.txt
+5 -1
View File
@@ -70,11 +70,15 @@ ci_test_reference: workspace_id(char), test_script_path(char), test_script_hash(
concurrency_settings: hash(bigint), concurrency_key(char), concurrent_limit(int), concurrency_time_window_s(int)
config: name(char), config(jsonb)
custom_concurrency_key_ended: key(char), ended_at(ts)
dbt_column_edge: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), parent_unique_id(text), parent_column(text), child_unique_id(text), child_column(text), lineage_kind(text), ingested_at(ts)
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
dbt_edge: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), parent_unique_id(text), child_unique_id(text), ingested_at(ts)
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
dbt_graph_snapshot: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), digest(text), relation_root_at_last_ingest(text), ingested_at(ts), permissioned_as(char)
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
dbt_node: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), unique_id(text), resource_type(text), name(text), asset_path(text), materialized(text), materialize_strategy(text), unique_key(text), tags(text[]), description(text), test_kind(text), test_column(text), test_args(jsonb), severity(text), attached_node(text), columns(jsonb), freshness(jsonb), raw_code(text), original_file_path(text), ingested_at(ts)
dbt_environment_state: workspace_id(char), script_path(char), environment(text), job_id(uuid), manifest(text), manifest_key(text), run_results(text), run_results_key(text), updated_at(ts)
FK: (workspace_id) -> workspace(id)
dbt_node: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), unique_id(text), resource_type(text), name(text), asset_path(text), materialized(text), materialize_strategy(text), unique_key(text), tags(text[]), description(text), test_kind(text), test_column(text), test_args(jsonb), severity(text), attached_node(text), columns(jsonb), column_schema(jsonb), freshness(jsonb), raw_code(text), original_file_path(text), ingested_at(ts)
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
dbt_run_progress: workspace_id(char), job_id(uuid), asset_kind(asset_kind), asset_path(char), status(materialization_status), row_count(bigint), error(text), updated_at(ts)
FK: (workspace_id) -> workspace(id)
backend/windmill-api-assets/src/lib.rs
+460 -4
View File
@@ -6,6 +6,7 @@ use axum::{
use serde::{Deserialize, Serialize};
use serde_json::Value;
use sqlx::Row;
use std::collections::{BTreeSet, HashMap, HashSet, VecDeque};
use windmill_common::{
assets::{parse_asset_trigger_ref, AssetKind, AssetUsageKind},
db::UserDB,
@@ -13,7 +14,9 @@ use windmill_common::{
utils::escape_ilike_pattern,
};
use windmill_api_auth::{build_scope_path_predicate, ApiAuthed};
use windmill_api_auth::{
build_scope_path_filter, build_scope_path_predicate, ApiAuthed, ScopePathFilter,
};
// Partition-range backfill preview. The logic (producer resolution, range
// enumeration, status join) is enterprise: the `private` build compiles the
@@ -33,6 +36,7 @@ pub fn workspaced_service() -> Router {
.route("/list_by_usages", post(list_assets_by_usages))
.route("/list_favorites", get(list_favorites))
.route("/graph", get(asset_graph))
.route("/column_lineage", get(dbt_column_lineage))
.route("/pipelines", get(list_pipeline_folders))
.route("/partitions", get(list_partitions))
.route("/partitions_in_range", get(list_partitions_in_range))
@@ -663,10 +667,21 @@ struct DbtAssetProvenance {
description: Option<String>,
#[serde(skip_serializing_if = "Vec::is_empty")]
data_tests: Vec<DbtDataTest>,
/// Declared column metadata (name -> description). NOT column lineage —
/// `manifest.json` carries none (docs/dbt-runtime.md, decision 14).
/// Declared column metadata (name -> description): what `manifest.json`
/// carries, which is only the columns an author wrote down.
#[serde(skip_serializing_if = "Option::is_none")]
columns: Option<serde_json::Value>,
/// Every column of the relation, typed and in order —
/// `[{"name": …, "type": …}]` — from the engine's static analysis. Present
/// only for a project that opted into it.
///
/// Gated exactly like `columns` and the model's SQL: a full column list is
/// the shape of what the author WROTE, one level finer than the `ref()`
/// graph, which is ungated only because it draws relations the caller
/// already sees in `asset`. Widening that boundary has to be a decision, not
/// a consequence of a project turning the analysis pass on.
#[serde(skip_serializing_if = "Option::is_none")]
column_schema: Option<serde_json::Value>,
/// A source's declared freshness policy, for the staleness chip.
#[serde(skip_serializing_if = "Option::is_none")]
freshness: Option<serde_json::Value>,
@@ -946,6 +961,442 @@ struct DbtLineageEdge {
to_asset_path: String,
}
/// One column-to-column edge, in the same terms: the two relations and the two
/// columns, never dbt's node ids.
#[derive(Serialize, Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
struct DbtColumnLineageEdge {
from_asset_path: String,
from_column: String,
to_asset_path: String,
to_column: String,
/// dbt's own word for how the value travelled: `copy` (passthrough), `mod`
/// (transformed), `scan` (read to produce the ROW rather than the value — a
/// join key, a predicate, a `group by`). Sent verbatim, including a kind
/// this engine version invented, because the renderer decides what a kind
/// means and the set is the engine's.
kind: String,
}
/// The dbt relations a view is tracing, and which stored graph to read them
/// from.
///
/// Several relations, answered as one union, because ONE selection reaches
/// several: a script's output column can be derived from columns of several dbt
/// models, and a model's columns can be consumed by scripts that feed others.
/// Asking per relation instead is a request per boundary plus the bookkeeping to
/// stitch the answers together and decide which of them is still current — which
/// is a cache, and is what taking them together exists to not need.
pub struct ColumnLineageQuery {
/// The `dbt://` relations whose lineage to return.
pub asset_paths: Vec<String>,
/// The deployed version a view is drawing, when it is drawing one — the dbt
/// editor, which shows a single project as of a single deploy.
///
/// A version-pinned answer is that version's project ALONE, the same as a
/// job-pinned one: the pin exists so the trace describes the stored graph on
/// screen, and another project's live graph is not part of it. Only the
/// unpinned answer crosses projects.
///
/// A run's or an editor buffer's graph is NOT reachable from here: it pins
/// to a job, and that costs the job-read gate.
pub dbt_script_hash: Option<windmill_common::scripts::ScriptHash>,
}
impl ColumnLineageQuery {
/// Built from the raw pairs rather than deserialized as a struct, because
/// `asset_path` REPEATS and `serde_urlencoded` — what `Query` deserializes
/// with — reads no sequence from a repeated key. A GET rather than a POST
/// body carrying the list: the method decides a scoped token's action, so a
/// POST would ask `assets:write` for a read and refuse a read-only token
/// outright.
pub fn from_query_pairs(pairs: Vec<(String, String)>) -> windmill_common::error::Result<Self> {
let mut asset_paths: Vec<String> = Vec::new();
let mut dbt_script_hash = None;
for (key, value) in pairs {
match key.as_str() {
"asset_path" => asset_paths.push(value),
// Hex, like every other script-hash parameter, so a page can
// pass `job.script_hash` verbatim.
"dbt_script_hash" => {
dbt_script_hash =
Some(serde_json::from_value(Value::String(value)).map_err(|_| {
windmill_common::error::Error::BadRequest(
"dbt_script_hash is not a script hash".to_string(),
)
})?)
}
_ => {}
}
}
// REFUSED, not answered empty. A caller that named no relation — or
// misspelled the parameter — asked for something, and an empty component
// is what a relation with no lineage returns, so answering that way says
// "this has none" for a question that was never asked.
if asset_paths.is_empty() {
return Err(windmill_common::error::Error::BadRequest(
"at least one asset_path is required".to_string(),
));
}
Ok(ColumnLineageQuery { asset_paths, dbt_script_hash })
}
}
#[derive(Serialize, Debug, PartialEq, Eq)]
pub struct ColumnLineageResponse {
/// Direct (`copy` / `mod`) column edges of the component the asked-for
/// relations' columns sit in, in the terms the canvas draws. Empty when no
/// project involved asked for the analysis pass, which is the ordinary case.
edges: Vec<DbtColumnLineageEdge>,
/// The component reaches further than what is here: `edges` holds the part
/// nearest the asked-for relations. Said rather than silently cut, because a
/// trace that stops short is otherwise indistinguishable from one that ends.
truncated: bool,
}
/// How many edges one trace may carry back. The renderer draws a box per column,
/// so a component past this is unreadable however it is served — a synthetic
/// 3000-model project whose models share a column returns 58k direct edges and
/// 7.3MB. Applied over the walk, which is the LAST filter, so what survives is
/// the part nearest the selection rather than an arbitrary slice of it.
const MAX_TRACE_EDGES: usize = 5_000;
/// How many times one trace may discover a project it has not read yet. Each
/// round costs a gate and a fetch, and a component crossing this many projects
/// has already outgrown what the canvas can show; stopping says what the edge
/// bound says.
const MAX_OWNER_ROUNDS: usize = 8;
/// How many edges one trace may HOLD while walking, as opposed to answer with.
/// A project's edges arrive whole — the walk decides what is in the component,
/// so a `LIMIT` on the fetch would cut an arbitrary set that need not even
/// contain the asked-for relation — and a project is bounded at ingest by
/// `MAX_COLUMN_EDGES`, which is 200k. Rounds are what this bounds: reading a
/// second project's worth on top of an already outsized first one buys nothing,
/// since the walk is going to stop at `MAX_TRACE_EDGES` regardless.
const MAX_HELD_EDGES: usize = 100_000;
/// A stored project graph: a deployed version, or one job's snapshot of it.
/// `script_hash` is NULL for an editor buffer's parse, which names no version.
type ProjectVersion = (String, Option<i64>, uuid::Uuid);
async fn dbt_column_lineage(
authed: ApiAuthed,
Path(w_id): Path<String>,
Extension(user_db): Extension<UserDB>,
Query(pairs): Query<Vec<(String, String)>>,
) -> JsonResult<ColumnLineageResponse> {
// `None`: pinning to one run is job-scoped and this endpoint is authorized
// as `assets:read`. See `dbt_column_lineage_for`.
let q = ColumnLineageQuery::from_query_pairs(pairs)?;
dbt_column_lineage_for(&authed, &w_id, user_db, q, None).await
}
/// The column-level lineage the asked-for relations sit in, optionally as one
/// run saw it.
///
/// AUTHORIZES NOTHING BY ITSELF, on the same contract as `asset_graph_for`:
/// `assets:read` always, and the job-read gate for `Some(pinned)`, whose path
/// and hash are then taken from that job's row rather than from the caller.
///
/// A column trace is transitive and a relation is not owned by one project, so
/// the answer grows a project at a time: resolve who owns the relations reached
/// so far, read their edges, walk, and repeat for the relations that walk newly
/// reached. Every round re-applies the caller's gate to the projects it
/// discovers — a relation being reachable from a project the caller may read
/// says nothing about the project on the other side of it.
pub async fn dbt_column_lineage_for(
authed: &ApiAuthed,
w_id: &str,
user_db: UserDB,
q: ColumnLineageQuery,
pinned: Option<PinnedRun>,
) -> JsonResult<ColumnLineageResponse> {
// A column-level view is the shape of what the author WROTE, so it takes the
// model's own gate rather than the relation's.
let (scope_all, scope_exact, scope_prefix) =
match build_scope_path_filter(authed, "scripts", "read") {
ScopePathFilter::AllowAll => (true, Vec::new(), Vec::new()),
ScopePathFilter::Restricted { exact, prefix } => (false, exact, prefix),
};
let (pinned_path, script_hash) = match pinned.as_ref() {
// The job's own version, so a pin cannot name one project's run while
// claiming another's version — including when it names NONE, which is
// the editor buffer.
Some(p) => (Some(p.script_path.as_str()), p.script_hash),
None => (None, q.dbt_script_hash.map(|h| h.0)),
};
let pinned_job_id = pinned.as_ref().map(|p| p.job_id);
let seeds: BTreeSet<String> = q.asset_paths.into_iter().collect();
let mut tx = user_db.begin(authed).await?;
// Relations whose owners have been asked for, project graphs already read,
// and the edges they yielded. These are what end the loop: a round asks only
// about relations not asked about before and reads only projects not read
// before, so it stops as soon as one of the two runs out.
let mut asked: HashSet<String> = HashSet::new();
let mut read: HashSet<ProjectVersion> = HashSet::new();
let mut edges: Vec<DbtColumnLineageEdge> = Vec::new();
let mut answer: Vec<DbtColumnLineageEdge> = Vec::new();
let mut pending: Vec<String> = seeds.iter().cloned().collect();
let mut truncated = false;
for _ in 0..MAX_OWNER_ROUNDS {
if pending.is_empty() {
break;
}
// Which project version owns each of these relations, under this
// caller's access. Usually one row per relation; a relation a second
// project declares as a source has two, and each answers for its own
// lineage.
let owners = sqlx::query!(
r#"SELECT DISTINCT n.script_path AS "script_path!", n.script_hash, n.job_id AS "job_id!"
FROM dbt_node n
WHERE n.workspace_id = $1 AND n.asset_path = ANY($2)
-- The run's snapshot, or the deployed graph when that job stored
-- none -- a build pins only if it wrote one.
AND n.job_id = CASE WHEN $5::uuid IS NOT NULL AND EXISTS (
SELECT 1 FROM dbt_graph_snapshot g
WHERE g.workspace_id = $1 AND g.job_id = $5)
THEN $5::uuid
ELSE '00000000-0000-0000-0000-000000000000'::uuid END
-- The gate, re-decided for every project the walk reaches. That
-- is what resolving owners in a loop is for: being entitled to
-- one project is not being entitled to the one that declares a
-- relation it hands over.
AND ( $6
OR n.script_path = ANY($7)
OR EXISTS ( SELECT 1 FROM unnest($8::text[]) AS pfx
WHERE n.script_path = pfx
OR left(n.script_path, length(pfx) + 1) = pfx || '/' ) )
AND CASE
-- Pinned: which version comes from a job this caller was
-- already granted, so `script` does not decide THAT -- but
-- it still decides whether the project may be read, the
-- same second gate `script_visible` is on the graph. Being
-- entitled to a run is not being entitled to the SQL
-- behind it, and column lineage is that SQL's shape. A
-- version-less row is exempt because it is an editor
-- buffer, which has no `script` row to ask and reaches
-- this only through the parse job that wrote it.
--
-- One project answers, so a pinned trace never crosses
-- into another: neither does the graph it annotates.
WHEN $4::text IS NOT NULL
THEN n.script_path = $4 AND n.script_hash IS NOT DISTINCT FROM $3::bigint
AND ($3::bigint IS NULL OR EXISTS (
SELECT 1 FROM script sc
WHERE sc.workspace_id = $1 AND sc.path = n.script_path
AND sc.hash = $3))
-- A named version: the deployed one an editor is drawing.
-- `script` is read under RLS, so this is the visibility
-- check as well as the existence one. A hash names one
-- script row, so this arm answers for one project too —
-- and deliberately: a pin says which stored graph is on
-- screen, and another project's live graph is not it.
WHEN $3::bigint IS NOT NULL
THEN n.script_hash = $3 AND EXISTS (
SELECT 1 FROM script sc
WHERE sc.workspace_id = $1 AND sc.path = n.script_path
AND sc.hash = $3)
-- Otherwise the version deployed now: an older one's rows
-- outlive it in `dbt_node` until the sweep, and describe a
-- project that is no longer what runs. `language` narrows
-- it the way the graph's own resolution does, so a path
-- that has since become a script of another kind draws and
-- explains the same version rather than disagreeing. Read
-- under RLS, so a project the caller cannot see resolves
-- to NULL and matches nothing.
ELSE n.script_hash = (
SELECT sc.hash FROM script sc
WHERE sc.workspace_id = $1 AND sc.path = n.script_path
AND sc.language = 'dbt'
AND sc.deleted = false AND sc.archived = false
ORDER BY sc.created_at DESC LIMIT 1)
END"#,
w_id,
&pending[..],
script_hash,
pinned_path,
pinned_job_id,
scope_all,
&scope_exact[..],
&scope_prefix[..],
)
.fetch_all(&mut *tx)
.await?;
asked.extend(pending.drain(..));
let fresh: Vec<ProjectVersion> = owners
.into_iter()
.map(|o| (o.script_path, o.script_hash, o.job_id))
.filter(|k| read.insert(k.clone()))
.collect();
if fresh.is_empty() {
break;
}
let fresh_paths: Vec<String> = fresh.iter().map(|k| k.0.clone()).collect();
let fresh_hashes: Vec<Option<i64>> = fresh.iter().map(|k| k.1).collect();
let fresh_jobs: Vec<uuid::Uuid> = fresh.iter().map(|k| k.2).collect();
// DIRECT kinds only. `scan` -- the column was read to produce the ROW,
// not the value -- reaches every output column of its model, so it is
// most of a project's stored lineage and none of what a trace draws.
// It stays in the table for a later view to ask for.
let rows = sqlx::query!(
r#"SELECT p.asset_path AS "from_path!", e.parent_column AS "from_column!",
c.asset_path AS "to_path!", e.child_column AS "to_column!",
e.lineage_kind AS "kind!"
FROM unnest($2::text[], $3::bigint[], $4::uuid[])
AS o(script_path, script_hash, job_id)
JOIN dbt_column_edge e ON e.workspace_id = $1
AND e.script_path = o.script_path
AND e.job_id = o.job_id
-- `=` still, with the NULL-to-NULL case
-- spelled out and gated on the pin: a
-- version-less row's hash is NULL on both
-- sides, which `=` never matches, but
-- `IS NOT DISTINCT FROM` would cost the
-- equality its index bound everywhere else.
AND (e.script_hash = o.script_hash
OR ($5::text IS NOT NULL
AND o.script_hash IS NULL
AND e.script_hash IS NULL))
JOIN dbt_node p ON p.workspace_id = e.workspace_id
AND p.script_path = e.script_path
AND p.script_hash IS NOT DISTINCT FROM e.script_hash
AND p.job_id = e.job_id
AND p.unique_id = e.parent_unique_id
JOIN dbt_node c ON c.workspace_id = e.workspace_id
AND c.script_path = e.script_path
AND c.script_hash IS NOT DISTINCT FROM e.script_hash
AND c.job_id = e.job_id
AND c.unique_id = e.child_unique_id
WHERE e.lineage_kind IN ('copy', 'mod')
AND p.asset_path IS NOT NULL AND c.asset_path IS NOT NULL"#,
w_id,
&fresh_paths[..],
&fresh_hashes[..] as &[Option<i64>],
&fresh_jobs[..],
pinned_path,
)
.fetch_all(&mut *tx)
.await?;
edges.extend(rows.into_iter().map(|r| DbtColumnLineageEdge {
from_asset_path: r.from_path,
from_column: r.from_column,
to_asset_path: r.to_path,
to_column: r.to_column,
kind: r.kind,
}));
// Two projects can describe one relation, so the same edge can arrive
// twice. Sorted as well as deduplicated: the walk reads the incidence
// lists in this order, so the answer does not depend on which round a
// project was discovered in.
edges.sort();
edges.dedup();
let walked = component(&edges, &seeds);
answer = walked.edges;
truncated = walked.truncated;
if truncated {
break;
}
// The relations the walk newly reached. Their owners are the next round's
// question: this project declares them, and so may another.
pending = answer
.iter()
.flat_map(|e| [&e.from_asset_path, &e.to_asset_path])
.filter(|p| !asked.contains(*p))
.cloned()
.collect::<BTreeSet<_>>()
.into_iter()
.collect();
// Stop discovering projects once the held set is outsized. The tail
// below reports what that leaves unresolved, and reports nothing when
// the walk had already reached everything.
if edges.len() >= MAX_HELD_EDGES {
break;
}
}
tx.commit().await?;
// Out of rounds with relations still unresolved: more of the component
// exists, which is what hitting the edge bound also means.
Ok(Json(ColumnLineageResponse {
truncated: truncated || !pending.is_empty(),
edges: answer,
}))
}
struct WalkedComponent {
edges: Vec<DbtColumnLineageEdge>,
truncated: bool,
}
/// The edges of the connected component the asked-for relations sit in, nearest
/// first and at most `MAX_TRACE_EDGES` of them.
///
/// The canvas lays out the component of the selected relation's columns, so a
/// project's other model families are edges nothing it draws can reach. Walked
/// here rather than in SQL: a recursive CTE has no index to walk, so it rescans
/// the whole edge set once per level — measured at 1.24s against 59ms for the
/// query alone on a 3000-model project, for a walk that is microseconds over a
/// map. Columns are keyed by relation, not by project, which is how the canvas
/// keys them too: two projects describing one relation draw one node.
///
/// Breadth-first, so the bound cuts the far end of the trace rather than an
/// arbitrary part of it.
fn component(edges: &[DbtColumnLineageEdge], seeds: &BTreeSet<String>) -> WalkedComponent {
let mut incident: HashMap<(&str, &str), Vec<usize>> = HashMap::new();
for (i, e) in edges.iter().enumerate() {
incident
.entry((&e.from_asset_path, &e.from_column))
.or_default()
.push(i);
incident
.entry((&e.to_asset_path, &e.to_column))
.or_default()
.push(i);
}
let mut start: Vec<(&str, &str)> = incident
.keys()
.filter(|(path, _)| seeds.contains(*path))
.copied()
.collect();
start.sort();
let mut seen_node: HashSet<(&str, &str)> = start.iter().copied().collect();
let mut queue: VecDeque<(&str, &str)> = start.into();
let mut taken = vec![false; edges.len()];
let mut kept: Vec<usize> = Vec::new();
let mut truncated = false;
'walk: while let Some(node) = queue.pop_front() {
for &i in incident.get(&node).map(Vec::as_slice).unwrap_or_default() {
if std::mem::replace(&mut taken[i], true) {
continue;
}
if kept.len() == MAX_TRACE_EDGES {
truncated = true;
break 'walk;
}
kept.push(i);
let e = &edges[i];
let ends = [
(e.from_asset_path.as_str(), e.from_column.as_str()),
(e.to_asset_path.as_str(), e.to_column.as_str()),
];
for end in ends {
if seen_node.insert(end) {
queue.push_back(end);
}
}
}
}
// Back into edge order, so a response does not carry the walk's shape.
kept.sort_unstable();
WalkedComponent { edges: kept.into_iter().map(|i| edges[i].clone()).collect(), truncated }
}
async fn asset_graph(
authed: ApiAuthed,
Path(w_id): Path<String>,
@@ -1323,7 +1774,7 @@ pub async fn asset_graph_for(
n.resource_type AS "resource_type!", n.name AS "name!", n.asset_path,
n.materialized, n.materialize_strategy, n.tags AS "tags!", n.description,
n.test_kind, n.test_column, n.test_args, n.severity, n.attached_node,
n.columns, n.freshness,
n.columns, n.column_schema, n.freshness,
n.raw_code, n.original_file_path,
-- Whether the caller may read the project this row describes.
-- The query deliberately reaches outside the requested folder
@@ -1379,6 +1830,10 @@ pub async fn asset_graph_for(
// `ref()` lineage between two models, resolved to the relations they
// produce. Joined to `dbt_node` on both key columns because a dbt
// `unique_id` is only unique within its project.
//
// Column lineage is NOT here. It is stored per relation and per column, and
// this response is folder-wide and polled by a run page, so it carries only
// what the canvas draws for every node at once.
let dbt_edge_rows = sqlx::query!(
r#"WITH live AS (
SELECT * FROM (
@@ -1604,6 +2059,7 @@ pub async fn asset_graph_for(
description: r.description.clone().filter(|_| source_allowed),
data_tests: vec![],
columns: r.columns.clone().filter(|_| source_allowed),
column_schema: r.column_schema.clone().filter(|_| source_allowed),
freshness: r.freshness.clone().filter(|_| source_allowed),
};
// One relation can carry rows from several projects — typically a model
backend/windmill-api-assets/tests/dbt_pinned_graph.rs
+491 -2
View File
@@ -8,7 +8,9 @@
//! whole dbt half, and every later fix in this area re-touched one of the two.
use sqlx::{Pool, Postgres};
use windmill_api_assets::{asset_graph_for, GraphQuery, PinnedRun};
use windmill_api_assets::{
asset_graph_for, dbt_column_lineage_for, ColumnLineageQuery, GraphQuery, PinnedRun,
};
use windmill_api_auth::ApiAuthed;
use windmill_common::db::UserDB;
@@ -84,6 +86,34 @@ async fn seed(db: &Pool<Postgres>, job: uuid::Uuid) {
.execute(db)
.await
.unwrap();
// The relation it reads, and the column edge between them.
sqlx::query!(
"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,
resource_type, name, asset_path, tags)
VALUES ($1, $2, $3, $4, 'model.p.raw_orders', 'model', 'raw_orders',
'u/a/wh/analytics/raw_orders', '{}')",
WS,
PATH,
HASH,
job
)
.execute(db)
.await
.unwrap();
sqlx::query!(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id,
child_column, lineage_kind)
VALUES ($1, $2, $3, $4, 'model.p.raw_orders', 'id', 'model.p.orders', 'order_id',
'copy')",
WS,
PATH,
HASH,
job
)
.execute(db)
.await
.unwrap();
// A test node, for the arguments it carries: `accepted_values` spells out a
// column's domain.
sqlx::query!(
@@ -365,7 +395,25 @@ async fn seed_editor_graph(db: &Pool<Postgres>, job: uuid::Uuid) {
r#"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,
resource_type, name, asset_path, raw_code, tags)
VALUES ($1, $2, NULL, $3, 'model.p.draft', 'model', 'draft',
'u/a/wh/analytics/draft', 'select 3', '{}')"#,
'u/a/wh/analytics/draft', 'select 3', '{}'),
($1, $2, NULL, $3, 'model.p.draft_src', 'model', 'draft_src',
'u/a/wh/analytics/draft_src', 'select 4', '{}')"#,
WS,
PATH,
job
)
.execute(db)
.await
.unwrap();
// A version-less row's `script_hash` is NULL on both sides of every join and
// every visibility check, and `= NULL` is never true — so the column edges
// need the same NULL arm the node query has, or a buffer parse renders its
// columns and none of their lineage.
sqlx::query!(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id,
child_column, lineage_kind)
VALUES ($1, $2, NULL, $3, 'model.p.draft_src', 'raw', 'model.p.draft', 'clean', 'mod')",
WS,
PATH,
job
@@ -451,3 +499,444 @@ async fn an_editor_graph_renders_only_through_its_own_job(db: Pool<Postgres>) {
"nor of a run of the deployed version: {deployed_run}"
);
}
async fn column_lineage_q(
db: &Pool<Postgres>,
authed: &ApiAuthed,
pairs: Vec<(String, String)>,
pinned: Option<PinnedRun>,
) -> serde_json::Value {
let res = dbt_column_lineage_for(
authed,
WS,
UserDB::new(db.clone()),
ColumnLineageQuery::from_query_pairs(pairs).unwrap(),
pinned,
)
.await
.unwrap();
serde_json::to_value(&res.0).unwrap()
}
async fn column_lineage(
db: &Pool<Postgres>,
authed: &ApiAuthed,
asset_paths: &[&str],
pinned: Option<PinnedRun>,
) -> serde_json::Value {
let pairs = asset_paths
.iter()
.map(|p| ("asset_path".to_string(), p.to_string()))
.collect();
column_lineage_q(db, authed, pairs, pinned).await
}
/// The buffer parse's own lineage, which is the case the versionless rows exist
/// for. Its `script_hash` is NULL on both sides of every join and every
/// visibility check, and `= NULL` is never true — so the versionless arm has to
/// be written for it, or a parse renders its columns and none of their lineage.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn an_editor_buffers_column_lineage_answers_through_its_job(db: Pool<Postgres>) {
let parse = uuid::Uuid::from_u128(9);
seed(&db, uuid::Uuid::from_u128(7)).await;
seed_editor_graph(&db, parse).await;
let admin = ApiAuthed { is_admin: true, ..outsider() };
let pinned = PinnedRun { job_id: parse, script_path: PATH.to_string(), script_hash: None };
assert_eq!(
column_lineage(&db, &admin, &["u/a/wh/analytics/draft"], Some(pinned)).await["edges"],
serde_json::json!([{
"from_asset_path": "u/a/wh/analytics/draft_src",
"from_column": "raw",
"to_asset_path": "u/a/wh/analytics/draft",
"to_column": "clean",
"kind": "mod",
}]),
);
// Unpinned, the same relation resolves through the deployed version, which
// never heard of the buffer's models.
assert_eq!(
column_lineage(&db, &admin, &["u/a/wh/analytics/draft"], None).await["edges"],
serde_json::json!([]),
"a buffer's lineage is reachable only through the job that parsed it"
);
}
/// Being entitled to a RUN is not being entitled to the SQL behind it, and
/// column lineage is that SQL's shape. The pinned graph draws the relations for
/// a share-link viewer and redacts what the author wrote; the lineage is the
/// second, and resolving the version from the job must not be mistaken for
/// deciding that too.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn a_pinned_run_does_not_hand_over_the_projects_column_lineage(db: Pool<Postgres>) {
let job = uuid::Uuid::from_u128(7);
seed(&db, job).await;
let pinned =
|| PinnedRun { job_id: job, script_path: PATH.to_string(), script_hash: Some(HASH) };
assert_eq!(
column_lineage(
&db,
&outsider(),
&["u/a/wh/analytics/orders"],
Some(pinned())
)
.await["edges"],
serde_json::json!([]),
"the run renders for them, its column-level shape does not"
);
assert_eq!(
column_lineage(
&db,
&ApiAuthed { is_admin: true, ..outsider() },
&["u/a/wh/analytics/orders"],
Some(pinned())
)
.await["edges"],
serde_json::json!([{
"from_asset_path": "u/a/wh/analytics/raw_orders",
"from_column": "id",
"to_asset_path": "u/a/wh/analytics/orders",
"to_column": "order_id",
"kind": "copy",
}]),
"while a reader of the project gets it"
);
}
/// The deployed version of the same two models, plus a `scan` edge beside the
/// direct one: `seed`'s rows are a run's snapshot, and the unpinned answer is
/// the version's own graph.
async fn seed_deployed_orders(db: &Pool<Postgres>) {
sqlx::query!(
"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,
resource_type, name, asset_path, tags)
VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'model.p.raw_orders',
'model', 'raw_orders', 'u/a/wh/analytics/raw_orders', '{}'),
($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'model.p.orders',
'model', 'orders', 'u/a/wh/analytics/orders', '{}')",
WS,
PATH,
HASH,
)
.execute(db)
.await
.unwrap();
sqlx::query!(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id,
child_column, lineage_kind)
VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000',
'model.p.raw_orders', 'id', 'model.p.orders', 'order_id', 'copy'),
($1, $2, $3, '00000000-0000-0000-0000-000000000000',
'model.p.raw_orders', 'status', 'model.p.orders', 'order_id', 'scan')",
WS,
PATH,
HASH,
)
.execute(db)
.await
.unwrap();
}
/// A column-level view is the shape of what the author WROTE, so it takes the
/// script's own gate — the same one that keeps `raw_code` behind access to the
/// project. `scan` says the column was read to produce the ROW rather than the
/// value, so it reaches every output column of its model and is never served.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn column_lineage_takes_the_scripts_gate_and_only_the_direct_kinds(db: Pool<Postgres>) {
seed(&db, uuid::Uuid::from_u128(7)).await;
seed_deployed_orders(&db).await;
let admin = ApiAuthed { is_admin: true, ..outsider() };
assert_eq!(
column_lineage(&db, &admin, &["u/a/wh/analytics/orders"], None).await["edges"],
serde_json::json!([{
"from_asset_path": "u/a/wh/analytics/raw_orders",
"from_column": "id",
"to_asset_path": "u/a/wh/analytics/orders",
"to_column": "order_id",
"kind": "copy",
}]),
"the direct edge, and not the `scan` one beside it"
);
assert_eq!(
column_lineage(&db, &outsider(), &["u/a/wh/analytics/orders"], None).await["edges"],
serde_json::json!([]),
"and nothing at all for a caller who cannot read the project"
);
}
/// One project routinely holds model families that share no column, and the
/// canvas lays out the connected component of the selected relation's columns.
/// Answering with the project's other components sends edges nothing can draw.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn column_lineage_stops_at_the_selected_relations_component(db: Pool<Postgres>) {
let job = uuid::Uuid::from_u128(7);
seed(&db, job).await;
// A second family in the same project version, reaching neither of the two
// relations `seed` wired together.
sqlx::query!(
"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,
resource_type, name, asset_path, tags)
VALUES ($1, $2, $3, $4, 'model.p.stock', 'model', 'stock',
'u/a/wh/analytics/stock', '{}'),
($1, $2, $3, $4, 'model.p.stock_daily', 'model', 'stock_daily',
'u/a/wh/analytics/stock_daily', '{}')",
WS,
PATH,
HASH,
job
)
.execute(&db)
.await
.unwrap();
sqlx::query!(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id,
child_column, lineage_kind)
VALUES ($1, $2, $3, $4, 'model.p.stock', 'sku', 'model.p.stock_daily', 'sku', 'copy')",
WS,
PATH,
HASH,
job
)
.execute(&db)
.await
.unwrap();
let admin = ApiAuthed { is_admin: true, ..outsider() };
let pinned =
|| PinnedRun { job_id: job, script_path: PATH.to_string(), script_hash: Some(HASH) };
assert_eq!(
column_lineage(&db, &admin, &["u/a/wh/analytics/orders"], Some(pinned())).await["edges"],
serde_json::json!([{
"from_asset_path": "u/a/wh/analytics/raw_orders",
"from_column": "id",
"to_asset_path": "u/a/wh/analytics/orders",
"to_column": "order_id",
"kind": "copy",
}]),
"the orders family, and not the stock one beside it in the same project"
);
assert_eq!(
column_lineage(
&db,
&admin,
&["u/a/wh/analytics/stock_daily"],
Some(pinned())
)
.await["edges"],
serde_json::json!([{
"from_asset_path": "u/a/wh/analytics/stock",
"from_column": "sku",
"to_asset_path": "u/a/wh/analytics/stock_daily",
"to_column": "sku",
"kind": "copy",
}]),
"and the other way round — reached from the child end, which is upstream"
);
}
/// A deployed dbt project in `folder`, declaring `parent`'s relation as a source
/// and deriving `child` from it. `orders → mart → secret_out` is three projects
/// chained through two shared relations.
async fn seed_neighbour_project(
db: &Pool<Postgres>,
folder: &str,
hash: i64,
parent: (&str, &str),
child: (&str, &str),
) {
let path = format!("f/{folder}/proj");
sqlx::query!(
"INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms)
VALUES ($1, $2, $2, '{}', '{}')",
WS,
folder
)
.execute(db)
.await
.unwrap();
sqlx::query!(
"INSERT INTO script (workspace_id, hash, path, summary, description, content,
created_by, language, lock)
VALUES ($1, $2, $3, '', '', 'profile: {}', 'test-user', 'dbt', '')",
WS,
hash,
path,
)
.execute(db)
.await
.unwrap();
sqlx::query!(
"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,
resource_type, name, asset_path, tags)
VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'source.q.' || $4,
'source', $4, $5, '{}'),
($1, $2, $3, '00000000-0000-0000-0000-000000000000', 'model.q.' || $6,
'model', $6, $7, '{}')",
WS,
path,
hash,
parent.0,
parent.1,
child.0,
child.1,
)
.execute(db)
.await
.unwrap();
sqlx::query!(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id,
child_column, lineage_kind)
VALUES ($1, $2, $3, '00000000-0000-0000-0000-000000000000',
'source.q.' || $4, 'k', 'model.q.' || $5, 'k', 'copy')",
WS,
path,
hash,
parent.0,
child.0,
)
.execute(db)
.await
.unwrap();
}
/// A trace crosses out of the project it started in, and the gate crosses with
/// it.
///
/// A relation one project produces is another's source, so the component reaches
/// edges the first project's owner set never named — that is what resolving
/// owners to a fixpoint is for. The other half is that the caller's access has
/// to be re-decided for each project discovered on the way: reaching a relation
/// says nothing about who may read the project on the far side of it.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn column_lineage_crosses_projects_only_where_the_caller_may_read_them(db: Pool<Postgres>) {
seed(&db, uuid::Uuid::from_u128(7)).await;
seed_deployed_orders(&db).await;
seed_neighbour_project(
&db,
"mid",
43,
("orders", "u/a/wh/analytics/orders"),
("mart", "u/a/wh/analytics/mart"),
)
.await;
seed_neighbour_project(
&db,
"secret",
44,
("mart", "u/a/wh/analytics/mart"),
("secret_out", "u/a/wh/analytics/secret_out"),
)
.await;
let admin = ApiAuthed { is_admin: true, ..outsider() };
let reached = |body: &serde_json::Value| {
body["edges"]
.as_array()
.unwrap()
.iter()
.map(|e| e["to_asset_path"].as_str().unwrap().to_string())
.collect::<std::collections::BTreeSet<_>>()
};
let all = column_lineage(&db, &admin, &["u/a/wh/analytics/orders"], None).await;
assert_eq!(
reached(&all),
[
"u/a/wh/analytics/mart",
"u/a/wh/analytics/orders",
"u/a/wh/analytics/secret_out"
]
.map(String::from)
.into(),
"two projects out from the one asked about, not one: {all}"
);
// Granted the first two folders and not the third. The edges of the project
// they may read are theirs; the one beyond it is not, even though the
// relation joining them is in the answer.
let partial = ApiAuthed {
folders: vec![
("private".to_string(), false, false),
("mid".to_string(), false, false),
],
..outsider()
};
let some = column_lineage(&db, &partial, &["u/a/wh/analytics/orders"], None).await;
assert_eq!(
reached(&some),
["u/a/wh/analytics/mart", "u/a/wh/analytics/orders"]
.map(String::from)
.into(),
"the trace stops where the caller's access does: {some}"
);
// The other half of the same gate, and the half that is hand-written SQL
// rather than RLS: a token scoped to one folder reaches the projects in it
// and no others, whatever its grants say.
let scoped = ApiAuthed {
is_admin: true,
scopes: Some(vec!["scripts:read:f/private/*".to_string()]),
..outsider()
};
let scoped = column_lineage(&db, &scoped, &["u/a/wh/analytics/orders"], None).await;
assert_eq!(
reached(&scoped),
["u/a/wh/analytics/orders"].map(String::from).into(),
"and where its scope does: {scoped}"
);
// A version pin answers for that version's project alone — the dbt editor,
// which draws one project as of one deploy. Crossing into `mid` here would
// annotate that canvas with relations it does not draw.
let pinned_version = column_lineage_q(
&db,
&admin,
vec![
(
"asset_path".to_string(),
"u/a/wh/analytics/orders".to_string(),
),
("dbt_script_hash".to_string(), format!("{:016x}", HASH)),
],
None,
)
.await;
assert_eq!(
reached(&pinned_version),
["u/a/wh/analytics/orders"].map(String::from).into(),
"a version pin does not cross into the project beside it: {pinned_version}"
);
}
/// The bound on the answer, and that hitting it is said rather than silently
/// cut: a trace that stops reads exactly like one that ends.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn a_component_past_the_bound_is_cut_and_says_so(db: Pool<Postgres>) {
seed(&db, uuid::Uuid::from_u128(7)).await;
seed_deployed_orders(&db).await;
// One direct edge per column pair, more of them than a trace may carry.
sqlx::query!(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id,
child_column, lineage_kind)
SELECT $1, $2, $3, '00000000-0000-0000-0000-000000000000',
'model.p.raw_orders', 'c' || i, 'model.p.orders', 'c' || i, 'copy'
FROM generate_series(1, 6000) i",
WS,
PATH,
HASH,
)
.execute(&db)
.await
.unwrap();
let admin = ApiAuthed { is_admin: true, ..outsider() };
let body = column_lineage(&db, &admin, &["u/a/wh/analytics/orders"], None).await;
assert_eq!(body["edges"].as_array().unwrap().len(), 5000);
assert_eq!(body["truncated"], serde_json::json!(true));
}
backend/windmill-api-groups/src/granular_acls.rs
+26
View File
@@ -318,6 +318,19 @@ async fn add_granular_acl(
)
.await?
}
"variable" => {
handle_deployment_metadata(
&authed.email,
&authed.username,
&db,
&w_id,
DeployedObject::Variable { path: path.to_string(), parent_path: None },
Some(format!("Variable '{}' changed permissions", path)),
true,
None,
)
.await?
}
_ => (),
}
@@ -528,6 +541,19 @@ async fn remove_granular_acl(
)
.await?
}
"variable" => {
handle_deployment_metadata(
&authed.email,
&authed.username,
&db,
&w_id,
DeployedObject::Variable { path: path.to_string(), parent_path: None },
Some(format!("Variable '{}' changed permissions", path)),
true,
None,
)
.await?
}
_ => (),
}
}
backend/windmill-api-integration-tests/tests/dbt_materialize_target.rs
+250
View File
@@ -0,0 +1,250 @@
use serde_json::json;
use sqlx::{Pool, Postgres};
use windmill_test_utils::*;
fn client() -> reqwest::Client {
reqwest::Client::new()
}
fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
builder.header("Authorization", "Bearer SECRET_TOKEN")
}
async fn deploy(port: u16, path: &str, content: &str) -> reqwest::Response {
authed(client().post(format!(
"http://localhost:{port}/api/w/test-workspace/scripts/create"
)))
.json(&json!({
"path": path,
"summary": "",
"description": "",
"content": content,
"language": "deno",
"schema": { "type": "object", "properties": {}, "required": [] }
}))
.send()
.await
.unwrap()
}
/// A `dbt://` relation is one graph node only while every side spells it the same
/// way, and three sides derive that spelling independently: the `// materialize`
/// target becomes an `asset.path`, a `// on` ref becomes a `script_trigger`, and
/// the deploy-time refusal joins the two. The unit tests on `sole_dbt_producer`
/// prove the predicate; only a deploy proves the handler feeds it the key the
/// table actually holds — so a canonicalization that drifted on one side would
/// pass those and split the node here.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_dbt_materialize_target_deploy_contract(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
sqlx::query!(
r#"UPDATE workspace_settings
SET dbt_warehouses = '{"main": {"resource_path": "u/test-user/wh"}}'::jsonb
WHERE workspace_id = 'test-workspace'"#
)
.execute(&db)
.await?;
// Nothing generates warehouse DDL, so a managed target is refused rather than
// degraded into the track-only mode it would silently become.
let resp = deploy(
port,
"u/test-user/managed",
"// materialize dbt://main/analytics/orders\nexport async function main() {}",
)
.await;
assert_eq!(resp.status(), 400);
assert!(resp.text().await?.contains("must be `manual`"));
// The warehouse segment is the identity a dbt model keys on; a name the
// workspace does not configure strands the write on an unreachable node.
let resp = deploy(
port,
"u/test-user/unknown_wh",
"// materialize manual dbt://nope/analytics/orders\nexport async function main() {}",
)
.await;
assert_eq!(resp.status(), 400);
assert!(resp.text().await?.contains("does not configure"));
// Only the DuckDB executor runs `// data_test` probes, and it runs them around
// a managed write — so a declarer in another language would deploy green with
// its assertions silently never executed.
let resp = deploy(
port,
"u/test-user/tested",
"// materialize manual dbt://main/analytics/orders\n// data_test not_null id\nexport async function main() {}",
)
.await;
assert_eq!(resp.status(), 400);
assert!(resp
.text()
.await?
.contains("`// data_test` is not supported"));
// Both halves are held to the same relation: every producer is a whole
// `<warehouse>/<schema>/<name>` under a configured warehouse, so a
// subscription to anything else names something nothing can ever write.
for (path, ref_, expected) in [
(
"u/test-user/partial_sub",
"dbt://main/analytics",
"not a whole warehouse relation",
),
(
"u/test-user/unknown_wh_sub",
"dbt://nope/analytics/orders",
"does not configure",
),
] {
let resp = deploy(
port,
path,
&format!("// on {ref_}\nexport async function main() {{}}"),
)
.await;
assert_eq!(resp.status(), 400);
assert!(resp.text().await?.contains(expected));
}
// Past `asset.path`'s column, where the manifest ingest drops the relation and
// no producer row can exist on either side — computed from the bound so it
// cannot drift under it.
let overlong = format!(
"main/analytics/{}",
"o".repeat(windmill_common::dbt_manifest::MAX_ASSET_PATH_LEN)
);
let resp = deploy(
port,
"u/test-user/overlong_sub",
&format!("// on dbt://{overlong}\nexport async function main() {{}}"),
)
.await;
assert_eq!(resp.status(), 400);
assert!(resp.text().await?.contains("characters an asset path holds"));
// Any language may declare the write — the DuckLake write engine is DuckDB's,
// this declaration is not — and the target is canonicalized on the way into
// `asset`, so a hand-written mixed-case spelling lands on the model's key.
let resp = deploy(
port,
"u/test-user/ingest",
"// materialize manual dbt://main/ANALYTICS/Orders\nexport async function main() {}",
)
.await;
assert_eq!(resp.status(), 201);
// The create response, not `{:x}` over the stored i64: `ScriptHash` decodes
// hex and demands 8 bytes, while `LowerHex` drops leading zeros, so a hash
// under 2^60 would 422 the rename below instead of reaching the refusal.
let ingest_hash = resp.text().await?;
let write = sqlx::query_scalar!(
"SELECT path FROM asset WHERE workspace_id = 'test-workspace' AND kind = 'dbt' \
AND usage_path = 'u/test-user/ingest' AND usage_access_type = 'w'"
)
.fetch_one(&db)
.await?;
assert_eq!(write, "main/analytics/orders");
// That producer is native, so subscribing to what it writes is accepted — and
// the `// on` ref has to canonicalize identically, or the row it stores names
// a relation nothing produces.
let resp = deploy(
port,
"u/test-user/consumer",
"// on dbt://main/\"Analytics\"/\"Orders\"\nexport async function main() {}",
)
.await;
assert_eq!(resp.status(), 201);
let trigger_ref = sqlx::query_scalar!(
"SELECT trigger_ref FROM script_trigger WHERE workspace_id = 'test-workspace' \
AND runnable_path = 'u/test-user/consumer' AND trigger_kind = 'asset'"
)
.fetch_one(&db)
.await?;
assert_eq!(trigger_ref, "dbt://main/analytics/orders");
// With dbt as the only producer the same subscription can never be woken — a
// dbt run does not dispatch — so the deploy refuses it and names the project.
sqlx::query!(
"INSERT INTO script (workspace_id, hash, path, summary, description, content, created_by,
language)
VALUES ('test-workspace', 1, 'u/test-user/project', '', '', '', 'test-user', 'dbt')"
)
.execute(&db)
.await?;
sqlx::query!(
"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind)
VALUES ('test-workspace', 'main/analytics/marts', 'dbt', 'w', 'u/test-user/project',
'script')"
)
.execute(&db)
.await?;
let resp = deploy(
port,
"u/test-user/mart_consumer",
"// on dbt://main/analytics/MARTS\nexport async function main() {}",
)
.await;
assert_eq!(resp.status(), 400);
assert!(resp.text().await?.contains("u/test-user/project"));
// A rename is the other half of that: the producer's write still sits at the
// OLD path in the committed snapshot this deploy reads, while the same
// transaction removes it — so it must not count as the producer that would
// wake the subscription the rename adds.
sqlx::query!(
"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind)
VALUES ('test-workspace', 'main/analytics/orders', 'dbt', 'w', 'u/test-user/project',
'script')"
)
.execute(&db)
.await?;
let resp = authed(client().post(format!(
"http://localhost:{port}/api/w/test-workspace/scripts/create"
)))
.json(&json!({
"path": "u/test-user/ingest_renamed",
"parent_hash": ingest_hash,
"summary": "",
"description": "",
"content": "// on dbt://main/analytics/orders\nexport async function main() {}",
"language": "deno",
"schema": { "type": "object", "properties": {}, "required": [] }
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 400);
assert!(resp.text().await?.contains("u/test-user/project"));
// Neither annotation is accepted on a dbt script: the graph ingest
// republishes that path's asset and trigger rows wholesale, so either would
// deploy something the dependency job then silently removes.
for content in [
"# materialize manual dbt://main/analytics/orders\nprofile:\n warehouse: main\n",
"# on dbt://main/analytics/orders\nprofile:\n warehouse: main\n",
] {
let resp = authed(client().post(format!(
"http://localhost:{port}/api/w/test-workspace/scripts/create"
)))
.json(&json!({
"path": "u/test-user/dbt_project",
"summary": "",
"description": "",
"content": content,
"language": "dbt",
"modules": { "dbt_project.yml": { "content": "name: p\n", "language": "dbt" } },
"schema": { "type": "object", "properties": {}, "required": [] }
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 400);
assert!(resp.text().await?.contains("a dbt script cannot"));
}
Ok(())
}
backend/windmill-api-integration-tests/tests/protection_rules.rs
+66
View File
@@ -123,6 +123,72 @@ async fn test_protection_rules(db: Pool<Postgres>) -> anyhow::Result<()> {
.await?;
assert!(!resp.status().is_success(), "Non-admin should be blocked from flows: {}", resp.status());
// ========================================
// 4b. ...but a draft-only resource stays deletable: nothing is deployed at
// its path, so its DELETE is a draft discard rather than a deployment.
// ========================================
let draft_only_path = "u/test-user-2/draft_only_resource";
let resp = authed(
client().post(format!("{base}/drafts/update/resource/{draft_only_path}")),
"SECRET_TOKEN_2",
)
.json(&json!({ "value": {
"path": draft_only_path,
"value": { "a": 1 },
"resource_type": "c_test",
"description": ""
}}))
.send()
.await?;
assert_eq!(
resp.status(),
200,
"Non-admin should save a draft: {}",
resp.text().await?
);
let resp = authed(
client().delete(format!("{base}/resources/delete/{draft_only_path}")),
"SECRET_TOKEN_2",
)
.send()
.await?;
assert_eq!(
resp.status(),
200,
"Draft-only delete should not be gated by the deploy rules: {}",
resp.text().await?
);
let remaining: i64 = sqlx::query_scalar(
"SELECT count(*) FROM draft WHERE workspace_id = 'test-workspace' AND path = $1 \
AND typ = 'resource'::DRAFT_KIND",
)
.bind(draft_only_path)
.fetch_one(&db)
.await?;
assert_eq!(remaining, 0, "the draft should be gone");
// The gate itself is still there for a DEPLOYED resource at the same path.
let resp = authed(
client().post(format!("{base}/resources/create")),
"SECRET_TOKEN_2",
)
.json(&json!({
"path": draft_only_path,
"value": { "a": 1 },
"resource_type": "c_test",
"description": ""
}))
.send()
.await?;
assert!(
!resp.status().is_success(),
"Non-admin should still be blocked from creating a resource: {}",
resp.status()
);
// ========================================
// 5. Admin bypasses protection rule
// ========================================
backend/windmill-api-integration-tests/tests/schedules.rs
+117 -31
View File
@@ -113,7 +113,9 @@ async fn test_schedule_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
"expected at least 2 schedules, got {}",
list.len()
);
assert!(list.iter().any(|s| s["path"] == "u/test-user/test_schedule"));
assert!(list
.iter()
.any(|s| s["path"] == "u/test-user/test_schedule"));
// --- list_with_jobs ---
let resp = authed(client().get(format!("{base}/list_with_jobs")))
@@ -125,18 +127,14 @@ async fn test_schedule_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
assert!(!list.is_empty());
// --- update ---
let resp = authed(client().post(schedule_url(
port,
"update",
"u/test-user/test_schedule",
)))
.json(&json!({
"schedule": "0 0 */12 * * *",
"timezone": "Europe/Paris"
}))
.send()
.await
.unwrap();
let resp = authed(client().post(schedule_url(port, "update", "u/test-user/test_schedule")))
.json(&json!({
"schedule": "0 0 */12 * * *",
"timezone": "Europe/Paris"
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "update: {}", resp.text().await?);
// verify update
@@ -204,14 +202,11 @@ async fn test_schedule_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
assert_eq!(resp.status(), 200);
// --- delete ---
let resp = authed(client().delete(schedule_url(
port,
"delete",
"u/test-user/another_schedule",
)))
.send()
.await
.unwrap();
let resp =
authed(client().delete(schedule_url(port, "delete", "u/test-user/another_schedule")))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let resp = authed_get(port, "exists", "u/test-user/another_schedule").await;
@@ -220,17 +215,108 @@ async fn test_schedule_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
// ===== Global endpoints =====
// --- preview ---
let resp = authed(client().post(format!(
"http://localhost:{port}/api/schedules/preview"
)))
.json(&json!({
"schedule": "0 0 */6 * * *",
"timezone": "UTC"
}))
.send()
.await
.unwrap();
let resp = authed(client().post(format!("http://localhost:{port}/api/schedules/preview")))
.json(&json!({
"schedule": "0 0 */6 * * *",
"timezone": "UTC"
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "preview: {}", resp.text().await?);
Ok(())
}
/// A schedule with no `schedule` row is listed from the `draft` table, so its
/// DELETE drops that draft, then 404s once nothing is left at the path. A legacy
/// (`email IS NULL`) draft is owned by nobody and stays put.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_delete_draft_only_schedule(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let path = "u/test-user/draft_only_schedule";
let resp = authed(client().post(format!(
"http://localhost:{port}/api/w/test-workspace/drafts/update/trigger_schedule/{path}"
)))
.json(&json!({ "value": {
"path": path,
"schedule": "0 0 */6 * * *",
"timezone": "UTC",
"script_path": "u/test-user/never_deployed",
"is_flow": false,
}}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "save draft: {}", resp.text().await?);
let resp = authed(client().get(format!(
"http://localhost:{port}/api/w/test-workspace/schedules/list?include_draft_only=true"
)))
.send()
.await
.unwrap();
let listed: Vec<serde_json::Value> = resp.json().await?;
assert!(
listed
.iter()
.any(|s| s["path"] == path && s["draft_only"] == json!(true)),
"draft-only schedule should be listed: {listed:?}"
);
let resp = authed(client().delete(schedule_url(port, "delete", path)))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "delete: {}", resp.text().await?);
let remaining: i64 = sqlx::query_scalar(
"SELECT count(*) FROM draft WHERE workspace_id = 'test-workspace' AND path = $1 \
AND typ = 'trigger_schedule'::DRAFT_KIND",
)
.bind(path)
.fetch_one(&db)
.await?;
assert_eq!(remaining, 0, "the draft should be gone");
let resp = authed(client().delete(schedule_url(port, "delete", path)))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 404, "nothing left at the path");
// A legacy (email IS NULL) draft is owned by nobody and keeps the write gate
// on the drafts routes, so this one must not become a second door to it.
let legacy_path = "u/test-user/legacy_draft_only_schedule";
sqlx::query(
"INSERT INTO draft (workspace_id, email, path, typ, value) \
VALUES ('test-workspace', NULL, $1, 'trigger_schedule'::DRAFT_KIND, '{}'::json)",
)
.bind(legacy_path)
.execute(&db)
.await?;
let resp = authed(client().delete(schedule_url(port, "delete", legacy_path)))
.send()
.await
.unwrap();
assert_eq!(
resp.status(),
404,
"legacy draft is not this route's to delete"
);
let legacy_remaining: i64 = sqlx::query_scalar(
"SELECT count(*) FROM draft WHERE workspace_id = 'test-workspace' AND path = $1 \
AND typ = 'trigger_schedule'::DRAFT_KIND",
)
.bind(legacy_path)
.fetch_one(&db)
.await?;
assert_eq!(legacy_remaining, 1, "the legacy draft should survive");
Ok(())
}
backend/windmill-api-schedule/src/lib.rs
+14 -2
View File
@@ -31,8 +31,8 @@ use windmill_common::{
self, TriggerHistoryEvent, TriggerOperation, TriggerSource, SCHEDULE_TRIGGER_KIND,
},
user_drafts::{
delete_all_drafts_for_path, fetch_draft_only_list_rows, overlay_or_draft_only,
UserDraftItemKind, WithDraftOverlay, WithDraftQuery,
delete_all_drafts_for_path, delete_draft_only_for_path, fetch_draft_only_list_rows,
overlay_or_draft_only, UserDraftItemKind, WithDraftOverlay, WithDraftQuery,
},
utils::{
escape_ilike_pattern, not_found_if_none, paginate, Pagination, ScheduleType, StripPath,
@@ -1331,6 +1331,18 @@ async fn delete_schedule(
.flatten();
if exists.is_none() {
drop(tx);
if delete_draft_only_for_path(
&db,
&w_id,
UserDraftItemKind::TriggerSchedule,
path,
&authed.email,
)
.await?
{
return Ok(format!("Draft-only schedule {} deleted", path));
}
return Err(windmill_common::error::Error::NotFound(format!(
"Schedule {} not found",
path
backend/windmill-api-scripts/src/scripts.rs
+203 -58
View File
@@ -39,8 +39,8 @@ use sqlx::{FromRow, Postgres, Transaction};
use std::{collections::HashMap, sync::Arc};
use windmill_audit::audit_oss::{audit_log, AuditAuthorable};
use windmill_audit::ActionKind;
use windmill_dep_map::{lock_hash::record_lock_hashes, process_relative_imports};
use windmill_dep_map::scoped_dependency_map::ScopedDependencyMap;
use windmill_dep_map::{lock_hash::record_lock_hashes, process_relative_imports};
use windmill_common::{
assets::{
@@ -1081,6 +1081,57 @@ fn lock_hash_entry(path: &str, lock: &str) -> [(String, i64); 1] {
[(path.to_string(), hash_script(lock))]
}
/// The `dbt://` relation both halves of a deploy have to agree on: a whole
/// `<warehouse>/<schema>/<name>`, under a warehouse this workspace configures.
///
/// Every producer is held to exactly this — a `// materialize` target here, a
/// descriptor's `profile.warehouse` in the worker — so a subscription to anything
/// else names a relation nothing can ever write. No later deploy fixes that and
/// no dormant-edge warning reports it, since the warning fires on a dbt project's
/// ingest and no project can claim a relation under a warehouse that isn't there.
/// Asking here rather than at each site is what keeps the two from drifting into
/// refusing and accepting the same string.
async fn validate_dbt_relation(
db: &sqlx::Pool<Postgres>,
w_id: &str,
relation: &str,
what: &str,
) -> Result<()> {
if !windmill_parser::asset_parser::is_full_relation_path(relation) {
return Err(Error::BadRequest(format!(
"{what} `dbt://{relation}` is not a whole warehouse relation \
(`dbt://<warehouse>/<schema>/<name>`)."
)));
}
// `asset.path` is VARCHAR(255) and the manifest ingest drops a relation that
// outgrows it rather than failing the whole graph, so past the column no
// producer row can exist on either side — a write would be rejected by
// Postgres mid-deploy, and `script_trigger.trigger_ref` is unbounded text
// that would take the subscription and keep it dormant for good.
let max = windmill_common::dbt_manifest::MAX_ASSET_PATH_LEN;
if relation.chars().count() > max {
return Err(Error::BadRequest(format!(
"{what} `dbt://{relation}` is longer than the {max} characters an asset path \
holds, so it cannot be recorded."
)));
}
let warehouse = relation.split('/').next().unwrap_or_default();
// Only the resolver's own "no such warehouse" is the annotation's fault. Its
// other failures — the query, and a setting entry with no `resource_path` —
// keep their own error: blaming the warehouse name for those misdescribes
// them, and flattening the malformed-setting one to a 400 hides a server
// fault behind a client one.
windmill_common::workspaces::dbt_warehouse_exists(db, w_id, warehouse)
.await
.map_err(|e| match e {
Error::NotFound(_) => Error::BadRequest(format!(
"{what} `dbt://{relation}` names a warehouse this workspace does not \
configure: {e}"
)),
other => other,
})
}
async fn create_script_internal<'c>(
mut ns: NewScript,
w_id: String,
@@ -1565,34 +1616,90 @@ async fn create_script_internal<'c>(
// membership; parsed writes tell us what is produced (we don't record
// them in auto_kind itself).
let pipeline_annotations = parse_pipeline_annotations(&ns.content);
// `// materialize` materializes a `ducklake://<name>/<table>` target from a
// DuckDB script. These two constraints hold for *both* modes: a non-DuckLake
// target would otherwise deploy, register a producer in the asset graph, then
// silently no-op at run time (`build_materialized_query` returns `Ok(None)`),
// and a non-DuckDB script never reaches the executor that records state. The
// managed-only checks (single trailing SELECT, no SQL args) come after — a
// `manual` script owns its DDL and skips them.
// `// materialize` names what this script produces. Two target kinds, and the
// runtime behind each is what constrains the annotation:
// • `ducklake://<name>/<table>` — the DuckDB executor generates the write
// (or, in `manual` mode, records the state the script wrote itself), so
// the script has to be a DuckDB one and the target has to name a table.
// A non-DuckDB script never reaches that executor.
// • `dbt://<warehouse>/<schema>/<name>` — a warehouse relation. Nothing
// generates warehouse DDL, so the declaration is track-only (`manual`)
// and any language but dbt's own may make it: the script writes the
// relation, the worker records the materialization, and the relation's
// asset node is shared with whatever dbt model reads it. A dbt project's
// own writes are read from its manifest, so it may not declare one.
// Any other kind would deploy, register a producer in the asset graph, then
// silently no-op at run time (`build_materialized_query` returns `Ok(None)`).
// The managed-only checks (single trailing SELECT, no SQL args) come after —
// a `manual` script owns its DDL and skips them.
if let Some(m) = pipeline_annotations.materialize.as_ref() {
if ns.language != ScriptLang::DuckDb {
return Err(Error::BadRequest(format!(
"`// materialize` is only supported for DuckDB scripts, not {}. Use the \
wmll.ducklake helpers to materialize from other languages.",
ns.language.as_str()
)));
}
if m.target_kind != windmill_parser::asset_parser::AssetKind::Ducklake {
use windmill_parser::asset_parser::AssetKind as PAssetKind;
// The producer half of the rule the trigger loop below applies to `// on`:
// a dbt project's writes come from its manifest, and the graph ingest
// republishes this path's asset rows wholesale, so a declared one would be
// wiped by the very deploy that accepted it while its runs kept stamping
// the relation.
if ns.language == ScriptLang::Dbt {
return Err(Error::BadRequest(
"`// materialize` only supports a DuckLake target \
(`ducklake://<name>/<table>`); other asset kinds aren't materializable."
"a dbt script cannot declare `// materialize`: what a project builds is read \
from its manifest and published by the graph ingest, not annotated."
.to_string(),
));
}
if !m.target_path.contains('/') {
return Err(Error::BadRequest(format!(
"`// materialize` needs a table in the target: \
`ducklake://{0}/<table>` (got `ducklake://{0}`).",
m.target_path
)));
match m.target_kind {
PAssetKind::Ducklake => {
if ns.language != ScriptLang::DuckDb {
return Err(Error::BadRequest(format!(
"`// materialize` is only supported for DuckDB scripts, not {}. Use the \
wmll.ducklake helpers to materialize from other languages, or declare a \
warehouse relation with `// materialize manual dbt://…`.",
ns.language.as_str()
)));
}
if !m.target_path.contains('/') {
return Err(Error::BadRequest(format!(
"`// materialize` needs a table in the target: \
`ducklake://{0}/<table>` (got `ducklake://{0}`).",
m.target_path
)));
}
}
PAssetKind::Dbt => {
// `// data_test` runs as verifier probes the DuckDB executor
// splices around a MANAGED write. Nothing generates a warehouse
// write, so nothing would run them — and unlike the DuckLake
// `manual` case, which at least fails loudly in that executor, a
// declarer in another language would deploy green with its
// data-quality assertions silently never executed.
if !pipeline_annotations.data_tests.is_empty() {
return Err(Error::BadRequest(
"`// data_test` is not supported with a `dbt://` target: the checks run \
against a managed materialization, and a warehouse relation is \
written by the script itself. Assert on the relation with a dbt \
test in the project that reads it."
.to_string(),
));
}
if !m.manual {
return Err(Error::BadRequest(
"`// materialize dbt://…` must be `manual`: Windmill generates no \
warehouse DDL, so the script issues its own write and only the outcome \
is recorded. Write \
`// materialize manual dbt://<warehouse>/<schema>/<name>`."
.to_string(),
));
}
validate_dbt_relation(&db, &w_id, &m.target_path, "`// materialize` target")
.await?;
}
_ => {
return Err(Error::BadRequest(
"`// materialize` only supports a DuckLake (`ducklake://<name>/<table>`) or \
warehouse-relation (`dbt://<warehouse>/<schema>/<name>`) target; other asset \
kinds aren't materializable."
.to_string(),
));
}
}
if !m.manual {
if let Err(e) = windmill_parser::sql_materialize::classify_wrap(&ns.content) {
@@ -2347,27 +2454,28 @@ async fn create_script_internal<'c>(
// while its own finished runs still render from them. Clearing by path
// would empty those run pages for good.
if ns.language != ScriptLang::Dbt {
// The saved retry state does go: nothing regenerates it, it is keyed by
// path alone, and it carries one user's failed invocation and its
// arguments. No dbt version is live at this path any more to resume it.
windmill_common::dbt_manifest::clear_dbt_run_state(&mut tx, &w_id, &ns.path).await?;
// The saved run and environment state do go: nothing regenerates them,
// both are keyed by path alone, and they carry one user's failed
// invocation with its arguments and the project's own manifest. No dbt
// version is live at this path any more to resume or defer to.
windmill_common::dbt_manifest::clear_dbt_script_state(&mut tx, &w_id, &ns.path).await?;
}
if let Some(ref old) = p_path_opt {
if old != &ns.path {
clear_script_triggers(&mut *tx, &w_id, old, AssetUsageKind::Script).await?;
clear_static_asset_usage(&mut *tx, &w_id, old, AssetUsageKind::Script).await?;
// The saved retry state travels rather than being cleared: nothing
// The saved state travels rather than being cleared: nothing
// regenerates it, so dropping it would throw away a resumable
// failure for what is only a rename. Only while the destination is
// still dbt — a rename that also converts the language would
// otherwise reinstate at the new path the state the branch above
// just cleared, leaving one user's arguments and results under a
// path no dbt script occupies.
// failure and every deferral until the next full run, for what is
// only a rename. Only while the destination is still dbt — a rename
// that also converts the language would otherwise reinstate at the
// new path the state the branch above just cleared, leaving one
// user's arguments and results under a path no dbt script occupies.
if ns.language == ScriptLang::Dbt {
windmill_common::dbt_manifest::move_dbt_run_state(&mut tx, &w_id, old, &ns.path)
windmill_common::dbt_manifest::move_dbt_script_state(&mut tx, &w_id, old, &ns.path)
.await?;
} else {
windmill_common::dbt_manifest::clear_dbt_run_state(&mut tx, &w_id, old).await?;
windmill_common::dbt_manifest::clear_dbt_script_state(&mut tx, &w_id, old).await?;
}
}
}
@@ -2375,16 +2483,43 @@ async fn create_script_internal<'c>(
let Some((trigger_kind, trigger_ref)) = trigger_spec_to_row(spec) else {
continue;
};
// A `dbt://` subscription can never fire: dbt is the only producer of a
// warehouse relation (`// materialize` takes DuckLake targets only) and a
// dbt run does not dispatch. Refusing beats persisting a row that draws a
// cascade arrow on the canvas and then never wakes anything.
if trigger_ref.starts_with("dbt://") {
return Err(Error::BadRequest(format!(
"`{trigger_ref}` cannot be subscribed to: a dbt run does not trigger downstream \
runs, and nothing else writes a warehouse relation. Declare the read without \
`on` to keep the lineage edge, or schedule this script."
)));
// A `dbt://` subscription fires only when a NON-dbt job materialized the
// relation: `// materialize manual dbt://…` declares such a write, while a
// dbt run records its models and does not dispatch. So refuse exactly the
// edge that cannot fire — one whose relation is already claimed by dbt and
// by nothing else — rather than every `dbt://` edge (`sole_dbt_producer`,
// which takes the workspace pool: under RLS an unreadable native producer
// would refuse a live subscription).
if let Some(relation) = trigger_ref.strip_prefix("dbt://") {
// The subscriber side of the same rule: a dbt project is not woken by
// the asset cascade. Its graph ingest clears these rows for its own
// path, so accepting one here would deploy an edge the dependency job
// then silently removes.
if ns.language == ScriptLang::Dbt {
return Err(Error::BadRequest(format!(
"a dbt script cannot subscribe to `{trigger_ref}`: dbt orders its own DAG \
and a project is run on its schedule, not woken by an asset cascade."
)));
}
validate_dbt_relation(&db, &w_id, relation, "subscription target").await?;
// Both paths under a rename: the old one's committed write row is
// still there and this transaction is about to remove it.
let deploying_paths = match p_path_opt.as_deref().filter(|old| *old != ns.path) {
Some(old) => vec![ns.path.clone(), old.to_string()],
None => vec![ns.path.clone()],
};
if let Some(dbt_owner) =
windmill_common::assets::sole_dbt_producer(&db, &w_id, relation, &deploying_paths)
.await?
{
return Err(Error::BadRequest(format!(
"`{trigger_ref}` cannot be subscribed to: it is built by the dbt project at \
`{dbt_owner}`, and a dbt run does not trigger downstream runs. Declare the \
read without `on` to keep the lineage edge, or schedule this script. A \
relation written by a `// materialize manual {trigger_ref}` script can be \
subscribed to."
)));
}
}
// Effective debounce for this edge: per-`// on debounce=` wins,
// else the script-level `// debounce` default. Debounce only
@@ -3578,7 +3713,11 @@ async fn archive_script_by_path(
path,
&w_id
)
.fetch_one(&db)
// In the SAME transaction as the cleanup below, as the by-hash routes are:
// committed on its own, a cleanup that then fails leaves dbt state at a path
// no live version occupies, for whatever is created there next to defer
// through.
.fetch_one(&mut *tx)
.await
.map_err(|e| Error::internal_err(format!("archiving script in {w_id}: {e:#}")))?;
@@ -3586,9 +3725,10 @@ async fn archive_script_by_path(
// The graph stays: the pinned read resolves versions through a CTE that
// already skips archived rows, so it stops answering for current relations
// either way, while deleting it would empty the Models panel of every
// completed run of the project. Retry state does go — nothing may resume a
// script that is no longer live.
windmill_common::dbt_manifest::clear_dbt_run_state(&mut tx, &w_id, path).await?;
// completed run of the project. The saved run and environment state do go —
// nothing may resume a script that is no longer live, and nothing may defer
// through what it last built.
windmill_common::dbt_manifest::clear_dbt_script_state(&mut tx, &w_id, path).await?;
// Pipeline event hygiene: an archived script must not be triggered by
// anything. Wipe declared `// on ...` edges (asset-event subscribers
// look these up).
@@ -3673,7 +3813,7 @@ async fn archive_script_by_hash(
clear_static_asset_usage_by_script_hash(&mut *tx, &w_id, hash).await?;
// The version's graph stays: its finished runs still render from it, and
// the live-version CTE already skips archived rows. Deletion clears it.
windmill_common::dbt_manifest::clear_dbt_run_state_if_path_retired(
windmill_common::dbt_manifest::clear_dbt_script_state_if_path_retired(
&mut tx,
&w_id,
&script.path,
@@ -3736,7 +3876,12 @@ async fn delete_script_by_hash(
)
.bind(&hash.0)
.bind(&w_id)
.fetch_one(&db)
// In the SAME transaction as the cleanup below, as `archive_script_by_hash`
// already does. Committed on its own, it opens a window where the path has
// no live version and a concurrent deploy can take it — and the retirement
// guard below then finds that new script live, keeps the old project's dbt
// state, and leaves the replacement able to defer through its manifest.
.fetch_one(&mut *tx)
.await
.map_err(|e| Error::internal_err(format!("deleting script by hash {w_id}: {e:#}")))?;
@@ -3749,7 +3894,7 @@ async fn delete_script_by_hash(
windmill_common::dbt_manifest::clear_dbt_manifest_version(&mut tx, &w_id, &script.path, hash.0)
.await?;
clear_static_asset_usage_by_script_hash(&mut *tx, &w_id, hash).await?;
windmill_common::dbt_manifest::clear_dbt_run_state_if_path_retired(
windmill_common::dbt_manifest::clear_dbt_script_state_if_path_retired(
&mut tx,
&w_id,
&script.path,
@@ -3850,11 +3995,11 @@ async fn delete_script_by_path(
// After the DELETE, never before: every dbt writer locks the `script` row
// first, so taking a sidecar ahead of it deadlocks one of the pair. The
// VERSIONED graph needs no clear at all, cascading off `script`; the retry
// state does, being keyed by path alone and so inherited by whatever is
// created here next, and so do the editor's own graphs, whose NULL
// VERSIONED graph needs no clear at all, cascading off `script`; the saved
// run and environment state do, being keyed by path alone and so inherited
// by whatever is created here next, and so do the editor's own graphs, whose NULL
// `script_hash` satisfies that foreign key without riding its cascade.
windmill_common::dbt_manifest::clear_dbt_run_state(&mut tx, &w_id, path).await?;
windmill_common::dbt_manifest::clear_dbt_script_state(&mut tx, &w_id, path).await?;
windmill_common::dbt_manifest::clear_dbt_editor_graphs(&mut tx, &w_id, path).await?;
if !trash_scripts.is_empty() {
@@ -4023,7 +4168,7 @@ async fn delete_scripts_bulk(
// Same reason as the single-path delete, over every requested path rather
// than the deleted ones: a path that had no script left can still hold state.
for p in &request.paths {
windmill_common::dbt_manifest::clear_dbt_run_state(&mut tx, &w_id, p).await?;
windmill_common::dbt_manifest::clear_dbt_script_state(&mut tx, &w_id, p).await?;
windmill_common::dbt_manifest::clear_dbt_editor_graphs(&mut tx, &w_id, p).await?;
}
backend/windmill-api-workspaces/src/workspaces.rs
+21 -3
View File
@@ -6627,7 +6627,7 @@ async fn clone_scripts(
}
/// The parsed dbt graph a deployed script carries: its models, their SQL and
/// tests, and the `ref()` lineage between them.
/// tests, and the `ref()` and column-level lineage between them.
///
/// Keyed on (workspace_id, script_path, script_hash), and the fork keeps every
/// script's hash, so each row moves across as itself.
@@ -6645,11 +6645,11 @@ async fn clone_dbt_graph(
"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,
resource_type, name, asset_path, materialized, materialize_strategy, unique_key,
tags, description, test_kind, test_column, test_args, severity, attached_node,
columns, freshness, raw_code, original_file_path, ingested_at)
columns, column_schema, freshness, raw_code, original_file_path, ingested_at)
SELECT $2, script_path, script_hash, job_id, unique_id,
resource_type, name, asset_path, materialized, materialize_strategy, unique_key,
tags, description, test_kind, test_column, test_args, severity, attached_node,
columns, freshness, raw_code, original_file_path, ingested_at
columns, column_schema, freshness, raw_code, original_file_path, ingested_at
FROM dbt_node
WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'",
source_workspace_id,
@@ -6669,6 +6669,24 @@ async fn clone_dbt_graph(
)
.execute(&mut **tx)
.await?;
// Column lineage travels with the rest of the graph, and it has to: the
// snapshot's digest covers it, so a fork missing these rows recomputes the
// digest the source stored, matches, and stores nothing — leaving the
// lineage gone until someone redeploys, which is the failure this whole
// function exists to prevent.
sqlx::query!(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id,
parent_unique_id, parent_column, child_unique_id, child_column, lineage_kind,
ingested_at)
SELECT $2, script_path, script_hash, job_id, parent_unique_id, parent_column,
child_unique_id, child_column, lineage_kind, ingested_at
FROM dbt_column_edge
WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'",
source_workspace_id,
target_workspace_id
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id,
digest, relation_root_at_last_ingest, ingested_at)
backend/windmill-api/openapi.yaml
+238 -2
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.804.0
version: 1.805.0
title: Windmill API
contact:
@@ -8528,6 +8528,84 @@ paths:
items:
type: string
/w/{workspace}/resources/type/resource_counts:
get:
summary: count the workspace's resources per resource type
operationId: listResourceCountsByType
tags:
- resource
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: resource count per resource_type
content:
application/json:
schema:
type: array
items:
type: object
properties:
resource_type:
type: string
count:
type: integer
required:
- resource_type
- count
/w/{workspace}/resources/type/hub/info:
get:
summary: list what the hub knows about its resource types
operationId: listHubResourceTypeInfo
tags:
- resource
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: each hub resource type with its integration and pick count, empty if the hub answers neither read
content:
application/json:
schema:
type: array
items:
type: object
properties:
name:
type: string
app:
description: the integration the resource type belongs to, which is not always its own name
type: string
picks:
type: integer
required:
- name
- app
- picks
/w/{workspace}/resources/type/hub/pick/{name}:
post:
summary: record a hub resource type pick
operationId: pickHubResourceType
tags:
- resource
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- $ref: "#/components/parameters/Name"
responses:
"200":
description: whether the hub recorded the pick
content:
application/json:
schema:
type: object
properties:
success:
type: boolean
required:
- success
/w/{workspace}/npm_proxy/config:
get:
summary: get npm proxy configuration
@@ -8777,6 +8855,9 @@ paths:
properties:
name:
type: string
picks:
description: how often the integration has been picked, absent on a hub that does not count picks
type: integer
required:
- name
@@ -24364,6 +24445,68 @@ paths:
schema:
$ref: "#/components/schemas/AssetGraph"
/w/{workspace}/assets/column_lineage:
get:
summary: Column-level lineage of a set of dbt relations
description: >
The direct (`copy` / `mod`) column-to-column lineage the given relations'
columns sit in — the connected component around them, from the engine's
static analysis. Not their own edges, which would stop one hop out since
a column trace walks transitively, and not a whole project's, which
carries model families the selection cannot reach.
Several relations, answered as one union, because one selection reaches
several: a script's output column can derive from columns of several dbt
models. Unpinned, the component crosses projects — a relation one project
produces is another's source — and the caller's access is decided again
for every project it reaches, so a trace ends where their grants do. A
pinned answer, by version here or by job on the run route, is one
project's.
Its own endpoint rather than a field on the asset graph: the graph is
folder-wide and polled by a run page, while this is rendered for one
selection at a time. Empty for projects that did not opt into the
analysis pass (`column_lineage: true`), which is the ordinary case. The
indirect `scan` kind is stored but never served: it reaches every output
column of its model.
operationId: getDbtColumnLineage
tags:
- asset
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: asset_path
in: query
required: true
description: >
The `dbt://` relations whose lineage to return. Repeated, once per
relation, and answered as one union.
schema:
type: array
items:
type: string
- name: dbt_script_hash
in: query
description: >
The deployed version a view is drawing, when it is drawing one — the
dbt editor, which shows a single project as of a single deploy. A
version-pinned answer is that version's project alone, the same as a
job-pinned one, and only the unpinned answer crosses projects: a pin
says which stored graph is on screen, and another project's live
graph is not part of it.
A run's or an editor buffer's own graph is not reachable here: that
pins to a job, and costs the job-read gate — see
`jobs/dbt_column_lineage/{id}`.
schema:
type: string
responses:
"200":
description: the relations' column-level lineage
content:
application/json:
schema:
$ref: "#/components/schemas/DbtColumnLineage"
/w/{workspace}/assets/macros:
get:
summary: List every workspace DuckDB macro (deployed `// macros` libraries)
@@ -24555,6 +24698,49 @@ paths:
schema:
$ref: "#/components/schemas/AssetGraph"
/w/{workspace}/jobs/dbt_column_lineage/{id}:
get:
summary: Get relations' project column lineage as one run saw it
description: >
The same answer as `assets/column_lineage`, for the project version a
single job ran — including the dbt editor's parse of its own buffer,
whose graph belongs to that job and is reachable no other way. One
project answers here, the one the run is of, since the graph this
annotates is that project's too. Authorized through the job, the same
gate as `dbt_graph`. Reaching the run is not on its own enough to read
the project: a caller with no access to the script gets its relations and
`ref()` edges from `dbt_graph` and an empty answer here, exactly as that
endpoint redacts the model's SQL.
operationId: getDbtRunColumnLineage
tags:
- job
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: id
in: path
required: true
description: The job whose graph the lineage is read from
schema:
type: string
format: uuid
- name: asset_path
in: query
required: true
description: >
The `dbt://` relations whose lineage to return. Repeated, once per
relation, and answered as one union.
schema:
type: array
items:
type: string
responses:
"200":
description: the relations' column-level lineage
content:
application/json:
schema:
$ref: "#/components/schemas/DbtColumnLineage"
/w/{workspace}/jobs/run_progress/{id}:
get:
summary: List the per-relation progress one job has recorded so far
@@ -26169,6 +26355,40 @@ components:
drawn identically and the ambiguity would otherwise just move into
the editor. Omitted for the unpinned workspace graph, which spans
every project and so has no one time.
DbtColumnLineage:
type: object
description: >-
The direct column-to-column lineage the asked-for relations' columns sit
in — the connected component around them — in the terms the canvas draws:
relations and columns, never dbt's node ids.
required: [edges, truncated]
properties:
edges:
type: array
items:
type: object
required: [from_asset_path, from_column, to_asset_path, to_column, kind]
properties:
from_asset_path:
type: string
from_column:
type: string
to_asset_path:
type: string
to_column:
type: string
kind:
type: string
description: >-
dbt's own word for how the value travelled — `copy`
(passthrough) or `mod` (transformed). Not an enum: the engine
treats the set as open.
truncated:
type: boolean
description: >-
The component reaches further than `edges`, which holds the part
nearest the asked-for relations. A trace that stops short is
otherwise indistinguishable from one that ends.
DbtAssetProvenance:
type: object
description: >-
@@ -26216,7 +26436,23 @@ components:
columns:
type: object
additionalProperties: true
description: Declared column metadata (name -> description). NOT column lineage — `manifest.json` carries none.
description: Declared column metadata (name -> description) — what `manifest.json` carries, which is only the columns an author wrote down. Omitted when the caller cannot read the script.
column_schema:
type: array
description: >-
Every column of the relation, typed and in the order the model
produces them, from the engine's static analysis. Present only for a
project that opted into it, and gated like `columns` and the model's
SQL: a full column list is the shape of what the author wrote.
items:
type: object
required: [name]
properties:
name:
type: string
type:
type: string
description: The declared type where `schema.yml` gives one, else the inferred one. Omitted when neither is known.
freshness:
type: object
additionalProperties: true
backend/windmill-api/src/apps.rs
+3
View File
@@ -5825,6 +5825,9 @@ mod embed_token_tests {
"GET",
),
("/api/w/test/resources/list_search", "GET"),
// The metadata allowlist matches `resources/type/` by prefix, so a route
// added under it that is not a read must be denied by its method.
("/api/w/test/resources/type/hub/pick/slack", "POST"),
// Workspace-wide job enumeration/export must NOT be reachable — an app
// reads only jobs it launched, by id (blocked via the app_embed sentinel).
("/api/w/test/jobs/list", "GET"),
backend/windmill-api/src/jobs.rs
+61 -26
View File
@@ -139,6 +139,7 @@ pub fn workspaced_service() -> Router {
.route("/run_progress/{id}", get(get_run_progress))
.route("/run_assets/{id}", get(list_run_assets))
.route("/dbt_graph/{id}", get(get_dbt_run_graph))
.route("/dbt_column_lineage/{id}", get(get_dbt_run_column_lineage))
.route("/dbt_resumable/{id}", get(get_dbt_resumable))
.route(
"/dbt_resumable_script/p/{*script_path}",
@@ -891,21 +892,27 @@ struct AssetProgress {
error: Option<String>,
}
/// The asset graph as one run saw it. Pinning to a job needs the full job-read
/// contract, so it lives on `require_job_read_access` here rather than as a
/// parameter on `/assets/graph`. See docs/dbt-runtime.md.
async fn get_dbt_run_graph(
authed: ApiAuthed,
OptViewToken(view_token): OptViewToken,
Extension(db): Extension<DB>,
Extension(user_db): Extension<UserDB>,
Path((w_id, job_id)): Path<(String, Uuid)>,
Query(q): Query<windmill_api_assets::GraphQuery>,
) -> error::JsonResult<windmill_api_assets::AssetGraphResponse> {
/// Which project version a dbt view pins to for this job, once the caller has
/// been shown to be entitled to it.
///
/// `Ok(None)` is "answer unpinned", not a refusal: a job that stored no graph of
/// its own — and one that has aged out of retention — is served the deployed
/// version rather than an error, so a run page keeps drawing after the run is
/// gone. Pinning needs the full job-read contract, which is why it lives on
/// `require_job_read_access` here rather than as a parameter on `/assets/*`.
/// See docs/dbt-runtime.md.
async fn dbt_pinned_run(
authed: &ApiAuthed,
db: &DB,
user_db: &UserDB,
w_id: &str,
job_id: Uuid,
view_token: Option<&str>,
) -> error::Result<Option<windmill_api_assets::PinnedRun>> {
// The scope domain comes from the URL segment, so `/jobs` asks a scoped token
// for `jobs:read` alone while the body returned is asset data. Both are
// required: the job gate below reaches this run, this reaches assets at all.
check_scopes(&authed, || "assets:read".to_string())?;
check_scopes(authed, || "assets:read".to_string())?;
let job = sqlx::query!(
r#"SELECT created_by, runnable_path,
CASE WHEN kind = 'script' THEN runnable_id END AS script_hash,
@@ -918,42 +925,70 @@ async fn get_dbt_run_graph(
AND g.script_hash IS NULL) AS "editor_graph!"
FROM v2_job WHERE id = $1 AND workspace_id = $2"#,
job_id,
&w_id
w_id
)
.fetch_optional(&db)
.fetch_optional(db)
.await?;
// No such job: answer the unpinned graph rather than 404, so a run page whose
// job has aged out of retention still draws the deployed version instead of
// an error. Reachable only with `assets:read`, which is exactly what
// `/assets/graph` would have cost for the same answer.
// Unpinned rather than 404 for a job that is gone. Reachable only with
// `assets:read`, which is exactly what the unpinned route would have cost
// for the same answer.
let Some(job) = job else {
return windmill_api_assets::asset_graph_for(&authed, &w_id, user_db, db, q, None).await;
return Ok(None);
};
require_job_read_access(
&db,
&user_db,
&authed,
&w_id,
db,
user_db,
authed,
w_id,
&job_id,
&job.created_by,
view_token.as_deref(),
view_token,
)
.await?;
// A preview or flow job names no deployed version, so there is usually no
// graph to pin to and the workspace one answers. The exception is a job that
// parsed one itself, which is what the dbt editor's refresh is: its graph
// belongs to that job alone and nothing else can reach it.
let pinned = job
Ok(job
.runnable_path
.filter(|_| job.script_hash.is_some() || job.editor_graph)
.map(|path| windmill_api_assets::PinnedRun {
job_id,
script_path: path,
script_hash: job.script_hash,
});
}))
}
/// The asset graph as one run saw it.
async fn get_dbt_run_graph(
authed: ApiAuthed,
OptViewToken(view_token): OptViewToken,
Extension(db): Extension<DB>,
Extension(user_db): Extension<UserDB>,
Path((w_id, job_id)): Path<(String, Uuid)>,
Query(q): Query<windmill_api_assets::GraphQuery>,
) -> error::JsonResult<windmill_api_assets::AssetGraphResponse> {
let pinned =
dbt_pinned_run(&authed, &db, &user_db, &w_id, job_id, view_token.as_deref()).await?;
windmill_api_assets::asset_graph_for(&authed, &w_id, user_db, db, q, pinned).await
}
/// The column lineage a set of relations sits in as one run saw it — the same
/// pin as `get_dbt_run_graph`, for the trace drawn beside a node of that graph.
async fn get_dbt_run_column_lineage(
authed: ApiAuthed,
OptViewToken(view_token): OptViewToken,
Extension(db): Extension<DB>,
Extension(user_db): Extension<UserDB>,
Path((w_id, job_id)): Path<(String, Uuid)>,
Query(pairs): Query<Vec<(String, String)>>,
) -> error::JsonResult<windmill_api_assets::ColumnLineageResponse> {
let q = windmill_api_assets::ColumnLineageQuery::from_query_pairs(pairs)?;
let pinned =
dbt_pinned_run(&authed, &db, &user_db, &w_id, job_id, view_token.as_deref()).await?;
windmill_api_assets::dbt_column_lineage_for(&authed, &w_id, user_db, q, pinned).await
}
/// Whether a `dbt retry` submitted by this caller would resume THIS run.
///
/// One failure is saved per script per execution principal, so a page showing an
backend/windmill-api/src/workspaces_export.rs
+5 -6
View File
@@ -346,7 +346,7 @@ pub(crate) struct ArchiveQueryParams {
default_ts: Option<String>,
/// Settings format version: "v1" (default) returns legacy flat format, "v2" returns grouped format
settings_version: Option<String>,
/// Opt-in: include `extra_perms` on flow / script / app rows. Default `false`
/// Opt-in: include `extra_perms` on script / flow / app / variable rows. Default `false`
/// so cross-workspace tarball imports do not carry over ACLs referring to
/// identities that may not exist in the target workspace. `wmill sync pull`
/// passes `true` to surface ACLs in the git-tracked yaml.
@@ -365,8 +365,8 @@ pub(crate) struct ArchiveQueryParams {
/// pre-existing serialization for folders and groups so
/// no customer sees a one-time noisy diff on upgrade.
/// * `KeepIfNonEmpty` — keep when there is at least one entry, drop when `{}`
/// or null. New surface for flow / script / app, which
/// never carried ACLs in source before this change.
/// or null. New surface for script / flow / app / variable,
/// which never carried ACLs in source before this change.
#[derive(Clone, Copy)]
pub enum ExtraPermsBehavior {
Drop,
@@ -665,7 +665,7 @@ pub(crate) async fn tarball_workspace(
check_scopes(&authed, || "variables:read".to_string())?;
}
// Opt-in behavior for surfacing per-resource ACLs on flow/app rows.
// Opt-in behavior for surfacing per-resource ACLs on script/flow/app/variable rows.
// Folder and group rows have always carried `extra_perms` in source and
// continue to do so unconditionally (`KeepEvenEmpty`) so existing
// customer git repos see no one-time noisy diff.
@@ -1002,8 +1002,7 @@ pub(crate) async fn tarball_workspace(
Error::internal_err(format!("Error decrypting variable {}: {}", var.path, e))
})?);
}
let var_str =
&to_string_without_metadata(&var, ExtraPermsBehavior::Drop, None).unwrap();
let var_str = &to_string_without_metadata(&var, new_kinds_extra_perms, None).unwrap();
archive
.write_to_archive(&var_str, &format!("{}.variable.json", var.path))
.await?;
backend/windmill-common/src/assets.rs
+134 -4
View File
@@ -175,10 +175,12 @@ fn is_write_access(access: Option<AssetUsageAccessType>) -> bool {
/// producers). Resource / datatable / volume reads stay explicit-`// on`:
/// a config/lookup read cascading is more often surprising than wanted.
fn is_auto_trigger_kind(kind: AssetKind) -> bool {
// `Dbt` is deliberately NOT here. dbt is the only thing that can produce a
// warehouse relation (`// materialize` takes DuckLake targets only) and a dbt
// run does not dispatch, so a derived `dbt://` edge could never fire — it
// would draw a cascade arrow into a script nothing can wake.
// `Dbt` is deliberately NOT here. A warehouse relation is usually built by
// the dbt project that reads it, and a dbt run does not dispatch, so deriving
// an edge from every `dbt://` read would draw cascade arrows that mostly never
// fire. The relations a native `// materialize manual dbt://…` script writes
// do wake subscribers, but only through an explicit `// on`, which is where
// the author states that this particular relation has such a producer.
matches!(kind, AssetKind::Ducklake | AssetKind::S3Object)
}
@@ -235,6 +237,134 @@ pub fn derive_pipeline_asset_trigger_refs(
out
}
/// A dbt script that builds the `dbt://` relation at `asset_path`, when dbt is
/// its ONLY producer.
///
/// That is the one shape in which subscribing to a warehouse relation can never
/// be woken: a dbt run records the models it built and does not dispatch
/// (`asset_dispatch` returns early for `ScriptLang::Dbt`), while a script that
/// declares `// materialize manual dbt://…` fans out on the ordinary path.
///
/// `None` covers both "some non-dbt script materializes it" and "nothing
/// produces it yet" — the second is the ordinary deploy-order case, identical to
/// every other asset kind, not a dormant edge.
///
/// **Give it the workspace pool, not an RLS-scoped transaction.** `script`
/// carries RLS while `asset` does not, so a scoped executor hides producers, and
/// the hidden ones fail in the harmful direction: a native producer the deployer
/// cannot read leaves a dbt-only set behind and refuses a subscription that would
/// have fired. What it discloses in exchange is the path of a dbt script building
/// a relation the caller already named, which the workspace asset graph hands out
/// for every `dbt://` node anyway (the source that script wrote stays gated).
/// Callers must therefore already be scoped to `workspace_id`.
///
/// `deploying_paths` is excluded from the producer set, and has to be: reading
/// committed rows means the deploying script's own are the version being
/// replaced, so one that just dropped its `// materialize` would still count as a
/// producer and let a now-dormant subscription through. Pass every path this
/// deploy is rewriting — under a rename that is the old path as well as the new
/// one, whose committed write row the transaction is about to remove. Excluding
/// them is free of the opposite error, because a script never wakes its own
/// subscription — the dispatcher skips that as a self-loop.
///
/// A producer another deploy is committing concurrently is invisible either way,
/// and the outcome depends on which side it is. An uncommitted NATIVE producer
/// leaves a dbt-only set and refuses, with a message the user can retry past. An
/// uncommitted DBT one leaves an empty set and accepts — and if that ingest then
/// commits and runs [`dormant_dbt_subscriptions`] before this deploy's trigger
/// row lands, neither side reports the edge it left dormant. Serializing the two
/// is not worth it: they would have to share a per-relation lock, and the ingest
/// takes `script … FOR UPDATE` before its own advisory lock, so a deploy holding
/// relation locks first inverts that order into a deadlock across the two
/// subsystems. The next deploy of that project warns (docs/dbt-runtime.md).
pub async fn sole_dbt_producer<'e>(
executor: impl PgExecutor<'e>,
workspace_id: &str,
asset_path: &str,
deploying_paths: &[String],
) -> error::Result<Option<String>> {
use crate::scripts::ScriptLang;
let producers = sqlx::query!(
r#"SELECT s.path AS "path!", s.language AS "language!: ScriptLang"
FROM asset a
JOIN script s ON s.workspace_id = a.workspace_id AND s.path = a.usage_path
AND s.archived = false AND s.deleted = false
WHERE a.workspace_id = $1 AND a.kind = 'dbt' AND a.path = $2
AND a.usage_kind = 'script' AND a.usage_access_type IN ('w', 'rw')
AND a.usage_path <> ALL($3)"#,
workspace_id,
asset_path,
deploying_paths
)
.fetch_all(executor)
.await?;
if producers
.iter()
.any(|p| !matches!(p.language, ScriptLang::Dbt))
{
return Ok(None);
}
Ok(producers.into_iter().next().map(|p| p.path))
}
/// The set form of [`sole_dbt_producer`], for asking about many relations at
/// once: every `// on dbt://<relation>` edge among `relations` whose producers
/// are all dbt scripts, rendered as `dbt://<relation> → <subscriber path>`.
///
/// A dbt deploy asks this about the relations it just ingested, because that
/// ingest is what can retroactively leave a subscription accepted earlier — when
/// nothing produced the relation — with dbt as its only producer.
///
/// Spells the predicate the same way its singular sibling does, per subscriber:
/// the producer set excludes the subscriber's own path (a script never wakes
/// itself) and has to be non-empty (nothing produces it yet is deploy order, not
/// a dormant edge). Two "is dbt the sole producer" rules that drifted apart would
/// silence this warning with nothing failing.
///
/// Same disclosure and executor contract as [`sole_dbt_producer`]: workspace
/// pool, caller already scoped to `workspace_id`.
pub async fn dormant_dbt_subscriptions<'e>(
executor: impl PgExecutor<'e>,
workspace_id: &str,
relations: &[String],
) -> error::Result<Vec<String>> {
if relations.is_empty() {
return Ok(vec![]);
}
let refs = relations
.iter()
.map(|r| format!("dbt://{r}"))
.collect::<Vec<_>>();
Ok(sqlx::query_scalar!(
r#"WITH producer AS (
SELECT 'dbt://' || a.path AS trigger_ref, a.usage_path, s.language
FROM asset a
JOIN script s ON s.workspace_id = a.workspace_id AND s.path = a.usage_path
AND s.archived = false AND s.deleted = false
WHERE a.workspace_id = $1 AND a.kind = 'dbt'
AND a.usage_kind = 'script' AND a.usage_access_type IN ('w', 'rw')
AND 'dbt://' || a.path = ANY($2)
)
SELECT DISTINCT st.trigger_ref || ' → ' || st.runnable_path AS "edge!"
FROM script_trigger st
WHERE st.workspace_id = $1 AND st.trigger_kind = 'asset'
AND st.trigger_ref = ANY($2)
AND EXISTS (SELECT 1 FROM producer p
WHERE p.trigger_ref = st.trigger_ref
AND p.usage_path <> st.runnable_path
AND p.language = 'dbt')
AND NOT EXISTS (SELECT 1 FROM producer p
WHERE p.trigger_ref = st.trigger_ref
AND p.usage_path <> st.runnable_path
AND p.language <> 'dbt')
ORDER BY 1"#,
workspace_id,
&refs
)
.fetch_all(executor)
.await?)
}
/// Clear and reinsert the full static-asset usage set of a script in one tx,
/// invalidating the producer-writes cache at most once and only on a real
/// change. The cache (asset_dispatch::ASSET_PRODUCER_WRITES_CACHE) keys a
backend/windmill-common/src/dbt_manifest.rs
+339 -32
View File
@@ -8,9 +8,11 @@
//! Two things this module is deliberate about:
//!
//! * **Asset identity is the physical relation.** A model becomes
//! `dbt://<warehouse>/<schema>/<name>`: the scheme names dbt, which is the
//! only thing that creates one, but the PATH is the relation and never dbt's
//! own `unique_id`. Two projects meet at a handoff — one materializes a mart,
//! `dbt://<warehouse>/<schema>/<name>`: the scheme names the namespace dbt
//! made — it is the only thing that DERIVES one, while any other language can
//! declare a write to one (decision 25) — but the PATH is the relation and
//! never dbt's own `unique_id`. Two projects meet at a handoff — one
//! materializes a mart,
//! the next declares it a `source` — where `model.a.orders` and
//! `source.b.analytics.orders` differ but the relation does not, so keying on
//! the node id would leave each project an island; a native script reading the
@@ -33,8 +35,8 @@
//! Every `pub` mutator in this module — the manifest ones
//! (`replace_dbt_manifest`, `clear_dbt_manifest_version`,
//! `clear_dbt_editor_graphs`),
//! the snapshot sweep, and the retry-state ones (`move_dbt_run_state`,
//! `clear_dbt_run_state`, `clear_dbt_run_state_if_path_retired`) — takes the
//! the snapshot sweep, and the script-state ones (`move_dbt_script_state`,
//! `clear_dbt_script_state`, `clear_dbt_script_state_if_path_retired`) — takes the
//! workspace and the script to act on as plain arguments and enforces nothing:
//! **the caller must already have verified write access to that script**,
//! exactly like the sibling `assets::replace_static_asset_usage` each is called
@@ -188,6 +190,19 @@ fn graph_digest(ingested: &IngestedManifest, relation_root: &str) -> String {
.unwrap_or_default()
.as_bytes(),
);
// Only when there are any, so a project that never asked for the analysis
// pass keeps the digest it already has. Hashing an empty section
// unconditionally would change every stored digest at once, and every
// dynamic run would then store a full snapshot until its script is
// redeployed — which reads exactly like the suppression above never working.
if !ingested.column_edges.is_empty() {
h.update(b"\0");
h.update(
serde_json::to_string(&ingested.column_edges)
.unwrap_or_default()
.as_bytes(),
);
}
format!("{:x}", h.finalize())
}
@@ -297,6 +312,14 @@ pub async fn prune_dbt_run_graphs(
)
.execute(db)
.await?;
sqlx::query!(
"DELETE FROM dbt_column_edge
WHERE job_id <> '00000000-0000-0000-0000-000000000000'
AND ingested_at < now() - make_interval(days => $1)",
RUN_GRAPH_RETENTION_DAYS,
)
.execute(db)
.await?;
// In ONE transaction with the orphan sweep: a restart in the gap leaves graph
// rows whose marker is gone, and since the sweep runs only when a marker went,
// every later call computes `retired == 0` and skips them for good.
@@ -325,7 +348,7 @@ pub async fn prune_dbt_run_graphs(
// partial index here — all of them `WHERE job_id <> DEPLOYED` — and past the
// keep-count is rare, so the ordinary run should pay for neither.
if retired > 0 {
for table in ["dbt_node", "dbt_edge"] {
for table in ["dbt_node", "dbt_edge", "dbt_column_edge"] {
sqlx::query(&format!(
"DELETE FROM {table} t
WHERE t.workspace_id = $1 AND t.script_path = $2
@@ -381,6 +404,18 @@ pub struct IngestedNode {
pub severity: Option<String>,
pub attached_node: Option<String>,
pub columns: Option<serde_json::Value>,
/// The node's real columns, typed and ordered — `[{"name": …, "type": …}]`,
/// from the engine's static analysis. `None` when the project did not ask
/// for it or the engine wrote none. Beside `columns` rather than merged into
/// it: that one is what the author DECLARED, and stays that.
///
/// Skipped when absent, unlike its neighbours, because `graph_digest`
/// serializes these nodes: emitting `"column_schema":null` would change
/// every stored digest at once, and every dynamic run of a project that
/// never asked for the pass would store a full snapshot until its script is
/// redeployed.
#[serde(skip_serializing_if = "Option::is_none")]
pub column_schema: Option<serde_json::Value>,
pub freshness: Option<serde_json::Value>,
/// The transform itself, for the graph to render. The copy taken at
/// deploy: the file itself is in the script's module bundle.
@@ -388,6 +423,40 @@ pub struct IngestedNode {
pub original_file_path: Option<String>,
}
/// One column-to-column edge of the ingested graph.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Default)]
#[serde(default)]
pub struct IngestedColumnEdge {
pub parent_unique_id: String,
pub parent_column: String,
pub child_unique_id: String,
pub child_column: String,
/// dbt's own word: `copy`, `mod` or `scan`. Kept verbatim — the engine's own
/// reader maps those three and passes anything else through, so the set is
/// open.
pub lineage_kind: String,
}
/// One column of a node, as the engine's static analysis resolved it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct IndexedColumn {
pub name: String,
/// The declared type where `schema.yml` gives one, else the inferred one.
/// Empty when neither is known; the column still belongs to the relation, so
/// only the type is left out.
pub column_type: String,
/// Position in the relation, which is the order the panel lists them in.
pub index: i64,
}
/// What one `--write-index` pass produced: the column edges of the whole
/// project and the real column schema per node.
#[derive(Debug, Default)]
pub struct ColumnIndex {
pub edges: Vec<IngestedColumnEdge>,
pub columns: HashMap<String, Vec<IndexedColumn>>,
}
// Serde: an agent worker cannot write these tables directly, so it posts the
// whole manifest to the server, which stores it with the same function the SQL
// path uses.
@@ -398,6 +467,9 @@ pub struct IngestedNode {
pub struct IngestedManifest {
pub nodes: Vec<IngestedNode>,
pub edges: Vec<(String, String)>,
/// Column-to-column lineage, when the project asked for it and the engine
/// produced it. Empty is the normal case — see `attach_column_index`.
pub column_edges: Vec<IngestedColumnEdge>,
/// The `asset` rows the owning script produces (models) and consumes
/// (sources) — what the lineage graph is drawn from.
pub assets: Vec<AssetWithAltAccessType>,
@@ -405,6 +477,88 @@ pub struct IngestedManifest {
pub adapter_type: String,
}
/// The most column edges one graph stores.
///
/// A `scan` edge — the column was read to produce the row, not the value — is
/// emitted from every join key and every predicate column to every output
/// column, so one wide model over a multi-column join contributes columns times
/// predicates edges on its own. The cap is what keeps a project shaped like that
/// from turning one deploy into a multi-million-row insert; past it the lineage
/// is truncated and the rest of the graph is unaffected.
pub const MAX_COLUMN_EDGES: usize = 200_000;
/// Whether the value travelled along this edge, as opposed to the column merely
/// being read to produce the row.
///
/// A `scan` edge reaches every output column of its model, so it is most of what
/// a wide project's index holds and the first thing `MAX_COLUMN_EDGES` gives up.
/// It is still stored, for a view that wants indirect influence.
pub fn is_direct(lineage_kind: &str) -> bool {
matches!(lineage_kind, "copy" | "mod")
}
impl IngestedManifest {
/// Fold one `--write-index` pass into the graph.
///
/// Both halves are scoped to the nodes this graph already kept: the index
/// describes the whole project, while the graph describes what this script's
/// selection builds plus the parents anchoring its edges, and an edge whose
/// endpoint is absent has nothing to draw.
pub fn attach_column_index(&mut self, index: ColumnIndex) {
let kept: std::collections::HashSet<&str> =
self.nodes.iter().map(|n| n.unique_id.as_str()).collect();
let mut edges: Vec<IngestedColumnEdge> = index
.edges
.into_iter()
.filter(|e| {
kept.contains(e.parent_unique_id.as_str())
&& kept.contains(e.child_unique_id.as_str())
})
.collect();
// Sorted and deduplicated for the digest, which decides whether a run
// stores a snapshot at all: parquet row order is the engine's and two
// passes over one project must not read as two different graphs.
//
// Direct kinds first, so what the truncation below gives up is `scan` —
// the bulk of a wide project's lineage, and the kind that says the column
// was read to produce the row rather than the value. The
// worker's reader already applies this order while decoding, because the
// memory bound has to; repeating it here is what makes the ordering a
// property of the manifest rather than of one caller's reader, and it is
// the only ordering an index assembled some other way would get.
edges.sort_by(|a, b| {
is_direct(&b.lineage_kind)
.cmp(&is_direct(&a.lineage_kind))
.then_with(|| a.cmp(b))
});
edges.dedup();
edges.truncate(MAX_COLUMN_EDGES);
self.column_edges = edges;
let mut columns = index.columns;
for node in self.nodes.iter_mut() {
let Some(mut cols) = columns.remove(&node.unique_id) else {
continue;
};
if cols.is_empty() {
continue;
}
cols.sort_by_key(|c| c.index);
node.column_schema = Some(serde_json::Value::Array(
cols.into_iter()
// A column the analysis typed as nothing still belongs in
// the list — that it exists is the half `manifest.json`
// could not answer.
.map(|c| match c.column_type.is_empty() {
true => serde_json::json!({ "name": c.name }),
false => serde_json::json!({ "name": c.name, "type": c.column_type }),
})
.collect(),
));
}
}
}
/// dbt's `materialized` mapped onto Windmill's write strategy.
///
/// The mapping is exact for the four strategies Windmill has, and deliberately
@@ -655,6 +809,9 @@ pub fn ingest_manifest(
.map(|(k, v)| (k.clone(), v.description.clone().unwrap_or_default()))
.collect::<BTreeMap<_, _>>())
}),
// Filled by `attach_column_index` when the project asked for it:
// the manifest carries declared columns only.
column_schema: None,
freshness: node.freshness.clone(),
// The transform the graph renders. Capped: a project can hold
// thousands of models and this is duplicated per deploy, so a
@@ -826,6 +983,16 @@ pub async fn replace_dbt_manifest(
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"DELETE FROM dbt_column_edge WHERE workspace_id = $1 AND script_path = $2
AND script_hash = $3 AND job_id = $4",
workspace_id,
script_path,
script_hash,
job_id
)
.execute(&mut **tx)
.await?;
// The marker, before the rows: a graph with no nodes at all is a legitimate
// answer for a dynamic run that disabled every model, and the reader must be
// able to tell it from a run that stored nothing.
@@ -881,7 +1048,7 @@ async fn insert_graph_rows(
"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id, \
resource_type, name, asset_path, materialized, materialize_strategy, unique_key, \
tags, description, test_kind, test_column, test_args, severity, attached_node, \
columns, freshness, raw_code, original_file_path) ",
columns, column_schema, freshness, raw_code, original_file_path) ",
);
q.push_values(chunk, |mut b, n| {
b.push_bind(workspace_id)
@@ -903,6 +1070,7 @@ async fn insert_graph_rows(
.push_bind(&n.severity)
.push_bind(&n.attached_node)
.push_bind(&n.columns)
.push_bind(&n.column_schema)
.push_bind(&n.freshness)
.push_bind(&n.raw_code)
.push_bind(&n.original_file_path);
@@ -926,6 +1094,26 @@ async fn insert_graph_rows(
q.push(" ON CONFLICT DO NOTHING");
q.build().execute(&mut **tx).await?;
}
for chunk in ingested.column_edges.chunks(COLUMN_EDGE_INSERT_CHUNK) {
let mut q = sqlx::QueryBuilder::new(
"INSERT INTO dbt_column_edge (workspace_id, script_path, script_hash, job_id, \
parent_unique_id, parent_column, child_unique_id, child_column, lineage_kind) ",
);
q.push_values(chunk, |mut b, e| {
b.push_bind(workspace_id)
.push_bind(script_path)
.push_bind(script_hash)
.push_bind(job_id)
.push_bind(&e.parent_unique_id)
.push_bind(&e.parent_column)
.push_bind(&e.child_unique_id)
.push_bind(&e.child_column)
.push_bind(&e.lineage_kind);
});
q.push(" ON CONFLICT DO NOTHING");
q.build().execute(&mut **tx).await?;
}
Ok(())
}
@@ -966,7 +1154,7 @@ pub async fn replace_dbt_editor_graph(
) -> Result<()> {
// By job alone, so re-executing one — a zombie recovered onto another
// worker — replaces its rows rather than colliding with them.
for table in ["dbt_node", "dbt_edge", "dbt_graph_snapshot"] {
for table in ["dbt_node", "dbt_edge", "dbt_column_edge", "dbt_graph_snapshot"] {
sqlx::query(&format!(
"DELETE FROM {table} WHERE workspace_id = $1 AND job_id = $2 AND script_hash IS NULL"
))
@@ -1016,7 +1204,7 @@ pub async fn replace_dbt_editor_graph(
.fetch_all(&mut **tx)
.await?;
if !retired.is_empty() {
for table in ["dbt_node", "dbt_edge"] {
for table in ["dbt_node", "dbt_edge", "dbt_column_edge"] {
sqlx::query(&format!(
"DELETE FROM {table} WHERE workspace_id = $1 AND job_id = ANY($2) \
AND script_hash IS NULL"
@@ -1035,6 +1223,8 @@ pub async fn replace_dbt_editor_graph(
const NODE_INSERT_CHUNK: usize = 2000;
/// Six columns, so the same ceiling allows far more.
const EDGE_INSERT_CHUNK: usize = 8000;
/// Nine columns, and by far the most numerous rows of the three.
const COLUMN_EDGE_INSERT_CHUNK: usize = 6000;
/// Clear one VERSION's graph: the delete-by-hash route, which only soft-deletes
/// its `script` row and so fires no cascade, and the ingest that finds no
@@ -1070,6 +1260,15 @@ pub async fn clear_dbt_manifest_version(
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"DELETE FROM dbt_column_edge
WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3",
workspace_id,
script_path,
script_hash
)
.execute(&mut **tx)
.await?;
// The marker too, and every job's: a marker left standing for rows that are
// gone is read as a snapshot, and its digest still answers the suppression
// check — so an identical run would write nothing and then render an empty
@@ -1105,7 +1304,7 @@ pub async fn clear_dbt_editor_graphs(
workspace_id: &str,
script_path: &str,
) -> Result<()> {
for table in ["dbt_node", "dbt_edge", "dbt_graph_snapshot"] {
for table in ["dbt_node", "dbt_edge", "dbt_column_edge", "dbt_graph_snapshot"] {
sqlx::query(&format!(
"DELETE FROM {table}
WHERE workspace_id = $1 AND script_path = $2 AND script_hash IS NULL"
@@ -1118,22 +1317,27 @@ pub async fn clear_dbt_editor_graphs(
Ok(())
}
/// Move a dbt script's saved retry state to its new path.
/// Move a dbt script's saved state to its new path: the run `dbt retry` resumes,
/// and the state each environment's deferrals resolve through.
///
/// Keyed by path like the sidecar, but unlike the sidecar it is not
/// regenerated by anything: the deploy re-ingests a manifest, while these are
/// the results of a run that already happened. Clearing on rename would throw
/// away a resumable failure for a cosmetic change, so it travels instead.
/// Keyed by path like the sidecar, but unlike the sidecar neither is regenerated
/// by anything: the deploy re-ingests a manifest, while these are the results of
/// runs that already happened. Clearing on rename would throw away a resumable
/// failure, and every deferral until the next full run, for a cosmetic change —
/// so they travel instead. An artifact too large for its row is unaffected: its
/// key is that publication's own, and the moved row is what names it.
///
/// See the mutator contract above: this authorizes nothing.
pub async fn move_dbt_run_state(
pub async fn move_dbt_script_state(
tx: &mut Transaction<'_, Postgres>,
workspace_id: &str,
old_path: &str,
new_path: &str,
) -> Result<()> {
// The destination may already hold state from a script that lived there
// before; the incoming row is the newer truth for this project.
// before; the incoming row is the newer truth for this project. What the
// displaced row named in object storage is left there, as a cleared one's is
// — see `clear_dbt_script_state`.
sqlx::query!(
"DELETE FROM dbt_run_state WHERE workspace_id = $1 AND script_path = $2",
workspace_id,
@@ -1149,22 +1353,38 @@ pub async fn move_dbt_run_state(
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"DELETE FROM dbt_environment_state WHERE workspace_id = $1 AND script_path = $2",
workspace_id,
new_path
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"UPDATE dbt_environment_state SET script_path = $3
WHERE workspace_id = $1 AND script_path = $2",
workspace_id,
old_path,
new_path
)
.execute(&mut **tx)
.await?;
Ok(())
}
/// Drop the saved retry state, but only once NO live version of the path is
/// left.
/// Drop the saved state, but only once NO live version of the path is left.
///
/// `dbt_run_state`'s key is the path and the principal — one saved run per script
/// per identity it executes as, not
/// one per version — so archiving or deleting a single version must not take it
/// with them: the live version's `dbt retry` would be refused and the
/// partial-failure resume lost. It does not need to be version-scoped either,
/// because `identity` already refuses a resume whose project, warehouse or
/// engine moved.
/// Neither table is keyed by version — `dbt_run_state` by path and principal,
/// `dbt_environment_state` by path and environment — so archiving or deleting a
/// single version must not take them with it: the live version's `dbt retry`
/// would be refused, its partial-failure resume lost, and every deferral would
/// have to wait for another full run to republish. Neither needs to be
/// version-scoped either: `identity` already refuses a resume whose project,
/// warehouse or engine moved, and a deferral resolves relation names, which a
/// new version of the same project spells the same way.
///
/// See the mutator contract above: this authorizes nothing.
pub async fn clear_dbt_run_state_if_path_retired(
pub async fn clear_dbt_script_state_if_path_retired(
tx: &mut Transaction<'_, Postgres>,
workspace_id: &str,
script_path: &str,
@@ -1179,17 +1399,34 @@ pub async fn clear_dbt_run_state_if_path_retired(
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"DELETE FROM dbt_environment_state WHERE workspace_id = $1 AND script_path = $2
AND NOT EXISTS (SELECT 1 FROM script
WHERE workspace_id = $1 AND path = $2
AND deleted = false AND archived = false)",
workspace_id,
script_path
)
.execute(&mut **tx)
.await?;
Ok(())
}
/// Drop a dbt script's saved retry state.
/// Drop a dbt script's saved state, both halves.
///
/// Archive and delete: `run_results` is not small, the invocation arguments it
/// carries are the user's, and a script later created at the same path would
/// otherwise inherit a stranger's resumable failure.
/// Archive and delete: neither is small, the invocation arguments and manifest
/// they carry are the user's, and a script later created at the same path would
/// otherwise inherit a stranger's resumable failure and defer to a project it
/// has nothing to do with.
///
/// An artifact too large for its row lives in the instance's object storage, and
/// this leaves it there — as a deleted script leaves its bundle. Reaching it from
/// here would mean an object-store client in this crate and a delete that has to
/// land after the caller's transaction commits, for one object per environment of
/// a script that is gone.
///
/// See the mutator contract above: this authorizes nothing.
pub async fn clear_dbt_run_state(
pub async fn clear_dbt_script_state(
tx: &mut Transaction<'_, Postgres>,
workspace_id: &str,
script_path: &str,
@@ -1201,6 +1438,13 @@ pub async fn clear_dbt_run_state(
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"DELETE FROM dbt_environment_state WHERE workspace_id = $1 AND script_path = $2",
workspace_id,
script_path
)
.execute(&mut **tx)
.await?;
Ok(())
}
@@ -1765,6 +2009,69 @@ mod tests {
assert_eq!(back.assets.len(), ingested.assets.len());
assert_eq!(back.assets[0].path, ingested.assets[0].path);
}
// The index describes the whole PROJECT while the graph describes what this
// script's selection builds, so an edge whose endpoint the graph does not
// hold has nothing to draw and must not be stored.
#[test]
fn column_lineage_is_scoped_to_the_nodes_the_graph_kept() {
let mut i = ingested();
let kept = "model.jaffle_shop.customers";
let dropped = "model.other_project.elsewhere";
i.attach_column_index(ColumnIndex {
edges: vec![
edge("model.jaffle_shop.orders_daily", "id", kept, "id", "copy"),
edge(dropped, "id", kept, "id", "copy"),
edge(kept, "id", dropped, "id", "copy"),
],
columns: [
(
kept.to_string(),
vec![
col("total", "Float64", 1),
col("id", "Int32", 0),
col("untyped", "", 2),
],
),
(dropped.to_string(), vec![col("id", "Int32", 0)]),
]
.into(),
});
assert_eq!(
i.column_edges
.iter()
.map(|e| (e.parent_unique_id.as_str(), e.child_unique_id.as_str()))
.collect::<Vec<_>>(),
vec![("model.jaffle_shop.orders_daily", kept)]
);
// In `column_index` order, and a column the analysis could not type still
// belongs to the relation.
assert_eq!(
node(&i, kept).column_schema,
Some(serde_json::json!([
{"name": "id", "type": "Int32"},
{"name": "total", "type": "Float64"},
{"name": "untyped"},
]))
);
assert!(node(&i, "model.jaffle_shop.orders_daily")
.column_schema
.is_none());
}
fn edge(from: &str, from_col: &str, to: &str, to_col: &str, kind: &str) -> IngestedColumnEdge {
IngestedColumnEdge {
parent_unique_id: from.into(),
parent_column: from_col.into(),
child_unique_id: to.into(),
child_column: to_col.into(),
lineage_kind: kind.into(),
}
}
fn col(name: &str, column_type: &str, index: i64) -> IndexedColumn {
IndexedColumn { name: name.into(), column_type: column_type.into(), index }
}
}
/// Record one model's state for THIS RUN.
backend/windmill-common/src/user_drafts.rs
+39
View File
@@ -401,6 +401,45 @@ pub async fn fetch_draft_only_list_rows(
Ok(rows)
}
/// Delete the caller's OWN draft at a path with no deployed row, for the DELETE
/// route of a kind whose list synthesizes such rows via
/// `fetch_draft_only_list_rows`. The `NOT EXISTS` leaves a deployed row's draft
/// alone, so a route may call this on its not-found branch whatever the reason
/// for the miss. `Ok(false)` means nothing matched: the caller reports its own error.
///
/// Takes no permission check and callers must not add one: an email-scoped row
/// belongs to the caller, who can always discard it, as `update_draft`'s
/// own-discard does. Legacy (`email IS NULL`) rows are owned by nobody and keep
/// their write gate, so discarding one stays on the `update_draft` route.
pub async fn delete_draft_only_for_path(
db: &DB,
w_id: &str,
kind: UserDraftItemKind,
path: &str,
email: &str,
) -> Result<bool> {
let Some(table) = kind.deployed_table() else {
return Ok(false);
};
// `table` is from the closed `deployed_table()` enum, never user input.
let sql = format!(
"DELETE FROM draft \
WHERE workspace_id = $1 AND typ = $2::text::DRAFT_KIND AND path = $3 \
AND email = $4 \
AND NOT EXISTS (SELECT 1 FROM {table} t \
WHERE t.workspace_id = draft.workspace_id AND t.path = draft.path)"
);
let deleted = sqlx::query(&sql)
.bind(w_id)
.bind(kind.as_str())
.bind(path)
.bind(email)
.execute(db)
.await?
.rows_affected();
Ok(deleted > 0)
}
/// The get-by-path draft choreography, shared by every entity's "get by path"
/// route. Given the deployed entity as an `Option` (caller maps its own "not
/// found" to `None`):
backend/windmill-common/tests/dbt_graph_storage.rs
+237 -9
View File
@@ -7,9 +7,10 @@
use sqlx::{Pool, Postgres};
use windmill_common::dbt_manifest::{
clear_dbt_editor_graphs, clear_dbt_manifest_version, prune_dbt_run_graphs,
replace_dbt_editor_graph, replace_dbt_manifest, IngestedManifest, IngestedNode,
DBT_EDITOR_GRAPHS_KEPT, DEPLOYED_GRAPH, DEPLOYED_GRAPH_VERSIONS_KEPT,
clear_dbt_editor_graphs, clear_dbt_manifest_version, clear_dbt_script_state,
clear_dbt_script_state_if_path_retired, move_dbt_script_state, prune_dbt_run_graphs,
replace_dbt_editor_graph, replace_dbt_manifest, IngestedColumnEdge, IngestedManifest,
IngestedNode, DBT_EDITOR_GRAPHS_KEPT, DEPLOYED_GRAPH, DEPLOYED_GRAPH_VERSIONS_KEPT,
};
const WS: &str = "test-workspace";
@@ -52,10 +53,36 @@ fn manifest(names: &[&str]) -> IngestedManifest {
.windows(2)
.map(|w| (format!("model.p.{}", w[0]), format!("model.p.{}", w[1])))
.collect(),
// Same reason: a project that opted into the analysis pass has these, and
// a fixture without them leaves every column-edge insert and sweep in
// this file unexecuted.
column_edges: names
.windows(2)
.map(|w| IngestedColumnEdge {
parent_unique_id: format!("model.p.{}", w[0]),
parent_column: w[0].to_string(),
child_unique_id: format!("model.p.{}", w[1]),
child_column: w[1].to_string(),
lineage_kind: "copy".to_string(),
})
.collect(),
..Default::default()
}
}
/// Column edges of one version, so the sweeps can be shown to reach them.
async fn column_edges_for(db: &Pool<Postgres>, hash: i64) -> i64 {
sqlx::query_scalar!(
"SELECT count(*) FROM dbt_column_edge WHERE workspace_id = $1 AND script_hash = $2",
WS,
hash
)
.fetch_one(db)
.await
.unwrap()
.unwrap_or(0)
}
/// Edges for one version, so a test can assert the batched insert ran at all.
async fn edges_for(db: &Pool<Postgres>, hash: i64) -> i64 {
sqlx::query_scalar!(
@@ -135,6 +162,33 @@ async fn an_identical_run_stores_no_snapshot(db: Pool<Postgres>) {
assert_eq!(markers(&db, 1).await, 1, "and leaves no marker of its own");
}
/// A column that is projected AND used as a predicate for the same output column
/// has both a `copy` edge and a `scan` one. They are two facts, and the digest
/// counts both — so the uniqueness key has to carry `lineage_kind`, or the
/// second is dropped by `ON CONFLICT DO NOTHING` while the digest still claims
/// it was stored.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn both_kinds_of_one_column_pair_are_stored(db: Pool<Postgres>) {
deploy_script(&db, 1).await;
let pair = |kind: &str| IngestedColumnEdge {
parent_unique_id: "model.p.a".to_string(),
parent_column: "id".to_string(),
child_unique_id: "model.p.b".to_string(),
child_column: "id".to_string(),
lineage_kind: kind.to_string(),
};
let mut m = manifest(&["a", "b"]);
m.column_edges = vec![pair("copy"), pair("scan")];
let mut tx = db.begin().await.unwrap();
replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &m, "root")
.await
.unwrap();
tx.commit().await.unwrap();
assert_eq!(column_edges_for(&db, 1).await, 2, "both kinds survive");
}
/// A run whose model set differs keeps its own, and the version's is untouched:
/// this is what lets an older run page render the project that run built.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
@@ -235,7 +289,11 @@ async fn clearing_one_version_leaves_the_others(db: Pool<Postgres>) {
// this is where two versions coexist: it pins the batched edge insert
// against a real database as well as the version scoping.
assert_eq!(edges_for(&db, 1).await, 0, "the cleared version's edges go");
assert_eq!(edges_for(&db, 2).await, 1, "the other version keeps its own");
assert_eq!(
edges_for(&db, 2).await,
1,
"the other version keeps its own"
);
}
/// The routes that hard-delete a path clear no graph rows: they delete the
@@ -275,6 +333,8 @@ async fn deleting_the_script_cascades_to_every_sidecar(db: Pool<Postgres>) {
assert_eq!(nodes_for(&db, 2, DEPLOYED_GRAPH).await, 0);
assert_eq!(edges_for(&db, 1).await, 0);
assert_eq!(edges_for(&db, 2).await, 0);
assert_eq!(column_edges_for(&db, 1).await, 0);
assert_eq!(column_edges_for(&db, 2).await, 0);
assert_eq!(markers_for_path(&db).await, 0);
}
@@ -315,7 +375,7 @@ async fn the_sweep_takes_old_snapshots_and_spares_the_version(db: Pool<Postgres>
tx.commit().await.unwrap();
// Age one snapshot past the window, rows and marker together.
for t in ["dbt_node", "dbt_edge", "dbt_graph_snapshot"] {
for t in ["dbt_node", "dbt_edge", "dbt_column_edge", "dbt_graph_snapshot"] {
sqlx::query(&format!(
"UPDATE {t} SET ingested_at = now() - interval '400 days' WHERE job_id = $1"
))
@@ -363,7 +423,11 @@ async fn only_the_newest_deploys_keep_their_graph(db: Pool<Postgres>) {
// The newest is always among them: losing the live version's graph would
// empty the page of every run of it.
assert_eq!(nodes_for(&db, over, DEPLOYED_GRAPH).await, 1);
assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 0, "the oldest is reclaimed");
assert_eq!(
nodes_for(&db, 1, DEPLOYED_GRAPH).await,
0,
"the oldest is reclaimed"
);
}
/// The third provenance: a `parse` of the EDITOR's buffer, which names no
@@ -480,17 +544,27 @@ async fn a_version_clear_spares_editor_graphs_and_a_path_clear_does_not(db: Pool
replace_dbt_editor_graph(&mut tx, WS, PATH, job, ME, &manifest(&["a"]), "root")
.await
.unwrap();
clear_dbt_manifest_version(&mut tx, WS, PATH, 1).await.unwrap();
clear_dbt_manifest_version(&mut tx, WS, PATH, 1)
.await
.unwrap();
tx.commit().await.unwrap();
assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 0);
assert_eq!(editor_nodes(&db, job).await, 1, "the buffer's graph survives");
assert_eq!(
editor_nodes(&db, job).await,
1,
"the buffer's graph survives"
);
let mut tx = db.begin().await.unwrap();
clear_dbt_editor_graphs(&mut tx, WS, PATH).await.unwrap();
tx.commit().await.unwrap();
assert_eq!(editor_nodes(&db, job).await, 0, "retiring the path takes it");
assert_eq!(
editor_nodes(&db, job).await,
0,
"retiring the path takes it"
);
}
/// A preview names its own PATH and needs only `jobs:run`, so a bound over the
@@ -552,3 +626,157 @@ async fn editor_markers(db: &Pool<Postgres>) -> i64 {
.unwrap()
.unwrap_or(0)
}
/// A deferral resolves a `ref()` through the manifest of the last successful run
/// at this path, so that state has to follow the script the way the retry state
/// does: a rename must not strand it, and a path no live dbt version occupies
/// must not hand its manifest to whatever is created there next.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn environment_state_follows_the_script(db: Pool<Postgres>) {
const MOVED: &str = "f/test/renamed";
deploy_script(&db, 1).await;
publish_environment_state(&db, PATH).await;
let mut tx = db.begin().await.unwrap();
move_dbt_script_state(&mut tx, WS, PATH, MOVED)
.await
.unwrap();
tx.commit().await.unwrap();
assert_eq!(environment_states(&db, PATH).await, 0);
assert_eq!(environment_states(&db, MOVED).await, 1);
let mut tx = db.begin().await.unwrap();
clear_dbt_script_state(&mut tx, WS, MOVED).await.unwrap();
tx.commit().await.unwrap();
assert_eq!(environment_states(&db, MOVED).await, 0);
}
/// Archiving or deleting ONE version must not take the path's state with it —
/// the live version's next deferral still needs it — while the last one leaving
/// must, or a script later created at that path inherits the previous project's
/// manifest. The condition is a `NOT EXISTS` in raw SQL, so both directions are
/// pinned against a real database.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn state_goes_only_once_no_live_version_is_left(db: Pool<Postgres>) {
deploy_script(&db, 1).await;
deploy_script(&db, 2).await;
publish_environment_state(&db, PATH).await;
retire(&db, 1).await;
let mut tx = db.begin().await.unwrap();
clear_dbt_script_state_if_path_retired(&mut tx, WS, PATH)
.await
.unwrap();
tx.commit().await.unwrap();
assert_eq!(
environment_states(&db, PATH).await,
1,
"another version is still live here"
);
retire(&db, 2).await;
let mut tx = db.begin().await.unwrap();
clear_dbt_script_state_if_path_retired(&mut tx, WS, PATH)
.await
.unwrap();
tx.commit().await.unwrap();
assert_eq!(
environment_states(&db, PATH).await,
0,
"the last one leaving takes it"
);
}
async fn retire(db: &Pool<Postgres>, hash: i64) {
sqlx::query!(
"UPDATE script SET archived = true WHERE workspace_id = $1 AND hash = $2",
WS,
hash
)
.execute(db)
.await
.unwrap();
}
/// The worker publishes under a guard naming the version that ran, and the whole
/// point of it is a job that finishes late: its script can be renamed away and an
/// unrelated one created at the same path while it runs, and that project must
/// not inherit this one's manifest as its deferral state. Enforced in raw SQL,
/// where a refactor can drop a predicate with no type error, so it is pinned
/// against a real database — the same shape `dbt_state::publish` issues.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn a_late_job_cannot_publish_for_a_path_it_no_longer_owns(db: Pool<Postgres>) {
deploy_script(&db, 1).await;
assert_eq!(guarded_publish(&db, PATH, 1).await, 1, "its own version");
assert_eq!(
guarded_publish(&db, PATH, 2).await,
0,
"a version that never lived here"
);
// The script is gone from this path and another one takes it.
sqlx::query!(
"DELETE FROM script WHERE workspace_id = $1 AND hash = 1",
WS
)
.execute(&db)
.await
.unwrap();
deploy_script(&db, 3).await;
assert_eq!(
guarded_publish(&db, PATH, 1).await,
0,
"the late job's version does not own this path any more"
);
}
/// The predicate `dbt_state::publish` locks the script row on, reduced to what it
/// decides. Keep the two in step — this file cannot call `publish` itself, which
/// is `pub(crate)` in `windmill-worker`.
async fn guarded_publish(db: &Pool<Postgres>, path: &str, ran: i64) -> u64 {
sqlx::query!(
"INSERT INTO dbt_environment_state (workspace_id, script_path, environment, job_id,
manifest)
SELECT $1::varchar, $2::varchar, 'main||analytics|wh'::text, $3::uuid, '{}'::text
WHERE EXISTS (SELECT 1 FROM script
WHERE workspace_id = $1 AND path = $2
AND deleted = false AND archived = false
AND language = 'dbt'
AND (hash = $4 OR $4 = ANY(parent_hashes)))
ON CONFLICT (workspace_id, script_path, environment) DO UPDATE SET job_id = EXCLUDED.job_id",
WS,
path,
uuid::Uuid::from_u128(9),
ran,
)
.execute(db)
.await
.unwrap()
.rows_affected()
}
async fn publish_environment_state(db: &Pool<Postgres>, path: &str) {
sqlx::query!(
"INSERT INTO dbt_environment_state (workspace_id, script_path, environment, job_id,
manifest)
VALUES ($1, $2, 'main||analytics|wh', $3, '{}')",
WS,
path,
uuid::Uuid::from_u128(9),
)
.execute(db)
.await
.unwrap();
}
async fn environment_states(db: &Pool<Postgres>, path: &str) -> i64 {
sqlx::query_scalar!(
"SELECT count(*) FROM dbt_environment_state WHERE workspace_id = $1 AND script_path = $2",
WS,
path
)
.fetch_one(db)
.await
.unwrap()
.unwrap_or(0)
}
backend/windmill-common/tests/dbt_producer_rules.rs
+196
View File
@@ -0,0 +1,196 @@
/*!
* One rule, two spellings: "is dbt the sole producer of this warehouse
* relation". `sole_dbt_producer` decides whether a `// on dbt://…` subscription
* is refused at deploy; `dormant_dbt_subscriptions` names the edges a dbt deploy
* retroactively leaves unwakeable. Both must answer "yes, dormant" only when
* every script writing the relation is a dbt one — a dbt run does not dispatch —
* and "no" both when a native `// materialize manual dbt://…` producer exists and
* when nothing produces the relation yet, which is the ordinary deploy-order
* case. Every way of getting this wrong is silent: a dormant edge on the canvas,
* a refused deploy of a valid pipeline, or a warning that stops appearing.
*/
use sqlx::{Pool, Postgres};
use windmill_common::assets::{dormant_dbt_subscriptions, sole_dbt_producer};
const WS: &str = "test-workspace";
const RELATION: &str = "main/analytics/orders";
const SUBSCRIBER: &str = "u/test-user/consumer";
async fn plant_producer(db: &Pool<Postgres>, path: &str, language: &str, hash: i64) {
sqlx::query(
"INSERT INTO script (workspace_id, hash, path, summary, description, content, created_by,
language)
VALUES ($1, $2, $3, '', '', '', 'test-user', $4::text::script_lang)",
)
.bind(WS)
.bind(hash)
.bind(path)
.bind(language)
.execute(db)
.await
.expect("insert script");
sqlx::query(
"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind)
VALUES ($1, $2, 'dbt', 'w', $3, 'script') ON CONFLICT DO NOTHING",
)
.bind(WS)
.bind(RELATION)
.bind(path)
.execute(db)
.await
.expect("insert asset");
}
async fn plant_subscriber(db: &Pool<Postgres>, path: &str) {
sqlx::query(
"INSERT INTO script_trigger (workspace_id, runnable_kind, runnable_path, trigger_kind,
trigger_ref)
VALUES ($1, 'script', $2, 'asset', 'dbt://' || $3)",
)
.bind(WS)
.bind(path)
.bind(RELATION)
.execute(db)
.await
.expect("insert script_trigger");
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn no_producer_is_not_dormant(db: Pool<Postgres>) {
assert_eq!(
sole_dbt_producer(&db, WS, RELATION, &[SUBSCRIBER.to_string()])
.await
.unwrap(),
None,
"a relation nothing produces yet must not refuse the subscription"
);
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn dbt_only_producer_is_dormant(db: Pool<Postgres>) {
plant_producer(&db, "u/test-user/project", "dbt", 1).await;
assert_eq!(
sole_dbt_producer(&db, WS, RELATION, &[SUBSCRIBER.to_string()])
.await
.unwrap(),
Some("u/test-user/project".to_string())
);
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn a_native_producer_beside_dbt_is_not_dormant(db: Pool<Postgres>) {
plant_producer(&db, "u/test-user/project", "dbt", 1).await;
plant_producer(&db, "u/test-user/ingest", "postgresql", 2).await;
assert_eq!(
sole_dbt_producer(&db, WS, RELATION, &[SUBSCRIBER.to_string()])
.await
.unwrap(),
None
);
}
/// `asset` is keyed by path while `script` holds every version of it, so the
/// language has to be read off the live one: a path converted to dbt still has
/// its old native versions sitting in `script`.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn a_superseded_native_version_does_not_count(db: Pool<Postgres>) {
plant_producer(&db, "u/test-user/project", "postgresql", 1).await;
sqlx::query("UPDATE script SET archived = true WHERE hash = 1")
.execute(&db)
.await
.expect("archive the old version");
plant_producer(&db, "u/test-user/project", "dbt", 2).await;
assert_eq!(
sole_dbt_producer(&db, WS, RELATION, &[SUBSCRIBER.to_string()])
.await
.unwrap(),
Some("u/test-user/project".to_string())
);
}
/// The rows of the script being deployed describe the version it replaces, so a
/// script dropping its `// materialize` while adding a subscription would
/// otherwise count itself as the producer that wakes it — and commit a dormant
/// edge. It can never be that producer anyway: the dispatcher skips self-loops.
/// Under a rename that write sits at the OLD path, which the deploy is removing
/// in the same uncommitted transaction, so both paths have to be excluded.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn the_subscriber_is_never_its_own_producer(db: Pool<Postgres>) {
plant_producer(&db, "u/test-user/project", "dbt", 1).await;
plant_producer(&db, SUBSCRIBER, "postgresql", 2).await;
assert_eq!(
sole_dbt_producer(&db, WS, RELATION, &[SUBSCRIBER.to_string()])
.await
.unwrap(),
Some("u/test-user/project".to_string())
);
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn a_renamed_producer_is_excluded_too(db: Pool<Postgres>) {
plant_producer(&db, "u/test-user/project", "dbt", 1).await;
plant_producer(&db, "u/test-user/old_ingest", "postgresql", 2).await;
assert_eq!(
sole_dbt_producer(
&db,
WS,
RELATION,
&[SUBSCRIBER.to_string(), "u/test-user/old_ingest".to_string()]
)
.await
.unwrap(),
Some("u/test-user/project".to_string()),
"the write this deploy is moving off the old path cannot wake the subscription"
);
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn the_set_form_agrees_with_the_singular_one(db: Pool<Postgres>) {
let relations = vec![RELATION.to_string()];
plant_subscriber(&db, "u/test-user/consumer").await;
plant_producer(&db, "u/test-user/project", "dbt", 1).await;
assert_eq!(
dormant_dbt_subscriptions(&db, WS, &relations)
.await
.unwrap(),
vec![format!("dbt://{RELATION} → u/test-user/consumer")],
"dbt alone builds it, so the subscription can never be woken"
);
plant_producer(&db, "u/test-user/ingest", "postgresql", 2).await;
assert!(
dormant_dbt_subscriptions(&db, WS, &relations)
.await
.unwrap()
.is_empty(),
"a native producer wakes it, so the edge is live"
);
}
/// The two ways the set form could stop meaning what the singular one means: a
/// relation nothing produces is deploy order rather than a dormant edge, and a
/// subscriber's own write is not a producer that can wake it — the dispatcher
/// skips self-loops, so that edge is dormant and has to be named.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn the_set_form_matches_on_the_edge_cases_too(db: Pool<Postgres>) {
let relations = vec![RELATION.to_string()];
plant_subscriber(&db, SUBSCRIBER).await;
assert!(
dormant_dbt_subscriptions(&db, WS, &relations)
.await
.unwrap()
.is_empty(),
"nothing produces it yet, so nothing is dormant"
);
plant_producer(&db, "u/test-user/project", "dbt", 1).await;
plant_producer(&db, SUBSCRIBER, "postgresql", 2).await;
assert_eq!(
dormant_dbt_subscriptions(&db, WS, &relations)
.await
.unwrap(),
vec![format!("dbt://{RELATION} → {SUBSCRIBER}")],
"the subscriber's own write cannot wake it, so dbt is still the sole producer"
);
}
backend/windmill-queue/src/asset_dispatch.rs
+7 -5
View File
@@ -248,11 +248,13 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result<DispatchResult>
// A dbt run records the relations it builds, so it looks like a producer
// here — but dbt does not trigger downstream runs. Its own DAG is dbt's to
// order; the only thing a cascade would add is waking Windmill scripts that
// read a mart, and nothing outside dbt can declare a `dbt://` write, so
// that edge exists in one direction only. Cascading from a project whose
// per-run selection can build any subset of itself needs a per-run write set
// to be correct, which is a design worth doing deliberately rather than
// inferring. Until then dbt materializes and reports; it does not dispatch.
// read a mart. Cascading from a project whose per-run selection can build any
// subset of itself needs a per-run write set to be correct, which is a design
// worth doing deliberately rather than inferring: the deploy-time write set is
// not what ran, and the per-relation state table keeps one row per relation.
// Until then dbt materializes and reports; it does not dispatch. The opposite
// direction does: a native `// materialize manual dbt://…` script reaches the
// fan-out below on the ordinary path, on the strength of its own asset rows.
if job.script_lang == Some(ScriptLang::Dbt) {
return Ok(DispatchResult::default());
}
backend/windmill-store/src/resources.rs
+368 -4
View File
@@ -50,9 +50,9 @@ use windmill_common::{
error::{self, Error, JsonResult, Result},
get_database_url,
user_drafts::{
delete_all_drafts_for_path, delete_own_draft_for_path, fetch_draft_only,
fetch_draft_only_list_rows, maybe_overlay_draft, UserDraftItemKind, WithDraftOverlay,
WithDraftQuery,
delete_all_drafts_for_path, delete_draft_only_for_path, delete_own_draft_for_path,
fetch_draft_only, fetch_draft_only_list_rows, maybe_overlay_draft, UserDraftItemKind,
WithDraftOverlay, WithDraftQuery,
},
utils::{not_found_if_none, paginate, require_admin, Pagination, StripPath},
variables,
@@ -89,6 +89,9 @@ pub fn workspaced_service() -> Router {
.route("/git_commit_hash/{*path}", get(get_git_commit_hash))
.route("/type/list", get(list_resource_types))
.route("/type/listnames", get(list_resource_types_names))
.route("/type/resource_counts", get(list_resource_counts_by_type))
.route("/type/hub/info", get(list_hub_resource_type_info))
.route("/type/hub/pick/{name}", post(pick_hub_resource_type))
.route("/type/get/{name}", get(get_resource_type))
.route("/type/exists/{name}", get(exists_resource_type))
.route("/type/update/{name}", post(update_resource_type))
@@ -134,7 +137,10 @@ pub struct EditResourceType {
/// `Option` conflates: an absent field leaves the extension alone, while an
/// explicit `null` clears it. A hub pull relies on both — a type that stops
/// being a file type has to stop being one locally too.
#[serde(default, deserialize_with = "windmill_common::more_serde::double_option")]
#[serde(
default,
deserialize_with = "windmill_common::more_serde::double_option"
)]
pub format_extension: Option<Option<String>>,
}
@@ -1309,6 +1315,17 @@ async fn delete_resource(
let path = path.to_path();
check_scopes(&authed, || format!("resources:write:{}", path))?;
// Ahead of the deploy rules: nothing is deployed at a draft-only path, so
// gating this discard on them would strand the row in a protected workspace.
// Ahead of the transaction too — the not-found branch other kinds hang this
// off is the `not_found_if_none` below, past the linked-variable cascade.
if delete_draft_only_for_path(&db, &w_id, UserDraftItemKind::Resource, path, &authed.email)
.await?
{
return Ok(format!("draft-only resource {} deleted", path));
}
if let RuleCheckResult::Blocked(msg) = check_deploy_rules(
&w_id,
AuditAuthorable::username(&authed),
@@ -1320,6 +1337,7 @@ async fn delete_resource(
{
return Err(Error::PermissionDenied(msg));
}
let mut tx = user_db.begin(&authed).await?;
// Capture resource data for trashbin before deleting
@@ -2566,6 +2584,352 @@ async fn list_resource_types_names(
Ok(Json(rows))
}
#[derive(Serialize)]
struct ResourceTypeCount {
resource_type: String,
count: i64,
}
/// How many resources of each type this workspace holds — how popular a type is *here*,
/// which is what the pickers rank on below the hub's own pick counts.
async fn list_resource_counts_by_type(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Path(w_id): Path<String>,
) -> JsonResult<Vec<ResourceTypeCount>> {
// A count per type is aggregate, so there is no path to narrow it by: a token scoped
// to individual resources gets nothing rather than a total spanning paths it cannot
// read. Callers treat the refusal as "no local signal".
check_scopes(&authed, || "resources:read".to_string())?;
let mut tx = user_db.begin(&authed).await?;
let rows = sqlx::query!(
"SELECT resource_type, count(*) as \"count!\" FROM resource WHERE workspace_id = $1 GROUP BY resource_type",
&w_id
)
.fetch_all(&mut *tx)
.await?;
tx.commit().await?;
Ok(Json(
rows.into_iter()
.map(|r| ResourceTypeCount { resource_type: r.resource_type, count: r.count })
.collect(),
))
}
/// A hub read, remembered with the hub it came from: `hub_base_url` is a live instance
/// setting, so a cache ignoring it would keep serving the previous hub's answers.
struct HubCached<T> {
hub_base_url: String,
fetched_at: std::time::Instant,
value: T,
}
/// The index changes only when a resource type is published to the hub; picks move slowly
/// and only reorder a list. Both are read on every drawer open, hence caching at all.
const HUB_RT_INDEX_TTL: std::time::Duration = std::time::Duration::from_secs(60 * 60);
const HUB_RT_PICKS_TTL: std::time::Duration = std::time::Duration::from_secs(5 * 60);
/// How long a *failed* index read is remembered. Short, and deliberately not the hour a
/// success is good for: this read is on the path of every picker open, so an unreachable
/// hub must not cost an outbound timeout each time, while one blip must not silence pick
/// reporting for an hour.
const HUB_RT_INDEX_FAILURE_TTL: std::time::Duration = std::time::Duration::from_secs(60);
static HUB_RT_INDEX: LazyLock<
std::sync::RwLock<Option<HubCached<Option<HashMap<String, HubResourceType>>>>>,
> = LazyLock::new(|| std::sync::RwLock::new(None));
static HUB_RT_PICKS: LazyLock<std::sync::RwLock<Option<HubCached<Vec<HubResourceTypePicks>>>>> =
LazyLock::new(|| std::sync::RwLock::new(None));
fn hub_cache_get<T: Clone>(
cache: &std::sync::RwLock<Option<HubCached<T>>>,
hub_base_url: &str,
ttl: std::time::Duration,
) -> Option<T> {
let guard = cache.read().ok()?;
let entry = guard.as_ref()?;
(entry.hub_base_url == hub_base_url && entry.fetched_at.elapsed() < ttl)
.then(|| entry.value.clone())
}
fn hub_cache_put<T>(cache: &std::sync::RwLock<Option<HubCached<T>>>, hub_base_url: &str, value: T) {
if let Ok(mut guard) = cache.write() {
*guard = Some(HubCached {
hub_base_url: hub_base_url.to_string(),
fetched_at: std::time::Instant::now(),
value,
});
}
}
#[derive(Deserialize)]
struct HubResourceTypeEntry {
id: i64,
name: String,
/// Optional so a hub that does not send it costs only the mapping. Required, it would
/// fail the whole parse and take pick reporting — which needs just the id — with it.
#[serde(default)]
app: Option<String>,
}
#[derive(Clone)]
struct HubResourceType {
id: i64,
/// The integration the type belongs to. Usually the type's own name, but not always:
/// `discord_webhook` and `discord_bot_configuration` are both `discord`, and only the
/// hub knows that. Without it a workspace holding a `discord_webhook` resource looks
/// like one that has never touched Discord.
app: String,
}
/// Reads the index cache, choosing the TTL by what is stored: a failure expires far sooner
/// than a success. `None` is a miss, `Some(None)` a remembered failure.
fn hub_index_cached(hub_base_url: &str) -> Option<Option<HashMap<String, HubResourceType>>> {
let guard = HUB_RT_INDEX.read().ok()?;
let entry = guard.as_ref()?;
if entry.hub_base_url != hub_base_url {
return None;
}
let ttl = if entry.value.is_some() {
HUB_RT_INDEX_TTL
} else {
HUB_RT_INDEX_FAILURE_TTL
};
(entry.fetched_at.elapsed() < ttl).then(|| entry.value.clone())
}
/// What the hub knows about every published resource type, keyed by the name Windmill
/// addresses it by. `None` when the hub cannot be reached or does not answer with a list.
async fn hub_resource_types(
db: &DB,
hub_base_url: &str,
) -> Option<HashMap<String, HubResourceType>> {
if let Some(cached) = hub_index_cached(hub_base_url) {
return cached;
}
let index = async {
let response = windmill_common::utils::http_get_from_hub(
&windmill_common::utils::HTTP_CLIENT,
&format!("{hub_base_url}/resource_types/list"),
false,
None,
Some(db),
)
.await
.ok()?;
if !response.status().is_success() {
return None;
}
// Only the id and the app are kept. That listing carries every type's schema —
// around a megabyte — and neither reporting a pick nor grouping types by
// integration needs it.
Some(
response
.json::<Vec<HubResourceTypeEntry>>()
.await
.ok()?
.into_iter()
.map(|rt| {
let app = rt.app.unwrap_or_else(|| rt.name.clone());
(rt.name, HubResourceType { id: rt.id, app })
})
.collect::<HashMap<String, HubResourceType>>(),
)
}
.await;
hub_cache_put(&HUB_RT_INDEX, hub_base_url, index.clone());
index
}
#[derive(Serialize)]
struct PickHubResourceTypeResult {
success: bool,
}
/// Tells the hub a resource type was taken into a workspace, which is the counter its
/// `/resource_types/picked` ranking reads.
///
/// Never fails the caller: a hub predating the route, an unreachable one, and a type that
/// is local-only all mean the same thing — not counted — and the request that reaches here
/// has already saved the user's resource.
///
/// POST, and scoped as a write, because it changes state on the hub under the instance's
/// own credentials. The sibling `/type/*` routes are metadata reads that a `resources:run`
/// app-embed token may make, and both the method and this check keep such a token — which
/// is untrusted app JavaScript — from driving hub counters through us.
async fn pick_hub_resource_type(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path((_w_id, name)): Path<(String, String)>,
) -> JsonResult<PickHubResourceTypeResult> {
check_scopes(&authed, || "resources:write".to_string())?;
let hub_base_url = (**windmill_common::HUB_BASE_URL.load()).clone();
let success = async {
let id = hub_resource_types(&db, &hub_base_url).await?.get(&name)?.id;
let response = windmill_common::utils::http_get_from_hub(
&windmill_common::utils::HTTP_CLIENT,
&format!("{hub_base_url}/resource_types/{id}/pick"),
false,
None,
Some(&db),
)
.await
.ok()?;
Some(response.status().is_success())
}
.await
.unwrap_or(false);
if !success {
tracing::debug!("hub did not record a pick for resource type {name}");
}
Ok(Json(PickHubResourceTypeResult { success }))
}
#[derive(Deserialize, Clone)]
struct HubResourceTypePicks {
name: String,
/// The hub counts picks in a bigint, which its driver serialises as a string.
#[serde(deserialize_with = "windmill_common::more_serde::maybe_number")]
picks: i64,
}
#[derive(Deserialize)]
struct HubPickedResourceTypes {
resource_types: Vec<HubResourceTypePicks>,
}
#[cfg(test)]
mod hub_picks_tests {
use super::*;
/// Two properties of the index cache that a later edit could quietly drop: it is keyed on
/// the hub it came from, and a failure is forgotten long before a success is.
#[test]
fn the_index_cache_is_keyed_on_the_hub_and_forgets_failures_sooner() {
let index = || {
Some(HashMap::from([(
"slack".to_string(),
HubResourceType { id: 1, app: "slack".to_string() },
)]))
};
hub_cache_put(&HUB_RT_INDEX, "https://hub.example", index());
assert!(hub_index_cached("https://hub.example").is_some_and(|v| v.is_some()));
// Switching hubs must miss rather than serve the previous hub's mapping.
assert!(hub_index_cached("https://other.example").is_none());
// A remembered failure reads as a hit (so the hub is not re-attempted) carrying
// nothing, and only until the shorter of the two TTLs.
hub_cache_put(&HUB_RT_INDEX, "https://hub.example", None);
assert!(hub_index_cached("https://hub.example").is_some_and(|v| v.is_none()));
assert!(HUB_RT_INDEX_FAILURE_TTL < HUB_RT_INDEX_TTL);
if let Ok(mut guard) = HUB_RT_INDEX.write() {
*guard = None;
}
}
/// The hub counts picks in a bigint, which postgres.js serialises as a string. Typing
/// the field as a plain i64 fails the whole response, and the ranking silently empties.
#[test]
fn picks_decode_from_a_string_or_a_number() {
let parsed: HubPickedResourceTypes = serde_json::from_str(
r#"{"resource_types":[{"name":"slack","picks":"42"},{"name":"github","picks":7}]}"#,
)
.unwrap();
assert_eq!(
parsed
.resource_types
.iter()
.map(|rt| (rt.name.as_str(), rt.picks))
.collect::<Vec<_>>(),
vec![("slack", 42), ("github", 7)]
);
}
}
/// One hub resource type as the pickers need it.
#[derive(Serialize)]
struct HubResourceTypeInfo {
name: String,
/// The integration it belongs to, so a caller can total a workspace's resources per
/// integration rather than per type.
app: String,
picks: i64,
}
/// What the hub knows about its resource types: which integration each belongs to, and how
/// often each has been picked.
///
/// Empty rather than an error when the hub answers neither read, so the pickers treat an
/// older or private hub as "no hub signal" and fall back to what the workspace itself uses.
/// The two reads degrade independently: a hub that lists types but has no `picked` route
/// still supplies the type-to-integration mapping, which is what decides whether a
/// workspace's resources are recognised as belonging to an integration at all.
async fn list_hub_resource_type_info(
Extension(db): Extension<DB>,
) -> JsonResult<Vec<HubResourceTypeInfo>> {
let hub_base_url = (**windmill_common::HUB_BASE_URL.load()).clone();
let picks = match hub_cache_get(&HUB_RT_PICKS, &hub_base_url, HUB_RT_PICKS_TTL) {
Some(picks) => picks,
None => {
let fetched = async {
let response = windmill_common::utils::http_get_from_hub(
&windmill_common::utils::HTTP_CLIENT,
&format!("{hub_base_url}/resource_types/picked"),
false,
Some(vec![("limit", "200".to_string())]),
Some(&db),
)
.await
.ok()?;
if !response.status().is_success() {
return None;
}
Some(
response
.json::<HubPickedResourceTypes>()
.await
.ok()?
.resource_types,
)
}
.await
.unwrap_or_default();
hub_cache_put(&HUB_RT_PICKS, &hub_base_url, fetched.clone());
fetched
}
};
let mut picks_by_name: HashMap<String, i64> =
picks.into_iter().map(|rt| (rt.name, rt.picks)).collect();
let index = hub_resource_types(&db, &hub_base_url)
.await
.unwrap_or_default();
let mut info: Vec<HubResourceTypeInfo> = index
.into_iter()
.map(|(name, rt)| HubResourceTypeInfo {
picks: picks_by_name.remove(&name).unwrap_or(0),
name,
app: rt.app,
})
.collect();
// What the index did not account for is a type the picks read knows and the listing does
// not — which is what a hub answering only the second read looks like. Its own name is
// the same guess a caller makes for any type the mapping misses.
info.extend(
picks_by_name
.into_iter()
.map(|(name, picks)| HubResourceTypeInfo { app: name.clone(), name, picks }),
);
Ok(Json(info))
}
async fn get_resource_type(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
backend/windmill-trigger/src/handler.rs
+15 -2
View File
@@ -18,8 +18,9 @@ use windmill_common::{
error::{Error, JsonResult, Result},
trigger_history::{self, TriggerHistoryEvent, TriggerOperation, TriggerSource},
user_drafts::{
delete_all_drafts_for_path, delete_own_draft_for_path, fetch_draft_only_list_rows,
overlay_or_draft_only, UserDraftItemKind, WithDraftOverlay, WithDraftQuery,
delete_all_drafts_for_path, delete_draft_only_for_path, delete_own_draft_for_path,
fetch_draft_only_list_rows, overlay_or_draft_only, UserDraftItemKind, WithDraftOverlay,
WithDraftQuery,
},
utils::{paginate, Pagination, StripPath},
worker::CLOUD_HOSTED,
@@ -990,6 +991,18 @@ async fn delete_trigger<T: TriggerCrud>(
.await?;
if !deleted {
drop(tx);
if delete_draft_only_for_path(
&db,
&workspace_id,
T::user_draft_item_kind(),
path,
&authed.email,
)
.await?
{
return Ok(format!("Draft-only trigger '{}' deleted", path));
}
return Err(Error::NotFound(format!(
"Trigger not found at path: {}",
path
backend/windmill-types/src/assets.rs
+13 -10
View File
@@ -14,17 +14,20 @@ pub enum AssetKind {
Ducklake,
DataTable,
Volume,
/// A warehouse relation a dbt project builds or reads,
/// `dbt://<warehouse>/<schema>/<name>`, where `<warehouse>` is the name the
/// workspace configures it under.
/// A warehouse relation, `dbt://<warehouse>/<schema>/<name>`, where
/// `<warehouse>` is the name the workspace configures it under.
///
/// The SCHEME names the producer — dbt is the only thing that creates one —
/// while the PATH stays the physical relation, because that is what two
/// projects agree on: a mart one builds is a `source` the next reads, and
/// their dbt `unique_id`s differ (`model.a.orders` vs
/// `source.b.analytics.orders`) where the relation does not
/// (docs/dbt-runtime.md, decision 11). A dbt run does not trigger that
/// reader — the shared node is lineage, not a cascade edge.
/// The SCHEME names the namespace dbt made rather than an exclusive
/// producer: dbt is what derives these relations from a project, and a script
/// in any language but dbt's own can DECLARE one it writes
/// (`// materialize manual dbt://…`) — a project's writes are read from its
/// manifest, never annotated. The PATH stays the physical relation,
/// because that is what two producers agree on: a mart one builds is a
/// `source` the next reads, and their dbt `unique_id`s differ
/// (`model.a.orders` vs `source.b.analytics.orders`) where the relation does
/// not (docs/dbt-runtime.md, decision 11). A dbt run does not trigger the
/// readers of what it built — that shared node is lineage, not a cascade
/// edge — while a declared write does (decision 25).
Dbt,
}
backend/windmill-worker/Cargo.toml
+4
View File
@@ -114,6 +114,10 @@ rust_decimal.workspace = true
jsonwebtoken.workspace = true
sha2.workspace = true
hmac.workspace = true
# Reads the dbt engine's `target/index/*.parquet`, which is where column-level
# lineage lives. Unconditional rather than behind the `parquet` FEATURE: that one
# is object storage, and a build without it still runs dbt jobs.
parquet.workspace = true
pem = { workspace = true, optional = true }
rsa = { workspace = true, optional = true }
urlencoding.workspace = true
backend/windmill-worker/src/ansible_executor.rs
+1 -11
View File
@@ -408,16 +408,6 @@ pub fn create_empty_dir(path: &PathBuf) -> std::io::Result<()> {
}
}
/// Signals a detached `spawn_blocking` task that the future awaiting it is
/// gone, so it can stop instead of running to completion in the background.
struct AbortOnDrop(std::sync::Arc<std::sync::atomic::AtomicBool>);
impl Drop for AbortOnDrop {
fn drop(&mut self) {
self.0.store(true, std::sync::atomic::Ordering::Relaxed);
}
}
/// Lay down the tree of an app-backed repository, which git can't clone
/// because its URL carries no credential.
///
@@ -490,7 +480,7 @@ async fn fetch_repo_archive(
// stopping it, so the flag is what a cancelled job uses to reach the
// extraction loop. The guard sets it when this future is dropped.
let aborted = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
let _abort_on_drop = AbortOnDrop(aborted.clone());
let _abort_on_drop = crate::common::AbortOnDrop(aborted.clone());
let unpack_archive = download_archive.clone();
tokio::task::spawn_blocking(move || {
unpack_repo_archive(&unpack_archive, &download_target, &aborted)
backend/windmill-worker/src/common.rs
+14
View File
@@ -67,6 +67,20 @@ mount {
#[cfg(not(debug_assertions))]
pub const DEV_CONF_NSJAIL: &str = "";
/// Tells a `spawn_blocking` task to stop when the future awaiting it goes away.
///
/// Dropping a `JoinHandle` detaches the task rather than cancelling it, so a
/// cancelled or timed-out phase otherwise leaves the blocking pool working on an
/// answer nobody will read. Hold one of these beside the handle and have the
/// blocking loop check the flag.
pub(crate) struct AbortOnDrop(pub(crate) std::sync::Arc<std::sync::atomic::AtomicBool>);
impl Drop for AbortOnDrop {
fn drop(&mut self) {
self.0.store(true, std::sync::atomic::Ordering::Relaxed);
}
}
/// Turn a JSON value into the string a shell/CLI arg should receive: a JSON string
/// becomes its inner value, anything else is re-serialized compactly.
pub(crate) fn raw_to_string(x: &str) -> String {
backend/windmill-worker/src/dbt_column_index.rs
+589
View File
@@ -0,0 +1,589 @@
//! Column-level lineage and real column schemas, from the engine's own static
//! analysis.
//!
//! `manifest.json` carries neither. What does is the parquet index an engine
//! writes under `dbt compile --static-analysis strict --write-index`:
//! `dbt.column_lineage.parquet` (column-to-column edges, each labelled `copy`,
//! `mod` or `scan`) and `dbt.node_columns.parquet` (every column of every node,
//! typed and ordered, rather than only the ones an author documented).
//!
//! Four properties shape everything here, all of them measured against the real
//! engines rather than assumed:
//!
//! - **Strict analysis rejects SQL the default accepts.** An unresolvable
//! identifier is an error under `strict` and compiles fine otherwise, so this
//! is a SEPARATE pass with its own `--target-path`, never a flag on the build,
//! and it is opt-in per project.
//! - **A failed pass still writes the index**, holding every edge of the models
//! that did analyze. So the artifact is read whatever the exit status.
//! - **The flag is not the capability.** `dbt-core` 2.0.0-alpha.5 accepts
//! `--write-index`, declares the views over these two tables in its own
//! `views.sql`, and writes neither file; only Fusion does today. Nothing here
//! asks which engine it is beyond "has the flag" — a release that starts
//! writing them is picked up with no change.
//! - **An incremental model has two shapes, and one ingest holds one of them.**
//! `is_incremental()` is false when the target does not exist or the build
//! is `--full-refresh`, so the `{{ this }}` self-join — and any `ref()` inside
//! that branch — compiles only in the other case. What this stores is
//! therefore what the compile in front of it saw: at DEPLOY, before the first
//! build, that is the cold shape, and a project deployed again after its
//! tables exist stores the incremental one for the same source. Nothing here
//! can reconcile that; dbt has no mode that emits both. The flag is taken from
//! the build so a per-run ingest matches its own run, and the version's graph
//! is honest about the compile that produced it rather than about every run
//! that will follow.
use std::collections::HashSet;
use std::ops::ControlFlow;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::path::Path;
use std::time::Duration;
use parquet::file::reader::{FileReader, SerializedFileReader};
use parquet::record::{Field, Row};
use uuid::Uuid;
use windmill_common::dbt_manifest::{
is_direct, ColumnIndex, IndexedColumn, IngestedColumnEdge, MAX_COLUMN_EDGES,
};
use windmill_common::error;
use windmill_common::worker::Connection;
use windmill_parser_yaml::dbt::DbtDescriptor;
use windmill_queue::append_logs;
use crate::dbt_executor::{dbt_command, Invocation, PreparedProject};
use crate::handle_child::JobCtx;
/// Where the lineage pass writes, relative to the project directory.
///
/// Its own tree, not the runtime's `wm_target`: a `dbt compile` writes
/// `manifest.json` and `run_results.json` like any other invocation, and after a
/// build those two are what the graph ingest and `dbt retry` read.
const CLL_ARTIFACTS_DIR: &str = "wm_target_cll";
const COLUMN_LINEAGE_PARQUET: &str = "dbt.column_lineage.parquet";
const NODE_COLUMNS_PARQUET: &str = "dbt.node_columns.parquet";
/// Run the lineage pass and read what it produced.
///
/// Two steps with deliberately different contracts, because conflating them is
/// what made a best-effort annotation able to fail the job it annotates:
///
/// - [`compile_index`] runs a subprocess and owns the JOB's semantics. Only a
/// cancellation or the job's own deadline can `Err` out of it; a non-zero exit
/// and an over-long output are outcomes, not failures.
/// - [`read_index`] owns the ARTIFACT's semantics. Reading it never fails the
/// job on the artifact's account: an absent, unreadable or partial index is a
/// value, not an error. It runs UNDER the poller all the same, so the job can
/// still end the phase — a cancel, a completion or the phase timeout — which
/// is the job's semantics reaching in, not the artifact's reaching out.
///
/// The phase budget wraps the compile alone, because it exists to leave the
/// BUILD its share of the clock and only the compile can spend that share
/// unboundedly. The decode's own end is the job's: the poller it runs under
/// stops it when the job stops.
pub(crate) async fn collect(
p: &PreparedProject,
descriptor: &DbtDescriptor,
inv: &Invocation,
// The dbt subcommand the job runs, which decides the effective
// `--full-refresh` — see `dbt_executor::full_refresh`.
command: &str,
ctx: &mut JobCtx<'_>,
job_id: &Uuid,
w_id: &str,
conn: &Connection,
kept: &HashSet<&str>,
) -> error::Result<Option<ColumnIndex>> {
if !descriptor.column_lineage {
return Ok(None);
}
if !p.engine.engine.writes_column_index() {
append_logs(
job_id,
w_id,
format!(
"\n`column_lineage` is set, but the {} engine has no `--write-index`: column \
lineage needs an engine that does static analysis. The rest of the graph is \
unaffected.\n",
p.engine.engine.as_str()
),
conn,
)
.await;
return Ok(None);
}
let index_dir = p.project_dir.join(CLL_ARTIFACTS_DIR).join("index");
let Some(compiled) = compile_index(p, descriptor, inv, command, ctx, job_id, w_id, conn).await? else {
return Ok(None);
};
let coverage = Coverage::of(&compiled);
// Decoded UNDER the poller, not followed by a check of its own. The decode is
// the one phase of this pass with no subprocess behind it, so nothing else
// heartbeats while it runs: left alone, a large index is a silent worker for
// as long as it takes, which the zombie sweep reads as a dead job and
// restarts. The poller pings throughout and ends this with an `Err` if the
// job is cancelled or completed meanwhile — the job's own semantics, which
// this module may always propagate.
let artifact = crate::handle_child::run_future_with_polling_update_job_poller(
*job_id,
ctx.timeout(),
conn,
ctx.mem_peak,
ctx.canceled_by,
async { Ok(read_index(&index_dir, kept).await) },
ctx.worker_name,
w_id,
&mut Some(ctx.occupancy_metrics),
Box::pin(futures::stream::empty()),
)
.await?;
// What only the pass knows. The COUNTS are logged where the index is folded
// into the graph, since the graph decides how much of it is kept.
let note = match artifact {
Artifact::Read(index) => {
if let Some(note) = coverage.caveat() {
log(job_id, w_id, note, &compiled.stderr, conn).await;
}
return Ok(Some(index));
}
// The truncated arms come first: a compile stopped part-way explains an
// absent or unreadable artifact, and blaming the engine's capability
// for it sends the reader to check the wrong thing entirely.
Artifact::Missing if matches!(coverage, Coverage::Truncated) => format!(
"No column lineage: the analysis pass printed more than this runtime reads and was \
stopped before it wrote `{COLUMN_LINEAGE_PARQUET}`."
),
Artifact::Unreadable(why) if matches!(coverage, Coverage::Truncated) => format!(
"No column lineage: the analysis pass was stopped for printing more than this \
runtime reads, and the `{COLUMN_LINEAGE_PARQUET}` it had written could not be read \
({why})."
),
// Said apart from the one below, because it sends the reader somewhere
// else entirely: the engine did its job and this runtime could not read
// what it wrote.
Artifact::Unreadable(why) => format!(
"No column lineage: `{COLUMN_LINEAGE_PARQUET}` was written but could not be read \
({why}). The graph is unaffected."
),
Artifact::Missing => format!(
"No column lineage: the analysis pass wrote no `{COLUMN_LINEAGE_PARQUET}`. Only an \
engine that computes it does, and only for the warehouses it analyzes natively — \
the flag alone is not the capability."
),
};
// The engine's own diagnostics come along. They are how a reader learns that
// this adapter turned static analysis off, which it reports as a warning on
// a SUCCESSFUL compile that nothing else would show.
log(job_id, w_id, &note, &compiled.stderr, conn).await;
Ok(None)
}
async fn log(job_id: &Uuid, w_id: &str, note: &str, stderr: &str, conn: &Connection) {
append_logs(
job_id,
w_id,
format!("\n{note}\n{}", diagnostics(stderr)),
conn,
)
.await;
}
/// How completely the analysis compile covered the project.
///
/// Every way the COMPILE can disappoint is a value here rather than an error. An
/// `Err` from `compile_index` is the JOB's — a cancellation or its deadline —
/// and must fail it; the pass giving up on its own terms is `Ok(None)` and has
/// already been logged.
enum Coverage {
/// Every model analyzed.
Whole,
/// `--static-analysis strict` rejected part of the project. Whatever it did
/// analyze is still in the index.
Partial,
/// The output ceiling killed the compile mid-run. Distinct from `Partial`:
/// nothing rejected the project, but the index is however far it had got, so
/// it is not `Whole` either.
Truncated,
}
impl Coverage {
fn of(c: &crate::dbt_executor::Captured) -> Self {
match (c.truncated, c.success) {
(true, _) => Coverage::Truncated,
(false, true) => Coverage::Whole,
(false, false) => Coverage::Partial,
}
}
/// What to tell the reader when an index WAS produced. `None` for a run that
/// covered everything, which needs no caveat.
fn caveat(&self) -> Option<&'static str> {
match self {
Coverage::Whole => None,
Coverage::Partial => Some(
"Column lineage: `--static-analysis strict` rejected part of the project, so \
the lineage covers only the models it could analyze.",
),
Coverage::Truncated => Some(
"Column lineage: the analysis pass printed more than this runtime reads and was \
stopped, so the lineage covers only the models it had reached.",
),
}
}
}
/// Run `dbt compile --static-analysis strict --write-index`, under this phase's
/// share of the job's clock.
///
/// `Ok(None)` is "the pass gave up and said so"; `Err` is the job's own
/// cancellation or deadline and must propagate. Nothing outlives this function:
/// the budget is a race around the child, and dropping that future kills it
/// through `run_captured`'s `kill_on_drop`.
async fn compile_index(
p: &PreparedProject,
descriptor: &DbtDescriptor,
inv: &Invocation,
command: &str,
ctx: &mut JobCtx<'_>,
job_id: &Uuid,
w_id: &str,
conn: &Connection,
) -> error::Result<Option<crate::dbt_executor::Captured>> {
// A previous pass in the same job directory — a retry's second attempt —
// would otherwise be read back as this one's answer.
tokio::fs::remove_dir_all(p.project_dir.join(CLL_ARTIFACTS_DIR))
.await
.ok();
let mut cmd = dbt_command(
p,
&[
"compile",
"--static-analysis",
"strict",
"--write-index",
// Documented as what builds the CLL graph, and `--write-index` alone
// happens to imply it on the engine probed. Passed explicitly so the
// pass does not depend on which of the two is doing the work.
"--write-lineage",
"--target-path",
CLL_ARTIFACTS_DIR,
],
);
// The flag already wins over the env var dbt_command sets, but setting both
// means this pass cannot write into the runtime's artifacts even if that
// precedence ever changes — and what is in there after a build is the
// `run_results.json` a `dbt retry` resumes from.
cmd.env("DBT_TARGET_PATH", CLL_ARTIFACTS_DIR);
crate::dbt_executor::add_vars(&mut cmd, descriptor, inv)?;
// The BUILD's answer, not the descriptor's default: `is_incremental()`
// branches on it, so a model reading `{{ this }}` compiles its self-join —
// and any `ref()` inside that branch — only when this is absent. Guessing
// here stores lineage for SQL the run never executed.
if crate::dbt_executor::full_refresh(descriptor, inv, command)? {
cmd.arg("--full-refresh");
}
// Captured rather than streamed: a strict-analysis failure is a wall of
// diagnostics about SQL the build itself accepts, and this pass decides
// nothing about whether that build runs.
// Read before the future below borrows `ctx` mutably.
let budget = phase_budget(ctx);
let run = crate::dbt_executor::run_captured(
cmd,
"dbt compile (column lineage)",
ctx,
job_id,
w_id,
conn,
CLL_MAX_OUTPUT_BYTES,
// The ceiling is this pass's, not the job's: a compile that prints more
// than it than we care to read has still analyzed the project, and the
// index it wrote is on disk either way.
crate::dbt_executor::Overflow::Truncate,
);
let Some(budget) = budget else {
return Ok(Some(run.await?));
};
match tokio::time::timeout(budget, run).await {
Ok(r) => Ok(Some(r?)),
Err(_) => {
append_logs(
job_id,
w_id,
format!(
"\nNo column lineage: the analysis pass did not finish within {}s, half of \
what was left of this job's time. The build below gets the rest.\n",
budget.as_secs()
),
conn,
)
.await;
Ok(None)
}
}
}
/// stdout the pass may produce. It is a compile, so this is diagnostics rather
/// than data.
const CLL_MAX_OUTPUT_BYTES: usize = 1 << 20;
/// The share of the job's remaining wall clock this pass may spend.
///
/// A per-run refresh ingests BEFORE the build and shares the job's one deadline,
/// so an unbounded pass on a slow project would hand `dbt build` an expired
/// budget and fail the run it exists only to annotate. Half leaves the build at
/// least as long as the annotation was allowed to take.
///
/// Spent as a race around the COMPILE rather than as a shortened deadline handed
/// to the runner: the runner reports its expiry as an `Err`, indistinguishable
/// from a cancellation or the job's own deadline, and those two MUST fail the
/// job. Expiring here is this budget and nothing else. The child dies with the
/// dropped future through `run_captured`'s `kill_on_drop`; the decode is outside
/// this race and answers to the poller instead.
fn phase_budget(ctx: &JobCtx<'_>) -> Option<Duration> {
ctx.timeout()
.map(|left| Duration::from_secs((left.max(0) as u64 / 2).max(1)))
}
/// The tail of what the engine said, bounded. The whole of it is every rendered
/// model on a large project, which is not what a job log is for.
const DIAGNOSTIC_LINES: usize = 40;
fn diagnostics(out: &str) -> String {
let lines: Vec<&str> = out.lines().filter(|l| !l.trim().is_empty()).collect();
let tail = &lines[lines.len().saturating_sub(DIAGNOSTIC_LINES)..];
match tail.is_empty() {
true => String::new(),
false => format!("{}\n", tail.join("\n")),
}
}
/// What came back from the artifact. Never an `Err`: nothing the file does or
/// fails to do is a reason to fail a job. `Unreadable` is separate from
/// `Missing` because the two send a reader looking in different places — one at
/// their engine and adapter, the other at a file that exists.
enum Artifact {
Read(ColumnIndex),
Missing,
Unreadable(String),
}
/// Read both parquets, if the lineage one is there.
///
/// The column schemas alone are not worth a graph: they arrive with the lineage
/// or not at all, and a node's declared columns already answer for the case
/// where the pass never ran.
async fn read_index(index_dir: &Path, kept: &HashSet<&str>) -> Artifact {
let lineage = index_dir.join(COLUMN_LINEAGE_PARQUET);
if !tokio::fs::try_exists(&lineage).await.unwrap_or(false) {
return Artifact::Missing;
}
let columns = index_dir.join(NODE_COLUMNS_PARQUET);
// Owned, because the decode moves to a blocking thread. The index describes
// the whole project while this graph describes one selection of it, so
// scoping HERE is what keeps the bound below from being spent on rows the
// graph would discard anyway.
let kept: HashSet<String> = kept.iter().map(|s| (*s).to_string()).collect();
// Dropping the handle of a blocking task does NOT stop it: the poller
// cancelling this phase would otherwise leave a thread decoding millions of
// rows for a job that is over. `abandoned` is set when this future is
// dropped, and the row loop reads it.
let abandoned = Arc::new(AtomicBool::new(false));
let _stop = crate::common::AbortOnDrop(abandoned.clone());
// Decompressing and decoding a parquet is CPU work on a file the engine just
// wrote, so it does not belong on the runtime's poll thread.
let read = tokio::task::spawn_blocking(move || {
read_index_blocking(&lineage, &columns, &kept, &abandoned)
})
.await;
match read {
Ok(Ok(index)) => Artifact::Read(index),
Ok(Err(e)) => Artifact::Unreadable(e.to_string()),
Err(e) => Artifact::Unreadable(e.to_string()),
}
}
fn read_index_blocking(
lineage: &Path,
columns: &Path,
kept: &HashSet<String>,
abandoned: &AtomicBool,
) -> error::Result<ColumnIndex> {
let mut out = ColumnIndex::default();
// ONE pass, with the two kinds bucketed as they arrive. `copy` and `mod` say
// the value itself travelled, so they get the whole budget; `scan` — the
// column was read to produce the ROW, which reaches every output column of
// its model and is the bulk of a wide project's index — fills only what is
// left over at the end. Reading the file twice to get that ordering would
// double the decode of exactly the large index this bound exists for.
let mut scan: Vec<IngestedColumnEdge> = Vec::new();
for_each_row(lineage, abandoned, |row| {
let lineage_kind = string(row, "lineage_kind");
let parent_unique_id = string(row, "from_node_unique_id");
let child_unique_id = string(row, "to_node_unique_id");
let parent_column = string(row, "from_column_name");
let child_column = string(row, "to_column_name");
// A column of a node the analysis could not name is not an endpoint the
// graph can draw, and neither is one outside this graph's nodes.
if parent_column.is_empty()
|| child_column.is_empty()
|| !kept.contains(&parent_unique_id)
|| !kept.contains(&child_unique_id)
{
return ControlFlow::Continue(());
}
let edge = IngestedColumnEdge {
parent_unique_id,
parent_column,
child_unique_id,
child_column,
lineage_kind,
};
// The bound covers BOTH buckets, so the pass never holds more than one
// budget's worth however the kinds are distributed.
let held = out.edges.len() + scan.len();
if is_direct(&edge.lineage_kind) {
// A direct edge displaces a `scan` one: the budget is spent on
// value flow first.
if held >= MAX_COLUMN_EDGES {
scan.pop();
}
out.edges.push(edge);
// The edge that FILLS the budget ends the read, not the next one to
// arrive: once the displacing kind is full nothing later in the file
// can be kept, and waiting for another direct edge to say so decodes
// a `scan`-only tail all the way to the backstop for nothing.
return match out.edges.len() >= MAX_COLUMN_EDGES {
true => ControlFlow::Break(()),
false => ControlFlow::Continue(()),
};
}
if held < MAX_COLUMN_EDGES {
scan.push(edge);
}
// Not a stopping point even when full: a direct edge still to come takes
// a `scan` entry's place.
ControlFlow::Continue(())
})?;
out.edges.append(&mut scan);
// Absent is normal — an engine can write the lineage table and not this one —
// and unreadable is not worth losing the lineage over.
let mut held = 0usize;
let _ = for_each_row(columns, abandoned, |row| {
let unique_id = string(row, "unique_id");
let name = string(row, "column_name");
if held >= MAX_INDEXED_COLUMNS {
return ControlFlow::Break(());
}
if name.is_empty() || !kept.contains(&unique_id) {
return ControlFlow::Continue(());
}
held += 1;
// The author's `data_type` where `schema.yml` gives one, since that is
// what the project calls the column; the analysis's own inference
// otherwise.
let column_type = match string(row, "declared_type") {
t if !t.is_empty() => t,
_ => string(row, "inferred_type"),
};
out.columns
.entry(unique_id)
.or_default()
.push(IndexedColumn {
name,
column_type,
index: int(row, "column_index").unwrap_or(i64::MAX),
});
ControlFlow::Continue(())
});
Ok(out)
}
/// The most rows of `dbt.node_columns.parquet` one pass keeps. One per column of
/// the project, so the same bound as the edges is far more than any project
/// reaches; it exists for the same reason.
const MAX_INDEXED_COLUMNS: usize = MAX_COLUMN_EDGES;
/// The most rows of an index one pass DECODES, whatever it keeps of them.
///
/// A bound on work rather than on memory, and the two are separate because the
/// input this defends against is the one that cannot be collected: `scan`
/// lineage is emitted from every predicate and join column to every output
/// column, so a project shaped that way writes an index whose row count is
/// quadratic in its widest model. This pass runs outside the phase budget, on a
/// blocking thread, and nothing the file contains may fail a deploy or a run —
/// so the file it walks needs an end even when almost nothing in it is
/// retained. The abandonment flag ends it sooner when the job is over; this is
/// the bound for a job that is not.
const MAX_INDEX_ROWS: usize = 4_000_000;
/// Decode a parquet a row at a time, handing each to `f` and never holding two.
///
/// Collecting first would put a `Vec<Row>` — each row carrying its own copy of
/// every column NAME — in front of the caller's own bound, which is what would
/// take the worker process down on the index described above.
///
/// `f` says when it has all it will take, and that is the ordinary end: this
/// runs outside the phase budget, so every row decoded past the point of being
/// able to keep one is wall clock the build below does not get.
fn for_each_row(
path: &Path,
abandoned: &AtomicBool,
mut f: impl FnMut(&Row) -> ControlFlow<()>,
) -> error::Result<()> {
let fail = |e: parquet::errors::ParquetError| {
error::Error::internal_err(format!("reading {}: {e}", path.display()))
};
let file = std::fs::File::open(path)
.map_err(|e| error::Error::internal_err(format!("opening {}: {e}", path.display())))?;
let reader = SerializedFileReader::new(file).map_err(fail)?;
for (n, row) in reader.get_row_iter(None).map_err(fail)?.enumerate() {
// Nobody is waiting for this any more — the job was cancelled, completed
// or ran out of time while it decoded.
if abandoned.load(Ordering::Relaxed) {
break;
}
if n >= MAX_INDEX_ROWS {
tracing::warn!(
"dbt column index: {} holds more than {MAX_INDEX_ROWS} rows; the rest is dropped",
path.display()
);
break;
}
if f(&row.map_err(fail)?).is_break() {
break;
}
}
Ok(())
}
/// By NAME, not by position: these tables are the engine's own schema and it
/// adds columns to them between releases.
fn field<'a>(row: &'a Row, name: &str) -> Option<&'a Field> {
row.get_column_iter()
.find(|(k, _)| k.as_str() == name)
.map(|(_, v)| v)
}
fn string(row: &Row, name: &str) -> String {
match field(row, name) {
Some(Field::Str(s)) => s.clone(),
Some(Field::Bytes(b)) => String::from_utf8_lossy(b.data()).into_owned(),
_ => String::new(),
}
}
fn int(row: &Row, name: &str) -> Option<i64> {
match field(row, name) {
Some(Field::Long(v)) => Some(*v),
Some(Field::Int(v)) => Some(*v as i64),
Some(Field::Short(v)) => Some(*v as i64),
Some(Field::UInt(v)) => Some(*v as i64),
Some(Field::ULong(v)) => i64::try_from(*v).ok(),
_ => None,
}
}
backend/windmill-worker/src/dbt_executor.rs
+991 -91
View File
File diff suppressed because it is too large Load Diff
backend/windmill-worker/src/dbt_state.rs
+748
View File
@@ -0,0 +1,748 @@
//! The dbt state a project last built into one environment, and the state
//! directory a run reads it back through.
//!
//! `dbt --defer --state <dir>` resolves a `ref()` the run does not build to the
//! relation the manifest in `<dir>` names, instead of to the schema this run
//! writes into. That makes the state a durable, per-environment artifact rather
//! than a cache: the next run of a project usually lands on a worker holding
//! neither the manifest nor the results, so anything worker-local answers for
//! one machine's history rather than for the environment.
//!
//! Two artifacts live in that directory and both are stored: `manifest.json`,
//! which is what a deferral resolves through, and `run_results.json`, which
//! `select`'s `result:` selectors read — and `select` reaches dbt verbatim, so a
//! state directory missing it fails a selection a user may legitimately write.
use std::path::{Path, PathBuf};
use uuid::Uuid;
use windmill_common::error::{self, Error};
use windmill_common::worker::Connection;
use crate::dbt_executor::{digest, PreparedProject, ARTIFACTS_DIR};
lazy_static::lazy_static! {
/// Above this, an artifact goes to the instance's object storage instead of
/// into the row. A manifest passes a few hundred KB on a handful of models
/// and grows with the project, so this ceiling is what decides whether a
/// large project needs storage configured at all; a small one stays in the
/// database, where it costs no round trip and needs nothing configured.
static ref DBT_STATE_INLINE_MAX_BYTES: usize = std::env::var("DBT_STATE_INLINE_MAX_BYTES")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(8 * 1024 * 1024);
}
/// The directory `--state` points at. Inside the job directory, so it sits in
/// the sandbox's one writable bind and goes away with the job, and prefixed like
/// the artifacts directory beside it so a project carrying a directory of this
/// name is not overwritten.
///
/// Passed to dbt RELATIVE, and that is load-bearing rather than tidiness. dbt
/// records the invocation's flags into `run_results.json` and a later
/// `dbt retry` restores them, so an absolute path would name the job directory
/// of the run being resumed — gone by then, leaving the retry to resolve a
/// deferred `ref()` against nothing. Relative, it resolves against the project
/// root, which is whichever job directory the retry landed in.
pub(crate) const STATE_DIR: &str = "wm_dbt_state";
/// Where this run's relations live, which is the only thing a deferral is about.
pub(crate) fn environment(p: &PreparedProject) -> String {
environment_key(
p.warehouse.as_deref(),
// The target dbt RUNS, not the descriptor's: it falls back to the
// workspace warehouse's and to the project's own default, so reading the
// descriptor's would put two inherited targets under one empty name.
p.effective_target.as_deref(),
// The pair `relation_root` reports to the graph's drift check, taken
// apart so neither can absorb the other's delimiter below.
p.default_schema.as_deref(),
p.default_database.as_deref(),
)
}
/// The warehouse and the target name the environment; the database and schema
/// they resolve to are in the key because a repointed warehouse resource or a
/// moved schema keeps both names while putting the relations somewhere else —
/// and a manifest is a list of relation names, so a deferral has no other way to
/// notice. A move therefore reads as an environment nothing has published yet.
///
/// Length-prefixed rather than joined on a separator. Every component but the
/// warehouse is spelled by the user — a dbt target name and a schema are both
/// arbitrary strings a profile may quote — so `prod|analytics` + `scratch` and
/// `prod` + `analytics|scratch` would otherwise be one key, and a profile moving
/// between them would read as the same environment rather than as one nothing
/// has published. Same reasoning as `stable_digest`, and still legible in a row:
/// `4:main|4:prod|9:analytics|12:dbt_wh_defer`. What a MESSAGE names is
/// `environment_label`, since this encoding is for storage.
fn environment_key(
warehouse: Option<&str>,
target: Option<&str>,
schema: Option<&str>,
database: Option<&str>,
) -> String {
[warehouse, target, schema, database]
.iter()
.map(|v| {
let v = v.unwrap_or("");
format!("{}:{v}", v.len())
})
.collect::<Vec<_>>()
.join("|")
}
/// The environment as a message names it: the key above is length-prefixed for
/// storage, which is not something to put in front of a caller.
pub(crate) fn environment_label(p: &PreparedProject) -> String {
format!(
"warehouse `{}`, target `{}`, relations in `{}`",
p.warehouse.as_deref().unwrap_or("(none)"),
p.effective_target
.as_deref()
.unwrap_or("(the profile's default)"),
match (p.default_database.as_deref(), p.default_schema.as_deref()) {
(Some(db), Some(schema)) => format!("{db}.{schema}"),
(None, Some(schema)) => schema.to_string(),
_ => "(the adapter's default)".to_string(),
}
)
}
/// The state one environment last published.
pub(crate) struct StoredState {
pub manifest: String,
pub run_results: Option<String>,
/// The run that published it, so a deferring run can say what it deferred to.
pub job_id: Uuid,
}
/// Publish this run's artifacts as the environment's state.
///
/// Called for a run that BUILT what the script's own descriptor selects and
/// succeeded (see `handle_dbt_job`). Best-effort in the same sense as the retry
/// state: losing it costs the next deferral, not the run that just finished.
///
/// **What the artifacts may carry follows from that condition.** A publishing run
/// added nothing of its own — no `select` or `vars` override, and a descriptor
/// interpolating a `{{ }}` placeholder into `vars` never publishes at all — so
/// dbt's `run_results.json` records the descriptor's own arguments, which are the
/// script's content. That is why this is keyed by environment where
/// `dbt_run_state` is keyed by principal: the retry state holds whatever a caller
/// submitted, this holds what the script says. Widen the publish condition and
/// that stops being true.
pub(crate) async fn publish(
p: &PreparedProject,
w_id: &str,
job_id: &Uuid,
// The version this job ran. `None` for a preview, which publishes nothing.
script_hash: Option<i64>,
// A build recovered by the automatic in-job node retry has a
// `run_results.json` naming only the nodes that retry redid. The manifest is
// unaffected — it is a function of the project, not of what ran — so the
// state is published without results rather than with a set describing some
// other slice of the build.
results_are_partial: bool,
conn: &Connection,
) -> error::Result<()> {
let Connection::Sql(db) = conn else {
// An agent worker reaches the database only through the API, which does
// not expose this table.
return Ok(());
};
if p.script_path.is_empty() {
// A preview has no path to key state on, and an empty one would be
// shared by every dbt script in the workspace.
return Ok(());
}
if p.templated_location {
// Refused on this side too, not only where a deferral reads. A template
// renders to one location per environment while the key sees the
// template, so publishing would file this run's manifest under a key a
// literal profile shares — and de-templating later would make that stale
// manifest readable as the new location's.
return Ok(());
}
let artifacts = p.project_dir.join(ARTIFACTS_DIR);
// The manifest is what a deferral resolves through, so there is no state
// without one. Every engine writes it beside the results of a build, so this
// is the invocation that built nothing rather than a case to report.
let Ok(manifest) = tokio::fs::read_to_string(artifacts.join("manifest.json")).await else {
return Ok(());
};
let run_results = match results_are_partial {
true => None,
false => tokio::fs::read_to_string(artifacts.join("run_results.json"))
.await
.ok(),
};
let environment = environment(p);
// Uploaded BEFORE the transaction, and to this publication's own keys, so two
// publishers cannot collide on them and nothing here can overwrite an
// artifact a committed row still names. A failure below has only its own
// objects to drop.
let nonce = Uuid::new_v4();
let (manifest, manifest_key) = store(
manifest,
"manifest.json",
&environment,
&p.script_path,
w_id,
job_id,
&nonce,
)
.await?;
let (run_results, run_results_key) = match run_results {
Some(r) => match store(
r,
"run_results.json",
&environment,
&p.script_path,
w_id,
job_id,
&nonce,
)
.await
{
Ok(stored) => stored,
Err(e) => {
forget_objects(&[manifest_key, None]).await;
return Err(e);
}
},
None => (None, None),
};
let mine = [manifest_key.clone(), run_results_key.clone()];
// One publisher per environment at a time, so the row and the objects it
// displaces are settled by one of them at a time. An advisory lock rather
// than the row's, because the first publish of an environment has no row to
// lock and is exactly when two runs of a newly deployed script are most
// likely to race.
let mut tx = match db.begin().await {
Ok(tx) => tx,
Err(e) => {
forget_objects(&mine).await;
return Err(e.into());
}
};
let staged = async {
sqlx::query_scalar!(
"SELECT pg_advisory_xact_lock($1)",
publication_lock(w_id, &p.script_path, &environment)
)
.execute(&mut *tx)
.await?;
// The script row FIRST, and held, so a rename, archive or delete of this
// path either waits for this publication or is seen by it. Reading it
// unlocked leaves a window where lifecycle cleanup finds no row to clear,
// finishes, and this transaction then commits state at a path a new
// script goes on to occupy. Script row before sidecar is also the order
// every other dbt writer takes, which is what keeps the two off a
// deadlock.
//
// The version, not just the path: "some live dbt script is here" is also
// satisfied by a script created at a path this one was renamed away from.
// A preview names no version, so `script_hash` is NULL and nothing
// matches — right for a run of content that was never deployed.
let owns_path = sqlx::query_scalar!(
"SELECT 1 FROM script
WHERE workspace_id = $1 AND path = $2
AND deleted = false AND archived = false AND language = 'dbt'
AND (hash = $3 OR $3 = ANY(parent_hashes))
FOR SHARE",
w_id,
&p.script_path,
script_hash,
)
.fetch_optional(&mut *tx)
.await?
.is_some();
if !owns_path {
return error::Result::Ok(None);
}
// What the row points at NOW, so those objects can go once this one is
// committed in their place — never before, since a reader that has
// already read the row is about to fetch them.
let displaced = sqlx::query!(
"SELECT manifest_key, run_results_key FROM dbt_environment_state
WHERE workspace_id = $1 AND script_path = $2 AND environment = $3",
w_id,
&p.script_path,
environment
)
.fetch_optional(&mut *tx)
.await?
.map(|r| [r.manifest_key, r.run_results_key])
.unwrap_or_default()
// Never a key this publication is about to commit. The keys carry a
// per-execution nonce so the two cannot coincide, and this is what says
// so rather than leaving it to be re-derived.
.map(|k| k.filter(|k| !mine.iter().flatten().any(|m| m == k)));
sqlx::query!(
"INSERT INTO dbt_environment_state (workspace_id, script_path, environment, job_id,
manifest, manifest_key, run_results,
run_results_key, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now())
ON CONFLICT (workspace_id, script_path, environment) DO UPDATE SET
job_id = EXCLUDED.job_id, manifest = EXCLUDED.manifest,
manifest_key = EXCLUDED.manifest_key, run_results = EXCLUDED.run_results,
run_results_key = EXCLUDED.run_results_key, updated_at = now()",
w_id,
&p.script_path,
environment,
job_id,
manifest,
manifest_key,
run_results,
run_results_key,
)
.execute(&mut *tx)
.await?;
error::Result::Ok(Some(displaced))
}
.await;
let displaced = match staged {
// Refused by the guard, or the write failed: nothing is committed and
// what was uploaded above has no row naming it.
Ok(None) | Err(_) => {
forget_objects(&mine).await;
return staged.map(|_| ());
}
Ok(Some(displaced)) => displaced,
};
// A commit that reports an error may still have committed — what was lost can
// be the acknowledgement. Dropping this run's objects would then leave the
// committed row naming objects that are gone, and every deferral would fail
// until the next publication; an orphan costs storage instead.
tx.commit().await?;
forget_objects(&displaced).await;
Ok(())
}
/// The environment's state, or `None` where nothing has published one.
pub(crate) async fn load(
p: &PreparedProject,
w_id: &str,
conn: &Connection,
) -> error::Result<Option<StoredState>> {
let Connection::Sql(db) = conn else {
return Err(Error::BadRequest(
"`defer` resolves a `ref()` through the dbt state stored for this environment, which \
an agent worker cannot read: it reaches the database only through the API. Run this \
script on a worker of the main group, or without `defer`"
.to_string(),
));
};
let environment = environment(p);
// A publication committing between the row and the objects it named drops
// those objects, so a miss is re-read rather than reported as a state that is
// not there. Re-read for as long as the row keeps MOVING: a reader takes no
// lock, so back-to-back publications can each overtake it, and a fixed one
// retry would report the second as missing. An unmoved row is the other
// answer — nothing republished, so the object really is gone.
let mut tried: Option<(Uuid, Option<String>, Option<String>)> = None;
for _ in 0..PUBLICATIONS_OUTRUN {
let Some(row) = sqlx::query!(
"SELECT job_id, manifest, manifest_key, run_results, run_results_key
FROM dbt_environment_state
WHERE workspace_id = $1 AND script_path = $2 AND environment = $3",
w_id,
&p.script_path,
environment
)
.fetch_optional(db)
.await?
else {
return Ok(None);
};
let seen = (
row.job_id,
row.manifest_key.clone(),
row.run_results_key.clone(),
);
let fetched = async {
let manifest = fetch(row.manifest, row.manifest_key).await?;
let run_results = fetch(row.run_results, row.run_results_key).await?;
error::Result::Ok((manifest, run_results))
}
.await;
match fetched {
Ok((Some(manifest), run_results)) => {
return Ok(Some(StoredState { manifest, run_results, job_id: seen.0 }))
}
Ok((None, _)) => return Ok(None),
Err(e) => {
if tried.as_ref() == Some(&seen) {
return Err(e);
}
tried = Some(seen);
}
}
}
Err(Error::internal_err(
"the dbt state for this environment was replaced faster than it could be read; run this \
script again"
.to_string(),
))
}
/// How many publications a read may lose to before it gives up. Each one costs a
/// re-read, and a project publishing this often while another run defers is
/// already contending for the same relations.
const PUBLICATIONS_OUTRUN: usize = 5;
/// A `manifest.json` for a state directory, whichever side it comes from.
///
/// One enum because the three restores — a deferral's stored state, a retry's
/// worker-local generation, a retry's database row — differ only in where the
/// bytes are, and a second copy of the directory layout is a second chance for
/// one of them to write a directory dbt reads differently.
pub(crate) enum StateManifest {
Bytes(String),
/// A file on this worker, copied rather than read into memory: a manifest
/// grows with the project.
CopyOf(PathBuf),
None,
}
/// Write the artifacts a dbt state directory holds, creating it if needed.
///
/// Returns whether a `manifest.json` ended up there — a worker-local generation
/// can be pruned out from under a restore, and the caller then owes a
/// `dbt parse` for one.
pub(crate) async fn write_state_dir(
dir: &Path,
run_results: Option<&str>,
manifest: StateManifest,
) -> error::Result<bool> {
tokio::fs::create_dir_all(dir)
.await
.map_err(|e| Error::internal_err(format!("preparing the dbt state directory: {e}")))?;
if let Some(run_results) = run_results {
tokio::fs::write(dir.join("run_results.json"), run_results)
.await
.map_err(|e| Error::internal_err(format!("writing run_results.json: {e}")))?;
}
Ok(match manifest {
StateManifest::Bytes(m) => {
tokio::fs::write(dir.join("manifest.json"), m)
.await
.map_err(|e| Error::internal_err(format!("writing manifest.json: {e}")))?;
true
}
StateManifest::CopyOf(from) => tokio::fs::copy(from, dir.join("manifest.json"))
.await
.is_ok(),
StateManifest::None => false,
})
}
/// The advisory lock one environment's publishers take, so only one of them
/// settles the row and the objects it displaces at a time.
///
/// Derived from the same three components as the row's key. Two environments
/// whose digests collide wait for each other, which costs a moment and nothing
/// else.
fn publication_lock(w_id: &str, script_path: &str, environment: &str) -> i64 {
let d = digest(&format!("{w_id}|{script_path}|{environment}"));
// Parsed unsigned and reinterpreted: half of all digests set the top bit,
// and read as `i64` those overflow and would collapse onto one key.
u64::from_str_radix(&d[..16], 16).unwrap_or_default() as i64
}
/// The object-storage key an artifact takes.
///
/// One key per PUBLICATION, so an upload never overwrites an artifact the
/// committed row still names: a run that fails between its two uploads, or
/// between them and its row, leaves the state pointing at the pair it already
/// had. The row switches to these in one statement and the objects it displaced
/// are dropped afterwards. The path and environment are only a prefix — the row
/// is what says where an artifact is, so state that moves with a renamed script
/// keeps naming objects under the old one. Digested because a Windmill path and a
/// schema name may both carry characters an object key gives meaning to.
///
/// The `nonce` is per EXECUTION rather than per job, because zombie recovery
/// re-runs a job under its own id: keyed on that alone, the second attempt would
/// overwrite the objects the first attempt's committed row still names, and then
/// read those same keys back as displaced and drop them.
fn object_key(
w_id: &str,
script_path: &str,
environment: &str,
job_id: &Uuid,
nonce: &Uuid,
artifact: &str,
) -> String {
format!(
"wmill_dbt_state/{w_id}/{}/{job_id}.{nonce}/{artifact}",
digest(&format!("{script_path}|{environment}"))
)
}
/// Put an artifact where its size says it belongs: `(inline, key)`, exactly one
/// of which is set.
#[allow(clippy::too_many_arguments)]
async fn store(
value: String,
artifact: &str,
environment: &str,
script_path: &str,
w_id: &str,
job_id: &Uuid,
nonce: &Uuid,
) -> error::Result<(Option<String>, Option<String>)> {
if value.len() <= *DBT_STATE_INLINE_MAX_BYTES {
return Ok((Some(value), None));
}
let key = object_key(w_id, script_path, environment, job_id, nonce, artifact);
let size = value.len();
if put_object(&key, value).await? {
return Ok((None, Some(key)));
}
Err(Error::BadRequest(format!(
"this project's {artifact} is {}, past the {} this instance keeps in the database, and \
this instance has no object storage configured to hold it. Configure instance object \
storage, or raise DBT_STATE_INLINE_MAX_BYTES",
mib(size),
mib(*DBT_STATE_INLINE_MAX_BYTES),
)))
}
fn mib(bytes: usize) -> String {
format!("{:.1} MiB", bytes as f64 / (1024.0 * 1024.0))
}
/// Read an artifact back from whichever home the row names.
async fn fetch(inline: Option<String>, key: Option<String>) -> error::Result<Option<String>> {
match (inline, key) {
(Some(inline), _) => Ok(Some(inline)),
(None, Some(key)) => get_object(&key).await.map(Some),
(None, None) => Ok(None),
}
}
/// Drop the objects nothing points at any more. Best-effort: an object left
/// behind costs storage, and there is nothing useful to do about it in the path
/// of a run that has already finished.
async fn forget_objects(keys: &[Option<String>; 2]) {
for key in keys.iter().flatten() {
delete_object(key).await;
}
}
/// Whether the artifact was stored. `false` means this instance has no object
/// storage to put it in.
///
/// The INSTANCE store, where every other internal worker artifact lives — bun
/// bundles, python wheels, job logs, the global cache. Not the workspace's:
/// that bucket is the one workspace members read and write through
/// `job_helpers/*` and `wmill.write_s3_file`, so a manifest there is one any
/// member could replace, and the next deferring run would hand dbt an
/// attacker-chosen `defer_relation` for every unbuilt `ref()` while holding the
/// script's warehouse credentials. Its compiled SQL would be readable there too,
/// for a project the reader may have no access to.
#[cfg(all(feature = "enterprise", feature = "parquet"))]
async fn put_object(key: &str, value: String) -> error::Result<bool> {
use windmill_object_store::object_store_reexports::Path as ObjectPath;
let Some(store) = windmill_object_store::get_object_store().await else {
return Ok(false);
};
store
.put(&ObjectPath::from(key), bytes::Bytes::from(value).into())
.await
.map_err(|e| Error::internal_err(format!("storing the dbt state at {key}: {e:#}")))?;
Ok(true)
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
async fn get_object(key: &str) -> error::Result<String> {
let Some(store) = windmill_object_store::get_object_store().await else {
return Err(missing_storage());
};
let bytes = windmill_object_store::attempt_fetch_bytes(store, key).await?;
String::from_utf8(bytes.to_vec())
.map_err(|e| Error::internal_err(format!("the stored dbt state is not valid UTF-8: {e}")))
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
async fn delete_object(key: &str) {
use windmill_object_store::object_store_reexports::Path as ObjectPath;
let Some(store) = windmill_object_store::get_object_store().await else {
return;
};
if let Err(e) = store.delete(&ObjectPath::from(key)).await {
tracing::warn!("dbt: could not drop the superseded state object {key}: {e:#}");
}
}
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
async fn delete_object(_key: &str) {}
/// A build without the instance store carries no client at all, so an oversized
/// artifact has nowhere but the row, and a row naming a key was written by a
/// worker that did have one.
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
async fn put_object(_key: &str, _value: String) -> error::Result<bool> {
Ok(false)
}
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
async fn get_object(_key: &str) -> error::Result<String> {
Err(missing_storage())
}
fn missing_storage() -> Error {
Error::BadRequest(
"the dbt state for this environment is in the instance's object storage, which this \
worker cannot reach: it is no longer configured, or this worker was built without \
object-storage support"
.to_string(),
)
}
/// The stored state this run resolves its unbuilt `ref()`s through, materialised
/// into the job directory at `STATE_DIR`.
#[derive(Clone, Debug)]
pub(crate) struct Deferral {
/// The run that published the state, so the job log and the result can say
/// what this one deferred to.
pub published_by: Uuid,
/// Whether the state carries `run_results.json` beside its manifest. A build
/// recovered by node retry publishes without one, and that is the only file a
/// `result:` selector reads.
pub has_run_results: bool,
}
/// Materialise the environment's state so `--state` has a directory to read.
///
/// Refused rather than run without deferral where nothing is published: the run
/// would build against a `ref()` resolving into the schema it writes, and fail
/// deep inside dbt with a relation-not-found the caller has no way to connect
/// back to a missing state.
pub(crate) async fn prepare_deferral(
p: &PreparedProject,
w_id: &str,
job_dir: &str,
conn: &Connection,
) -> error::Result<Deferral> {
if p.script_path.is_empty() {
return Err(Error::BadRequest(
"`defer` resolves a `ref()` through the state a previous run of this script \
published, so it needs a deployed script; a preview run has no environment to have \
published one"
.to_string(),
));
}
// An environment is the warehouse, the target and where they RESOLVE to, and
// a `profiles.yml` that templates its schema or database resolves somewhere
// this runtime does not render. Two renderings would then share one
// environment, and a deferral after the value changed would resolve every
// unbuilt `ref()` through the previous location's manifest.
if p.templated_location {
return Err(Error::BadRequest(
"this project's profile selects its schema or database with a template, which dbt \
renders and Windmill does not — so two environments cannot be told apart and a \
deferral could resolve through the wrong one's manifest. Spell the target's schema \
and database literally to use `defer`"
.to_string(),
));
}
let Some(state) = load(p, w_id, conn).await? else {
return Err(Error::BadRequest(format!(
"no dbt state is stored for this environment ({}), so a `ref()` this run does not \
build has no relation to resolve to. It is published by a successful run that adds \
nothing of its own: one overriding `select` or `vars` does not publish, and neither \
does any run of a descriptor that interpolates a `{{{{ }}}}` placeholder into `vars` \
or a `$var:` into `env` — those describe a model set the caller's arguments decided. \
Run this script once without `defer` and without overrides",
environment_label(p)
)));
};
let has_run_results = state.run_results.is_some();
write_state_dir(
&PathBuf::from(job_dir).join(STATE_DIR),
state.run_results.as_deref(),
StateManifest::Bytes(state.manifest),
)
.await?;
Ok(Deferral { published_by: state.job_id, has_run_results })
}
#[cfg(test)]
mod tests {
use super::*;
// Every component earns its place: a deferral resolves relation NAMES, so
// state published where those names meant something else has to read as no
// state at all rather than as state that silently no longer fits.
#[test]
fn a_moved_profile_is_another_environment() {
let here = environment_key(Some("main"), Some("prod"), Some("analytics"), Some("wh"));
assert_eq!(
here,
environment_key(Some("main"), Some("prod"), Some("analytics"), Some("wh"))
);
assert_ne!(
here,
environment_key(Some("other"), Some("prod"), Some("analytics"), Some("wh"))
);
assert_ne!(
here,
environment_key(Some("main"), Some("dev"), Some("analytics"), Some("wh"))
);
assert_ne!(
here,
environment_key(Some("main"), Some("prod"), Some("marts"), Some("wh"))
);
assert_ne!(
here,
environment_key(
Some("main"),
Some("prod"),
Some("analytics"),
Some("other_db")
)
);
}
// A target name and a schema are both the user's own strings, so a component
// carrying the separator must not be able to spell another tuple's key: a
// profile moving between the two would read as the same environment and
// defer through the manifest of relations that are somewhere else.
#[test]
fn a_component_cannot_spell_another_environments_key() {
assert_ne!(
environment_key(Some("main"), Some("prod|analytics"), Some("scratch"), None),
environment_key(Some("main"), Some("prod"), Some("analytics|scratch"), None)
);
assert_ne!(
environment_key(Some("main"), Some("prod"), Some("a"), Some("b|c")),
environment_key(Some("main"), Some("prod"), Some("a|b"), Some("c"))
);
// A component the profile leaves out is the same environment as one it
// spells empty: there is no target named "".
assert_eq!(
environment_key(Some("main"), None, Some("a"), None),
environment_key(Some("main"), Some(""), Some("a"), Some(""))
);
}
// Two environments must not queue behind one advisory lock, which is what a
// digest folded through a signed parse did for every one whose top bit is
// set — half of them.
#[test]
fn each_environment_gets_its_own_publication_lock() {
let mut seen = std::collections::HashSet::new();
for i in 0..64 {
seen.insert(publication_lock(
"ws",
"f/a/p",
&format!("main|prod|s{i}|db"),
));
}
assert_eq!(seen.len(), 64);
assert_eq!(
publication_lock("ws", "f/a/p", "e"),
publication_lock("ws", "f/a/p", "e")
);
}
}
backend/windmill-worker/src/duckdb_executor.rs
+3 -1
View File
@@ -112,7 +112,9 @@ async fn fetch_custom_test_body(conn: &Connection, w_id: &str, path: &str) -> Re
// the user's own ATTACH. `// data_test` lines append verifier probes that run
// against the freshly-materialized target and raise (failing the run) on
// violation. Returns `None` when there is no materialize annotation or the
// target isn't a ducklake (only ducklake is materialized in v1).
// target isn't a ducklake: only ducklake has a write engine, and a `dbt://`
// target is recorded by the generic job path instead (worker.rs,
// `record_declared_warehouse_write`).
fn build_materialized_query(
query: &str,
partition_value: Option<&str>,
backend/windmill-worker/src/lib.rs
+2
View File
@@ -33,9 +33,11 @@ pub mod common;
mod config;
mod csharp_executor;
mod dbt_column_index;
mod dbt_engine;
mod dbt_executor;
mod dbt_profiles;
mod dbt_state;
#[cfg(feature = "private")]
mod dedicated_worker_ee;
mod dedicated_worker_oss;
backend/windmill-worker/src/worker.rs
+105 -2
View File
@@ -5527,7 +5527,7 @@ async fn handle_code_execution_job(
.await?;
let language = language.clone();
run_language_executor(
let result = run_language_executor(
job,
conn,
client,
@@ -5553,7 +5553,110 @@ async fn handle_code_execution_job(
false,
in_pipeline,
)
.await
.await;
record_declared_warehouse_write(job, conn, code, &result).await;
result
}
/// Record the outcome of a `// materialize manual dbt://<warehouse>/<schema>/<name>`
/// declaration, the way the DuckDB executor records a DuckLake target.
///
/// Nothing generates warehouse DDL, so the script issues its own write and this
/// is the only thing that turns it into a `materialized_partition` row — the
/// relation's last writer on the run page and the graph. Language-agnostic on
/// purpose — the DuckLake write engine is DuckDB's, this declaration is anyone's
/// — except dbt's own, which is refused at deploy.
///
/// Best-effort, and it can be: the cascade fans out from the deploy-time `asset`
/// rows, not from this one, so a lost row costs the relation its last writer and
/// nothing else. It must not fail a job whose write already landed.
///
/// Shares the reach of every other runtime pipeline annotation, which is this
/// function's caller: a job handed to a dedicated worker or a flow runner never
/// passes through it, so — exactly as `// partitioned` is not resolved there —
/// such a run performs its write and records no row.
async fn record_declared_warehouse_write(
job: &MiniPulledJob,
conn: &Connection,
code: &str,
result: &error::Result<Box<RawValue>>,
) {
use windmill_common::materialization::{
MaterializationStatus, RecordMaterializationRequest, UNPARTITIONED,
};
// A DEPLOYED script only. The annotation is a deploy-time contract — `manual`,
// a three-segment relation, a configured warehouse — checked in
// `create_script_internal`, which also required write access to the path. A
// preview, hub or inline-flow body reaches this function without any of that,
// so honouring it there would let `jobs:run` alone restamp any relation's last
// writer from a script that never touched it.
if job.kind != JobKind::Script {
return;
}
// Cheap guard: the annotation scan is skipped for the overwhelming majority
// of jobs, which carry no `materialize` line at all.
if !code.contains("materialize") {
return;
}
let Some(m) = windmill_parser::asset_parser::parse_pipeline_annotations(code)
.materialize
.filter(|m| m.target_kind == windmill_parser::asset_parser::AssetKind::Dbt)
else {
return;
};
// The slice this run wrote, resolved once upstream (`resolve_partition_for_job`)
// and carried in the args the cascade reads too, so a partitioned producer
// records the same identity everything else propagates.
let partition = job
.args
.as_ref()
.and_then(|a| a.0.get(windmill_common::partition::PARTITION_ARG))
.and_then(|v| serde_json::from_str::<String>(v.get()).ok())
.unwrap_or_else(|| UNPARTITIONED.to_string());
let (status, error) = match result {
Ok(_) => (MaterializationStatus::Materialized, None),
Err(e) => (MaterializationStatus::Failed, Some(e.to_string())),
};
let recorded = match conn {
Connection::Sql(db) => windmill_common::materialization::record_materialization(
db,
&job.workspace_id,
windmill_common::assets::AssetKind::Dbt,
&m.target_path,
&partition,
status,
None,
None,
Some(job.id),
error.as_deref(),
)
.await
.map_err(|e| anyhow::anyhow!("{e:#}")),
Connection::Http(client) => {
crate::agent_workers::record_materialization_from_agent_http(
client,
&job.workspace_id,
&RecordMaterializationRequest {
asset_kind: windmill_common::assets::AssetKind::Dbt,
asset_path: m.target_path.clone(),
partition,
status,
snapshot_id: None,
row_count: None,
job_id: Some(job.id),
error,
schema: None,
},
)
.await
}
};
if let Err(e) = recorded {
tracing::warn!(
"recording the materialization of dbt://{} failed: {e:#}",
m.target_path
);
}
}
/// True when `path` contains only `Normal`/`CurDir` components, i.e. it cannot
benchmarks/lib.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
export const VERSION = "v1.804.0";
export const VERSION = "v1.805.0";
export async function login(email: string, password: string): Promise<string> {
return await windmill.UserService.login({
cli/src/commands/app/app.ts
+18 -2
View File
@@ -12,7 +12,11 @@ import * as wmill from "../../../gen/services.gen.ts";
import { ListableApp, Policy } from "../../../gen/types.gen.ts";
import { GlobalOptions, isSuperset } from "../../types.ts";
import { getWmillYamlPath, mergeConfigWithConfigFile } from "../../core/conf.ts";
import {
getWmillYamlPath,
mergeConfigWithConfigFile,
readEffectiveSyncBehavior,
} from "../../core/conf.ts";
import { readInlinePathSync } from "../../utils/utils.ts";
import devCommand from "./dev.ts";
import lintCommand from "./lint.ts";
@@ -21,6 +25,7 @@ import newCommand from "./new.ts";
import generateAgentsCommand from "./generate_agents.ts";
import { isVersionsGeq1585 } from "../sync/global.ts";
import type { PermissionedAsContext } from "../../core/permissioned_as.ts";
import { buildPermissionedAsContext } from "../../core/permissioned_as.ts";
import { applyExtraPermsDiff } from "../../core/extra_perms.ts";
export interface AppFile {
@@ -420,6 +425,8 @@ async function push(
if (isRawAppByName || hasRawAppYaml) {
const { pushRawApp } = await import("./raw_apps.ts");
const merged = await mergeConfigWithConfigFile(opts);
// Raw-app ownership preservation is not implemented on either push
// path: sync push hands pushRawApp no context either.
await pushRawApp(
workspace.workspaceId,
remotePath,
@@ -429,7 +436,16 @@ async function push(
);
log.info(colors.bold.underline.green("Raw app pushed"));
} else {
await pushApp(workspace.workspaceId, remotePath, absoluteFilePath);
await pushApp(
workspace.workspaceId,
remotePath,
absoluteFilePath,
undefined,
await buildPermissionedAsContext(
workspace.workspaceId,
await readEffectiveSyncBehavior(opts, workspace),
),
);
log.info(colors.bold.underline.green("App pushed"));
}
}
cli/src/commands/flow/flow.ts
+18 -3
View File
@@ -4,7 +4,7 @@ import { Command } from "@cliffy/command";
import { Confirm } from "@cliffy/prompt/confirm";
import { Table } from "@cliffy/table";
import * as log from "../../core/log.ts";
import { dirname, sep as SEP } from "node:path";
import { dirname, sep as SEP, resolve as pathResolve } from "node:path";
import { stringify as yamlStringify } from "yaml";
import { yamlParseFile } from "../../utils/yaml.ts";
import { readTextFile, validateRequiredArgs } from "../../utils/utils.ts";
@@ -21,11 +21,16 @@ import {
} from "../../core/context.ts";
import { resolve, track_job, pollForJobResult } from "../script/script.ts";
import { defaultFlowDefinition } from "../../../bootstrap/flow_bootstrap.ts";
import { SyncOptions, mergeConfigWithConfigFile } from "../../core/conf.ts";
import {
SyncOptions,
mergeConfigWithConfigFile,
readEffectiveSyncBehavior,
} from "../../core/conf.ts";
import { FSFSElement, elementsToMap, ignoreF } from "../sync/sync.ts";
import { Flow } from "../../../gen/types.gen.ts";
import { applyExtraPermsDiff } from "../../core/extra_perms.ts";
import type { PermissionedAsContext } from "../../core/permissioned_as.ts";
import { buildPermissionedAsContext } from "../../core/permissioned_as.ts";
import {
collectPathScriptPaths,
replaceInlineScripts,
@@ -327,10 +332,20 @@ async function push(opts: Options & { message?: string }, filePath: string, remo
if (!validatePath(remotePath)) {
return;
}
// Reading the config moves the cwd to the wmill.yaml root when it sits in a
// parent directory, so pin the file against the invocation cwd first.
filePath = pathResolve(filePath);
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const syncBehavior = await readEffectiveSyncBehavior(opts, workspace);
await pushFlow(workspace.workspaceId, remotePath, filePath, opts.message);
await pushFlow(
workspace.workspaceId,
remotePath,
filePath,
opts.message,
await buildPermissionedAsContext(workspace.workspaceId, syncBehavior)
);
log.info(colors.bold.underline.green("Flow pushed"));
}
cli/src/commands/schedule/schedule.ts
+15 -4
View File
@@ -6,13 +6,19 @@ import { Command } from "@cliffy/command";
import { Table } from "@cliffy/table";
import { colors } from "@cliffy/ansi/colors";
import * as log from "../../core/log.ts";
import { sep as SEP } from "node:path";
import { sep as SEP, resolve as pathResolve } from "node:path";
import { requireLogin } from "../../core/auth.ts";
import { resolveWorkspace, validatePath } from "../../core/context.ts";
import { mergeConfigWithConfigFile } from "../../core/conf.ts";
import {
mergeConfigWithConfigFile,
readEffectiveSyncBehavior,
} from "../../core/conf.ts";
import * as wmill from "../../../gen/services.gen.ts";
import type { PermissionedAsContext } from "../../core/permissioned_as.ts";
import { lookupUsernameByEmail } from "../../core/permissioned_as.ts";
import {
buildPermissionedAsContext,
lookupUsernameByEmail,
} from "../../core/permissioned_as.ts";
import {
GlobalOptions,
@@ -299,8 +305,12 @@ async function disable(opts: GlobalOptions, path: string) {
}
async function push(opts: GlobalOptions, filePath: string, remotePath: string) {
// Reading the config moves the cwd to the wmill.yaml root when it sits in a
// parent directory, so pin the file against the invocation cwd first.
filePath = pathResolve(filePath);
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const syncBehavior = await readEffectiveSyncBehavior(opts, workspace);
if (!validatePath(remotePath)) {
return;
@@ -317,7 +327,8 @@ async function push(opts: GlobalOptions, filePath: string, remotePath: string) {
workspace.workspaceId,
remotePath,
undefined,
parseFromFile(filePath)
parseFromFile(filePath),
await buildPermissionedAsContext(workspace.workspaceId, syncBehavior)
);
console.log(colors.bold.underline.green("Schedule pushed"));
}
cli/src/commands/script/script.ts
+7 -1
View File
@@ -7,6 +7,7 @@ import {
validatePath,
} from "../../core/context.ts";
import type { PermissionedAsContext } from "../../core/permissioned_as.ts";
import { buildPermissionedAsContext } from "../../core/permissioned_as.ts";
import { applyExtraPermsDiff } from "../../core/extra_perms.ts";
import { writeFile, stat, mkdir } from "node:fs/promises";
import { Buffer } from "node:buffer";
@@ -58,6 +59,7 @@ import {
SyncOptions,
mergeConfigWithConfigFile,
readConfigFile,
readEffectiveSyncBehavior,
} from "../../core/conf.ts";
import { SyncCodebase, listSyncCodebases } from "../../utils/codebase.ts";
import { pollJobWithQueueLogging } from "../../utils/job_polling.ts";
@@ -231,7 +233,11 @@ async function push(opts: PushOptions, filePath: string) {
opts.message,
opts,
await getRawWorkspaceDependencies(true),
codebases
codebases,
await buildPermissionedAsContext(
workspace.workspaceId,
await readEffectiveSyncBehavior(opts, workspace)
)
);
log.info(colors.bold.underline.green(`Script ${filePath} pushed`));
}
cli/src/commands/sync/pull.ts
+13 -2
View File
@@ -104,7 +104,7 @@ export async function downloadZip(
// from v1 the on-behalf-of address is stripped below, so the tarball sends the
// `has_on_behalf_of` marker instead and never resolves an address.
// `preserve_extra_perms=true` opts the tarball into surfacing granular ACLs
// on flow / script / app rows. Default-off on the server protects cross-
// on script / flow / app / variable rows. Default-off on the server protects cross-
// workspace tarball imports from carrying ACLs that reference identities
// missing in the target workspace; the CLI sync flow explicitly wants them.
const baseParams = `&plain_secret=${plainSecrets ?? false
@@ -150,7 +150,18 @@ export async function downloadZip(
}
if (zipResponse.status === 404 || body.includes("no rows returned")) {
log.info(colors.red(`Workspace '${workspace.workspaceId}' not found on ${workspace.remote}. Please check your --workspace and try again.`));
log.info(
colors.red(
`Workspace id '${workspace.workspaceId}' not found on ${workspace.remote}` +
(workspace.name !== workspace.workspaceId
? ` (resolved from profile '${workspace.name}')`
: "") +
`.\n` +
`Note this is the workspace *id* sent to the API, which is not necessarily what you passed to --workspace:\n` +
` - check 'wmill workspace list' (the 'workspace id' column)\n` +
` - check the 'workspaces' block of wmill.yaml ('workspaceId' overrides the workspace name)`
)
);
} else {
log.info(colors.red(`Failed to request tarball from API: ${zipResponse.status} ${zipResponse.statusText}`));
if (body) log.info(colors.red(body));
cli/src/commands/sync/sync.ts
+13 -75
View File
@@ -76,7 +76,8 @@ import {
} from "../../utils/utils.ts";
import {
getEffectiveSettings,
getWorkspaceNames,
inferWsNameFromProfile,
resolveWsNameForConfigFromFlags,
mergeConfigWithConfigFile,
parseSyncBehavior,
SyncOptions,
@@ -85,7 +86,10 @@ import {
WorkspaceEntryConfig,
} from "../../core/conf.ts";
import type { PermissionedAsContext } from "../../core/permissioned_as.ts";
import { preCheckPermissionedAs } from "../../core/permissioned_as.ts";
import {
buildPermissionedAsContext,
preCheckPermissionedAs,
} from "../../core/permissioned_as.ts";
import {
fromWorkspaceSpecificPath,
toWorkspaceSpecificPath,
@@ -429,37 +433,6 @@ export function computeWsSpecificFlagOnlyPushes(
return out;
}
// Resolve workspace name from a --branch override (git branch → workspace name).
// Falls back to using the branch value as-is (backward compat: old key = branch name).
function resolveWsNameFromBranch(
opts: SyncOptions,
branchName: string,
): string {
const match = findWorkspaceByGitBranch(opts.workspaces, branchName);
return match ? match[0] : branchName;
}
// Resolve wsNameForConfig from CLI flags. Prefers --branch → matching config key,
// then --workspace → matching config key (incl. when --base-url is set). Returns
// undefined when no flag-based resolution applies; callers then fall back to
// inferWsNameFromProfile on the resolved workspace profile.
export function resolveWsNameForConfigFromFlags(
opts: SyncOptions & { branch?: string; workspace?: string },
): string | undefined {
if (opts.branch) {
return resolveWsNameFromBranch(opts, opts.branch);
}
if (opts.workspace) {
// Use getWorkspaceNames so reserved keys (e.g. commonSpecificItems) are filtered out,
// matching the behavior of findWorkspaceByGitBranch / inferWsNameFromProfile.
const validKeys = getWorkspaceNames(opts.workspaces);
if (validKeys.includes(opts.workspace)) {
return opts.workspace;
}
}
return undefined;
}
// Warn if --workspace overrides auto-detected branch or if workspace not in config.
function warnWorkspaceOverride(
opts: SyncOptions,
@@ -507,33 +480,6 @@ function resolveWsNameForFiles(_opts: SyncOptions, wsName: string): string {
return wsName;
}
// After resolveWorkspace, infer the workspace config name from the resolved profile
// by matching baseUrl + workspaceId against the workspaces config entries.
function inferWsNameFromProfile(
opts: SyncOptions,
profile: { remote: string; workspaceId: string },
): string | undefined {
if (!opts.workspaces) return undefined;
const wsNames = Object.keys(opts.workspaces).filter(
(k) => k !== "commonSpecificItems",
);
for (const name of wsNames) {
const entry = (opts.workspaces as any)[name] as WorkspaceEntryConfig;
if (!entry?.baseUrl) continue;
try {
const entryUrl = new URL(entry.baseUrl).toString();
const profileUrl = new URL(profile.remote).toString();
const entryWsId = entry.workspaceId ?? name;
if (entryUrl === profileUrl && entryWsId === profile.workspaceId) {
return name;
}
} catch {
continue;
}
}
return undefined;
}
// Merge CLI options with effective settings, preserving CLI flags as overrides
function mergeCliWithEffectiveOptions<
T extends GlobalOptions & SyncOptions & { repository?: string },
@@ -5540,27 +5486,19 @@ export async function push(
return;
}
let permissionedAsContext: PermissionedAsContext | undefined = undefined;
if (parseSyncBehavior(opts.syncBehavior) >= 1) {
const user = await wmill.whoami({ workspace: workspace.workspaceId });
const userIsAdminOrDeployer =
user.is_admin || (user.groups ?? []).includes("wm_deployers");
log.debug(
`permissioned_as: user=${user.email}, is_admin=${user.is_admin}, groups=${JSON.stringify(user.groups)}, isAdminOrDeployer=${userIsAdminOrDeployer}`,
const permissionedAsContext: PermissionedAsContext | undefined =
await buildPermissionedAsContext(
workspace.workspaceId,
opts.syncBehavior,
);
permissionedAsContext = {
userCache: new Map(),
userIsAdminOrDeployer,
userEmail: user.email,
};
if (permissionedAsContext) {
// ws_specific_flag changes have no content payload, so they don't
// affect permissioned_as resolution — filter them out before the
// pre-check (which expects only added/edited/deleted).
await preCheckPermissionedAs(
changes.filter((c) => c.name !== "ws_specific_flag"),
user.email,
userIsAdminOrDeployer,
permissionedAsContext.userEmail,
permissionedAsContext.userIsAdminOrDeployer,
opts.acceptOverridingPermissionedAsWithSelf ?? false,
!!process.stdin.isTTY,
);

Some files were not shown because too many files have changed in this diff Show More