diff --git a/.agents/skills/update-sqlx/SKILL.md b/.agents/skills/update-sqlx/SKILL.md index ce4eaa5ff7..ccbd7baf3d 100644 --- a/.agents/skills/update-sqlx/SKILL.md +++ b/.agents/skills/update-sqlx/SKILL.md @@ -9,11 +9,75 @@ Windmill uses `SQLX_OFFLINE=true` in CI, which requires all `sqlx::query!` / `sq ## When to Run -Run after any change to SQL queries in Rust source files. Without it, CI will fail with: +Run after **adding or editing** a SQL query in Rust source. Without it, CI fails with: ``` error: `SQLX_OFFLINE=true` but there is no cached data for this query ``` +**Do NOT run it when a change only *removes* queries.** The cache is already complete for +CI; all that is left are orphaned entries, which are cosmetic and never break a build. +Running `prepare` to tidy them risks destroying the cache for no gain. Delete them +offline instead: for each `.sqlx/query-*.json`, normalize its `query` field (strip `\` +line-continuations, collapse whitespace) and check whether it still appears in any `.rs` +file. That detector reports ~48 false positives in a CE checkout — EE queries live in +`*_ee.rs` symlinks it cannot read — so **filter to the tables your change touched** and +delete only those. + +## Before You Run Anything + +1. **Back the cache up.** `prepare` deletes `.sqlx/` *before* regenerating, so any compile + failure leaves it gutted (observed: 2350 → 142 entries). + ```bash + cp -r backend/.sqlx /tmp/sqlx_backup # restore with: rm -rf backend/.sqlx && cp -r /tmp/sqlx_backup backend/.sqlx + ``` +2. **Point `DATABASE_URL` at THIS worktree's database.** `prepare` compiles every + `sqlx::query!` against the **live** database. Another worktree's DB lacks your + migrations, so every new-table query fails and takes the cache down with it. The + symptom is `relation "" does not exist` — that is a wrong + `DATABASE_URL`, not a broken query. See AGENTS.md → "Per-worktree ports and database". + +## Queries Inside Tests Need `--all-targets`, Which Fails In A CE Checkout + +`prepare` only caches queries in code it compiles, and `--workspace` alone does **not** +compile test targets. A `sqlx::query!` inside `tests/*.rs` therefore gets no entry, and CI +fails on the test target with the usual "no cached data" error even though the lib built +clean. `SQLX_OFFLINE=true cargo check --workspace --all-targets` is what reproduces it. + +Adding `--all-targets` caches them — and, in a CE checkout, **aborts partway through**: +`backend/tests/otel.rs` imports `windmill_common::otel_ee`, which exists only behind the +`private` feature, so the compile dies after `prepare` has already emptied `.sqlx/`. +Observed: 2435 → 4 entries, `error: cargo check failed with status: exit status: 101`. + +Do not fight it — the abort is a pre-existing EE gap, not something your change caused. +Take the entries you need and put the backup back: + +```bash +cd backend +cp -r .sqlx /tmp/sqlx_backup +ls /tmp/sqlx_backup | sort > /tmp/before.txt + +DATABASE_URL= \ + cargo sqlx prepare --workspace -- --workspace --features all_sqlx_features --all-targets +# expected to fail; it still wrote the entries it got to before dying + +ls .sqlx | sort > /tmp/after.txt +mkdir -p /tmp/newq +comm -13 /tmp/before.txt /tmp/after.txt | while read f; do cp ".sqlx/$f" /tmp/newq/; done + +rm -rf .sqlx && cp -r /tmp/sqlx_backup .sqlx && cp /tmp/newq/*.json .sqlx/ +``` + +**Read every file in `/tmp/newq` before copying it in** — print each one's `query` field and +confirm it is one of yours. The set is small (one per new test query), and anything else in +there means the run got further than you think. + +Then verify both targets, since the lib passing says nothing about the tests: + +```bash +SQLX_OFFLINE=true cargo check --workspace --features all_sqlx_features # lib +SQLX_OFFLINE=true cargo check -p --all-targets # tests +``` + ## The Problem `cargo sqlx prepare --workspace` **deletes all existing cache files** and regenerates only the ones found in the current compilation. If you don't compile with every feature flag (especially `private` for EE files), you will **silently delete EE query caches**, breaking CI for enterprise tests. @@ -68,7 +132,16 @@ But if it fails with EE compilation errors, use the safe procedure above. - **Never** run `cargo sqlx prepare --workspace` with only OSS features and commit the result — it will delete EE caches. - **Never** set `SQLX_OFFLINE=true` for local `cargo sqlx prepare` — use a live database per CLAUDE.md. (CI runs with `SQLX_OFFLINE=true`, which is why the cache must be complete.) +- **Never** run `prepare` without a `.sqlx` backup, or against a `DATABASE_URL` you have not confirmed belongs to this worktree. +- **Never** run `prepare` at all for a removal-only change. - **Never** skip the verification step (step 4 above). +- **Never** leave a `--all-targets` run's output in place after it aborts — it is a + near-empty cache. Restore the backup and graft on only the entries you verified. + +Step 4 compares against `origin/main` because step 1 restored from it, so the two agree. +If you did **not** run step 1 — auditing a branch's cache on its own, say — compare +against `git merge-base HEAD origin/main` instead: `origin/main` advances, so its newer +entries would read as losses on your branch. ## Verification diff --git a/.claude/hooks/allow-fileops-in-tmp.sh b/.claude/hooks/allow-fileops-in-tmp.sh new file mode 100755 index 0000000000..ce2ce81831 --- /dev/null +++ b/.claude/hooks/allow-fileops-in-tmp.sh @@ -0,0 +1,179 @@ +#!/usr/bin/env bash +# PreToolUse allowance for scratch file ops: auto-allow a single, plain, single-line +# `mkdir` / `cp` / `mv` / `touch` / `chmod` / `tar` / `unzip` whose every path operand +# resolves under /tmp. Anything else makes no decision (exit 0) and falls back to the normal +# permission flow — where `Bash(mv:*)` and `Bash(chmod:*)` in the `ask` list prompt. A +# PreToolUse `allow` overrides those ask rules, which is why this is a hook and not an allow +# rule: permission rules match a command prefix, so they can only constrain the FIRST operand. +# `cp /tmp/x ~/.zshrc` matches a `cp /tmp/` prefix, and requiring every operand is the point. +# +# Requiring the sources under /tmp too (not just the destination) keeps this from becoming a +# read-exfiltration path around the `Read(**/.env)` / `Read(**/secrets/**)` deny rules: a copy +# out of the project into /tmp would land the content somewhere `Read(/tmp/**)` allows. +# +# Deny-by-default tokenizing, in the same spirit as guard-rm-outside-tmp.sh: every path token +# must consist only of alphanumerics and `. _ / -`. That set contains none of the characters +# bash uses for quoting, expansion, or command separation ($ ` ~ { } ( ) ' " \ ; & | < >), nor +# any glob character, so all of those forms fail by construction. `realpath -m` then resolves +# `..` and existing symlinks, so `/tmp/link` pointing at /etc/passwd is caught. +# +# `tar` and `unzip` get their own parser: their write destination arrives as a flag VALUE +# (`-C`, `-d`) rather than a positional, and a bundle like `-xzf` consumes the token after it. +# Flags are an allowlist, not a denylist, so `-P` / `--absolute-names` — which turn off tar's +# refusal to extract `..` and absolute member paths — defer rather than needing enumeration. +# Extraction additionally requires an explicit destination under /tmp, or a cwd already under +# /tmp, since otherwise members land in the project checkout. +# +# Residual risk accepted: an archive whose members include a symlink pointing out of /tmp +# followed by a write through it can still escape, because tar applies member symlinks as it +# extracts. The archive itself must be under /tmp to get here, so this is a hazard only for +# archives fetched from an untrusted source into the scratch dir. +# +# Assumes GNU `realpath` (-m) and `jq`, both present in this repo's Linux dev env. +set -uo pipefail + +input=$(cat) +command -v jq >/dev/null 2>&1 || exit 0 +cmd=$(printf '%s' "$input" | jq -r '.tool_input.command // empty' 2>/dev/null) +[ -z "$cmd" ] && exit 0 +cwd=$(printf '%s' "$input" | jq -r '.cwd // empty' 2>/dev/null) + +# A newline separates commands, and the tokenizer below only reads the first line — defer. +case "$cmd" in *$'\n'*) exit 0 ;; esac + +read -r -a toks <<< "$cmd" + +# 0 iff the token is charset-safe and resolves to a path strictly inside /tmp. +under_tmp() { + local t="$1" canon + # Globs never auto-allow. Bash expands them only after this hook has decided, so realpath + # sees the unexpanded pattern: `/tmp/link*` canonicalizes to itself and passes, then + # expands onto a symlink whose target is outside /tmp. chmod and cp follow command-line + # symlinks, so that is a write to the target. guard-rm-outside-tmp.sh can allow globs + # because `rm` unlinks the symlink itself rather than following it. + case "$t" in *[*?[]*) return 1 ;; esac + [ -n "$(printf '%s' "$t" | tr -d 'A-Za-z0-9._/-')" ] && return 1 + # Absolute only. Resolving a relative operand against the cwd makes any bare word look like + # a safe path whenever the cwd is under /tmp, while the tool itself reads it as an option: + # `tar P -xf ...` is --absolute-names, not ./P, and `cp /tmp/t -RL /tmp/o` is a + # dereferencing recursive copy, not a file named -RL. + case "$t" in /*) ;; *) return 1 ;; esac + canon=$(realpath -m -- "$t" 2>/dev/null) + [ -n "$canon" ] || return 1 + # /tmp itself is never a target — only paths strictly inside it. + case "$canon" in /tmp/?*) return 0 ;; esac + return 1 +} + +allow() { + jq -nc --arg r "$1" '{hookSpecificOutput:{hookEventName:"PreToolUse",permissionDecision:"allow",permissionDecisionReason:$r}}' + exit 0 +} + +# Bare command word only; wrappers (`timeout cp`), env prefixes, and `/bin/cp` defer. +# Options are an allowlist per command, so anything that changes how symlinks are followed +# defers instead of needing enumeration. `cp -L` / `-H` matter most: they dereference while +# recursing, which copies the CONTENT of a symlink target from outside /tmp into a scratch +# dir that `Read(/tmp/**)` then exposes. Plain `-r` and `-a` (which implies `-d`) recreate +# such a symlink as a symlink instead, so no outside content is materialized. +case "${toks[0]:-}" in + mkdir) takes_mode=0; ok_opts='pv' ;; + cp) takes_mode=0; ok_opts='rRvfnpa' ;; + mv) takes_mode=0; ok_opts='vfn' ;; + touch) takes_mode=0; ok_opts='acmv' ;; + chmod) takes_mode=1; ok_opts='Rvfc' ;; # chmod's first operand is a mode, not a path + tar) ok_flags='xctzjJavfC'; val_flags='fC' ;; + unzip) ok_flags='oqnljvd'; val_flags='d' ;; + *) exit 0 ;; +esac + +# ---------------------------------------------------------------- tar / unzip +if [ -n "${ok_flags:-}" ]; then + saw_archive=0 saw_dest=0 extracting=0 listing=0 end_opts=0 + i=1 + while [ "$i" -lt "${#toks[@]}" ]; do + t="${toks[$i]}" + i=$((i + 1)) + if [ "$end_opts" = 0 ]; then + [ "$t" = "--" ] && { end_opts=1; continue; } + case "$t" in + -?*) + flags="${t#-}" + # Allowlist: a long option, -P/--absolute-names, --transform, -I and friends all + # leave a residue here and defer rather than being enumerated as denials. + [ -n "$(printf '%s' "$flags" | tr -d "$ok_flags")" ] && exit 0 + case "$flags" in *x*) extracting=1 ;; esac + case "${toks[0]}$flags" in unzip*[lv]*) listing=1 ;; esac + # A flag consuming the next token must be alone in its bundle's final position + # (`-xzf a.tar`), else the token it eats is ambiguous. + case "${flags%?}" in *[$val_flags]*) exit 0 ;; esac + case "${flags: -1}" in + [$val_flags]) + val="${toks[$i]:-}" + i=$((i + 1)) + [ -n "$val" ] || exit 0 + under_tmp "$val" || exit 0 + case "${flags: -1}" in + f) saw_archive=1 ;; + C | d) saw_dest=1 ;; + esac + ;; + esac + continue + ;; + esac + fi + # Positional. For tar these are sources (create) or member names (extract); for unzip the + # first is the archive. Requiring every one under /tmp is conservative for member names, + # which are not filesystem paths — those defer rather than being wrongly allowed. + under_tmp "$t" || exit 0 + [ "${toks[0]}" = "unzip" ] && saw_archive=1 + done + + [ "$saw_archive" = 1 ] || exit 0 # tar without -f reads a tape/stdin; unzip needs an archive + # Writes land relative to the working directory unless a destination was given. `unzip -l` + # and `-v` only list, so they need no destination. + if [ "$extracting" = 1 ] || { [ "${toks[0]}" = "unzip" ] && [ "$listing" = 0 ]; }; then + [ "$saw_dest" = 1 ] || under_tmp "${cwd:-$PWD}" || exit 0 + fi + allow "archive paths and extraction target are under /tmp" +fi + +# ------------------------------------------- mkdir / cp / mv / touch / chmod +path_operand=0 +seen_mode=0 +end_opts=0 +i=1 +while [ "$i" -lt "${#toks[@]}" ]; do + t="${toks[$i]}" + i=$((i + 1)) + + if [ "$end_opts" = 0 ]; then + [ "$t" = "--" ] && { end_opts=1; continue; } + # Checked at any position, not just before the first operand: GNU utils permute, so + # `cp /tmp/tree -RL /tmp/out` still enables dereferencing recursion. + case "$t" in + -?*) + # Allowlist: long options and the dereferencing flags leave a residue and defer. + [ -n "$(printf '%s' "${t#-}" | tr -d "$ok_opts")" ] && exit 0 + continue + ;; + esac + fi + + # chmod: consume the mode operand without a path check. Octal, or symbolic clauses. + if [ "$takes_mode" = 1 ] && [ "$seen_mode" = 0 ]; then + case "$t" in + [0-7] | [0-7][0-7] | [0-7][0-7][0-7] | [0-7][0-7][0-7][0-7]) ;; + *) printf '%s' "$t" | grep -Eq '^[ugoa]*[+=-][rwxXst]*(,[ugoa]*[+=-][rwxXst]*)*$' || exit 0 ;; + esac + seen_mode=1 + continue + fi + + under_tmp "$t" || exit 0 + path_operand=1 +done + +[ "$path_operand" = 1 ] || exit 0 +allow "every path operand is under /tmp" diff --git a/.claude/hooks/format-frontend.sh b/.claude/hooks/format-frontend.sh index 37c3f8d4ec..14b2c2d97c 100755 --- a/.claude/hooks/format-frontend.sh +++ b/.claude/hooks/format-frontend.sh @@ -10,8 +10,12 @@ if [ -z "$FILE_PATH" ]; then exit 0 fi -# Check if the file is in the frontend directory -if [[ "$FILE_PATH" == *"/frontend/"* ]]; then +# Only the frontend app itself, i.e. a "frontend" directory sitting at a repo root. +# A bare */frontend/* substring also matches ai_evals/adapters/frontend and the +# ai_evals app fixtures, which no prettier config governs — prettier then falls back +# to its defaults and rewrites the whole file. Anchoring to $CLAUDE_PROJECT_DIR +# instead would skip worktrees edited from a session rooted elsewhere. +if [[ "$FILE_PATH" == *"/frontend/"* ]] && [[ -e "${FILE_PATH%%/frontend/*}/.git" ]]; then # Check if it's a formattable file type if [[ "$FILE_PATH" =~ \.(ts|js|svelte|json|css|html|md)$ ]]; then cd "$CLAUDE_PROJECT_DIR/frontend" || exit 0 diff --git a/.claude/settings.json b/.claude/settings.json index eeb3683842..641ea70b53 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -48,18 +48,6 @@ "Read(/tmp/**)", "Write(/tmp/**)", "Edit(/tmp/**)", - "Bash(mkdir:/tmp/*)", - "Bash(mkdir:/tmp/**)", - "Bash(cp:/tmp/*)", - "Bash(cp:/tmp/**)", - "Bash(mv:/tmp/*)", - "Bash(mv:/tmp/**)", - "Bash(touch:/tmp/*)", - "Bash(touch:/tmp/**)", - "Bash(chmod:/tmp/*)", - "Bash(chmod:/tmp/**)", - "Bash(tar * /tmp/*)", - "Bash(unzip * /tmp/*)", "mcp__claude_ai_Gmail__search_threads", "mcp__claude_ai_Gmail__get_thread", "mcp__claude_ai_Gmail__get_message", @@ -124,6 +112,11 @@ "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/guard-rm-outside-tmp.sh", "timeout": 5 + }, + { + "type": "command", + "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/allow-fileops-in-tmp.sh", + "timeout": 5 } ] } diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000..8ceb8e722a --- /dev/null +++ b/.gitattributes @@ -0,0 +1,3 @@ +# Files a generator owns. Collapsed in review diffs and left out of language +# stats: reviewing them means reviewing the generator instead. +*.gen.ts linguist-generated=true diff --git a/.github/scripts/check-docs-links.mjs b/.github/scripts/check-docs-links.mjs index 122d098197..f03b956d9f 100644 --- a/.github/scripts/check-docs-links.mjs +++ b/.github/scripts/check-docs-links.mjs @@ -15,6 +15,11 @@ const CONCURRENCY = 24 const TIMEOUT_MS = 20000 const RETRIES = 2 +// Links whose target page is written but not yet deployed on windmill.dev: the app +// link is already the final slug, so a 404 is expected until the docs side ships. +// The value is why the entry exists, for whoever has to judge whether it still should. +const PENDING_DEPLOY = new Map() + async function walk(dir) { const out = [] for (const entry of await readdir(dir, { withFileTypes: true })) { @@ -111,16 +116,45 @@ async function worker() { } await Promise.all(Array.from({ length: CONCURRENCY }, worker)) -const failures = results.filter((r) => !r.ok) -if (failures.length === 0) { - console.log(`\n✅ All ${allUrls.length} docs links are reachable.`) +// An entry claims one thing — the page is not published yet — and 404 is the only +// answer that means it. A timeout, 403 or 5xx on the same URL is a real fault, and +// suppressing it would also read as "still waiting" and defer the staleness check. +const isPendingDeploy = (r) => PENDING_DEPLOY.has(r.url) && r.status === 404 + +const pending = results.filter((r) => PENDING_DEPLOY.has(r.url)) +const waiting = results.filter(isPendingDeploy) +if (waiting.length) { + console.log(`\n⏳ ${waiting.length} link(s) waiting on a docs deploy:`) + for (const p of waiting.sort((a, b) => a.url.localeCompare(b.url))) { + console.log(` ${p.url}\n ${PENDING_DEPLOY.get(p.url)} — not live yet (${p.status})`) + } +} + +// An entry that outlived its reason exempts a URL from the check forever, so a stale +// one has to fail the job: a line in a green log is not read at release time. +const stale = [ + ...pending.filter((p) => p.ok).map((p) => [p.url, 'the page is live']), + ...[...PENDING_DEPLOY.keys()].filter((u) => !urls.has(u)).map((u) => [u, 'nothing references it']) +] + +const failures = results.filter((r) => !r.ok && !isPendingDeploy(r)) +if (failures.length === 0 && stale.length === 0) { + console.log(`\n✅ No broken docs links (${allUrls.length} checked).`) process.exit(0) } -console.log(`\n❌ ${failures.length} broken docs link(s):`) -for (const f of failures.sort((a, b) => a.url.localeCompare(b.url))) { - console.log(`\n ${f.url}`) - console.log(` status: ${f.error ? `error (${f.error})` : f.status}`) - for (const file of urls.get(f.url)) console.log(` ↳ ${file}`) +if (failures.length) { + console.log(`\n❌ ${failures.length} broken docs link(s):`) + for (const f of failures.sort((a, b) => a.url.localeCompare(b.url))) { + console.log(`\n ${f.url}`) + console.log(` status: ${f.error ? `error (${f.error})` : f.status}`) + for (const file of urls.get(f.url)) console.log(` ↳ ${file}`) + } +} +if (stale.length) { + console.log(`\n❌ ${stale.length} PENDING_DEPLOY entr(ies) to delete from this script:`) + for (const [url, why] of stale.sort((a, b) => a[0].localeCompare(b[0]))) { + console.log(`\n ${url}\n ${why}`) + } } process.exit(1) diff --git a/.github/workflows/ai-evals-test.yml b/.github/workflows/ai-evals-test.yml index 6bf7e50f3c..3ee7aed876 100644 --- a/.github/workflows/ai-evals-test.yml +++ b/.github/workflows/ai-evals-test.yml @@ -77,9 +77,11 @@ jobs: - uses: actions/setup-node@v4 with: - # Node 22.19+ is required by the frontend's undici 8.x, which the + # Node 24 ships npm 11, which frontend/package-lock.json is authored + # with; npm 10 rejects it ("Missing: picomatch@4.0.5 from lock file"). + # Node must also stay >= 22.19 for the frontend's undici 8.x, which the # Vitest bridge loads; Node 20 fails with markAsUncloneable. - node-version: "22" + node-version: "24" # CE build used only as the AI proxy (login, workspace, provider resource, # /ai/proxy). No worker execution or MCP needed — global tools/drafts run @@ -116,6 +118,12 @@ jobs: npm ci npm run generate-backend-client + - name: Run harness unit tests + working-directory: ./ai_evals + run: | + bun install + bun test adapters/ + - name: Run global AI evals timeout-minutes: 20 working-directory: ./ai_evals diff --git a/.github/workflows/check-write-access.yml b/.github/workflows/check-write-access.yml index 5e9494e74a..097fc3b3dd 100644 --- a/.github/workflows/check-write-access.yml +++ b/.github/workflows/check-write-access.yml @@ -30,8 +30,14 @@ jobs: outputs: authorized: ${{ steps.check.outputs.authorized }} steps: + # This check is purely additive: callers OR it with author_association, so it must + # never fail the job. Failing here would block every dependent reviewer job through + # `needs`, turning an unconfigured or misconfigured app into a total review outage + # rather than a fallback to the author_association path. - name: Mint internal app token id: app + if: vars.INTERNAL_APP_ID != '' + continue-on-error: true uses: actions/create-github-app-token@v2 with: app-id: ${{ vars.INTERNAL_APP_ID }} @@ -41,7 +47,9 @@ jobs: - name: Resolve authorization id: check env: - GH_TOKEN: ${{ steps.app.outputs.token }} + # Without the app token, the default token still resolves public members and + # repo collaborators; private members simply fall through to author_association. + GH_TOKEN: ${{ steps.app.outputs.token || github.token }} USERNAME: ${{ inputs.username }} TRUSTED_BOT: ${{ inputs.trusted_bot }} REPO: ${{ github.repository }} diff --git a/.github/workflows/frontend-check.yml b/.github/workflows/frontend-check.yml index 90d839aca5..a675c17444 100644 --- a/.github/workflows/frontend-check.yml +++ b/.github/workflows/frontend-check.yml @@ -23,5 +23,8 @@ jobs: cache-dependency-path: "frontend/package-lock.json" - name: "npm check" timeout-minutes: 5 + env: + # svelte-check peaks past node's ~4GB default ceiling on this runner and aborts. + NODE_OPTIONS: --max-old-space-size=8192 run: cd frontend && npm ci && npm run generate-backend-client && npm run check diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml index 110baacda3..2f7dcde204 100644 --- a/.github/workflows/release-please.yml +++ b/.github/workflows/release-please.yml @@ -8,8 +8,10 @@ jobs: name: "Release please" runs-on: ubicloud steps: - - uses: GoogleCloudPlatform/release-please-action@v3 + # Config lives in release-please-config.json / .release-please-manifest.json: + # a `release-type` input instead re-derives the last released version by + # paginating every GitHub release, which on a repo this size is slow enough + # to fail intermittently. + - uses: googleapis/release-please-action@v5 with: - release-type: simple - package-name: windmill token: ${{ secrets.PAT_TOKEN }} diff --git a/.mcp.json b/.mcp.json index f6929bbd71..f2a745614b 100644 --- a/.mcp.json +++ b/.mcp.json @@ -7,12 +7,12 @@ "playwright": { "type": "stdio", "command": "npx", - "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--headless"] + "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--headless", "--output-dir", "/tmp/playwright-mcp-${USER:-shared}"] }, "playwright-headed": { "type": "stdio", "command": "npx", - "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium"] + "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--output-dir", "/tmp/playwright-mcp-${USER:-shared}"] } } } \ No newline at end of file diff --git a/.release-please-manifest.json b/.release-please-manifest.json new file mode 100644 index 0000000000..9a978495de --- /dev/null +++ b/.release-please-manifest.json @@ -0,0 +1,3 @@ +{ + ".": "1.783.0" +} diff --git a/AGENTS.md b/AGENTS.md index 90c0060ae2..e39d4ebede 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,6 +13,9 @@ Open-source platform for internal tools, workflows, API integrations, background - **Validation**: `docs/validation.md` — what checks to run based on what you changed - **Unreleased SDK changes**: `docs/wac-sdk-e2e.md` — exercising a client change on a real worker +- **Agent workers**: `docs/agent-worker-e2e.md` — building and running one locally. An agent + reaches the DB only through the API, so `Connection::Http` paths are never taken by a plain + `cargo run`; a normal build cannot start one at all. - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow - **Backend patterns**: use the `rust-backend` skill when writing Rust code - **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill. @@ -20,10 +23,17 @@ Open-source platform for internal tools, workflows, API integrations, background - **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi. For a Codex-driven pass that mirrors the `codex-pr-review` GitHub action against your unpushed work (committed + uncommitted) before you push, use `/local-review-codex` (`.agents/skills/local-review-codex/`) — same `REVIEW.md` policy, `gpt-5.6-sol`, `xhigh` reasoning; requires the `codex` CLI >= 0.144.1. - **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc` - **Brand/UI guidelines**: `frontend/brand-guidelines.md` +- **Domain vocabulary**: `CONTEXT.md` — the words this codebase uses for its own concepts (step, step setting, trigger step, …). Name things the way it does. - **CLI commands**: when adding/modifying/removing a command, subcommand, option, or description in `cli/src/commands/`, run `python system_prompts/generate.py` to refresh `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`. The CLI docs the agents use to operate `wmill` are derived from the source — stale generated files give agents the wrong flags. +- **Session recorder**: `frontend/src/lib/components/recording/` is also the recorder `wmill app dev --recording` serves, vendored into the CLI as `cli/src/commands/app/devRecorderBundle.gen.ts`. After changing `rawAppSnapshot.ts` or `rawAppRecording.svelte.ts`, run `bun run gen:dev-recorder` from `cli/` (`cli/test/dev_recorder_bundle_unit.test.ts` fails otherwise). ## Dev Environment +> **In a git worktree, the ports and database below are NOT the ones to use.** Each +> worktree gets its own backend port, frontend port and Postgres database, so the +> defaults in this section apply only to a plain single checkout. **Discover the real +> values before running anything** — see "Per-worktree ports and database" below. + - **Backend**: `cargo run` from `backend/` (API at http://localhost:8000) - **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing. The bundled DuckDB compile (~2min) is cached in a per-user dir shared across worktrees, so a fresh worktree reuses it and the build is near-instant. - **Data pipelines (DuckLake) from source**: a plain `cargo run` (even `--features quickjs`) advertises a `duckdb` worker tag but **cannot** execute DuckDB scripts and has **no** working S3 proxy (DuckLake writes 404). Build CE DuckLake with `cargo run --features quickjs,duckdb,parquet,private` (add `,python` for Python scripts, `,enterprise,license` for EE) **and** build the FFI (bullet above). See `backend/CLAUDE.md` → "Running data pipelines (DuckLake) from source" for the exact feature sets and the two feature-gate gotchas. @@ -33,6 +43,37 @@ Open-source platform for internal tools, workflows, API integrations, background - **Instance settings**: navigate to `/#superadmin-settings` - **Migrations**: use `cargo sqlx migrate add -r ` from `backend/` to create new migrations (never generate timestamps manually) +### Per-worktree ports and database + +A worktree's `.env` / `.env.local` (repo root) and `backend/.env` hold its own +`DATABASE_URL` and `PORT`; the database is typically `windmill_` +(branch `dbt-runtime` → `windmill_dbt_runtime`). Read them, or discover from what is +already running: + +```bash +psql postgres://postgres:changeme@localhost:5432/postgres -tAc \ + "select datname from pg_database where datname like 'windmill%'" | grep "$(git branch --show-current | tr - _)" +# the port the frontend actually proxies to (REMOTE of this worktree's vite): +for p in $(pgrep -f vite); do case "$(readlink /proc/$p/cwd)" in *"$(basename "$(git rev-parse --show-toplevel)")"*) + tr '\0' '\n' < /proc/$p/environ | grep -E '^REMOTE=|^PORT=';; esac; done +``` + +Getting these wrong is not a cheap mistake: + +- **`DATABASE_URL` pointed at another worktree's database silently destroys the sqlx + cache.** `cargo run` and `cargo sqlx prepare` both compile `sqlx::query!` against the + **live** database, so the wrong one fails with `relation "" does not + exist` — and `prepare` deletes the whole `.sqlx/` directory *before* it fails, leaving + it gutted. Always `cp -r backend/.sqlx /sqlx_backup` first (see the `update-sqlx` + skill). +- **The frontend proxies to its own worktree's backend port, not 8000.** Starting a + backend on the wrong port leaves the UI up but every API call 502s, which reads like an + application bug rather than a misconfiguration. +- **Kill backends by pid scoped to this worktree's cwd** (`readlink /proc//cwd`), + never `pkill -f target/debug/windmill` — that kills every sibling worktree's backend. + Beware that a `pgrep -f ""` in a shell whose own command line contains + `` matches the shell itself. + ## Verifying Frontend Changes After modifying frontend code, drive the running dev server with the **Playwright MCP** to verify the change in a real browser — don't claim a UI change works without exercising it. @@ -51,10 +92,34 @@ Typical flow: 5. `mcp__playwright__browser_take_screenshot` for visual confirmation 6. `mcp__playwright__browser_console_messages` / `browser_network_requests` to surface errors +Write screenshots to an absolute path under `/tmp` (the MCP servers already do; standalone +Playwright scripts must be told): moving a PNG out of the checkout afterwards needs a `mv` the +permission hooks always prompt on. Same reason to run `rm`/`mv`/`cp` as one plain command per Bash +call: those hooks defer on `&&`, `;`, redirects, quotes and `$VAR`. + **Attach the screenshots to the PR.** For any change under `frontend/`, embed screenshots of the affected UI in the PR body — the `pr` skill requires this and carries the upload recipe. If you cannot exercise a UI change (no dev server, etc.), say so explicitly rather than claiming success. +## Verifying Backend Changes + +`cargo check` and the unit tests do not exercise a worker code path. **If you changed how +a job runs — an executor, `handle_child`, anything spawning or reading from a +subprocess — run an actual job of that kind** and confirm it completed, then say so. +Whole classes of defect compile and unit-test clean: + +- **Stack overflow from a large buffer in an async block.** An array declared across an + `.await` is baked into the future's state; once that future is boxed a few layers deep + by the job poller, two 16 KB arrays abort the worker *process* (`thread + 'tokio-runtime-worker' has overflowed its stack`). Heap-allocate read buffers + (`vec![0u8; N]`, not `[0u8; N]`). +- Deadlocks from draining only one of a child's pipes, missed cancellation or timeout + propagation, and anything depending on the real engine's output format. + +A crash like this takes down every job on that worker, not just yours, so check the +backend log after the run rather than only the job's own status. If you cannot run one, +say which path went unexercised instead of implying it was verified. + ## Banned Patterns ### `$bindable(default_value)` on optional props diff --git a/CHANGELOG.md b/CHANGELOG.md index a69d7b0389..1b3283588c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,221 @@ # Changelog +## [1.783.0](https://github.com/windmill-labs/windmill/compare/v1.782.0...v1.783.0) (2026-08-07) + + +### Features + +* add public sharing option for job pages ([#10573](https://github.com/windmill-labs/windmill/issues/10573)) ([5ce29b3](https://github.com/windmill-labs/windmill/commit/5ce29b34364d6429411621752c64e306426aa2dd)) +* offer more dev workspace environment labels ([#10570](https://github.com/windmill-labs/windmill/issues/10570)) ([8c6211c](https://github.com/windmill-labs/windmill/commit/8c6211c27718912ad1e1a29f770747ea07fdb161)) +* open a session edit in the preview panel from the edits list ([#10486](https://github.com/windmill-labs/windmill/issues/10486)) ([d008975](https://github.com/windmill-labs/windmill/commit/d0089758e6091b07009dee5b8387f3f4b6d97455)) +* preview merge result in git-sync PR diff check ([#10542](https://github.com/windmill-labs/windmill/issues/10542)) ([c61404a](https://github.com/windmill-labs/windmill/commit/c61404a0f4008cd1de3c977c653c712ef3d9e0a3)) + + +### Bug Fixes + +* **frontend:** collapse the dev-workspace edit notice into a badge ([#10576](https://github.com/windmill-labs/windmill/issues/10576)) ([fdd76a6](https://github.com/windmill-labs/windmill/commit/fdd76a6f1358404b13326c68eed1eac14097bd88)) +* **frontend:** hide the fork workspace banner from operators ([#10575](https://github.com/windmill-labs/windmill/issues/10575)) ([57ed0f7](https://github.com/windmill-labs/windmill/commit/57ed0f77e1f83751ea04820b365fb53f098d38bf)) +* scope a fork's cloned app policy and custom path to its creator ([#10589](https://github.com/windmill-labs/windmill/issues/10589)) ([8e95bfe](https://github.com/windmill-labs/windmill/commit/8e95bfe6157ebe4f8fc5f6e13c36a6f0bdb9969d)) + +## [1.782.0](https://github.com/windmill-labs/windmill/compare/v1.781.3...v1.782.0) (2026-08-06) + + +### Features + +* improve duckdb isolation ([#10565](https://github.com/windmill-labs/windmill/issues/10565)) ([956210e](https://github.com/windmill-labs/windmill/commit/956210ea06224f7c6cd3e5082a96d3f80039404b)) + + +### Bug Fixes + +* compact ai chat context for models with unknown context windows ([#10564](https://github.com/windmill-labs/windmill/issues/10564)) ([fc1e11c](https://github.com/windmill-labs/windmill/commit/fc1e11cb3d056af6a3779cf17b23f9bf7152d355)) +* keep the mermaid fullscreen dialog in its pane and its emoji vector ([#10541](https://github.com/windmill-labs/windmill/issues/10541)) ([c2a6936](https://github.com/windmill-labs/windmill/commit/c2a6936e7d3fe724f968b10dd689abb6ff8f75ad)) + +## [1.781.3](https://github.com/windmill-labs/windmill/compare/v1.781.2...v1.781.3) (2026-08-06) + + +### Bug Fixes + +* **frontend:** take the editor's post-edit content, not setCode's argument ([#10562](https://github.com/windmill-labs/windmill/issues/10562)) ([1846bd5](https://github.com/windmill-labs/windmill/commit/1846bd5ce59ee71a4262297513673d662f6a38b6)) + +## [1.781.2](https://github.com/windmill-labs/windmill/compare/v1.781.1...v1.781.2) (2026-08-06) + + +### Bug Fixes + +* **frontend:** reset the editor content when the script language changes ([#10560](https://github.com/windmill-labs/windmill/issues/10560)) ([56ea133](https://github.com/windmill-labs/windmill/commit/56ea133366875d8bd13347b5002f4ef351a8e4e6)) + +## [1.781.1](https://github.com/windmill-labs/windmill/compare/v1.781.0...v1.781.1) (2026-08-06) + + +### Bug Fixes + +* open the expression property column on demand, not from focus ([#10558](https://github.com/windmill-labs/windmill/issues/10558)) ([386c66b](https://github.com/windmill-labs/windmill/commit/386c66bef0534d4a63b3220050e98b9654ef8f34)) +* restore the flow expression editor's property side panel ([#10555](https://github.com/windmill-labs/windmill/issues/10555)) ([e4e7782](https://github.com/windmill-labs/windmill/commit/e4e7782517a9fd0fbbec020694b263a948ed1c0c)) + + +### Performance Improvements + +* keep run status out of flow graph node and edge data ([#10554](https://github.com/windmill-labs/windmill/issues/10554)) ([b2d38e0](https://github.com/windmill-labs/windmill/commit/b2d38e0391f26d92e8f9245a3db099129b6c3977)) + +## [1.781.0](https://github.com/windmill-labs/windmill/compare/v1.780.0...v1.781.0) (2026-08-05) + + +### Features + +* add a load all to the tree view's per-folder pager ([#10548](https://github.com/windmill-labs/windmill/issues/10548)) ([a3b79d7](https://github.com/windmill-labs/windmill/commit/a3b79d7732deba54fd6344229d5b91a80a3e0dd6)) +* allow a dev workspace to have its own dev workspace ([#10534](https://github.com/windmill-labs/windmill/issues/10534)) ([e203ab0](https://github.com/windmill-labs/windmill/commit/e203ab087a4ce885a1d424267f96db861207a0e6)) +* **debugger:** install debug session deps from the instance registry settings ([#10550](https://github.com/windmill-labs/windmill/issues/10550)) ([154f8f4](https://github.com/windmill-labs/windmill/commit/154f8f461ef01d60da73f9633d76bed45381a035)) +* **mcp:** serve the 2026-07-28 spec alongside the legacy protocol ([#10535](https://github.com/windmill-labs/windmill/issues/10535)) ([4fe4fac](https://github.com/windmill-labs/windmill/commit/4fe4fac358e67ee6e20dbff8bfa1de0e92e5f3c7)) +* redesign flow step, loop and branch settings panels ([#10026](https://github.com/windmill-labs/windmill/issues/10026)) ([09c8f3b](https://github.com/windmill-labs/windmill/commit/09c8f3b1f3eef2409e212cdb26c7bd2f830dae89)) +* register mounted CA certificates in windmill_extra at startup ([#10545](https://github.com/windmill-labs/windmill/issues/10545)) ([d9d6ec8](https://github.com/windmill-labs/windmill/commit/d9d6ec82ab7ad5279ba8ebc059f58bd663bad87f)) + + +### Bug Fixes + +* **ai:** collapse thinking to a status row with a thought-for duration ([#10515](https://github.com/windmill-labs/windmill/issues/10515)) ([d9b10e7](https://github.com/windmill-labs/windmill/commit/d9b10e7b0a1ba6702ecf05f64b80ac8bb7c79326)) +* **debugger:** confine prepare-deps under nsjail in both language paths ([#10546](https://github.com/windmill-labs/windmill/issues/10546)) ([a5423a8](https://github.com/windmill-labs/windmill/commit/a5423a81ca5d7c1e88ebed4573952e858ef1546c)) +* **debugger:** pass python index settings to prepare-deps and report failures ([#10533](https://github.com/windmill-labs/windmill/issues/10533)) ([7d153d5](https://github.com/windmill-labs/windmill/commit/7d153d5750db0ea17812d6b1f9a63ceaefddc2b9)) +* **debugger:** report python debugger dependency install failures instead of timing out ([#10531](https://github.com/windmill-labs/windmill/issues/10531)) ([29e179f](https://github.com/windmill-labs/windmill/commit/29e179f7879845f01c6813ad5dfbb19d08dc030e)) +* edit-in-dev-workspace dead-ends, wraps, and misses the tree view ([#10354](https://github.com/windmill-labs/windmill/issues/10354)) ([616d4fe](https://github.com/windmill-labs/windmill/commit/616d4fe167e1c99f8f5104c7b66dbc5fd502e0be)) +* flow step picker layout and single hover/keyboard highlight ([#10488](https://github.com/windmill-labs/windmill/issues/10488)) ([aa91619](https://github.com/windmill-labs/windmill/commit/aa91619bb67b7ae0cbd3a3848b6171d36967bd1f)) +* forward TLS trust roots to debug sessions and honor INIT_SCRIPT on windmill_extra ([#10532](https://github.com/windmill-labs/windmill/issues/10532)) ([74c4185](https://github.com/windmill-labs/windmill/commit/74c418570bb5bf1af54f367cfecae3e8b4ede764)) +* **frontend:** draw the tab strip's scroll bar instead of the native one ([#10547](https://github.com/windmill-labs/windmill/issues/10547)) ([2c189fe](https://github.com/windmill-labs/windmill/commit/2c189fea14749b8bb4604b1363eef907abc3b129)) +* **frontend:** filter the AI Sandbox entry by the flow insert search ([#10529](https://github.com/windmill-labs/windmill/issues/10529)) ([1dcb6bb](https://github.com/windmill-labs/windmill/commit/1dcb6bb9005c1e97f1fca6470eb2d2105a973918)) +* keep the same_worker pin across a flow module that spawns no job ([#10551](https://github.com/windmill-labs/windmill/issues/10551)) ([1aee222](https://github.com/windmill-labs/windmill/commit/1aee22296e228c54d4ca7251e9ae7e04229adb34)) +* keep the same_worker pin when a suspend ends without approval ([#10552](https://github.com/windmill-labs/windmill/issues/10552)) ([c59b60c](https://github.com/windmill-labs/windmill/commit/c59b60c729b03c5be66738b1f4cbf0d989a7b438)) +* point re-opened previews at the tab already showing them ([#10538](https://github.com/windmill-labs/windmill/issues/10538)) ([f11e883](https://github.com/windmill-labs/windmill/commit/f11e8835fdc762fccb521de48049ea43838d7f3c)) +* **triggers:** stop one failing trigger count from zeroing the rest ([#10549](https://github.com/windmill-labs/windmill/issues/10549)) ([0e42381](https://github.com/windmill-labs/windmill/commit/0e42381df086d4463e5914d8a194fe5141793bd1)) + +## [1.780.0](https://github.com/windmill-labs/windmill/compare/v1.779.0...v1.780.0) (2026-08-04) + + +### Features + +* **dbt:** reach any dbt adapter through a dbt_profile resource, and constrain the warehouse picker ([#10525](https://github.com/windmill-labs/windmill/issues/10525)) ([340d3cd](https://github.com/windmill-labs/windmill/commit/340d3cd5653fb7f23ba349517c1ab9a2a74bba58)) + + +### Bug Fixes + +* reflect custom tag add/remove in the manage-tags drawer immediately ([#10526](https://github.com/windmill-labs/windmill/issues/10526)) ([552ad9c](https://github.com/windmill-labs/windmill/commit/552ad9c859497d3b2d6d20ecf3103b06758cada7)) + +## [1.779.0](https://github.com/windmill-labs/windmill/compare/v1.778.0...v1.779.0) (2026-08-04) + + +### Features + +* add multi-select and bulk actions to the Home page ([#10499](https://github.com/windmill-labs/windmill/issues/10499)) ([c57045d](https://github.com/windmill-labs/windmill/commit/c57045dbb30aa40a212a9ee179f5da78c0f1a405)) +* deploy a raw app from its sources, bundling them on a worker ([#10500](https://github.com/windmill-labs/windmill/issues/10500)) ([81ba961](https://github.com/windmill-labs/windmill/commit/81ba9611eb7cdbb67a285337d2069c3909d10b69)) +* **frontend:** pick workspace members from a searchable instance user list ([#10474](https://github.com/windmill-labs/windmill/issues/10474)) ([d63315e](https://github.com/windmill-labs/windmill/commit/d63315ed356afaedc60e3fcf8427cd2c2e822ba0)) +* let the database manager run its jobs on a custom worker tag ([#10516](https://github.com/windmill-labs/windmill/issues/10516)) ([221566d](https://github.com/windmill-labs/windmill/commit/221566d2824efbb3f1ac6f3c543633a5783c085a)) +* make createApp and updateApp the full-code app tools over MCP ([#10510](https://github.com/windmill-labs/windmill/issues/10510)) ([f3e73fb](https://github.com/windmill-labs/windmill/commit/f3e73fb006ab316c422a6b318fe21946bff34d13)) +* preview and edit steps inside expanded subflows ([#10520](https://github.com/windmill-labs/windmill/issues/10520)) ([0d1cb81](https://github.com/windmill-labs/windmill/commit/0d1cb818eec35de9f1af8963b0f9138dc89b0192)) +* send prompt_cache_key on the openai responses api ([#10507](https://github.com/windmill-labs/windmill/issues/10507)) ([8364dd9](https://github.com/windmill-labs/windmill/commit/8364dd96ef355aa01516aa53aac1a9f01fadaa80)) +* show far more in the home tree view and say what is not loaded ([#10519](https://github.com/windmill-labs/windmill/issues/10519)) ([4254686](https://github.com/windmill-labs/windmill/commit/425468675808d12b71f5fefde3bccce38e74869d)) + + +### Bug Fixes + +* bound ScopeSelector badge heights and correct three scope-chip defects ([#10523](https://github.com/windmill-labs/windmill/issues/10523)) ([6d21d30](https://github.com/windmill-labs/windmill/commit/6d21d30242632549a1698b3e82ed4f55895ea51e)) +* bundle a vector emoji font so emoji scale with flow graph zoom ([#10498](https://github.com/windmill-labs/windmill/issues/10498)) ([62b2f4b](https://github.com/windmill-labs/windmill/commit/62b2f4b06768b5559b59761760899076476b1e47)) +* keep the token scope builder inside its panel when scopes get long ([#10517](https://github.com/windmill-labs/windmill/issues/10517)) ([ae2f584](https://github.com/windmill-labs/windmill/commit/ae2f584de88f0b0eb34ec270c9dfb109cab54938)) +* let admins edit the dev workspace lock ruleset ([#10512](https://github.com/windmill-labs/windmill/issues/10512)) ([ecae932](https://github.com/windmill-labs/windmill/commit/ecae9320d0f752d01af12da1834e15d741b443d0)) +* load the workspace AI config even when the docked chat is disabled ([#10493](https://github.com/windmill-labs/windmill/issues/10493)) ([99e7661](https://github.com/windmill-labs/windmill/commit/99e7661a1b1b24a148a07f7dd37d44e3ae963ba2)) +* locate coursier artifacts by coordinate so private maven registries work ([#10501](https://github.com/windmill-labs/windmill/issues/10501)) ([56d53d3](https://github.com/windmill-labs/windmill/commit/56d53d3f328cc5a7002dbc1d31e2a90821590663)) +* log the db auth mode used and hint at the ms_entraid sentinel ([#10508](https://github.com/windmill-labs/windmill/issues/10508)) ([9f3d155](https://github.com/windmill-labs/windmill/commit/9f3d15583a863c50d60925e97fe98f35f9e4c9e2)) +* match openrouter model ids by parsed vendor, not raw prefix ([#10497](https://github.com/windmill-labs/windmill/issues/10497)) ([546b8d0](https://github.com/windmill-labs/windmill/commit/546b8d076961398a40e7e53fb1e3e561a39a135b)) +* open an AI session from the editor bar's AI button, on the step ([#10504](https://github.com/windmill-labs/windmill/issues/10504)) ([ea4f3ec](https://github.com/windmill-labs/windmill/commit/ea4f3ecc6e54f24a0899470c2efb9850d3ed2d48)) +* show which MCP endpoint tools a token scope will actually expose ([#10514](https://github.com/windmill-labs/windmill/issues/10514)) ([71e68e8](https://github.com/windmill-labs/windmill/commit/71e68e87dcc34939f9262d3fdf2e0e5495332491)) +* stop app updates from silently converting an app between raw and low-code ([#10495](https://github.com/windmill-labs/windmill/issues/10495)) ([d9dd036](https://github.com/windmill-labs/windmill/commit/d9dd036edc88124f2424f13e0ed3697cf03bdf99)) + +## [1.778.0](https://github.com/windmill-labs/windmill/compare/v1.777.1...v1.778.0) (2026-08-03) + + +### Features + +* **apps:** use the windmill-client SDK from raw app frontend code ([#10377](https://github.com/windmill-labs/windmill/issues/10377)) ([7e1c1fa](https://github.com/windmill-labs/windmill/commit/7e1c1fa3a455b8187886692567e0045658ed7013)) +* azure workload identity auth for mssql and postgres resources ([#10470](https://github.com/windmill-labs/windmill/issues/10470)) ([beef6e2](https://github.com/windmill-labs/windmill/commit/beef6e295cb4e01af00c279f406012a82d91c8c4)) +* **fork:** merge a fork deletion on evidence, not on the counters ([#10484](https://github.com/windmill-labs/windmill/issues/10484)) ([f365929](https://github.com/windmill-labs/windmill/commit/f365929eaaebf546c58ee1641a010d5f1f8b555f)) +* label ai session preview tabs by item summary ([#10475](https://github.com/windmill-labs/windmill/issues/10475)) ([b93ee66](https://github.com/windmill-labs/windmill/commit/b93ee66bba72250c17d4c7f567436b348504c2b1)) +* lazily expand s3 explorer folders one level at a time ([#10420](https://github.com/windmill-labs/windmill/issues/10420)) ([386e115](https://github.com/windmill-labs/windmill/commit/386e115daaf850ac45d10f2240bacb5758d238e6)) + + +### Bug Fixes + +* anchor overlays to their preview tab instead of the viewport ([#10477](https://github.com/windmill-labs/windmill/issues/10477)) ([b718ea8](https://github.com/windmill-labs/windmill/commit/b718ea84956995d96f4056cdbd03bd8634883921)) +* base a new fork on the dev workspace when forking from one ([#10489](https://github.com/windmill-labs/windmill/issues/10489)) ([db46d34](https://github.com/windmill-labs/windmill/commit/db46d347272c940057dca1aa7fb4b4456eea8a9e)) +* **cli:** stop a deleted raw-app .lock from dropping the whole app push ([#10473](https://github.com/windmill-labs/windmill/issues/10473)) ([4254324](https://github.com/windmill-labs/windmill/commit/42543240e87ad232f6dffde61d47cfaced766b20)) +* confirm step delete consequences in a single dialog ([#10485](https://github.com/windmill-labs/windmill/issues/10485)) ([5aeee56](https://github.com/windmill-labs/windmill/commit/5aeee564beaab5b398b0eff1ae9db8e9ef1d9bfe)) +* normalize the pipeline folder so AI node paths aren't double-prefixed ([#10479](https://github.com/windmill-labs/windmill/issues/10479)) ([4db432f](https://github.com/windmill-labs/windmill/commit/4db432f0048c9295e93ff7da0753eef7145d3623)) +* refresh the dev-workspace pairing after attach and detach ([#10491](https://github.com/windmill-labs/windmill/issues/10491)) ([fadbe07](https://github.com/windmill-labs/windmill/commit/fadbe079e4e747147861c6df495270cf2f390700)) +* report why a native trigger service refused instead of a 500 ([#10463](https://github.com/windmill-labs/windmill/issues/10463)) ([4f03aa9](https://github.com/windmill-labs/windmill/commit/4f03aa91a714eb0132cf01a6cd4f37a1a11cfe02)) +* show the fork banner to a superadmin who is not a workspace member ([#10465](https://github.com/windmill-labs/windmill/issues/10465)) ([3cef678](https://github.com/windmill-labs/windmill/commit/3cef678b5f247ec431e89cfc2a6a97e79dce2b51)) +* stop reading a parent-only fork item as deleted in the fork ([#10467](https://github.com/windmill-labs/windmill/issues/10467)) ([689f5d7](https://github.com/windmill-labs/windmill/commit/689f5d7c751f5fd8d4ceb7a79004a22afb8263b8)) +* stop the chat deleting drafts as deployed workspace items ([#10476](https://github.com/windmill-labs/windmill/issues/10476)) ([ed9fb47](https://github.com/windmill-labs/windmill/commit/ed9fb47f5a9e05aa8380d288d892f8068eaf1d0c)) + + +### Performance Improvements + +* cut ai chat request size and cache the prefix on openrouter ([#10481](https://github.com/windmill-labs/windmill/issues/10481)) ([e15bea4](https://github.com/windmill-labs/windmill/commit/e15bea419a1a65f6d9b577e918d5e617e0345da2)) +* serve the rare schedule options on demand instead of inlining them ([#10487](https://github.com/windmill-labs/windmill/issues/10487)) ([a3c59c7](https://github.com/windmill-labs/windmill/commit/a3c59c7320be961a88b581ab06ded24b80e1d544)) + +## [1.777.1](https://github.com/windmill-labs/windmill/compare/v1.777.0...v1.777.1) (2026-08-03) + + +### Bug Fixes + +* return result.json and stdout results from sandboxed containers ([#10460](https://github.com/windmill-labs/windmill/issues/10460)) ([d509551](https://github.com/windmill-labs/windmill/commit/d5095515ed007d4e7fbfc1a15582c550d3293ece)) + +## [1.777.0](https://github.com/windmill-labs/windmill/compare/v1.776.0...v1.777.0) (2026-08-03) + + +### Features + +* add session recording to wmill app dev ([#10457](https://github.com/windmill-labs/windmill/issues/10457)) ([2105540](https://github.com/windmill-labs/windmill/commit/2105540cca7cc7bdced9e06ef3ad1ed54732feef)) +* give dbt its own editor with an explicitly refreshed model graph ([#10448](https://github.com/windmill-labs/windmill/issues/10448)) ([baefa13](https://github.com/windmill-labs/windmill/commit/baefa1345b7b7a4110257a53a417d4fc229d0149)) +* show an on-behalf-of badge on the script and flow detail pages ([#10452](https://github.com/windmill-labs/windmill/issues/10452)) ([9bafb7e](https://github.com/windmill-labs/windmill/commit/9bafb7ebd2b8571f1adf5a86b681af8d07660a50)) + + +### Bug Fixes + +* keep uri and method on request logs under RUST_LOG=error ([#10462](https://github.com/windmill-labs/windmill/issues/10462)) ([0466ea2](https://github.com/windmill-labs/windmill/commit/0466ea201952bd382d893bd973eedaa7183b597e)) +* report a missing worker tag instead of spinning in data table UIs ([#10456](https://github.com/windmill-labs/windmill/issues/10456)) ([eca24bd](https://github.com/windmill-labs/windmill/commit/eca24bdfb5c7a49c428d541ca06407d1635cc3b6)) + +## [1.776.0](https://github.com/windmill-labs/windmill/compare/v1.775.2...v1.776.0) (2026-08-01) + + +### Features + +* **frontend:** add missing resource type icons and show them in the resource picker ([#10407](https://github.com/windmill-labs/windmill/issues/10407)) ([705c90d](https://github.com/windmill-labs/windmill/commit/705c90debda87451fcdc32f15f086080ab7dc5f9)) +* **git-sync:** dedicated base url for GitHub webhook delivery ([#10411](https://github.com/windmill-labs/windmill/issues/10411)) ([318c9f0](https://github.com/windmill-labs/windmill/commit/318c9f00739bfd43bd5d6afed4ae12f3f1a565d1)) +* let the merge UI target an arbitrary workspace ([#10417](https://github.com/windmill-labs/windmill/issues/10417)) ([61f2d8d](https://github.com/windmill-labs/windmill/commit/61f2d8dc6ab980c1aba959b853d505f09299ed9c)) +* make job subprocess oom_score_adj configurable ([#10443](https://github.com/windmill-labs/windmill/issues/10443)) ([2508417](https://github.com/windmill-labs/windmill/commit/25084170d7449dfa06dd758bc3783580916cfa95)) +* make the fork lineage the only deploy relationship ([#10410](https://github.com/windmill-labs/windmill/issues/10410)) ([81b23a2](https://github.com/windmill-labs/windmill/commit/81b23a2ba0ee1001241e4fe6066c07526ab640f6)) +* run dbt projects as a first-class Windmill runtime ([#10326](https://github.com/windmill-labs/windmill/issues/10326)) ([032300e](https://github.com/windmill-labs/windmill/commit/032300e28eba9f8e790f16e894bb00fff22eb296)) +* stamp webhook trigger_kind on token-driven job runs ([#10431](https://github.com/windmill-labs/windmill/issues/10431)) ([dda5976](https://github.com/windmill-labs/windmill/commit/dda59767c2b997e675ffe799bc6b1dcc79c2a52d)) +* sync data table migrations to git, gated by a new object type ([#10436](https://github.com/windmill-labs/windmill/issues/10436)) ([bfc3f52](https://github.com/windmill-labs/windmill/commit/bfc3f5242a0d9a833d12859f96ea8f1aa67d362b)) + + +### Bug Fixes + +* add apps:run to the token scope picker and confine path-scoped app tokens ([#10428](https://github.com/windmill-labs/windmill/issues/10428)) ([c69f080](https://github.com/windmill-labs/windmill/commit/c69f08073a657ee6d91bfdbd19e8639a392fd77a)) +* **ai:** pass only the output of a nested agent tool to the parent ([#10416](https://github.com/windmill-labs/windmill/issues/10416)) ([7d097d2](https://github.com/windmill-labs/windmill/commit/7d097d25c3bba89d708c21a2fc3a1a8e0099a5c5)) +* **ai:** route Azure OpenAI agent steps through the Responses API ([#10404](https://github.com/windmill-labs/windmill/issues/10404)) ([94bcc00](https://github.com/windmill-labs/windmill/commit/94bcc00554423eb9e4056dd33ceff1b07263e9ec)) +* app progress bar stuck on running, and misreporting queued/canceled jobs as errors ([#10409](https://github.com/windmill-labs/windmill/issues/10409)) ([5579913](https://github.com/windmill-labs/windmill/commit/557991360a5c920909cac45e335c11b35bd2d880)) +* apply default workspace dependencies to raw app runnables ([#10427](https://github.com/windmill-labs/windmill/issues/10427)) ([38b6099](https://github.com/windmill-labs/windmill/commit/38b6099b4c5039232cf306a01ee814b3ec5dc04c)) +* carry the token label into job-run audit rows ([#10433](https://github.com/windmill-labs/windmill/issues/10433)) ([02c4a9e](https://github.com/windmill-labs/windmill/commit/02c4a9e515b3ef9c0e759211a6b7cbd874a778b0)) +* **cli:** lint against the checkout's schema, not the published validator ([#10418](https://github.com/windmill-labs/windmill/issues/10418)) ([a372ae0](https://github.com/windmill-labs/windmill/commit/a372ae0c04d0849932d83a76d38b5a9e507077af)) +* credit the token owner instead of the token label in the audit trail ([#10423](https://github.com/windmill-labs/windmill/issues/10423)) ([3716a71](https://github.com/windmill-labs/windmill/commit/3716a71fd76f66b58bc29977b4f6a10ed97cea16)) +* **flows:** mint fresh orchestration token so long steps don't expire the result-fetch JWT ([#10415](https://github.com/windmill-labs/windmill/issues/10415)) ([2e249ff](https://github.com/windmill-labs/windmill/commit/2e249ff8922c152f410cd40b88514f5dad875b85)) +* **forks:** record fork changes that never reached the diff tally ([#10403](https://github.com/windmill-labs/windmill/issues/10403)) ([f9a547b](https://github.com/windmill-labs/windmill/commit/f9a547b8b8e4982346607acae4e8e7646529c374)) +* harden flow-orchestration token refresh (mint from job_perms) ([#10419](https://github.com/windmill-labs/windmill/issues/10419)) ([e0d6dc1](https://github.com/windmill-labs/windmill/commit/e0d6dc1a1997514bfcaa8615de61daff4a2f81ca)) +* honor on-behalf-of when a workflow step dispatches a script or flow ([#10437](https://github.com/windmill-labs/windmill/issues/10437)) ([2b525d2](https://github.com/windmill-labs/windmill/commit/2b525d28dbcda4a4b0f626ad40b0c4d79cfbcd53)) +* keep native triggers attached when a runnable is renamed ([#10432](https://github.com/windmill-labs/windmill/issues/10432)) ([bd71566](https://github.com/windmill-labs/windmill/commit/bd7156682d4d4b78e01fa316580e632152cc5b62)) +* make /usr/bin/coursier self-contained so java jobs work air-gapped ([#10414](https://github.com/windmill-labs/windmill/issues/10414)) ([7e64960](https://github.com/windmill-labs/windmill/commit/7e649604db2047bf8587b46628aa74051e1b9409)) +* make on_behalf_of control permissions for scripts and flows ([#10438](https://github.com/windmill-labs/windmill/issues/10438)) ([fb82748](https://github.com/windmill-labs/windmill/commit/fb82748296cd0f81ad8d21c30c12e172a6477173)) +* pre-warm coursier bootstrap cache at the worker's cache path ([#10413](https://github.com/windmill-labs/windmill/issues/10413)) ([ed9dfc5](https://github.com/windmill-labs/windmill/commit/ed9dfc5de684197e14f3c4a3a6873c0e1a55229a)) +* run the init script before dedicated workers install dependencies ([#10412](https://github.com/windmill-labs/windmill/issues/10412)) ([43a684d](https://github.com/windmill-labs/windmill/commit/43a684d7432837fcf00a97712f28d7c474a58402)) +* sidebar workspace toggle navigates home when already in workspace mode ([#10405](https://github.com/windmill-labs/windmill/issues/10405)) ([9cd6a70](https://github.com/windmill-labs/windmill/commit/9cd6a70f6ace956f679c1ce1fb46543cda569183)) + ## [1.775.2](https://github.com/windmill-labs/windmill/compare/v1.775.1...v1.775.2) (2026-07-29) diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000000..6efb92b669 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,38 @@ +# Windmill + +Open-source platform for internal tools, workflows, API integrations, background jobs and UIs. This file pins the vocabulary that is specific to Windmill's domain, so that code, docs and reviews name the same thing the same way. + +## Language + +### Flows + +**Step**: +One node of a flow — the unit a user selects in the graph and configures in the right-hand panel. Typed as `FlowModule` in code. +_Avoid_: module (ambiguous with the architectural sense), node, action + +**Step setting**: +A per-step runtime option stored on the step itself: retries, error handling, timeout, concurrency limit, priority, cache, debounce, early stop, skip, suspend, sleep, lifetime. Distinct from the step's inputs and its code. The panel that edits them is the **run settings** tab; a single setting is still a step setting. +_Avoid_: advanced setting, step config, flow option + +**Configured**: +Said of a step setting whose config object is present on the step. Deliberately not the same as "would change the runtime's behaviour" — a setting can be configured and still be a no-op (`sleep` of `0`). Every surface that answers "is this setting on?" answers it this way. +_Avoid_: enabled, active, effective + +**Trigger step**: +The first step of a polling flow. It runs on a schedule and returns the items found since its last run; an empty return means there is nothing to process and the flow stops early, marked skipped rather than failed. +_Avoid_: poll script, trigger node, schedule step + +**Default predicate**: +The `stop_after_if` expression seeded onto a trigger step at creation, encoding what "nothing new" looks like. One value, owned in one place, shared by every path that creates a trigger step. + +**Connect**: +Arming an input so that the next property picked fills it. A property can be picked from the prop picker or, when the panel is docked beside the graph, by clicking a step node's output. At most one input is armed per panel, so a pick always has exactly one destination. +_Avoid_: link, bind, plug (the icon is a plug; the action is connecting) + +**Step input**: +One argument of a step, edited in the step's input form. Its prop picker is a pane beside the form, always visible, so previous results can be browsed without connecting. +_Avoid_: argument field, param + +**Expression input**: +Any other place a property can be picked into: the loop iterator, skip and early-stop predicates, the retry condition, a branch predicate, timeout. Its prop picker opens in a popover from the connect button rather than taking a pane. +_Avoid_: JS field, code input diff --git a/ai_evals/adapters/frontend/core/global/evalArtifactHelpers.test.ts b/ai_evals/adapters/frontend/core/global/evalArtifactHelpers.test.ts new file mode 100644 index 0000000000..c74f341735 --- /dev/null +++ b/ai_evals/adapters/frontend/core/global/evalArtifactHelpers.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, it } from "bun:test"; +import { createEvalArtifactHelpers } from "./evalArtifactStore"; + +// A hand-written stand-in for SessionArtifactsStore (bun has no IndexedDB), so nothing +// makes it follow that class. A method missing from it surfaces as a tool throwing +// part-way through an eval run, which reads as a model failure rather than a harness one. +describe("eval artifact store", () => { + it("exposes every method the artifact tools call", () => { + const { helpers } = createEvalArtifactHelpers(); + for (const method of [ + "create", + "get", + "update", + "remove", + "listForSession", + "listVersions", + "getVersion", + ]) { + expect(typeof (helpers.artifacts as any)[method]).toBe("function"); + } + }); +}); diff --git a/ai_evals/adapters/frontend/core/global/evalArtifactStore.ts b/ai_evals/adapters/frontend/core/global/evalArtifactStore.ts new file mode 100644 index 0000000000..ed67cce468 --- /dev/null +++ b/ai_evals/adapters/frontend/core/global/evalArtifactStore.ts @@ -0,0 +1,93 @@ +// SessionArtifactsStore can't run here (bun has no IndexedDB, nor the compiled $state runes), +// so mirror only the shape the artifact tools call, not its scoping or race handling. +export const EVAL_SESSION_ID = "eval-session"; +export function createEvalArtifactHelpers() { + const items = new Map>(); + // Snapshots per artifact id, oldest first — the version tools read history from here. + const history = new Map>>(); + let seq = 0; + const snapshotOf = ( + artifact: Record, + version: number, + note?: string, + ) => ({ + key: `${artifact.id}:${version}`, + artifactId: artifact.id, + version, + name: artifact.name, + content: artifact.content, + savedAt: artifact.updatedAt, + note, + }); + const store = { + create: async (sessionId: string, input: Record) => { + const now = seq++; + const artifact = { + id: `eval-artifact-${now}`, + sessionId, + chatId: input.chatId, + kind: input.kind ?? "md", + name: input.name, + content: input.content, + createdAt: now, + updatedAt: now, + version: 1, + }; + items.set(artifact.id, artifact); + history.set(artifact.id, [snapshotOf(artifact, 1)]); + return artifact; + }, + get: async (id: string) => items.get(id), + update: async ( + id: string, + input: Record, + opts?: { sessionId?: string }, + ) => { + const existing = items.get(id); + if (!existing) return undefined; + if ( + opts?.sessionId !== undefined && + existing.sessionId !== opts.sessionId + ) + return undefined; + // Only a content change earns a version, as in SessionArtifactsStore. + const contentChanged = + input.content !== undefined && input.content !== existing.content; + const version = (existing.version ?? 1) + (contentChanged ? 1 : 0); + const updated = { + ...existing, + name: input.name ?? existing.name, + content: input.content ?? existing.content, + updatedAt: seq++, + version, + }; + items.set(id, updated); + if (contentChanged) { + history.set(id, [ + ...(history.get(id) ?? []), + snapshotOf(updated, version, input.note), + ]); + } + return updated; + }, + remove: async (id: string) => { + items.delete(id); + history.delete(id); + }, + listForSession: async (sessionId: string) => + [...items.values()].filter((a) => a.sessionId === sessionId), + listVersions: async (id: string) => + [...(history.get(id) ?? [])].sort((a, b) => b.version - a.version), + getVersion: async (id: string, version: number) => + (history.get(id) ?? []).find((v) => v.version === version), + }; + return { + helpers: { + artifacts: store, + sessionId: EVAL_SESSION_ID, + getChatId: () => "eval-chat", + openArtifact: () => {}, + }, + snapshot: () => [...items.values()], + }; +} diff --git a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts index 8fe28d022a..dcd771663a 100644 --- a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts @@ -14,6 +14,7 @@ import { } from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter"; import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared"; import { UserDraft } from "../../../../../frontend/src/lib/userDraft.svelte"; +import { createEvalArtifactHelpers } from "./evalArtifactStore"; import type { ModeRunContext } from "../../../../core/types"; import type { GlobalDraftState } from "../../../../core/validators"; import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings"; @@ -43,67 +44,6 @@ const LIVE_EDITOR_ITEM_KINDS = { app: "raw_app", } as const; -// SessionArtifactsStore can't run here (bun has no IndexedDB, nor the compiled $state runes), -// so mirror only its tool-facing shape; its own logic (scoping, race guard) is unit-tested. -const EVAL_SESSION_ID = "eval-session"; -function createEvalArtifactHelpers() { - const items = new Map>(); - let seq = 0; - const store = { - create: async (sessionId: string, input: Record) => { - const now = seq++; - const artifact = { - id: `eval-artifact-${now}`, - sessionId, - chatId: input.chatId, - kind: input.kind ?? "md", - name: input.name, - content: input.content, - createdAt: now, - updatedAt: now, - }; - items.set(artifact.id, artifact); - return artifact; - }, - get: async (id: string) => items.get(id), - update: async ( - id: string, - input: Record, - opts?: { sessionId?: string }, - ) => { - const existing = items.get(id); - if (!existing) return undefined; - if ( - opts?.sessionId !== undefined && - existing.sessionId !== opts.sessionId - ) - return undefined; - const updated = { - ...existing, - name: input.name ?? existing.name, - content: input.content ?? existing.content, - updatedAt: seq++, - }; - items.set(id, updated); - return updated; - }, - remove: async (id: string) => { - items.delete(id); - }, - listForSession: async (sessionId: string) => - [...items.values()].filter((a) => a.sessionId === sessionId), - }; - return { - helpers: { - artifacts: store, - sessionId: EVAL_SESSION_ID, - getChatId: () => "eval-chat", - openArtifact: () => {}, - }, - snapshot: () => [...items.values()], - }; -} - export interface GlobalLiveEditorDraftFixture { type: keyof typeof LIVE_EDITOR_ITEM_KINDS; storagePath?: string; diff --git a/ai_evals/adapters/frontend/mockBackend.ts b/ai_evals/adapters/frontend/mockBackend.ts index 40a6dfa4ad..b0b5bbabe2 100644 --- a/ai_evals/adapters/frontend/mockBackend.ts +++ b/ai_evals/adapters/frontend/mockBackend.ts @@ -561,6 +561,35 @@ export function runBenchmarkScriptPreview(input: { }) } +export function runBenchmarkScriptByPath(input: { + workspace: string + path: string + args?: Record +}): string { + const script = getBenchmarkScriptByPath(input.workspace, input.path) + return createBenchmarkCompletedJob({ + workspace: input.workspace, + jobKind: 'script', + success: script !== null, + scriptPath: input.path, + args: input.args, + result: + script !== null + ? { + path: input.path, + args: input.args ?? {}, + mocked: true + } + : { + error: `Script "${input.path}" not found in benchmark workspace` + }, + logs: + script !== null + ? 'Mock benchmark script run completed successfully.' + : `Script "${input.path}" not found in benchmark workspace.` + }) +} + export function runBenchmarkFlowByPath(input: { workspace: string path: string @@ -747,6 +776,27 @@ const BENCHMARK_MCP_TOOLS: EndpointTool[] = [ required: ['workspace'] } }, + { + name: 'getJob', + description: 'get job', + instructions: '', + path: '/w/{workspace}/jobs_u/get/{id}', + method: 'GET', + path_params_schema: { + type: 'object', + properties: { workspace: { type: 'string' }, id: { type: 'string', format: 'uuid' } }, + required: ['workspace', 'id'] + }, + query_params_schema: { + type: 'object', + properties: { + no_logs: { type: 'boolean' }, + no_code: { type: 'boolean' }, + approval_token: { type: 'string' } + }, + required: [] + } + }, { name: 'runScriptByPath', description: 'Run the deployed version of a script by path', @@ -968,6 +1018,26 @@ const BENCHMARK_WORKERS = [ } ] +const BENCHMARK_JOB_GET_PATH = /^\/api\/w\/([^/]+)\/jobs_u\/get\/([^/]+)$/ +const BENCHMARK_RUN_BY_PATH = /^\/api\/w\/([^/]+)\/jobs\/run\/(p|f)\/([^/]+)$/ + +/** `executeEndpoint` sends a JSON string; anything else means no args were supplied. */ +function parseBenchmarkRequestBody( + body: BodyInit | null | undefined +): Record | undefined { + if (typeof body !== 'string') { + return undefined + } + try { + const parsed = JSON.parse(body) + return typeof parsed === 'object' && parsed !== null + ? (parsed as Record) + : undefined + } catch { + return undefined + } +} + /** True when `handleBenchmarkApiFetch` has an answer for this `/api/...` url. * Any other relative fetch must keep its normal (non-benchmark) behavior — * intercepting it with a synthetic 404 sends the model into retry loops. */ @@ -975,6 +1045,8 @@ export function hasBenchmarkApiHandler(url: string): boolean { const path = url.split('?')[0] return ( path === '/api/workers/list' || + BENCHMARK_JOB_GET_PATH.test(path) || + BENCHMARK_RUN_BY_PATH.test(path) || /^\/api\/w\/[^/]+\/jobs\/queue\/list$/.test(path) || path === '/api/embeddings/query_hub_scripts' || path.startsWith('/api/scripts/hub/get_full/') @@ -983,7 +1055,7 @@ export function hasBenchmarkApiHandler(url: string): boolean { /** Answer a relative `/api/...` fetch — from the API catalog executor, or from the * chat's hub tools. */ -export function handleBenchmarkApiFetch(url: string): Response { +export function handleBenchmarkApiFetch(url: string, init?: RequestInit): Response { const path = url.split('?')[0] if (path === '/api/workers/list') { return Response.json(BENCHMARK_WORKERS) @@ -991,6 +1063,36 @@ export function handleBenchmarkApiFetch(url: string): Response { if (/^\/api\/w\/[^/]+\/jobs\/queue\/list$/.test(path)) { return Response.json([]) } + const jobGet = BENCHMARK_JOB_GET_PATH.exec(path) + if (jobGet) { + const id = decodeURIComponent(jobGet[2]) + const job = getBenchmarkCompletedJob(decodeURIComponent(jobGet[1]), id) + if (!job) { + return Response.json({ error: `Job not found for "${id}"` }, { status: 404 }) + } + // The real endpoint lets a caller drop the bulky fields. Ignoring that here would + // size the model's context off a payload it explicitly asked to shrink. + const query = new URLSearchParams(url.split('?')[1] ?? '') + if (query.get('no_logs') === 'true') { + delete job.logs + } + if (query.get('no_code') === 'true') { + delete job.raw_code + } + return Response.json(job) + } + const runByPath = BENCHMARK_RUN_BY_PATH.exec(path) + if (runByPath) { + const workspace = decodeURIComponent(runByPath[1]) + const runnablePath = decodeURIComponent(runByPath[3]) + const args = parseBenchmarkRequestBody(init?.body) + // The real endpoint answers with the bare job id as text, not JSON. + return new Response( + runByPath[2] === 'f' + ? runBenchmarkFlowByPath({ workspace, path: runnablePath, args }) + : runBenchmarkScriptByPath({ workspace, path: runnablePath, args }) + ) + } if (path === '/api/embeddings/query_hub_scripts') { const text = new URLSearchParams(url.split('?')[1] ?? '').get('text') ?? '' return Response.json(searchBenchmarkHubScripts(text)) diff --git a/ai_evals/adapters/frontend/mockBackendApi.test.ts b/ai_evals/adapters/frontend/mockBackendApi.test.ts new file mode 100644 index 0000000000..d9cfa08bf4 --- /dev/null +++ b/ai_evals/adapters/frontend/mockBackendApi.test.ts @@ -0,0 +1,83 @@ +import { afterEach, beforeEach, describe, expect, it } from 'bun:test' +import { + createBenchmarkCompletedJob, + getBenchmarkCompletedJob, + handleBenchmarkApiFetch, + hasBenchmarkApiHandler, + listBenchmarkMcpTools, + resetBenchmarkMockBackend, + registerBenchmarkWorkspaceRunnables +} from './mockBackend' + +const WORKSPACE = 'benchmark-api-ws' + +// A catalog entry with no fetch handler is a dead end: the catalog executor builds a +// relative `/api/...` url, the stub declines it, and node's fetch throws on the relative +// url instead of returning a result the model can act on. Mutating entries are reachable +// too — the eval runners define no `requestConfirmation`, so `call_api_endpoint` executes +// unconfirmed. +describe('benchmark API catalog', () => { + beforeEach(() => resetBenchmarkMockBackend()) + afterEach(() => resetBenchmarkMockBackend()) + + it('answers every endpoint it advertises', () => { + const unanswered = listBenchmarkMcpTools() + .map((tool) => + `/api${tool.path.replace('{workspace}', WORKSPACE)}`.replace(/\{[^}]+\}/g, 'x') + ) + .filter((url) => !hasBenchmarkApiHandler(url)) + + // The draft-covered entries are refused by name before any fetch, so they are + // advertised without a handler on purpose. + expect(unanswered).toEqual([ + `/api/w/${WORKSPACE}/scripts/get/p/x`, + `/api/w/${WORKSPACE}/flows/create`, + `/api/w/${WORKSPACE}/schedules/delete/x`, + `/api/w/${WORKSPACE}/variables/get/x` + ]) + }) + + it('runs a deployed script by path, the way call_api_endpoint reaches it', async () => { + registerBenchmarkWorkspaceRunnables(WORKSPACE, { + scripts: [ + { + path: 'f/evals/greet', + summary: 'Greet', + language: 'bun', + content: 'export async function main() {}' + } + ] + }) + + const res = handleBenchmarkApiFetch( + `/api/w/${WORKSPACE}/jobs/run/p/${encodeURIComponent('f/evals/greet')}`, + { method: 'POST', body: JSON.stringify({ name: 'ada' }) } + ) + + expect(res.status).toBe(200) + const job = getBenchmarkCompletedJob(WORKSPACE, (await res.text()).trim()) + expect(job).toMatchObject({ success: true, args: { name: 'ada' } }) + }) + + it('serves a recorded job so a model can check the run it just started', async () => { + const id = createBenchmarkCompletedJob({ + workspace: WORKSPACE, + jobKind: 'preview', + result: 'Hello, World!' + }) + + const res = handleBenchmarkApiFetch(`/api/w/${WORKSPACE}/jobs_u/get/${id}`) + + expect(res.status).toBe(200) + expect(await res.json()).toMatchObject({ + id, + success: true, + result: 'Hello, World!' + }) + }) + + it('404s an unknown job id instead of letting the fetch fall through', () => { + expect(hasBenchmarkApiHandler(`/api/w/${WORKSPACE}/jobs_u/get/missing`)).toBe(true) + expect(handleBenchmarkApiFetch(`/api/w/${WORKSPACE}/jobs_u/get/missing`).status).toBe(404) + }) +}) diff --git a/ai_evals/adapters/frontend/vitestAdapter.test.ts b/ai_evals/adapters/frontend/vitestAdapter.test.ts index c07d104f47..e9c33b5632 100644 --- a/ai_evals/adapters/frontend/vitestAdapter.test.ts +++ b/ai_evals/adapters/frontend/vitestAdapter.test.ts @@ -13,7 +13,7 @@ const ORIGINAL_FETCH = globalThis.fetch globalThis.fetch = (async (input: unknown, init?: RequestInit) => { const url = typeof input === 'string' ? input : ((input as Request | URL | null)?.url ?? '') if (typeof url === 'string' && hasBenchmarkApiHandler(url)) { - return handleBenchmarkApiFetch(url) + return handleBenchmarkApiFetch(url, init) } return ORIGINAL_FETCH(input as Parameters[0], init) }) as typeof fetch diff --git a/ai_evals/cases/global.yaml b/ai_evals/cases/global.yaml index 005c677e79..b2716671b1 100644 --- a/ai_evals/cases/global.yaml +++ b/ai_evals/cases/global.yaml @@ -567,6 +567,40 @@ - passes dry_run as true - leaves only the schedule draft for review +- id: global-test29-schedule-with-retry-and-error-handler + prompt: |- + The workspace already has a report digest helper. + Schedule it every morning at 6am UTC with `dry_run` on, and have it retry twice + if it fails. Run it on our `nightly` worker group. + Draft only, I'll review before deploying. + initial: ai_evals/fixtures/frontend/global/initial/report_digest_script.json + runtime: + maxTurns: 8 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: schedule + pathIncludes: + - digest + valueIncludes: + - f/evals/global/send_report_digest + - attempts + - nightly + toolExpect: + requiredToolsUsed: + - write_schedule + - get_schedule_schema + forbiddenToolsUsed: + - write_script + - write_flow + - deploy_workspace_item + judgeChecklist: + - finds the existing report digest helper rather than creating a new script or flow + - creates one schedule draft for that helper running daily around 06:00 UTC + - configures a retry policy with two attempts + - routes the schedule to the nightly worker tag + - leaves the schedule as a draft without deploying + - id: global-test18-human-slack-resource-with-secret prompt: |- I'm preparing Slack notifications for eval failures. @@ -1106,6 +1140,38 @@ - closes the runs preview tab in the side panel - does not write, deploy, or delete anything +# --- Artifact version history --- +# Every content change to an artifact is snapshotted, and update_artifact requires a +# change_note that the user reads in the version picker. A blank note makes the history +# unreadable, so pin that the model fills it on every edit. + +- id: global-artifact-note-on-each-edit + prompt: |- + Write up a short rollout plan for me as a doc I can come back to, covering a staged + rollout in three phases. Then add a rollback section to it, and after that tighten + the wording of phase 2. + runtime: + maxTurns: 12 + sessionChat: true + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - create_artifact + - update_artifact + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + # The note is what the version picker shows; a blank one makes history unreadable. + - tool: update_artifact + field: change_note + nonEmpty: true + skipJudge: true + judgeChecklist: + - creates one artifact and revises it rather than creating a second artifact + - each revision carries a short description of what changed + # --- Documentation search (search_docs) --- # Pure product-knowledge questions: the assistant should consult the docs via # search_docs and answer conversationally, not draft or mutate anything. No @@ -1773,6 +1839,27 @@ judgeChecklist: - deletes the deployed script via delete_workspace_item rather than a raw API endpoint +- id: global-undo-created-draft + prompt: |- + Create a draft Postgres resource at `u/admin/scratch_db` for host db.example.com port 5432, database `orders`, user `app`, and tell me what fields it ended up with. + Once you've shown me that, delete it from the workspace again — I only wanted to see the shape of it. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - write_resource + - discard_local_draft + forbiddenToolsUsed: + - delete_workspace_item + - deploy_workspace_item + # Undoing a draft leaves no draft behind; validate via tool use. + skipJudge: true + judgeChecklist: + - undoes its own never-deployed resource with discard_local_draft rather than delete_workspace_item + - id: global-draft-diff-report prompt: |- Update the existing workspace script at `f/evals/global/format_greeting` so the returned message ends with an exclamation mark, keeping everything else the same. diff --git a/ai_evals/core/types.ts b/ai_evals/core/types.ts index 83e9a218f3..f269c5c4eb 100644 --- a/ai_evals/core/types.ts +++ b/ai_evals/core/types.ts @@ -166,6 +166,12 @@ export interface ToolCallArgumentRule { * tool — e.g. SQL where a mutation is mixed with verification SELECTs. */ stringIncludesAnyOf?: string[]; + /** + * Universal over calls: every recorded call to `tool` must carry `field` as a + * non-blank string. Use for a required argument whose value is free text, where + * the point is that the model filled it in at all rather than what it said. + */ + nonEmpty?: boolean; } export interface ToolValidationSpec { diff --git a/ai_evals/core/validators.test.ts b/ai_evals/core/validators.test.ts index d603dae626..96107397b7 100644 --- a/ai_evals/core/validators.test.ts +++ b/ai_evals/core/validators.test.ts @@ -174,6 +174,52 @@ describe("validateToolExpectations", () => { expect(checks.every((check) => check.passed)).toBe(true); }); + it("fails nonEmpty when any call left the field blank", () => { + const checks = validateToolExpectations({ + run: { + success: true, + actual: {}, + assistantMessageCount: 1, + toolCallCount: 2, + toolsUsed: ["update_artifact"], + toolCallDetails: [ + { name: "update_artifact", arguments: { change_note: "Added a rollback section" } }, + // A whitespace-only note is as unreadable in the picker as a missing one. + { name: "update_artifact", arguments: { change_note: " " } }, + ], + skillsInvoked: [], + }, + toolExpect: { + toolCallArgs: [{ tool: "update_artifact", field: "change_note", nonEmpty: true }], + }, + }); + + const nonEmptyCheck = checks.find((c) => c.name.includes("is filled in on every call")); + expect(nonEmptyCheck?.passed).toBe(false); + expect(nonEmptyCheck?.details).toContain("blank on 1 of 2"); + }); + + it("passes nonEmpty when every call filled the field", () => { + const checks = validateToolExpectations({ + run: { + success: true, + actual: {}, + assistantMessageCount: 1, + toolCallCount: 1, + toolsUsed: ["update_artifact"], + toolCallDetails: [ + { name: "update_artifact", arguments: { change_note: "Tightened phase 2" } }, + ], + skillsInvoked: [], + }, + toolExpect: { + toolCallArgs: [{ tool: "update_artifact", field: "change_note", nonEmpty: true }], + }, + }); + + expect(checks.every((check) => check.passed)).toBe(true); + }); + it("accepts a stringIncludesAnyOf substring inside an array-valued field", () => { const checks = validateToolExpectations({ run: { diff --git a/ai_evals/core/validators.ts b/ai_evals/core/validators.ts index 32ff225fc1..fa39d8611a 100644 --- a/ai_evals/core/validators.ts +++ b/ai_evals/core/validators.ts @@ -233,6 +233,19 @@ export function validateToolExpectations(input: { ); } + if (rule.nonEmpty) { + const blankValues = values.filter( + (value) => typeof value !== "string" || value.trim().length === 0 + ); + checks.push( + check( + `${rule.tool}.${rule.field} is filled in on every call`, + blankValues.length === 0, + `blank on ${blankValues.length} of ${values.length} call(s); values: ${summarizeToolValues(values)}` + ) + ); + } + if (rule.stringIncludesAnyOf && rule.stringIncludesAnyOf.length > 0) { // Existential: at least one call must contain one of the substrings. // Other calls to the same tool may do anything — this suits SQL, where a diff --git a/backend/.sqlx/query-00a7fccde8bc2075642ba02f4015d752ba7d67966ed03c0bf3414c842c049b3a.json b/backend/.sqlx/query-00a7fccde8bc2075642ba02f4015d752ba7d67966ed03c0bf3414c842c049b3a.json new file mode 100644 index 0000000000..2d3292c34b --- /dev/null +++ b/backend/.sqlx/query-00a7fccde8bc2075642ba02f4015d752ba7d67966ed03c0bf3414c842c049b3a.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow (workspace_id, path, summary, description, value, edited_by, edited_at, on_behalf_of, on_behalf_of_email)\n VALUES ('test-workspace', 'u/test-user/obo_flow', '', '', $1, 'test-user', NOW(), 'u/test-user-2', 'test2@windmill.dev')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "00a7fccde8bc2075642ba02f4015d752ba7d67966ed03c0bf3414c842c049b3a" +} diff --git a/backend/.sqlx/query-00e9fc3fed9379264881c58df9404ab3569a0611b33c261602d62e9c847c583e.json b/backend/.sqlx/query-00e9fc3fed9379264881c58df9404ab3569a0611b33c261602d62e9c847c583e.json deleted file mode 100644 index 13536a244e..0000000000 --- a/backend/.sqlx/query-00e9fc3fed9379264881c58df9404ab3569a0611b33c261602d62e9c847c583e.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = ANY($2::text[])", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "TextArray" - ] - }, - "nullable": [] - }, - "hash": "00e9fc3fed9379264881c58df9404ab3569a0611b33c261602d62e9c847c583e" -} diff --git a/backend/.sqlx/query-015b63b1fbdf95fc76138fcf0aed03ac8948cfcf071c7f5df574c5f7003545cd.json b/backend/.sqlx/query-015b63b1fbdf95fc76138fcf0aed03ac8948cfcf071c7f5df574c5f7003545cd.json new file mode 100644 index 0000000000..9d3b9e9287 --- /dev/null +++ b/backend/.sqlx/query-015b63b1fbdf95fc76138fcf0aed03ac8948cfcf071c7f5df574c5f7003545cd.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM flow WHERE on_behalf_of = $1 AND NOT path LIKE $2 AND workspace_id = $3 AND NOT archived", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "015b63b1fbdf95fc76138fcf0aed03ac8948cfcf071c7f5df574c5f7003545cd" +} diff --git a/backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json b/backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json index a0d8b66a26..6c282c7db8 100644 --- a/backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json +++ b/backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json @@ -16,7 +16,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-0da2425ff8ee737104cd9f2650f1ffba1511715ac1f9270939d79fb82f8e88d0.json b/backend/.sqlx/query-024fa8a99a7967a56cd5ac9486ef728f2de0800eeb877fd29aca2fba3029aa55.json similarity index 54% rename from backend/.sqlx/query-0da2425ff8ee737104cd9f2650f1ffba1511715ac1f9270939d79fb82f8e88d0.json rename to backend/.sqlx/query-024fa8a99a7967a56cd5ac9486ef728f2de0800eeb877fd29aca2fba3029aa55.json index 36e9ac3e04..d44cd42986 100644 --- a/backend/.sqlx/query-0da2425ff8ee737104cd9f2650f1ffba1511715ac1f9270939d79fb82f8e88d0.json +++ b/backend/.sqlx/query-024fa8a99a7967a56cd5ac9486ef728f2de0800eeb877fd29aca2fba3029aa55.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "UPDATE azure_trigger SET email = $1 WHERE email = $2", + "query": "UPDATE flow SET on_behalf_of = $1 WHERE on_behalf_of = $2", "describe": { "columns": [], "parameters": { @@ -11,5 +11,5 @@ }, "nullable": [] }, - "hash": "0da2425ff8ee737104cd9f2650f1ffba1511715ac1f9270939d79fb82f8e88d0" + "hash": "024fa8a99a7967a56cd5ac9486ef728f2de0800eeb877fd29aca2fba3029aa55" } diff --git a/backend/.sqlx/query-03238f44f8d4db3ed406f853b5afd945796a2657e58cf1c5086d971e652803d5.json b/backend/.sqlx/query-03238f44f8d4db3ed406f853b5afd945796a2657e58cf1c5086d971e652803d5.json new file mode 100644 index 0000000000..f99e53856c --- /dev/null +++ b/backend/.sqlx/query-03238f44f8d4db3ed406f853b5afd945796a2657e58cf1c5086d971e652803d5.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id, digest)\n VALUES ($1, $2, $3, $4, 'd2')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "03238f44f8d4db3ed406f853b5afd945796a2657e58cf1c5086d971e652803d5" +} diff --git a/backend/.sqlx/query-04896f435eee78f5ac604f5a57c8d35a313fc321df788e1c701e11f60da46480.json b/backend/.sqlx/query-04896f435eee78f5ac604f5a57c8d35a313fc321df788e1c701e11f60da46480.json new file mode 100644 index 0000000000..4a632d868e --- /dev/null +++ b/backend/.sqlx/query-04896f435eee78f5ac604f5a57c8d35a313fc321df788e1c701e11f60da46480.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT created_by, runnable_path FROM v2_job\n WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "runnable_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + false, + true + ] + }, + "hash": "04896f435eee78f5ac604f5a57c8d35a313fc321df788e1c701e11f60da46480" +} diff --git a/backend/.sqlx/query-051a18ab1720ffff792a843be93e8c01515be10b2e40292d904d20e7d53bf289.json b/backend/.sqlx/query-051a18ab1720ffff792a843be93e8c01515be10b2e40292d904d20e7d53bf289.json new file mode 100644 index 0000000000..c436cb3ed8 --- /dev/null +++ b/backend/.sqlx/query-051a18ab1720ffff792a843be93e8c01515be10b2e40292d904d20e7d53bf289.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH RECURSIVE tree AS (\n SELECT id, is_dev_workspace, dev_workspace_label, deleted, 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.is_dev_workspace, w.dev_workspace_label, w.deleted,\n tree.depth + 1\n FROM workspace w JOIN tree ON w.parent_workspace_id = tree.id\n WHERE tree.depth < 20\n )\n SELECT id AS \"id!\", dev_workspace_label FROM tree\n WHERE depth > 0 AND is_dev_workspace AND NOT deleted", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "dev_workspace_label", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "051a18ab1720ffff792a843be93e8c01515be10b2e40292d904d20e7d53bf289" +} diff --git a/backend/.sqlx/query-058df18b148867dbb8bcf9e10c485d276b5af4b2b972fb335e41077a029c3196.json b/backend/.sqlx/query-058df18b148867dbb8bcf9e10c485d276b5af4b2b972fb335e41077a029c3196.json new file mode 100644 index 0000000000..1dd8e03e43 --- /dev/null +++ b/backend/.sqlx/query-058df18b148867dbb8bcf9e10c485d276b5af4b2b972fb335e41077a029c3196.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "058df18b148867dbb8bcf9e10c485d276b5af4b2b972fb335e41077a029c3196" +} diff --git a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json b/backend/.sqlx/query-09ae123099f05fe3db8bd67ca2c0d778a27619dd7fad71ba656f6b54c4915d1e.json similarity index 92% rename from backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json rename to backend/.sqlx/query-09ae123099f05fe3db8bd67ca2c0d778a27619dd7fad71ba656f6b54c4915d1e.json index 0eb26d5dc0..8840ce4958 100644 --- a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json +++ b/backend/.sqlx/query-09ae123099f05fe3db8bd67ca2c0d778a27619dd7fad71ba656f6b54c4915d1e.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT\n j.id, j.workspace_id, j.runnable_id AS \"runnable_id: ScriptHash\", q.scheduled_for, q.started_at, j.parent_job, j.flow_innermost_root_job, j.runnable_path, j.kind as \"kind!: JobKind\", j.permissioned_as,\n j.created_by, j.script_lang AS \"script_lang: ScriptLang\", j.permissioned_as_email, j.flow_step_id, j.trigger_kind AS \"trigger_kind: JobTriggerKind\", j.trigger, j.priority, j.concurrent_limit, j.tag, j.cache_ttl, q.cache_ignore_s3_path, q.runnable_settings_handle\n FROM v2_job j LEFT JOIN v2_job_queue q ON j.id = q.id\n WHERE j.id = $1 AND j.workspace_id = $2", + "query": "SELECT\n j.id, j.workspace_id, j.runnable_id AS \"runnable_id: ScriptHash\", q.scheduled_for, q.started_at, j.parent_job, j.flow_innermost_root_job, j.runnable_path, j.kind as \"kind!: JobKind\", j.permissioned_as,\n j.created_by, j.script_lang AS \"script_lang: ScriptLang\", j.permissioned_as_email, j.flow_step_id, j.trigger_kind AS \"trigger_kind: TriggerKindLabel\", j.trigger, j.priority, j.concurrent_limit, j.tag, j.cache_ttl, q.cache_ignore_s3_path, q.runnable_settings_handle\n FROM v2_job j LEFT JOIN v2_job_queue q ON j.id = q.id\n WHERE j.id = $1 AND j.workspace_id = $2", "describe": { "columns": [ { @@ -119,7 +119,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } @@ -137,7 +138,7 @@ }, { "ordinal": 14, - "name": "trigger_kind: JobTriggerKind", + "name": "trigger_kind: TriggerKindLabel", "type_info": { "Custom": { "name": "job_trigger_kind", @@ -236,5 +237,5 @@ true ] }, - "hash": "67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8" + "hash": "09ae123099f05fe3db8bd67ca2c0d778a27619dd7fad71ba656f6b54c4915d1e" } diff --git a/backend/.sqlx/query-0b3754e11677390837ec7ab00ba5443dc634b8628e90cce879952c8c4319a6a6.json b/backend/.sqlx/query-0b3754e11677390837ec7ab00ba5443dc634b8628e90cce879952c8c4319a6a6.json new file mode 100644 index 0000000000..99bf8ee009 --- /dev/null +++ b/backend/.sqlx/query-0b3754e11677390837ec7ab00ba5443dc634b8628e90cce879952c8c4319a6a6.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_edge WHERE workspace_id = $1 AND script_path = $2\n AND script_hash = $3 AND job_id = $4", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "0b3754e11677390837ec7ab00ba5443dc634b8628e90cce879952c8c4319a6a6" +} diff --git a/backend/.sqlx/query-0b9088064d2a61fd9df91269ec95cab2539a701109727f997a3e5a0f4b0f1f22.json b/backend/.sqlx/query-0b9088064d2a61fd9df91269ec95cab2539a701109727f997a3e5a0f4b0f1f22.json new file mode 100644 index 0000000000..25bfaf16d3 --- /dev/null +++ b/backend/.sqlx/query-0b9088064d2a61fd9df91269ec95cab2539a701109727f997a3e5a0f4b0f1f22.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext('dev_workspace_pairing:' || $1))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0b9088064d2a61fd9df91269ec95cab2539a701109727f997a3e5a0f4b0f1f22" +} diff --git a/backend/.sqlx/query-0c416eee574142b7a709a6e3e9a47d436f407781ff2bee820d3234317f46f2a6.json b/backend/.sqlx/query-0c416eee574142b7a709a6e3e9a47d436f407781ff2bee820d3234317f46f2a6.json new file mode 100644 index 0000000000..6f1603894b --- /dev/null +++ b/backend/.sqlx/query-0c416eee574142b7a709a6e3e9a47d436f407781ff2bee820d3234317f46f2a6.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE draft SET value = to_json(jsonb_set(to_jsonb(value), ARRAY['on_behalf_of'], to_jsonb('u/' || $1))) WHERE value->>'on_behalf_of' = ('u/' || $2) AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "0c416eee574142b7a709a6e3e9a47d436f407781ff2bee820d3234317f46f2a6" +} diff --git a/backend/.sqlx/query-0c5b02b6b70fb8fd2ab3e6c57897038750a44a67360d342b6ef705ef2e4d3007.json b/backend/.sqlx/query-0c5b02b6b70fb8fd2ab3e6c57897038750a44a67360d342b6ef705ef2e4d3007.json new file mode 100644 index 0000000000..e74bd5de0e --- /dev/null +++ b/backend/.sqlx/query-0c5b02b6b70fb8fd2ab3e6c57897038750a44a67360d342b6ef705ef2e4d3007.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_settings (workspace_id, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts) SELECT $1, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts FROM workspace_settings WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "0c5b02b6b70fb8fd2ab3e6c57897038750a44a67360d342b6ef705ef2e4d3007" +} diff --git a/backend/.sqlx/query-0d9c4c350db3ef8f47c0951d901e9b87a9e3ee8c157b3e9a33053d2f0eeb8069.json b/backend/.sqlx/query-0d9c4c350db3ef8f47c0951d901e9b87a9e3ee8c157b3e9a33053d2f0eeb8069.json new file mode 100644 index 0000000000..be32bf94a2 --- /dev/null +++ b/backend/.sqlx/query-0d9c4c350db3ef8f47c0951d901e9b87a9e3ee8c157b3e9a33053d2f0eeb8069.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM app WHERE workspace_id = $1 AND path = $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0d9c4c350db3ef8f47c0951d901e9b87a9e3ee8c157b3e9a33053d2f0eeb8069" +} diff --git a/backend/.sqlx/query-0dbac8411e9fdaacd11980ac548c8cb957ff989aa70dae1223630f48d9949b44.json b/backend/.sqlx/query-0dbac8411e9fdaacd11980ac548c8cb957ff989aa70dae1223630f48d9949b44.json new file mode 100644 index 0000000000..b6e0da69eb --- /dev/null +++ b/backend/.sqlx/query-0dbac8411e9fdaacd11980ac548c8cb957ff989aa70dae1223630f48d9949b44.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = $1 AND name = $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0dbac8411e9fdaacd11980ac548c8cb957ff989aa70dae1223630f48d9949b44" +} diff --git a/backend/.sqlx/query-0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285.json b/backend/.sqlx/query-0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285.json index 238522a3ff..a9e75189af 100644 --- a/backend/.sqlx/query-0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285.json +++ b/backend/.sqlx/query-0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285.json @@ -16,7 +16,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } @@ -55,7 +56,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-0f58d4e7e6f3e962e8a86a2d9feb921c308b6a047530eccd8966513ffdc722d0.json b/backend/.sqlx/query-0f58d4e7e6f3e962e8a86a2d9feb921c308b6a047530eccd8966513ffdc722d0.json index 4194e7e9cc..0bc0c3ce03 100644 --- a/backend/.sqlx/query-0f58d4e7e6f3e962e8a86a2d9feb921c308b6a047530eccd8966513ffdc722d0.json +++ b/backend/.sqlx/query-0f58d4e7e6f3e962e8a86a2d9feb921c308b6a047530eccd8966513ffdc722d0.json @@ -35,7 +35,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-10eba371c513030f165a6c8d3a32f7e33828247ab8108ffe304d7548aab5a42d.json b/backend/.sqlx/query-10eba371c513030f165a6c8d3a32f7e33828247ab8108ffe304d7548aab5a42d.json new file mode 100644 index 0000000000..df470bbf61 --- /dev/null +++ b/backend/.sqlx/query-10eba371c513030f165a6c8d3a32f7e33828247ab8108ffe304d7548aab5a42d.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM resource WHERE workspace_id = $1 AND path = ANY($2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false + ] + }, + "hash": "10eba371c513030f165a6c8d3a32f7e33828247ab8108ffe304d7548aab5a42d" +} diff --git a/backend/.sqlx/query-117b6de99cd75e279f738226d9455c506ac1d40163b4f3c50279a93e4cded5fa.json b/backend/.sqlx/query-117b6de99cd75e279f738226d9455c506ac1d40163b4f3c50279a93e4cded5fa.json new file mode 100644 index 0000000000..c7df8b7610 --- /dev/null +++ b/backend/.sqlx/query-117b6de99cd75e279f738226d9455c506ac1d40163b4f3c50279a93e4cded5fa.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE draft SET value = to_json(jsonb_set(to_jsonb(value), ARRAY['on_behalf_of_email'], to_jsonb($1::text))) WHERE typ IN ('script', 'flow') AND value->>'on_behalf_of_email' = $2 AND (value->>'on_behalf_of' IS NULL OR value->>'on_behalf_of' NOT LIKE 'g/%')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "117b6de99cd75e279f738226d9455c506ac1d40163b4f3c50279a93e4cded5fa" +} diff --git a/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json index cef272c219..b8e3bb8927 100644 --- a/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json +++ b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json @@ -34,7 +34,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-1253d0f62254b30979a1c8b41d5dc1a57a0253f9a76721e7b8965905613f2ace.json b/backend/.sqlx/query-1253d0f62254b30979a1c8b41d5dc1a57a0253f9a76721e7b8965905613f2ace.json new file mode 100644 index 0000000000..21dbc92b79 --- /dev/null +++ b/backend/.sqlx/query-1253d0f62254b30979a1c8b41d5dc1a57a0253f9a76721e7b8965905613f2ace.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of, on_behalf_of_email)\n VALUES ('test-workspace', 'f/shared/obo', 1099, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2', 'test2@windmill.dev')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "1253d0f62254b30979a1c8b41d5dc1a57a0253f9a76721e7b8965905613f2ace" +} diff --git a/backend/.sqlx/query-12912ebe5df4eda15976e7c3a2f0501ed3098f33a234f283def6932ce491eb66.json b/backend/.sqlx/query-12912ebe5df4eda15976e7c3a2f0501ed3098f33a234f283def6932ce491eb66.json new file mode 100644 index 0000000000..a797527421 --- /dev/null +++ b/backend/.sqlx/query-12912ebe5df4eda15976e7c3a2f0501ed3098f33a234f283def6932ce491eb66.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id, digest)\n VALUES ($1, $2, $3, $4, 'd')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "12912ebe5df4eda15976e7c3a2f0501ed3098f33a234f283def6932ce491eb66" +} diff --git a/backend/.sqlx/query-13ab0cda87d04b9c1fb52b46e0211d02b24375643a030efdbfab331cdc248349.json b/backend/.sqlx/query-13ab0cda87d04b9c1fb52b46e0211d02b24375643a030efdbfab331cdc248349.json new file mode 100644 index 0000000000..66432001da --- /dev/null +++ b/backend/.sqlx/query-13ab0cda87d04b9c1fb52b46e0211d02b24375643a030efdbfab331cdc248349.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin)\n VALUES (encode(sha256('LONG_TOKEN'::bytea), 'hex'), 'LONG_TOKEN', 'LONG_TOKEN', $1, 'long', true)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "13ab0cda87d04b9c1fb52b46e0211d02b24375643a030efdbfab331cdc248349" +} diff --git a/backend/.sqlx/query-14a826d794da720981368c77a4f9dac27307833fafaa0ea232819aecd4d62773.json b/backend/.sqlx/query-14a826d794da720981368c77a4f9dac27307833fafaa0ea232819aecd4d62773.json new file mode 100644 index 0000000000..6604c44c97 --- /dev/null +++ b/backend/.sqlx/query-14a826d794da720981368c77a4f9dac27307833fafaa0ea232819aecd4d62773.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE native_trigger\n SET webhook_token_hash = $1, service_config = $2, error = NULL, updated_at = NOW()\n WHERE\n workspace_id = $3\n AND service_name = $4\n AND external_id = $5\n AND updated_at = $6\n RETURNING 1 AS \"applied!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "applied!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Jsonb", + "Text", + { + "Custom": { + "name": "native_trigger_service", + "kind": { + "Enum": [ + "nextcloud", + "google", + "github" + ] + } + } + }, + "Text", + "Timestamptz" + ] + }, + "nullable": [ + null + ] + }, + "hash": "14a826d794da720981368c77a4f9dac27307833fafaa0ea232819aecd4d62773" +} diff --git a/backend/.sqlx/query-14f024e99b3f7129294435671eba84c3d8cd9e36ff773c0cf2843509cce4fdf9.json b/backend/.sqlx/query-14f024e99b3f7129294435671eba84c3d8cd9e36ff773c0cf2843509cce4fdf9.json new file mode 100644 index 0000000000..fd1769a44b --- /dev/null +++ b/backend/.sqlx/query-14f024e99b3f7129294435671eba84c3d8cd9e36ff773c0cf2843509cce4fdf9.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, source_last_event_kind, source_last_event_origin)\n SELECT $1, unnest($2::varchar[]), $3, $4, 0, 1, NULL, $5, $6\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n behind = workspace_diff.behind + 1,\n has_changes = NULL,\n source_last_event_kind = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.source_last_event_origin IS NOT NULL) THEN workspace_diff.source_last_event_kind ELSE $5 END,\n source_last_event_origin = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.source_last_event_origin IS NOT NULL) THEN workspace_diff.source_last_event_origin ELSE $6 END", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "VarcharArray", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "14f024e99b3f7129294435671eba84c3d8cd9e36ff773c0cf2843509cce4fdf9" +} diff --git a/backend/.sqlx/query-164d94369014ce77642984818a0e436a87a1bd1c56fcaafbfae2ee45f62fe4e1.json b/backend/.sqlx/query-164d94369014ce77642984818a0e436a87a1bd1c56fcaafbfae2ee45f62fe4e1.json new file mode 100644 index 0000000000..875e8b823c --- /dev/null +++ b/backend/.sqlx/query-164d94369014ce77642984818a0e436a87a1bd1c56fcaafbfae2ee45f62fe4e1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE app SET policy = jsonb_set(policy, ARRAY['on_behalf_of_email'], to_jsonb($1::text)) WHERE policy->>'on_behalf_of_email' = $2 AND (policy->>'on_behalf_of' IS NULL OR policy->>'on_behalf_of' NOT LIKE 'g/%')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "164d94369014ce77642984818a0e436a87a1bd1c56fcaafbfae2ee45f62fe4e1" +} diff --git a/backend/.sqlx/query-16e8d3f75ce4b5b18cefc25ebb670b506677681394606426964491488d64c62c.json b/backend/.sqlx/query-16e8d3f75ce4b5b18cefc25ebb670b506677681394606426964491488d64c62c.json deleted file mode 100644 index 06a3ee417e..0000000000 --- a/backend/.sqlx/query-16e8d3f75ce4b5b18cefc25ebb670b506677681394606426964491488d64c62c.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO azure_trigger (\n azure_resource_path, azure_mode, scope_resource_id, topic_name,\n subscription_name, event_type_filters, push_auth_config, path, script_path,\n is_flow, workspace_id, edited_by, email, edited_at, extra_perms, server_id,\n last_server_ping, error, mode, permissioned_as, error_handler_path,\n error_handler_args, retry, labels\n )\n SELECT\n azure_resource_path, azure_mode, scope_resource_id, topic_name,\n subscription_name, event_type_filters, push_auth_config, path, script_path,\n is_flow, $1, edited_by, email, edited_at, extra_perms, NULL,\n NULL, NULL, 'disabled'::TRIGGER_MODE, permissioned_as, error_handler_path,\n error_handler_args, retry, labels\n FROM azure_trigger WHERE workspace_id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Text" - ] - }, - "nullable": [] - }, - "hash": "16e8d3f75ce4b5b18cefc25ebb670b506677681394606426964491488d64c62c" -} diff --git a/backend/.sqlx/query-175a732180336a801d1c2f41854269a2ce2160349f163b714cee77b5ed4b92f5.json b/backend/.sqlx/query-175a732180336a801d1c2f41854269a2ce2160349f163b714cee77b5ed4b92f5.json new file mode 100644 index 0000000000..a25e9afac0 --- /dev/null +++ b/backend/.sqlx/query-175a732180336a801d1c2f41854269a2ce2160349f163b714cee77b5ed4b92f5.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE script SET on_behalf_of = $1, on_behalf_of_email = $4 WHERE on_behalf_of = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "175a732180336a801d1c2f41854269a2ce2160349f163b714cee77b5ed4b92f5" +} diff --git a/backend/.sqlx/query-19d19f3ed995ac68d9692c8fdf96836099dd325fa25d3660eb766a895b07c1da.json b/backend/.sqlx/query-19d19f3ed995ac68d9692c8fdf96836099dd325fa25d3660eb766a895b07c1da.json new file mode 100644 index 0000000000..76404ffce5 --- /dev/null +++ b/backend/.sqlx/query-19d19f3ed995ac68d9692c8fdf96836099dd325fa25d3660eb766a895b07c1da.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_run_state WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "19d19f3ed995ac68d9692c8fdf96836099dd325fa25d3660eb766a895b07c1da" +} diff --git a/backend/.sqlx/query-1a3fb9f51abd42f1ac4c7597f985a687bed32d2905a446c4e4b33f0b853e8e54.json b/backend/.sqlx/query-1a3fb9f51abd42f1ac4c7597f985a687bed32d2905a446c4e4b33f0b853e8e54.json new file mode 100644 index 0000000000..1c37708a50 --- /dev/null +++ b/backend/.sqlx/query-1a3fb9f51abd42f1ac4c7597f985a687bed32d2905a446c4e4b33f0b853e8e54.json @@ -0,0 +1,25 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_graph_snapshot g\n WHERE g.workspace_id = $1 AND g.script_path = $2 AND g.script_hash IS NULL\n AND g.permissioned_as IS NOT DISTINCT FROM $4\n AND g.job_id NOT IN (\n SELECT job_id FROM dbt_graph_snapshot\n WHERE workspace_id = $1 AND script_path = $2 AND script_hash IS NULL\n AND permissioned_as IS NOT DISTINCT FROM $4\n ORDER BY ingested_at DESC LIMIT $3)\n RETURNING g.job_id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "job_id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "1a3fb9f51abd42f1ac4c7597f985a687bed32d2905a446c4e4b33f0b853e8e54" +} diff --git a/backend/.sqlx/query-1b18f9fdcc0fbc3d5a1328776a804e2943193ad7d28b67df4e80baf38fc19271.json b/backend/.sqlx/query-1b18f9fdcc0fbc3d5a1328776a804e2943193ad7d28b67df4e80baf38fc19271.json new file mode 100644 index 0000000000..2dd66d2df6 --- /dev/null +++ b/backend/.sqlx/query-1b18f9fdcc0fbc3d5a1328776a804e2943193ad7d28b67df4e80baf38fc19271.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, on_behalf_of_email AS email FROM script WHERE workspace_id = 'test-workspace' AND path LIKE 'u/test-user/s%' ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "email", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true + ] + }, + "hash": "1b18f9fdcc0fbc3d5a1328776a804e2943193ad7d28b67df4e80baf38fc19271" +} diff --git a/backend/.sqlx/query-1c472a4740a41ae8985f865e15b0c4edb4d5af2281c22955121cb794428c3cde.json b/backend/.sqlx/query-1c472a4740a41ae8985f865e15b0c4edb4d5af2281c22955121cb794428c3cde.json new file mode 100644 index 0000000000..4dbd7ef8da --- /dev/null +++ b/backend/.sqlx/query-1c472a4740a41ae8985f865e15b0c4edb4d5af2281c22955121cb794428c3cde.json @@ -0,0 +1,70 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n workspace.id AS \"id!\",\n workspace.name AS \"name!\",\n workspace.owner AS \"owner!\",\n workspace.deleted AS \"deleted!\",\n workspace.premium AS \"premium!\",\n workspace_settings.color AS \"color\",\n workspace.parent_workspace_id AS \"parent_workspace_id\",\n workspace.is_dev_workspace AS \"is_dev_workspace!\",\n workspace.dev_workspace_label AS \"dev_workspace_label\"\n FROM workspace\n LEFT JOIN workspace_settings ON workspace.id = workspace_settings.workspace_id\n WHERE workspace.id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "name!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "owner!", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "deleted!", + "type_info": "Bool" + }, + { + "ordinal": 4, + "name": "premium!", + "type_info": "Bool" + }, + { + "ordinal": 5, + "name": "color", + "type_info": "Varchar" + }, + { + "ordinal": 6, + "name": "parent_workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 7, + "name": "is_dev_workspace!", + "type_info": "Bool" + }, + { + "ordinal": 8, + "name": "dev_workspace_label", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + true, + true, + false, + true + ] + }, + "hash": "1c472a4740a41ae8985f865e15b0c4edb4d5af2281c22955121cb794428c3cde" +} diff --git a/backend/.sqlx/query-1d4fb43c4856679371abfa5eeea8b92697ae3640a2e680395c2958c5d8664e27.json b/backend/.sqlx/query-1d4fb43c4856679371abfa5eeea8b92697ae3640a2e680395c2958c5d8664e27.json new file mode 100644 index 0000000000..7f1618b3ad --- /dev/null +++ b/backend/.sqlx/query-1d4fb43c4856679371abfa5eeea8b92697ae3640a2e680395c2958c5d8664e27.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_run_progress\n WHERE workspace_id = $1 AND job_id = $2 AND status = 'running'\n AND asset_path = ANY($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Uuid", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "1d4fb43c4856679371abfa5eeea8b92697ae3640a2e680395c2958c5d8664e27" +} diff --git a/backend/.sqlx/query-1ddb83c0941de9ca80f867b57ad20c3ee0952d118d339f94d16102d9edb23ec0.json b/backend/.sqlx/query-1ddb83c0941de9ca80f867b57ad20c3ee0952d118d339f94d16102d9edb23ec0.json new file mode 100644 index 0000000000..123c9fc89b --- /dev/null +++ b/backend/.sqlx/query-1ddb83c0941de9ca80f867b57ad20c3ee0952d118d339f94d16102d9edb23ec0.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT resource_type FROM resource WHERE workspace_id = $1 AND path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "resource_type", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "1ddb83c0941de9ca80f867b57ad20c3ee0952d118d339f94d16102d9edb23ec0" +} diff --git a/backend/.sqlx/query-1e044b8f64184393953dfeadbeda8af703060a604e68f32e714610513aa284eb.json b/backend/.sqlx/query-1e044b8f64184393953dfeadbeda8af703060a604e68f32e714610513aa284eb.json new file mode 100644 index 0000000000..b22608a039 --- /dev/null +++ b/backend/.sqlx/query-1e044b8f64184393953dfeadbeda8af703060a604e68f32e714610513aa284eb.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of FROM flow WHERE workspace_id = 'wm-fork-obo'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + true + ] + }, + "hash": "1e044b8f64184393953dfeadbeda8af703060a604e68f32e714610513aa284eb" +} diff --git a/backend/.sqlx/query-1f1e477b27f38f410b7e6f436ced0751f77f9cb41055481bc06af6827d647041.json b/backend/.sqlx/query-1f1e477b27f38f410b7e6f436ced0751f77f9cb41055481bc06af6827d647041.json index 9149698131..84ce6720d6 100644 --- a/backend/.sqlx/query-1f1e477b27f38f410b7e6f436ced0751f77f9cb41055481bc06af6827d647041.json +++ b/backend/.sqlx/query-1f1e477b27f38f410b7e6f436ced0751f77f9cb41055481bc06af6827d647041.json @@ -44,7 +44,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-1f39f7ca0c04e825d3648906eed0c98c1dc8c913dc95b691cc6f92ffcd57989e.json b/backend/.sqlx/query-1f39f7ca0c04e825d3648906eed0c98c1dc8c913dc95b691cc6f92ffcd57989e.json new file mode 100644 index 0000000000..57861d6518 --- /dev/null +++ b/backend/.sqlx/query-1f39f7ca0c04e825d3648906eed0c98c1dc8c913dc95b691cc6f92ffcd57989e.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE script SET archived = true WHERE workspace_id = $1 AND hash = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "1f39f7ca0c04e825d3648906eed0c98c1dc8c913dc95b691cc6f92ffcd57989e" +} diff --git a/backend/.sqlx/query-1fb5590cb1fe706b0d2dffcd04a41bb2fce6012b812d16d37c895b2eb9aeca4c.json b/backend/.sqlx/query-1fb5590cb1fe706b0d2dffcd04a41bb2fce6012b812d16d37c895b2eb9aeca4c.json new file mode 100644 index 0000000000..d58ac0c4e4 --- /dev/null +++ b/backend/.sqlx/query-1fb5590cb1fe706b0d2dffcd04a41bb2fce6012b812d16d37c895b2eb9aeca4c.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, raw_code, tags)\n VALUES ($1, $2, NULL, $3, 'model.p.draft', 'model', 'draft',\n 'u/a/wh/analytics/draft', 'select 3', '{}')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "1fb5590cb1fe706b0d2dffcd04a41bb2fce6012b812d16d37c895b2eb9aeca4c" +} diff --git a/backend/.sqlx/query-2213dc4b594d27b3788c48b19e13c600224681e11c016eea5cee3086a7bea8d0.json b/backend/.sqlx/query-2213dc4b594d27b3788c48b19e13c600224681e11c016eea5cee3086a7bea8d0.json deleted file mode 100644 index 14f17d871e..0000000000 --- a/backend/.sqlx/query-2213dc4b594d27b3788c48b19e13c600224681e11c016eea5cee3086a7bea8d0.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n UPDATE workspace_settings\n SET\n ai_config = source_ws.ai_config,\n large_file_storage = source_ws.large_file_storage,\n ducklake = source_ws.ducklake,\n datatable = source_ws.datatable,\n git_app_installations = source_ws.git_app_installations\n FROM workspace_settings source_ws\n WHERE source_ws.workspace_id = $1\n AND workspace_settings.workspace_id = $2\n ", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "2213dc4b594d27b3788c48b19e13c600224681e11c016eea5cee3086a7bea8d0" -} diff --git a/backend/.sqlx/query-229cc3975b7f579c35ac0daa9204ff3ff25fb57954dd1ba2d552c7b263696c07.json b/backend/.sqlx/query-229cc3975b7f579c35ac0daa9204ff3ff25fb57954dd1ba2d552c7b263696c07.json new file mode 100644 index 0000000000..1e92b86bb2 --- /dev/null +++ b/backend/.sqlx/query-229cc3975b7f579c35ac0daa9204ff3ff25fb57954dd1ba2d552c7b263696c07.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, raw_code, tags, description,\n columns, freshness)\n VALUES ($1, $2, $3, $4, 'model.p.orders', 'model', 'orders',\n 'u/a/wh/analytics/orders', 'select 1', '{finance}', 'daily order facts',\n '{\"order_id\": {\"description\": \"natural key\"}}'::jsonb,\n '{\"warn_after\": {\"count\": 12, \"period\": \"hour\"}}'::jsonb)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "229cc3975b7f579c35ac0daa9204ff3ff25fb57954dd1ba2d552c7b263696c07" +} diff --git a/backend/.sqlx/query-22be2359b966bd92d8327f142281ec620fc98c5fb77061ed192a6f75bea2b049.json b/backend/.sqlx/query-22be2359b966bd92d8327f142281ec620fc98c5fb77061ed192a6f75bea2b049.json new file mode 100644 index 0000000000..baef76a508 --- /dev/null +++ b/backend/.sqlx/query-22be2359b966bd92d8327f142281ec620fc98c5fb77061ed192a6f75bea2b049.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH keep AS (\n SELECT hash FROM script\n WHERE workspace_id = $2 AND path = $3 AND language = 'dbt'\n ORDER BY created_at DESC LIMIT $1\n )\n DELETE FROM dbt_graph_snapshot g\n WHERE g.workspace_id = $2 AND g.script_path = $3\n AND g.job_id = '00000000-0000-0000-0000-000000000000'\n AND NOT EXISTS (SELECT 1 FROM keep k WHERE k.hash = g.script_hash)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "22be2359b966bd92d8327f142281ec620fc98c5fb77061ed192a6f75bea2b049" +} diff --git a/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json index e3f72d9c3a..cfcb14ce72 100644 --- a/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json +++ b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json @@ -27,7 +27,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-2458d56c20942e9d13578dbf0e898cda51a2fc652fb008f4e57f07f0922fdcdd.json b/backend/.sqlx/query-2458d56c20942e9d13578dbf0e898cda51a2fc652fb008f4e57f07f0922fdcdd.json new file mode 100644 index 0000000000..8f28923f19 --- /dev/null +++ b/backend/.sqlx/query-2458d56c20942e9d13578dbf0e898cda51a2fc652fb008f4e57f07f0922fdcdd.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, fork_last_event_kind, fork_last_event_origin)\n VALUES ($1, $2, $3, $4, 1, 0, NULL, $5, $6)\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n ahead = workspace_diff.ahead + 1,\n has_changes = NULL,\n fork_last_event_kind = EXCLUDED.fork_last_event_kind,\n fork_last_event_origin = EXCLUDED.fork_last_event_origin", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "2458d56c20942e9d13578dbf0e898cda51a2fc652fb008f4e57f07f0922fdcdd" +} diff --git a/backend/.sqlx/query-24a8eba06c31e09a49ee2ced0748c6ced8664fc7f5278d0673d6134b1ab54bad.json b/backend/.sqlx/query-24a8eba06c31e09a49ee2ced0748c6ced8664fc7f5278d0673d6134b1ab54bad.json new file mode 100644 index 0000000000..1d27f23be4 --- /dev/null +++ b/backend/.sqlx/query-24a8eba06c31e09a49ee2ced0748c6ced8664fc7f5278d0673d6134b1ab54bad.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) AS \"count!\" FROM workspace_diff\n WHERE source_workspace_id = 'test-workspace'\n AND fork_workspace_id = 'wm-fork-removal'\n AND kind = 'script'\n AND fork_last_event_origin IS NOT NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "24a8eba06c31e09a49ee2ced0748c6ced8664fc7f5278d0673d6134b1ab54bad" +} diff --git a/backend/.sqlx/query-25763b3f6f9ef04f879a49407b65d4fa95129bd95418cf908088d9744fc22d4b.json b/backend/.sqlx/query-25763b3f6f9ef04f879a49407b65d4fa95129bd95418cf908088d9744fc22d4b.json new file mode 100644 index 0000000000..991e8c6351 --- /dev/null +++ b/backend/.sqlx/query-25763b3f6f9ef04f879a49407b65d4fa95129bd95418cf908088d9744fc22d4b.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ahead, fork_last_event_kind, fork_last_event_origin FROM workspace_diff\n WHERE source_workspace_id = 'test-workspace' AND fork_workspace_id = 'wm-fork-removal'\n AND kind = 'script' AND path = 'u/admin/dropped_by_sync'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ahead", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "fork_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "fork_last_event_origin", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true, + true + ] + }, + "hash": "25763b3f6f9ef04f879a49407b65d4fa95129bd95418cf908088d9744fc22d4b" +} diff --git a/backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json b/backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json index 02a419de06..16f26c4b78 100644 --- a/backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json +++ b/backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json @@ -21,7 +21,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-295325886239353a27bfb8807a59952b81141a2544fc5340e6526b8fa0fb06c5.json b/backend/.sqlx/query-295325886239353a27bfb8807a59952b81141a2544fc5340e6526b8fa0fb06c5.json new file mode 100644 index 0000000000..1a5cb5dbce --- /dev/null +++ b/backend/.sqlx/query-295325886239353a27bfb8807a59952b81141a2544fc5340e6526b8fa0fb06c5.json @@ -0,0 +1,101 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels, on_behalf_of, on_behalf_of_email) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Int8", + "Varchar", + "Int8Array", + "Text", + "Text", + "Text", + "Varchar", + "Text", + "Bool", + "Jsonb", + "Text", + { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang", + "dbt" + ] + } + } + }, + { + "Custom": { + "name": "script_kind", + "kind": { + "Enum": [ + "script", + "trigger", + "failure", + "command", + "approval", + "preprocessor" + ] + } + } + }, + "Varchar", + "VarcharArray", + "Int4", + "Int4", + "Int4", + "Bool", + "Bool", + "Int2", + "Bool", + "Bool", + "Int4", + "Int4", + "Varchar", + "Bool", + "Varchar", + "Varchar", + "Bool", + "Bool", + "Jsonb", + "Varchar", + "Int4", + "Bool", + "Int8", + "Jsonb", + "TextArray", + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "295325886239353a27bfb8807a59952b81141a2544fc5340e6526b8fa0fb06c5" +} diff --git a/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json index f940f25b4a..99850586d3 100644 --- a/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json +++ b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json @@ -42,7 +42,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-2a89648b28c40dfb9ebd4aa0b5ffa772d88c5cf97de840b5147deaeb95c98209.json b/backend/.sqlx/query-2a89648b28c40dfb9ebd4aa0b5ffa772d88c5cf97de840b5147deaeb95c98209.json new file mode 100644 index 0000000000..3ef757423e --- /dev/null +++ b/backend/.sqlx/query-2a89648b28c40dfb9ebd4aa0b5ffa772d88c5cf97de840b5147deaeb95c98209.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE flow SET on_behalf_of = $1, on_behalf_of_email = $4 WHERE on_behalf_of = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "2a89648b28c40dfb9ebd4aa0b5ffa772d88c5cf97de840b5147deaeb95c98209" +} diff --git a/backend/.sqlx/query-2a8b9b199a6d383ed39a64dad54014f869ade4a139956dcb082f0524779f7680.json b/backend/.sqlx/query-2a8b9b199a6d383ed39a64dad54014f869ade4a139956dcb082f0524779f7680.json new file mode 100644 index 0000000000..f770ee58df --- /dev/null +++ b/backend/.sqlx/query-2a8b9b199a6d383ed39a64dad54014f869ade4a139956dcb082f0524779f7680.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT relation_root_at_last_ingest FROM dbt_graph_snapshot\n WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3\n AND job_id = '00000000-0000-0000-0000-000000000000'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "relation_root_at_last_ingest", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + true + ] + }, + "hash": "2a8b9b199a6d383ed39a64dad54014f869ade4a139956dcb082f0524779f7680" +} diff --git a/backend/.sqlx/query-2c125b9655cf3d61bee135b8efbfad1d2a395d15f34cfaa4e1438b36afdd4043.json b/backend/.sqlx/query-2c125b9655cf3d61bee135b8efbfad1d2a395d15f34cfaa4e1438b36afdd4043.json new file mode 100644 index 0000000000..4694af020e --- /dev/null +++ b/backend/.sqlx/query-2c125b9655cf3d61bee135b8efbfad1d2a395d15f34cfaa4e1438b36afdd4043.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_diff SET fork_last_event_kind = 'delete', fork_last_event_origin = 'sync'\n WHERE fork_workspace_id = 'wm-fork-deferred' AND kind = 'flow'\n AND path = 'u/admin/moved_by_user' RETURNING 1 AS \"one!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "one!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "2c125b9655cf3d61bee135b8efbfad1d2a395d15f34cfaa4e1438b36afdd4043" +} diff --git a/backend/.sqlx/query-2c56da59ac2d7b410235a3181cf2db835776ff3928236dc26033a9f577ddad61.json b/backend/.sqlx/query-2c56da59ac2d7b410235a3181cf2db835776ff3928236dc26033a9f577ddad61.json new file mode 100644 index 0000000000..76c8a429fc --- /dev/null +++ b/backend/.sqlx/query-2c56da59ac2d7b410235a3181cf2db835776ff3928236dc26033a9f577ddad61.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_try_advisory_lock(hashtextextended($1, 0)) AS \"acquired!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "acquired!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "2c56da59ac2d7b410235a3181cf2db835776ff3928236dc26033a9f577ddad61" +} diff --git a/backend/.sqlx/query-2d943b22eb69010f3c2388b9443dc8c5729cb2b828cae94c047b896ebff37fd3.json b/backend/.sqlx/query-2d943b22eb69010f3c2388b9443dc8c5729cb2b828cae94c047b896ebff37fd3.json new file mode 100644 index 0000000000..558e5e466e --- /dev/null +++ b/backend/.sqlx/query-2d943b22eb69010f3c2388b9443dc8c5729cb2b828cae94c047b896ebff37fd3.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT job_id FROM dbt_run_state\n WHERE workspace_id = $1 AND script_path = $2 AND permissioned_as = $3 AND retryable", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "job_id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "2d943b22eb69010f3c2388b9443dc8c5729cb2b828cae94c047b896ebff37fd3" +} diff --git a/backend/.sqlx/query-2db7e748ce92c8f300c1d7519900dbfa39304167e58c096186c7f9ee47207a05.json b/backend/.sqlx/query-2db7e748ce92c8f300c1d7519900dbfa39304167e58c096186c7f9ee47207a05.json new file mode 100644 index 0000000000..00725cc52f --- /dev/null +++ b/backend/.sqlx/query-2db7e748ce92c8f300c1d7519900dbfa39304167e58c096186c7f9ee47207a05.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, fork_last_event_kind, fork_last_event_origin)\n VALUES ($1, $2, $3, $4, 1, 0, NULL, $5, $6)\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n ahead = workspace_diff.ahead + 1,\n has_changes = NULL,\n fork_last_event_kind = CASE WHEN $6::varchar IS NULL THEN workspace_diff.fork_last_event_kind ELSE $5 END,\n fork_last_event_origin = COALESCE($6, workspace_diff.fork_last_event_origin)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "2db7e748ce92c8f300c1d7519900dbfa39304167e58c096186c7f9ee47207a05" +} diff --git a/backend/.sqlx/query-31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9.json b/backend/.sqlx/query-31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9.json index ded7b65a83..346a46d150 100644 --- a/backend/.sqlx/query-31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9.json +++ b/backend/.sqlx/query-31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9.json @@ -21,7 +21,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-31f38820514b1d51459ad93dd0e59d96d4e8cc79c8f3d3c4228f3bc7585c1839.json b/backend/.sqlx/query-31f38820514b1d51459ad93dd0e59d96d4e8cc79c8f3d3c4228f3bc7585c1839.json new file mode 100644 index 0000000000..69c34a9bc3 --- /dev/null +++ b/backend/.sqlx/query-31f38820514b1d51459ad93dd0e59d96d4e8cc79c8f3d3c4228f3bc7585c1839.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of)\n VALUES ('test-workspace', 'u/test-user/s', 93001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'ext@windmill.dev')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "31f38820514b1d51459ad93dd0e59d96d4e8cc79c8f3d3c4228f3bc7585c1839" +} diff --git a/backend/.sqlx/query-3200de51b6fedd925ab3e9e471a6719b7b6de68976e9f90d18c5b74f43763ece.json b/backend/.sqlx/query-3200de51b6fedd925ab3e9e471a6719b7b6de68976e9f90d18c5b74f43763ece.json new file mode 100644 index 0000000000..adbdc13ea5 --- /dev/null +++ b/backend/.sqlx/query-3200de51b6fedd925ab3e9e471a6719b7b6de68976e9f90d18c5b74f43763ece.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT req.id AS \"id!\",\n (CASE\n WHEN workspace.id IS NULL THEN 'deleted'\n WHEN usr.email IS NULL AND NOT $3 THEN 'deleted'\n WHEN workspace.deleted THEN 'archived'\n ELSE 'active'\n END) AS \"status!\"\n FROM unnest($1::text[]) AS req(id)\n LEFT JOIN workspace ON workspace.id = req.id\n LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "status!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "TextArray", + "Text", + "Bool" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "3200de51b6fedd925ab3e9e471a6719b7b6de68976e9f90d18c5b74f43763ece" +} diff --git a/backend/.sqlx/query-33740249a429a5dcbb399459e6b92e865838992c21da78264b8f5d1fe71bc2bf.json b/backend/.sqlx/query-33740249a429a5dcbb399459e6b92e865838992c21da78264b8f5d1fe71bc2bf.json new file mode 100644 index 0000000000..fa90e5300c --- /dev/null +++ b/backend/.sqlx/query-33740249a429a5dcbb399459e6b92e865838992c21da78264b8f5d1fe71bc2bf.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM resource_version rv\n WHERE rv.workspace_id = $1 AND rv.path = $2\n AND rv.id != (\n SELECT max(id) FROM resource_version l\n WHERE l.workspace_id = rv.workspace_id AND l.path = rv.path\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "33740249a429a5dcbb399459e6b92e865838992c21da78264b8f5d1fe71bc2bf" +} diff --git a/backend/.sqlx/query-33b9ef6fabbb5f0de64246b51f0ce702603b7ae170054cda7ed5f6f0a9f44a28.json b/backend/.sqlx/query-33b9ef6fabbb5f0de64246b51f0ce702603b7ae170054cda7ed5f6f0a9f44a28.json new file mode 100644 index 0000000000..8316ec5fab --- /dev/null +++ b/backend/.sqlx/query-33b9ef6fabbb5f0de64246b51f0ce702603b7ae170054cda7ed5f6f0a9f44a28.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_node WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "33b9ef6fabbb5f0de64246b51f0ce702603b7ae170054cda7ed5f6f0a9f44a28" +} diff --git a/backend/.sqlx/query-34a0dd9de9495f80a07e568323c5b94e33dfcacba9466bd870152b130d400794.json b/backend/.sqlx/query-34a0dd9de9495f80a07e568323c5b94e33dfcacba9466bd870152b130d400794.json new file mode 100644 index 0000000000..706486c23c --- /dev/null +++ b/backend/.sqlx/query-34a0dd9de9495f80a07e568323c5b94e33dfcacba9466bd870152b130d400794.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT j.runnable_path, j.runnable_id, j.permissioned_as\n FROM v2_job_queue q JOIN v2_job j ON j.id = q.id\n WHERE q.id = $1 AND j.workspace_id = $2 AND q.tag = ANY($3)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "runnable_id", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "permissioned_as", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text", + "TextArray" + ] + }, + "nullable": [ + true, + true, + false + ] + }, + "hash": "34a0dd9de9495f80a07e568323c5b94e33dfcacba9466bd870152b130d400794" +} diff --git a/backend/.sqlx/query-34fbd17a412a11779fe54505a8c933a2884dbb4ab59161881fdf121ab48f1b35.json b/backend/.sqlx/query-34fbd17a412a11779fe54505a8c933a2884dbb4ab59161881fdf121ab48f1b35.json new file mode 100644 index 0000000000..c1df08d9b0 --- /dev/null +++ b/backend/.sqlx/query-34fbd17a412a11779fe54505a8c933a2884dbb4ab59161881fdf121ab48f1b35.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH RECURSIVE ancestors AS (\n SELECT id, parent_workspace_id, is_dev_workspace, dev_workspace_label, deleted,\n 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, w.is_dev_workspace, w.dev_workspace_label,\n w.deleted, ancestors.depth + 1\n FROM workspace w JOIN ancestors ON w.id = ancestors.parent_workspace_id\n WHERE ancestors.depth < 20\n )\n SELECT id AS \"id!\", dev_workspace_label FROM ancestors\n WHERE is_dev_workspace AND NOT deleted", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "dev_workspace_label", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "34fbd17a412a11779fe54505a8c933a2884dbb4ab59161881fdf121ab48f1b35" +} diff --git a/backend/.sqlx/query-359b85d7940b82bb0e40ebf09e054f9191672e92fff6b22c2130bb04c7009665.json b/backend/.sqlx/query-359b85d7940b82bb0e40ebf09e054f9191672e92fff6b22c2130bb04c7009665.json new file mode 100644 index 0000000000..e62e71a89a --- /dev/null +++ b/backend/.sqlx/query-359b85d7940b82bb0e40ebf09e054f9191672e92fff6b22c2130bb04c7009665.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, on_behalf_of, on_behalf_of_email FROM script WHERE workspace_id = 'wm-fork-obo' ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "on_behalf_of", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "on_behalf_of_email", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true, + true + ] + }, + "hash": "359b85d7940b82bb0e40ebf09e054f9191672e92fff6b22c2130bb04c7009665" +} diff --git a/backend/.sqlx/query-36d0ace456b6022e5311491e0ef457b33679b6ffe75d068349638f9345693348.json b/backend/.sqlx/query-36d0ace456b6022e5311491e0ef457b33679b6ffe75d068349638f9345693348.json new file mode 100644 index 0000000000..6cd95a08ac --- /dev/null +++ b/backend/.sqlx/query-36d0ace456b6022e5311491e0ef457b33679b6ffe75d068349638f9345693348.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE script SET on_behalf_of = $1 WHERE on_behalf_of = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "36d0ace456b6022e5311491e0ef457b33679b6ffe75d068349638f9345693348" +} diff --git a/backend/.sqlx/query-38125105da39bfedbdc1396fe37a8835f2630389dc339be0fda614ddb4d773df.json b/backend/.sqlx/query-38125105da39bfedbdc1396fe37a8835f2630389dc339be0fda614ddb4d773df.json new file mode 100644 index 0000000000..a779a3c995 --- /dev/null +++ b/backend/.sqlx/query-38125105da39bfedbdc1396fe37a8835f2630389dc339be0fda614ddb4d773df.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_node WHERE workspace_id = $1 AND script_path = $2\n AND script_hash = $3 AND job_id = $4", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "38125105da39bfedbdc1396fe37a8835f2630389dc339be0fda614ddb4d773df" +} diff --git a/backend/.sqlx/query-381d59e7ab08e2ae7147e4c277a2d96f22375fa2b1eccafb7f09928a8b0cb549.json b/backend/.sqlx/query-381d59e7ab08e2ae7147e4c277a2d96f22375fa2b1eccafb7f09928a8b0cb549.json new file mode 100644 index 0000000000..18df498bba --- /dev/null +++ b/backend/.sqlx/query-381d59e7ab08e2ae7147e4c277a2d96f22375fa2b1eccafb7f09928a8b0cb549.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE script SET archived = true, extra_perms = '{\"u/outsider\": true}'::jsonb\n WHERE workspace_id = $1 AND hash = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "381d59e7ab08e2ae7147e4c277a2d96f22375fa2b1eccafb7f09928a8b0cb549" +} diff --git a/backend/.sqlx/query-38c20dba51b1e2b4b28e5eed51f9071e1d3cf78e89ec467927823177c5a439cc.json b/backend/.sqlx/query-38c20dba51b1e2b4b28e5eed51f9071e1d3cf78e89ec467927823177c5a439cc.json new file mode 100644 index 0000000000..66681c2377 --- /dev/null +++ b/backend/.sqlx/query-38c20dba51b1e2b4b28e5eed51f9071e1d3cf78e89ec467927823177c5a439cc.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT workspace_id FROM usr WHERE email = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "workspace_id", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "38c20dba51b1e2b4b28e5eed51f9071e1d3cf78e89ec467927823177c5a439cc" +} diff --git a/backend/.sqlx/query-3b20f17e0fadb619de37cda93b361f73d964ae4562e0cb862c7402ae779ed13a.json b/backend/.sqlx/query-3b20f17e0fadb619de37cda93b361f73d964ae4562e0cb862c7402ae779ed13a.json new file mode 100644 index 0000000000..03b6042099 --- /dev/null +++ b/backend/.sqlx/query-3b20f17e0fadb619de37cda93b361f73d964ae4562e0cb862c7402ae779ed13a.json @@ -0,0 +1,153 @@ +{ + "db_name": "PostgreSQL", + "query": "select hash, tag, concurrency_key, concurrent_limit, concurrency_time_window_s, debounce_key, debounce_delay_s, cache_ttl, cache_ignore_s3_path, runnable_settings_handle, language as \"language: ScriptLang\", dedicated_worker, priority, timeout, on_behalf_of, created_by, labels FROM script\n WHERE path = $1 AND workspace_id = $2 AND archived = false AND (lock IS NOT NULL OR $3 = false)\n ORDER BY created_at DESC LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "hash", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "tag", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "concurrency_key", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "concurrent_limit", + "type_info": "Int4" + }, + { + "ordinal": 4, + "name": "concurrency_time_window_s", + "type_info": "Int4" + }, + { + "ordinal": 5, + "name": "debounce_key", + "type_info": "Varchar" + }, + { + "ordinal": 6, + "name": "debounce_delay_s", + "type_info": "Int4" + }, + { + "ordinal": 7, + "name": "cache_ttl", + "type_info": "Int4" + }, + { + "ordinal": 8, + "name": "cache_ignore_s3_path", + "type_info": "Bool" + }, + { + "ordinal": 9, + "name": "runnable_settings_handle", + "type_info": "Int8" + }, + { + "ordinal": 10, + "name": "language: ScriptLang", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang", + "dbt" + ] + } + } + } + }, + { + "ordinal": 11, + "name": "dedicated_worker", + "type_info": "Bool" + }, + { + "ordinal": 12, + "name": "priority", + "type_info": "Int2" + }, + { + "ordinal": 13, + "name": "timeout", + "type_info": "Int4" + }, + { + "ordinal": 14, + "name": "on_behalf_of", + "type_info": "Varchar" + }, + { + "ordinal": 15, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 16, + "name": "labels", + "type_info": "TextArray" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Bool" + ] + }, + "nullable": [ + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + false, + true, + true, + true, + true, + false, + true + ] + }, + "hash": "3b20f17e0fadb619de37cda93b361f73d964ae4562e0cb862c7402ae779ed13a" +} diff --git a/backend/.sqlx/query-3b40cccf059fbd28ff9c8d0004daee26c81a8fe909e12ae997eaaba83ade634d.json b/backend/.sqlx/query-3b40cccf059fbd28ff9c8d0004daee26c81a8fe909e12ae997eaaba83ade634d.json new file mode 100644 index 0000000000..d8afb95017 --- /dev/null +++ b/backend/.sqlx/query-3b40cccf059fbd28ff9c8d0004daee26c81a8fe909e12ae997eaaba83ade634d.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, raw_code, tags)\n VALUES ($1, $2, $3, $4, 'model.p.orders', 'model', 'orders',\n 'u/a/wh/analytics/orders', 'select 2', '{}')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "3b40cccf059fbd28ff9c8d0004daee26c81a8fe909e12ae997eaaba83ade634d" +} diff --git a/backend/.sqlx/query-3c44de953b08ecfd8a962490d21b2dd0bfb66b7631acc958742523cce63793a0.json b/backend/.sqlx/query-3c44de953b08ecfd8a962490d21b2dd0bfb66b7631acc958742523cce63793a0.json new file mode 100644 index 0000000000..4a6a4f0cb6 --- /dev/null +++ b/backend/.sqlx/query-3c44de953b08ecfd8a962490d21b2dd0bfb66b7631acc958742523cce63793a0.json @@ -0,0 +1,27 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow (\n workspace_id, path, summary, description,\n dependency_job, lock_error_logs, tag,\n dedicated_worker, visible_to_runner_only,\n ws_error_handler_muted,\n value, schema, edited_by, edited_at, labels,\n on_behalf_of, on_behalf_of_email\n ) VALUES (\n $1, $2, $3, $4,\n NULL, '', $5,\n $6, $7,\n $8,\n $9, $10::text::json, $11, now(), $12,\n $13, $14\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Text", + "Varchar", + "Bool", + "Bool", + "Bool", + "Jsonb", + "Text", + "Varchar", + "TextArray", + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "3c44de953b08ecfd8a962490d21b2dd0bfb66b7631acc958742523cce63793a0" +} diff --git a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json index 257e1e528f..2a02e5a8ce 100644 --- a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json +++ b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json @@ -77,7 +77,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-3cc398db9c2f8f698a45cb763036c554a9d7525ad34f9ec1bd4ac3dc7afaee38.json b/backend/.sqlx/query-3cc398db9c2f8f698a45cb763036c554a9d7525ad34f9ec1bd4ac3dc7afaee38.json new file mode 100644 index 0000000000..3905a1c7d1 --- /dev/null +++ b/backend/.sqlx/query-3cc398db9c2f8f698a45cb763036c554a9d7525ad34f9ec1bd4ac3dc7afaee38.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow (\n workspace_id, path, summary, description, value, edited_by, edited_at,\n archived, schema, extra_perms, dependency_job, tag,\n ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only,\n concurrency_key, versions, on_behalf_of, on_behalf_of_email, lock_error_logs\n )\n SELECT $2, path, summary, description, value, edited_by, edited_at,\n archived, schema, extra_perms, NULL, tag,\n ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only,\n concurrency_key, ARRAY[]::bigint[],\n -- Same predicate as clone_scripts.\n CASE WHEN on_behalf_of LIKE 'u/%' THEN\n (SELECT on_behalf_of WHERE EXISTS (\n SELECT 1 FROM usr u WHERE u.workspace_id = $2::varchar\n AND u.username = substring(on_behalf_of from 3)\n UNION ALL\n SELECT 1 FROM password p WHERE p.super_admin\n AND (p.username = substring(on_behalf_of from 3)\n OR p.email = substring(on_behalf_of from 3))))\n WHEN on_behalf_of LIKE 'g/%' THEN\n (SELECT on_behalf_of WHERE EXISTS (\n SELECT 1 FROM group_ g WHERE g.workspace_id = $2::varchar\n AND g.name = substring(on_behalf_of from 3)))\n ELSE\n (SELECT on_behalf_of WHERE EXISTS (\n SELECT 1 FROM usr u WHERE u.workspace_id = $2::varchar\n AND u.username = on_behalf_of\n UNION ALL\n SELECT 1 FROM password p WHERE p.email = on_behalf_of\n AND p.super_admin))\n END, on_behalf_of_email, lock_error_logs\n FROM flow\n WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "3cc398db9c2f8f698a45cb763036c554a9d7525ad34f9ec1bd4ac3dc7afaee38" +} diff --git a/backend/.sqlx/query-ce5c081bbf8322371a6ec95b6a0b033837d574a1ec72dfe42b666172daf10880.json b/backend/.sqlx/query-3ce57583042b1daa2bd9d88502f6f4ab76c509dafee7371ac620c85e1b5d2498.json similarity index 87% rename from backend/.sqlx/query-ce5c081bbf8322371a6ec95b6a0b033837d574a1ec72dfe42b666172daf10880.json rename to backend/.sqlx/query-3ce57583042b1daa2bd9d88502f6f4ab76c509dafee7371ac620c85e1b5d2498.json index f2b40e0184..c5a7dd6079 100644 --- a/backend/.sqlx/query-ce5c081bbf8322371a6ec95b6a0b033837d574a1ec72dfe42b666172daf10880.json +++ b/backend/.sqlx/query-3ce57583042b1daa2bd9d88502f6f4ab76c509dafee7371ac620c85e1b5d2498.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT\n workspace_invite.workspace_id,\n workspace_invite.email,\n workspace_invite.is_admin,\n workspace_invite.operator,\n workspace.parent_workspace_id\n FROM workspace_invite JOIN workspace ON workspace_invite.workspace_id = workspace.id\n WHERE workspace_id = $1", + "query": "SELECT\n workspace_invite.workspace_id,\n workspace_invite.email,\n workspace_invite.is_admin,\n workspace_invite.operator,\n workspace.parent_workspace_id\n FROM workspace_invite JOIN workspace ON workspace_invite.workspace_id = workspace.id\n WHERE workspace_id = $1\n ORDER BY workspace_invite.email", "describe": { "columns": [ { @@ -42,5 +42,5 @@ true ] }, - "hash": "ce5c081bbf8322371a6ec95b6a0b033837d574a1ec72dfe42b666172daf10880" + "hash": "3ce57583042b1daa2bd9d88502f6f4ab76c509dafee7371ac620c85e1b5d2498" } diff --git a/backend/.sqlx/query-3e474475c05ff5fc137a070b149dd3a5a77e1640e03f2964d5143c1063fd3d83.json b/backend/.sqlx/query-3e474475c05ff5fc137a070b149dd3a5a77e1640e03f2964d5143c1063fd3d83.json new file mode 100644 index 0000000000..77d9c6d71b --- /dev/null +++ b/backend/.sqlx/query-3e474475c05ff5fc137a070b149dd3a5a77e1640e03f2964d5143c1063fd3d83.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = ANY($2::text[]) RETURNING datatable, timestamp, name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "datatable", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "timestamp", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "3e474475c05ff5fc137a070b149dd3a5a77e1640e03f2964d5143c1063fd3d83" +} diff --git a/backend/.sqlx/query-2a510a8bec98055796f987d86c344ca116895d71de09db338ec09e425dcebe5e.json b/backend/.sqlx/query-3e869b95799300dff973a3c858f823a2872e33a360ff5a02169e4e80ce3d3b7b.json similarity index 75% rename from backend/.sqlx/query-2a510a8bec98055796f987d86c344ca116895d71de09db338ec09e425dcebe5e.json rename to backend/.sqlx/query-3e869b95799300dff973a3c858f823a2872e33a360ff5a02169e4e80ce3d3b7b.json index 23337708f8..ce3b4fcdf8 100644 --- a/backend/.sqlx/query-2a510a8bec98055796f987d86c344ca116895d71de09db338ec09e425dcebe5e.json +++ b/backend/.sqlx/query-3e869b95799300dff973a3c858f823a2872e33a360ff5a02169e4e80ce3d3b7b.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT email, username, is_admin, is_operator, groups, folders FROM job_perms WHERE job_id = $1 AND workspace_id = $2", + "query": "SELECT email, username, is_admin, is_operator, groups, folders, end_user_email FROM job_perms WHERE job_id = $1 AND workspace_id = $2", "describe": { "columns": [ { @@ -32,6 +32,11 @@ "ordinal": 5, "name": "folders", "type_info": "JsonbArray" + }, + { + "ordinal": 6, + "name": "end_user_email", + "type_info": "Varchar" } ], "parameters": { @@ -46,8 +51,9 @@ false, false, false, - false + false, + true ] }, - "hash": "2a510a8bec98055796f987d86c344ca116895d71de09db338ec09e425dcebe5e" + "hash": "3e869b95799300dff973a3c858f823a2872e33a360ff5a02169e4e80ce3d3b7b" } diff --git a/backend/.sqlx/query-3ec280ad74cf63dc3cebb312c620e7a5c2a3b75051618fc3c9635b4cd8d48e25.json b/backend/.sqlx/query-3ec280ad74cf63dc3cebb312c620e7a5c2a3b75051618fc3c9635b4cd8d48e25.json new file mode 100644 index 0000000000..7cbc8115c9 --- /dev/null +++ b/backend/.sqlx/query-3ec280ad74cf63dc3cebb312c620e7a5c2a3b75051618fc3c9635b4cd8d48e25.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM script WHERE workspace_id = $1 AND path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "3ec280ad74cf63dc3cebb312c620e7a5c2a3b75051618fc3c9635b4cd8d48e25" +} diff --git a/backend/.sqlx/query-407fdc870c39138811beb17e125bff3946d829f3d482a9b732ae3b37a9f7f2b5.json b/backend/.sqlx/query-407fdc870c39138811beb17e125bff3946d829f3d482a9b732ae3b37a9f7f2b5.json new file mode 100644 index 0000000000..3db0bad5a7 --- /dev/null +++ b/backend/.sqlx/query-407fdc870c39138811beb17e125bff3946d829f3d482a9b732ae3b37a9f7f2b5.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "407fdc870c39138811beb17e125bff3946d829f3d482a9b732ae3b37a9f7f2b5" +} diff --git a/backend/.sqlx/query-41428ecb8940a8fb4043b71eacb554314177fc11bc82212d95182e3723e64c6b.json b/backend/.sqlx/query-41428ecb8940a8fb4043b71eacb554314177fc11bc82212d95182e3723e64c6b.json new file mode 100644 index 0000000000..c812a91912 --- /dev/null +++ b/backend/.sqlx/query-41428ecb8940a8fb4043b71eacb554314177fc11bc82212d95182e3723e64c6b.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_run_progress\n WHERE workspace_id = $1 AND updated_at < now() - make_interval(days => $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Int4" + ] + }, + "nullable": [] + }, + "hash": "41428ecb8940a8fb4043b71eacb554314177fc11bc82212d95182e3723e64c6b" +} diff --git a/backend/.sqlx/query-42f2cf3165850524804842c51965c4020193a613357c7449644e09a89d6d5346.json b/backend/.sqlx/query-42f2cf3165850524804842c51965c4020193a613357c7449644e09a89d6d5346.json new file mode 100644 index 0000000000..45031a9f99 --- /dev/null +++ b/backend/.sqlx/query-42f2cf3165850524804842c51965c4020193a613357c7449644e09a89d6d5346.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_graph_snapshot\n (workspace_id, script_path, script_hash, job_id, permissioned_as, digest, ingested_at)\n VALUES ($1, $2, NULL, $3, $4, $5, now())", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Uuid", + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "42f2cf3165850524804842c51965c4020193a613357c7449644e09a89d6d5346" +} diff --git a/backend/.sqlx/query-4301a7ad1f7c09e26f5dc7044e0a270be78e44c167fe3f6a73e0ff4b1e3baa35.json b/backend/.sqlx/query-4301a7ad1f7c09e26f5dc7044e0a270be78e44c167fe3f6a73e0ff4b1e3baa35.json new file mode 100644 index 0000000000..3c41b3fb85 --- /dev/null +++ b/backend/.sqlx/query-4301a7ad1f7c09e26f5dc7044e0a270be78e44c167fe3f6a73e0ff4b1e3baa35.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow (workspace_id, path, summary, description, value, edited_by, edited_at, on_behalf_of)\n VALUES ('test-workspace', 'u/test-user/obo_flow', '', '', $1, 'test-user', NOW(), 'u/test-user-2')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "4301a7ad1f7c09e26f5dc7044e0a270be78e44c167fe3f6a73e0ff4b1e3baa35" +} diff --git a/backend/.sqlx/query-4330e66917f04dc6597d323e9102de473e3f319ba13fa0784dd5bef24224233c.json b/backend/.sqlx/query-4330e66917f04dc6597d323e9102de473e3f319ba13fa0784dd5bef24224233c.json new file mode 100644 index 0000000000..4762ed3279 --- /dev/null +++ b/backend/.sqlx/query-4330e66917f04dc6597d323e9102de473e3f319ba13fa0784dd5bef24224233c.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE dbt_run_state SET workspace_id = $1 WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "4330e66917f04dc6597d323e9102de473e3f319ba13fa0784dd5bef24224233c" +} diff --git a/backend/.sqlx/query-4374167ebf64c4e8dd683e31451800c2d068cbafbd279454724451d7f57e108e.json b/backend/.sqlx/query-4374167ebf64c4e8dd683e31451800c2d068cbafbd279454724451d7f57e108e.json new file mode 100644 index 0000000000..a2e8f4127c --- /dev/null +++ b/backend/.sqlx/query-4374167ebf64c4e8dd683e31451800c2d068cbafbd279454724451d7f57e108e.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, hash, path, summary, description, content,\n created_by, language, lock)\n VALUES ($1, $2, $3, '', '', 'profile: {}', 'test-user', 'dbt', '')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Int8", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "4374167ebf64c4e8dd683e31451800c2d068cbafbd279454724451d7f57e108e" +} diff --git a/backend/.sqlx/query-471edf439cb293aa824440a2ef2f0afc28f425b061e90e1dfc6133b426a7f339.json b/backend/.sqlx/query-471edf439cb293aa824440a2ef2f0afc28f425b061e90e1dfc6133b426a7f339.json new file mode 100644 index 0000000000..5903661c1a --- /dev/null +++ b/backend/.sqlx/query-471edf439cb293aa824440a2ef2f0afc28f425b061e90e1dfc6133b426a7f339.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_id FROM v2_job WHERE id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_id", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + true + ] + }, + "hash": "471edf439cb293aa824440a2ef2f0afc28f425b061e90e1dfc6133b426a7f339" +} diff --git a/backend/.sqlx/query-47f34b18306f043bdbd7dc74c3c83ac80b669a910eada527c2a75f5cbf7d830d.json b/backend/.sqlx/query-47f34b18306f043bdbd7dc74c3c83ac80b669a910eada527c2a75f5cbf7d830d.json new file mode 100644 index 0000000000..28aa09d18d --- /dev/null +++ b/backend/.sqlx/query-47f34b18306f043bdbd7dc74c3c83ac80b669a910eada527c2a75f5cbf7d830d.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, value FROM global_settings WHERE name = ANY($1)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "value", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "47f34b18306f043bdbd7dc74c3c83ac80b669a910eada527c2a75f5cbf7d830d" +} diff --git a/backend/.sqlx/query-48969b648d36da8ef422d8f936eb5324655def916a0ac6f6a65eb649b17f102d.json b/backend/.sqlx/query-48969b648d36da8ef422d8f936eb5324655def916a0ac6f6a65eb649b17f102d.json new file mode 100644 index 0000000000..ba9ccbbb77 --- /dev/null +++ b/backend/.sqlx/query-48969b648d36da8ef422d8f936eb5324655def916a0ac6f6a65eb649b17f102d.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT EXISTS(\n SELECT 1\n FROM workspace_settings ws\n JOIN workspace w ON w.id = ws.workspace_id\n WHERE w.deleted = false AND w.id NOT LIKE 'wm-fork%'\n AND jsonb_exists(ws.large_file_storage->'secondary_storage', 'main')\n ) AS \"exists!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "48969b648d36da8ef422d8f936eb5324655def916a0ac6f6a65eb649b17f102d" +} diff --git a/backend/.sqlx/query-493b29e093bb2b8f8ed6947843e26fe8fb6aefe54ad5862cad9b8fd9868cdd91.json b/backend/.sqlx/query-493b29e093bb2b8f8ed6947843e26fe8fb6aefe54ad5862cad9b8fd9868cdd91.json new file mode 100644 index 0000000000..2f36ff7969 --- /dev/null +++ b/backend/.sqlx/query-493b29e093bb2b8f8ed6947843e26fe8fb6aefe54ad5862cad9b8fd9868cdd91.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings SET dbt_warehouses = $1 WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Jsonb", + "Text" + ] + }, + "nullable": [] + }, + "hash": "493b29e093bb2b8f8ed6947843e26fe8fb6aefe54ad5862cad9b8fd9868cdd91" +} diff --git a/backend/.sqlx/query-4956c7bb3312520e0ba0ae330037056f9a6157c1c448e46f502a17615ec25d87.json b/backend/.sqlx/query-4956c7bb3312520e0ba0ae330037056f9a6157c1c448e46f502a17615ec25d87.json new file mode 100644 index 0000000000..c43bb800bb --- /dev/null +++ b/backend/.sqlx/query-4956c7bb3312520e0ba0ae330037056f9a6157c1c448e46f502a17615ec25d87.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH t1 AS (UPDATE http_trigger SET script_path = $1 WHERE script_path = $2 AND workspace_id = $3 AND is_flow = $4), t2 AS (UPDATE email_trigger SET script_path = $1 WHERE script_path = $2 AND workspace_id = $3 AND is_flow = $4) UPDATE native_trigger SET script_path = $1, updated_at = NOW(), error = $5 WHERE script_path = $2 AND workspace_id = $3 AND is_flow = $4 RETURNING service_name::text AS \"service_name!\", external_id, script_path, is_flow", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "service_name!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "external_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "script_path", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "is_flow", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text", + "Bool", + "Text" + ] + }, + "nullable": [ + null, + false, + false, + false + ] + }, + "hash": "4956c7bb3312520e0ba0ae330037056f9a6157c1c448e46f502a17615ec25d87" +} diff --git a/backend/.sqlx/query-4e3f3202e762aad38707315c05c9c3b7cebdbd9be8b5cc9a14b325c4ad1f5b4f.json b/backend/.sqlx/query-4e3f3202e762aad38707315c05c9c3b7cebdbd9be8b5cc9a14b325c4ad1f5b4f.json new file mode 100644 index 0000000000..cf360c9dd5 --- /dev/null +++ b/backend/.sqlx/query-4e3f3202e762aad38707315c05c9c3b7cebdbd9be8b5cc9a14b325c4ad1f5b4f.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_lock(hashtextextended($1, 0))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "4e3f3202e762aad38707315c05c9c3b7cebdbd9be8b5cc9a14b325c4ad1f5b4f" +} diff --git a/backend/.sqlx/query-4ed961992cd6c99e2efe889ab1cf06e6ad12305a5b845948dc200f304fb87a61.json b/backend/.sqlx/query-4ed961992cd6c99e2efe889ab1cf06e6ad12305a5b845948dc200f304fb87a61.json new file mode 100644 index 0000000000..ee7535ffc9 --- /dev/null +++ b/backend/.sqlx/query-4ed961992cd6c99e2efe889ab1cf06e6ad12305a5b845948dc200f304fb87a61.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT dbt_warehouses FROM workspace_settings WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "dbt_warehouses", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "4ed961992cd6c99e2efe889ab1cf06e6ad12305a5b845948dc200f304fb87a61" +} diff --git a/backend/.sqlx/query-4f266373d5acdbcedc81e8ed3eb1fb95861ef17ca5570ef06b9903f04e9adb5e.json b/backend/.sqlx/query-4f266373d5acdbcedc81e8ed3eb1fb95861ef17ca5570ef06b9903f04e9adb5e.json new file mode 100644 index 0000000000..922a187344 --- /dev/null +++ b/backend/.sqlx/query-4f266373d5acdbcedc81e8ed3eb1fb95861ef17ca5570ef06b9903f04e9adb5e.json @@ -0,0 +1,70 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT workspace.id, workspace.name, workspace.owner, workspace.deleted, workspace.premium, workspace_settings.color, workspace.parent_workspace_id, workspace.is_dev_workspace, workspace.dev_workspace_label\n FROM workspace\n LEFT JOIN workspace_settings ON workspace.id = workspace_settings.workspace_id\n JOIN usr ON usr.workspace_id = workspace.id\n WHERE usr.email = $1 AND workspace.deleted = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "owner", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "deleted", + "type_info": "Bool" + }, + { + "ordinal": 4, + "name": "premium", + "type_info": "Bool" + }, + { + "ordinal": 5, + "name": "color", + "type_info": "Varchar" + }, + { + "ordinal": 6, + "name": "parent_workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 7, + "name": "is_dev_workspace", + "type_info": "Bool" + }, + { + "ordinal": 8, + "name": "dev_workspace_label", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + true, + true, + false, + true + ] + }, + "hash": "4f266373d5acdbcedc81e8ed3eb1fb95861ef17ca5570ef06b9903f04e9adb5e" +} diff --git a/backend/.sqlx/query-50f81c128d644c60837e603099e169fbc7a500c11e79d2c78d6eccccf6491aec.json b/backend/.sqlx/query-50f81c128d644c60837e603099e169fbc7a500c11e79d2c78d6eccccf6491aec.json index a7f5d2f785..e4e2c240c7 100644 --- a/backend/.sqlx/query-50f81c128d644c60837e603099e169fbc7a500c11e79d2c78d6eccccf6491aec.json +++ b/backend/.sqlx/query-50f81c128d644c60837e603099e169fbc7a500c11e79d2c78d6eccccf6491aec.json @@ -71,7 +71,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-5145e0505df1de2aba1740dcef647f74748e5b08d49ad0f2cdb9153e3f59b79c.json b/backend/.sqlx/query-5145e0505df1de2aba1740dcef647f74748e5b08d49ad0f2cdb9153e3f59b79c.json new file mode 100644 index 0000000000..2510e42835 --- /dev/null +++ b/backend/.sqlx/query-5145e0505df1de2aba1740dcef647f74748e5b08d49ad0f2cdb9153e3f59b79c.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, fork_last_event_kind, fork_last_event_origin FROM workspace_diff\n WHERE fork_workspace_id = 'wm-fork-deferred' AND kind = 'flow'\n AND fork_last_event_origin IS NOT NULL ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "fork_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "fork_last_event_origin", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true, + true + ] + }, + "hash": "5145e0505df1de2aba1740dcef647f74748e5b08d49ad0f2cdb9153e3f59b79c" +} diff --git a/backend/.sqlx/query-5369258383098062e454539be369ea2f0a41771376720fb72c6b3c1fa4972eab.json b/backend/.sqlx/query-5369258383098062e454539be369ea2f0a41771376720fb72c6b3c1fa4972eab.json index 6a010dd1f4..f1941c012c 100644 --- a/backend/.sqlx/query-5369258383098062e454539be369ea2f0a41771376720fb72c6b3c1fa4972eab.json +++ b/backend/.sqlx/query-5369258383098062e454539be369ea2f0a41771376720fb72c6b3c1fa4972eab.json @@ -35,7 +35,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-5387fbfd4be55674dcbe9f9b3ce9649d0f5db125776fbdffea993a53a09708de.json b/backend/.sqlx/query-5387fbfd4be55674dcbe9f9b3ce9649d0f5db125776fbdffea993a53a09708de.json index c617944b91..b00c398445 100644 --- a/backend/.sqlx/query-5387fbfd4be55674dcbe9f9b3ce9649d0f5db125776fbdffea993a53a09708de.json +++ b/backend/.sqlx/query-5387fbfd4be55674dcbe9f9b3ce9649d0f5db125776fbdffea993a53a09708de.json @@ -41,7 +41,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-55d11300ccbe3869c42e7f1f9f7bdaaf4eb6cbec3bf5cf755aab45cbce4dbf84.json b/backend/.sqlx/query-55d11300ccbe3869c42e7f1f9f7bdaaf4eb6cbec3bf5cf755aab45cbce4dbf84.json new file mode 100644 index 0000000000..a4831a626a --- /dev/null +++ b/backend/.sqlx/query-55d11300ccbe3869c42e7f1f9f7bdaaf4eb6cbec3bf5cf755aab45cbce4dbf84.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) - COUNT(DISTINCT path) FROM resource_version WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "55d11300ccbe3869c42e7f1f9f7bdaaf4eb6cbec3bf5cf755aab45cbce4dbf84" +} diff --git a/backend/.sqlx/query-56badc145d03676b6bd80a3830ce0a0401517a9b60e5689d3c9cdc9a87aaf2a8.json b/backend/.sqlx/query-56badc145d03676b6bd80a3830ce0a0401517a9b60e5689d3c9cdc9a87aaf2a8.json new file mode 100644 index 0000000000..b467a2c39b --- /dev/null +++ b/backend/.sqlx/query-56badc145d03676b6bd80a3830ce0a0401517a9b60e5689d3c9cdc9a87aaf2a8.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_edge (workspace_id, script_path, script_hash, job_id,\n parent_unique_id, child_unique_id, ingested_at)\n SELECT $2, script_path, script_hash, job_id, parent_unique_id, child_unique_id,\n ingested_at\n FROM dbt_edge\n WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "56badc145d03676b6bd80a3830ce0a0401517a9b60e5689d3c9cdc9a87aaf2a8" +} diff --git a/backend/.sqlx/query-58e7d590631294899025c2cc279c4352708f65a6c094a89b4cafa1e6450a4e6a.json b/backend/.sqlx/query-58e7d590631294899025c2cc279c4352708f65a6c094a89b4cafa1e6450a4e6a.json new file mode 100644 index 0000000000..1edc69b002 --- /dev/null +++ b/backend/.sqlx/query-58e7d590631294899025c2cc279c4352708f65a6c094a89b4cafa1e6450a4e6a.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT result->'materialized' FROM v2_job_completed WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "58e7d590631294899025c2cc279c4352708f65a6c094a89b4cafa1e6450a4e6a" +} diff --git a/backend/.sqlx/query-59a913a29c3f32d2d0771f46f0d408e686283c8a6737ad4ededa019158e2cb8d.json b/backend/.sqlx/query-59a913a29c3f32d2d0771f46f0d408e686283c8a6737ad4ededa019158e2cb8d.json new file mode 100644 index 0000000000..b89b1ff7f9 --- /dev/null +++ b/backend/.sqlx/query-59a913a29c3f32d2d0771f46f0d408e686283c8a6737ad4ededa019158e2cb8d.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO password(email, password_hash, login_type, super_admin, verified, name)\n VALUES ($1, '', 'password', true, true, '')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "59a913a29c3f32d2d0771f46f0d408e686283c8a6737ad4ededa019158e2cb8d" +} diff --git a/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json b/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json index 36ddb8ab9f..713ccb9dd3 100644 --- a/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json +++ b/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json @@ -15,7 +15,7 @@ ] }, "nullable": [ - true + null ] }, "hash": "5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55" diff --git a/backend/.sqlx/query-5a4e42f3bfa96bb1184adeccf14580c46c1e77cb8dd9b36523965d79eb5416bb.json b/backend/.sqlx/query-5a4e42f3bfa96bb1184adeccf14580c46c1e77cb8dd9b36523965d79eb5416bb.json new file mode 100644 index 0000000000..c1f6558712 --- /dev/null +++ b/backend/.sqlx/query-5a4e42f3bfa96bb1184adeccf14580c46c1e77cb8dd9b36523965d79eb5416bb.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of, on_behalf_of_email)\n VALUES ('test-workspace', 'u/test-user/sg', 95001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'g/ops', 'group-ops@windmill.dev'),\n ('test-workspace', 'u/test-user/su', 95002, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2', 'group-ops@windmill.dev')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "5a4e42f3bfa96bb1184adeccf14580c46c1e77cb8dd9b36523965d79eb5416bb" +} diff --git a/backend/.sqlx/query-5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800.json b/backend/.sqlx/query-5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800.json index c126b6371a..40c66a93f9 100644 --- a/backend/.sqlx/query-5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800.json +++ b/backend/.sqlx/query-5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800.json @@ -16,7 +16,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } @@ -71,7 +72,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-5c0d0c7111b70b119b3182c9195361388ead36dc6ab03c6a7c17fb97d2b60a67.json b/backend/.sqlx/query-5c0d0c7111b70b119b3182c9195361388ead36dc6ab03c6a7c17fb97d2b60a67.json index e72722d53e..be7d99f4a6 100644 --- a/backend/.sqlx/query-5c0d0c7111b70b119b3182c9195361388ead36dc6ab03c6a7c17fb97d2b60a67.json +++ b/backend/.sqlx/query-5c0d0c7111b70b119b3182c9195361388ead36dc6ab03c6a7c17fb97d2b60a67.json @@ -34,7 +34,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389.json b/backend/.sqlx/query-5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389.json index 03de8472ad..fc8ed63f51 100644 --- a/backend/.sqlx/query-5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389.json +++ b/backend/.sqlx/query-5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389.json @@ -21,7 +21,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-5deb93bef71327582d997f3681b60e79b82f57c9c15c0870539ff057931c8f0a.json b/backend/.sqlx/query-5deb93bef71327582d997f3681b60e79b82f57c9c15c0870539ff057931c8f0a.json new file mode 100644 index 0000000000..01f7f523ed --- /dev/null +++ b/backend/.sqlx/query-5deb93bef71327582d997f3681b60e79b82f57c9c15c0870539ff057931c8f0a.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) FROM dbt_graph_snapshot\n WHERE workspace_id = $1 AND script_path = $2 AND script_hash IS NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "5deb93bef71327582d997f3681b60e79b82f57c9c15c0870539ff057931c8f0a" +} diff --git a/backend/.sqlx/query-5ec262094e8ddf420b9098f2b3ef6d2c2caa94f39a0e57fe015e19fd1d0ab0ea.json b/backend/.sqlx/query-5ec262094e8ddf420b9098f2b3ef6d2c2caa94f39a0e57fe015e19fd1d0ab0ea.json index 8ff04d2fc1..34f1870b7a 100644 --- a/backend/.sqlx/query-5ec262094e8ddf420b9098f2b3ef6d2c2caa94f39a0e57fe015e19fd1d0ab0ea.json +++ b/backend/.sqlx/query-5ec262094e8ddf420b9098f2b3ef6d2c2caa94f39a0e57fe015e19fd1d0ab0ea.json @@ -44,7 +44,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-5f95454c29587117f7a482ca74506cda7ee52f5a85aea5c2186353f8b1decc46.json b/backend/.sqlx/query-5f95454c29587117f7a482ca74506cda7ee52f5a85aea5c2186353f8b1decc46.json new file mode 100644 index 0000000000..d1b9e3c858 --- /dev/null +++ b/backend/.sqlx/query-5f95454c29587117f7a482ca74506cda7ee52f5a85aea5c2186353f8b1decc46.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "5f95454c29587117f7a482ca74506cda7ee52f5a85aea5c2186353f8b1decc46" +} diff --git a/backend/.sqlx/query-5fd6467e3316b64e0451614d6bb9101644d1869af85687ab55067308644a2523.json b/backend/.sqlx/query-5fd6467e3316b64e0451614d6bb9101644d1869af85687ab55067308644a2523.json new file mode 100644 index 0000000000..418083acf9 --- /dev/null +++ b/backend/.sqlx/query-5fd6467e3316b64e0451614d6bb9101644d1869af85687ab55067308644a2523.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT script_lang = 'dbt' FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "5fd6467e3316b64e0451614d6bb9101644d1869af85687ab55067308644a2523" +} diff --git a/backend/.sqlx/query-608d487e6f3313d57c1f2f043b6942c9ab6f58f9d7ffecb13a9de2c71b9bb709.json b/backend/.sqlx/query-608d487e6f3313d57c1f2f043b6942c9ab6f58f9d7ffecb13a9de2c71b9bb709.json new file mode 100644 index 0000000000..69951a8cb2 --- /dev/null +++ b/backend/.sqlx/query-608d487e6f3313d57c1f2f043b6942c9ab6f58f9d7ffecb13a9de2c71b9bb709.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, on_behalf_of FROM script WHERE workspace_id = 'wm-fork-obo' ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "on_behalf_of", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true + ] + }, + "hash": "608d487e6f3313d57c1f2f043b6942c9ab6f58f9d7ffecb13a9de2c71b9bb709" +} diff --git a/backend/.sqlx/query-617c19043d445ab1366630633e6afbeeea28b1d17dfacecb30dde65304cd9799.json b/backend/.sqlx/query-617c19043d445ab1366630633e6afbeeea28b1d17dfacecb30dde65304cd9799.json new file mode 100644 index 0000000000..9f41bf73e0 --- /dev/null +++ b/backend/.sqlx/query-617c19043d445ab1366630633e6afbeeea28b1d17dfacecb30dde65304cd9799.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO azure_trigger (\n azure_resource_path, azure_mode, scope_resource_id, topic_name,\n subscription_name, event_type_filters, push_auth_config, path, script_path,\n is_flow, workspace_id, edited_by, edited_at, extra_perms, server_id,\n last_server_ping, error, mode, permissioned_as, error_handler_path,\n error_handler_args, retry, labels\n )\n SELECT\n azure_resource_path, azure_mode, scope_resource_id, topic_name,\n subscription_name, event_type_filters, push_auth_config, path, script_path,\n is_flow, $1, edited_by, edited_at, extra_perms, NULL,\n NULL, NULL, 'disabled'::TRIGGER_MODE, permissioned_as, error_handler_path,\n error_handler_args, retry, labels\n FROM azure_trigger WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "617c19043d445ab1366630633e6afbeeea28b1d17dfacecb30dde65304cd9799" +} diff --git a/backend/.sqlx/query-620e8c417d60a8c86b359673da943df36d9348bf14dcb7b97263b226f0f3b337.json b/backend/.sqlx/query-620e8c417d60a8c86b359673da943df36d9348bf14dcb7b97263b226f0f3b337.json new file mode 100644 index 0000000000..9305fd955c --- /dev/null +++ b/backend/.sqlx/query-620e8c417d60a8c86b359673da943df36d9348bf14dcb7b97263b226f0f3b337.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, fork_last_event_kind, fork_last_event_origin FROM workspace_diff\n WHERE fork_workspace_id = 'wm-fork-renamed' AND kind = 'flow'\n AND fork_last_event_origin IS NOT NULL ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "fork_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "fork_last_event_origin", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true, + true + ] + }, + "hash": "620e8c417d60a8c86b359673da943df36d9348bf14dcb7b97263b226f0f3b337" +} diff --git a/backend/.sqlx/query-6348061d79b0b6b25bda7425f7a37344e04107f060f949cc197b4df9bca18fb8.json b/backend/.sqlx/query-6348061d79b0b6b25bda7425f7a37344e04107f060f949cc197b4df9bca18fb8.json index f3d518eff0..3f1729e29a 100644 --- a/backend/.sqlx/query-6348061d79b0b6b25bda7425f7a37344e04107f060f949cc197b4df9bca18fb8.json +++ b/backend/.sqlx/query-6348061d79b0b6b25bda7425f7a37344e04107f060f949cc197b4df9bca18fb8.json @@ -34,7 +34,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-652c7a8adefc03f06485a7e223f54ebcab136711b89671192f379bf06bf3ca9b.json b/backend/.sqlx/query-652c7a8adefc03f06485a7e223f54ebcab136711b89671192f379bf06bf3ca9b.json new file mode 100644 index 0000000000..43a2c8952d --- /dev/null +++ b/backend/.sqlx/query-652c7a8adefc03f06485a7e223f54ebcab136711b89671192f379bf06bf3ca9b.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO password(email, password_hash, login_type, super_admin, verified, name, username)\n VALUES ('sa@windmill.dev', 'not-a-real-hash', 'password', true, true, 'Ext', 'ext-sa')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "652c7a8adefc03f06485a7e223f54ebcab136711b89671192f379bf06bf3ca9b" +} diff --git a/backend/.sqlx/query-66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd.json b/backend/.sqlx/query-66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd.json index 80f172aa40..f6e000ba6f 100644 --- a/backend/.sqlx/query-66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd.json +++ b/backend/.sqlx/query-66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd.json @@ -45,7 +45,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-680f38ee8b7fd09aa3d0f521bce5ff3c9bf42c7a51aadbe7d7680827989211d0.json b/backend/.sqlx/query-680f38ee8b7fd09aa3d0f521bce5ff3c9bf42c7a51aadbe7d7680827989211d0.json new file mode 100644 index 0000000000..22dfb381a0 --- /dev/null +++ b/backend/.sqlx/query-680f38ee8b7fd09aa3d0f521bce5ff3c9bf42c7a51aadbe7d7680827989211d0.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of FROM script\n WHERE path = $1 AND workspace_id = $2 AND archived = false AND deleted = false\n ORDER BY created_at DESC LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "680f38ee8b7fd09aa3d0f521bce5ff3c9bf42c7a51aadbe7d7680827989211d0" +} diff --git a/backend/.sqlx/query-68aebbc8a7798abab3163aaffe298f6ea2722c68efff0c2a1baaf5de4fc247b2.json b/backend/.sqlx/query-68aebbc8a7798abab3163aaffe298f6ea2722c68efff0c2a1baaf5de4fc247b2.json new file mode 100644 index 0000000000..0b19784e3a --- /dev/null +++ b/backend/.sqlx/query-68aebbc8a7798abab3163aaffe298f6ea2722c68efff0c2a1baaf5de4fc247b2.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of, on_behalf_of_email)\n VALUES\n ('test-workspace', 'u/test-user/obo_member', 91001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user', 'test@windmill.dev'),\n ('test-workspace', 'u/test-user/obo_stranger', 91002, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2', 'test2@windmill.dev'),\n ('test-workspace', 'u/test-user/obo_group', 91003, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'g/all', 'group-all@windmill.dev'),\n ('test-workspace', 'u/test-user/obo_superadmin', 91004, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/ext-sa', 'sa@windmill.dev'),\n ('test-workspace', 'u/test-user/obo_address_only', 91005, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), NULL, 'test2@windmill.dev')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "68aebbc8a7798abab3163aaffe298f6ea2722c68efff0c2a1baaf5de4fc247b2" +} diff --git a/backend/.sqlx/query-6a0912f2cee5434ffecbdb7ea7ad0fe6800283a8694df86f335013a3d6397996.json b/backend/.sqlx/query-6a0912f2cee5434ffecbdb7ea7ad0fe6800283a8694df86f335013a3d6397996.json new file mode 100644 index 0000000000..b284ed202a --- /dev/null +++ b/backend/.sqlx/query-6a0912f2cee5434ffecbdb7ea7ad0fe6800283a8694df86f335013a3d6397996.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, fork_last_event_kind, fork_last_event_origin)\n VALUES ($1, $2, $3, $4, 1, 0, NULL, $5, $6)\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n ahead = workspace_diff.ahead + 1,\n has_changes = NULL,\n fork_last_event_kind = EXCLUDED.fork_last_event_kind,\n fork_last_event_origin = EXCLUDED.fork_last_event_origin", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "6a0912f2cee5434ffecbdb7ea7ad0fe6800283a8694df86f335013a3d6397996" +} diff --git a/backend/.sqlx/query-6a8865e80306a6deb68d0ff8ee79b179827ec89d4a2043b38cb96843f4675ccc.json b/backend/.sqlx/query-6a8865e80306a6deb68d0ff8ee79b179827ec89d4a2043b38cb96843f4675ccc.json new file mode 100644 index 0000000000..59e1b8f3a5 --- /dev/null +++ b/backend/.sqlx/query-6a8865e80306a6deb68d0ff8ee79b179827ec89d4a2043b38cb96843f4675ccc.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ahead, fork_last_event_kind, fork_last_event_origin FROM workspace_diff\n WHERE source_workspace_id = 'test-workspace' AND fork_workspace_id = 'wm-fork-removal'\n AND kind = 'script' AND path = 'u/admin/dropped_by_sync'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ahead", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "fork_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "fork_last_event_origin", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true, + true + ] + }, + "hash": "6a8865e80306a6deb68d0ff8ee79b179827ec89d4a2043b38cb96843f4675ccc" +} diff --git a/backend/.sqlx/query-6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f.json b/backend/.sqlx/query-6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f.json index 144bcb13f2..b54e73c157 100644 --- a/backend/.sqlx/query-6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f.json +++ b/backend/.sqlx/query-6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f.json @@ -22,7 +22,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-6d89436b268157453784087c076ef5fb7c5ae46bede607cca308e1d5778e6d1e.json b/backend/.sqlx/query-6d89436b268157453784087c076ef5fb7c5ae46bede607cca308e1d5778e6d1e.json new file mode 100644 index 0000000000..8e6b0372b1 --- /dev/null +++ b/backend/.sqlx/query-6d89436b268157453784087c076ef5fb7c5ae46bede607cca308e1d5778e6d1e.json @@ -0,0 +1,62 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT asset_kind AS \"asset_kind: windmill_common::assets::AssetKind\", asset_path,\n status::text AS \"status!\", row_count, error\n FROM dbt_run_progress\n WHERE workspace_id = $1 AND job_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_kind: windmill_common::assets::AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume", + "dbt" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "asset_path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "status!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "row_count", + "type_info": "Int8" + }, + { + "ordinal": 4, + "name": "error", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [ + false, + false, + null, + true, + true + ] + }, + "hash": "6d89436b268157453784087c076ef5fb7c5ae46bede607cca308e1d5778e6d1e" +} diff --git a/backend/.sqlx/query-6f1489b2c52e08566ffa9450709df2b9404a30197b05554a868bd06b1ed8ccaa.json b/backend/.sqlx/query-6f1489b2c52e08566ffa9450709df2b9404a30197b05554a868bd06b1ed8ccaa.json new file mode 100644 index 0000000000..e71f76301e --- /dev/null +++ b/backend/.sqlx/query-6f1489b2c52e08566ffa9450709df2b9404a30197b05554a868bd06b1ed8ccaa.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "6f1489b2c52e08566ffa9450709df2b9404a30197b05554a868bd06b1ed8ccaa" +} diff --git a/backend/.sqlx/query-717188b993d9c87f2de51d8edbafec074f8e02408efc6da2c742d81fd47c14f0.json b/backend/.sqlx/query-717188b993d9c87f2de51d8edbafec074f8e02408efc6da2c742d81fd47c14f0.json new file mode 100644 index 0000000000..6dc8e9acc2 --- /dev/null +++ b/backend/.sqlx/query-717188b993d9c87f2de51d8edbafec074f8e02408efc6da2c742d81fd47c14f0.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) FROM dbt_node WHERE workspace_id = $1 AND script_hash = $2 AND job_id = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8", + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "717188b993d9c87f2de51d8edbafec074f8e02408efc6da2c742d81fd47c14f0" +} diff --git a/backend/.sqlx/query-71767e6684957af5dff56a1bb64f980f712f92798d11feaeb0832962ad9ddb2e.json b/backend/.sqlx/query-71767e6684957af5dff56a1bb64f980f712f92798d11feaeb0832962ad9ddb2e.json index 96a68f827d..64c6611843 100644 --- a/backend/.sqlx/query-71767e6684957af5dff56a1bb64f980f712f92798d11feaeb0832962ad9ddb2e.json +++ b/backend/.sqlx/query-71767e6684957af5dff56a1bb64f980f712f92798d11feaeb0832962ad9ddb2e.json @@ -84,7 +84,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-71b341ba06d588a7fb425d97921f16a8fc4fdf0bcebecfd21c9b7154dedea25f.json b/backend/.sqlx/query-71b341ba06d588a7fb425d97921f16a8fc4fdf0bcebecfd21c9b7154dedea25f.json new file mode 100644 index 0000000000..397b310807 --- /dev/null +++ b/backend/.sqlx/query-71b341ba06d588a7fb425d97921f16a8fc4fdf0bcebecfd21c9b7154dedea25f.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE materialized_partition\n SET status = 'failed',\n error = COALESCE(error, 'the run ended before this model finished')\n WHERE workspace_id = $1 AND job_id = $2 AND status = 'running'\n AND NOT (asset_path = ANY($3))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Uuid", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "71b341ba06d588a7fb425d97921f16a8fc4fdf0bcebecfd21c9b7154dedea25f" +} diff --git a/backend/.sqlx/query-72fa27780070363e092a19dbea4b63195863245bfb271a946d2aa67c59da7b3c.json b/backend/.sqlx/query-72fa27780070363e092a19dbea4b63195863245bfb271a946d2aa67c59da7b3c.json new file mode 100644 index 0000000000..d97f8be4ac --- /dev/null +++ b/backend/.sqlx/query-72fa27780070363e092a19dbea4b63195863245bfb271a946d2aa67c59da7b3c.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT email, workspace_id, super_admin, owner FROM token WHERE token_hash = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "email", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "super_admin", + "type_info": "Bool" + }, + { + "ordinal": 3, + "name": "owner", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + true, + false, + true + ] + }, + "hash": "72fa27780070363e092a19dbea4b63195863245bfb271a946d2aa67c59da7b3c" +} diff --git a/backend/.sqlx/query-74cab23105c5892b5df86ba1e81ac3d575db3d3542341988917f68165174f133.json b/backend/.sqlx/query-74cab23105c5892b5df86ba1e81ac3d575db3d3542341988917f68165174f133.json new file mode 100644 index 0000000000..a58ee643d0 --- /dev/null +++ b/backend/.sqlx/query-74cab23105c5892b5df86ba1e81ac3d575db3d3542341988917f68165174f133.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE draft SET value = to_json(jsonb_set(to_jsonb(value), ARRAY['on_behalf_of'], to_jsonb($1::text))) WHERE value->>'on_behalf_of' = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "74cab23105c5892b5df86ba1e81ac3d575db3d3542341988917f68165174f133" +} diff --git a/backend/.sqlx/query-77146f3ebe59007b7608c88ba35dddc63e9fde5b301e7f935def7e1d0d41e672.json b/backend/.sqlx/query-77146f3ebe59007b7608c88ba35dddc63e9fde5b301e7f935def7e1d0d41e672.json new file mode 100644 index 0000000000..35a2e8deda --- /dev/null +++ b/backend/.sqlx/query-77146f3ebe59007b7608c88ba35dddc63e9fde5b301e7f935def7e1d0d41e672.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT label, expiration, scopes FROM token WHERE token_hash = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "label", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "expiration", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "scopes", + "type_info": "TextArray" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + true, + true + ] + }, + "hash": "77146f3ebe59007b7608c88ba35dddc63e9fde5b301e7f935def7e1d0d41e672" +} diff --git a/backend/.sqlx/query-79db50d906264ab96dffa134673f9171b943e959eb5401a496af850aa686391b.json b/backend/.sqlx/query-79db50d906264ab96dffa134673f9171b943e959eb5401a496af850aa686391b.json new file mode 100644 index 0000000000..75871561e9 --- /dev/null +++ b/backend/.sqlx/query-79db50d906264ab96dffa134673f9171b943e959eb5401a496af850aa686391b.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM workspace WHERE id = $1) AS \"exists!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "79db50d906264ab96dffa134673f9171b943e959eb5401a496af850aa686391b" +} diff --git a/backend/.sqlx/query-7ac16d499a630454d9327304d30ad667529f720e2a570b664679564cacec6708.json b/backend/.sqlx/query-7ac16d499a630454d9327304d30ad667529f720e2a570b664679564cacec6708.json new file mode 100644 index 0000000000..22182b5470 --- /dev/null +++ b/backend/.sqlx/query-7ac16d499a630454d9327304d30ad667529f720e2a570b664679564cacec6708.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at, labels)\n SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at, labels\n FROM flow\n WHERE path = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "7ac16d499a630454d9327304d30ad667529f720e2a570b664679564cacec6708" +} diff --git a/backend/.sqlx/query-7bb8ff3426ec73672fb572cd1ce36f495a84e0ad7d60fd39eddcccbc129d43c8.json b/backend/.sqlx/query-7bb8ff3426ec73672fb572cd1ce36f495a84e0ad7d60fd39eddcccbc129d43c8.json index b38e22ad67..d26e9f7c87 100644 --- a/backend/.sqlx/query-7bb8ff3426ec73672fb572cd1ce36f495a84e0ad7d60fd39eddcccbc129d43c8.json +++ b/backend/.sqlx/query-7bb8ff3426ec73672fb572cd1ce36f495a84e0ad7d60fd39eddcccbc129d43c8.json @@ -77,7 +77,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-7be4645fe51c8a9bd13645a245846cb485c593b6df2194ae91ba2d8c584c0cce.json b/backend/.sqlx/query-7be4645fe51c8a9bd13645a245846cb485c593b6df2194ae91ba2d8c584c0cce.json new file mode 100644 index 0000000000..936a85b611 --- /dev/null +++ b/backend/.sqlx/query-7be4645fe51c8a9bd13645a245846cb485c593b6df2194ae91ba2d8c584c0cce.json @@ -0,0 +1,42 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT identity, args, run_results, job_id FROM dbt_run_state\n WHERE workspace_id = $1 AND script_path = $2 AND permissioned_as = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "identity", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "args", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "run_results", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "job_id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false, + true + ] + }, + "hash": "7be4645fe51c8a9bd13645a245846cb485c593b6df2194ae91ba2d8c584c0cce" +} diff --git a/backend/.sqlx/query-7c19d92d25b799e8872452c5c77ae110e2c0628b2721a44735f4fc06e59b5c77.json b/backend/.sqlx/query-7c19d92d25b799e8872452c5c77ae110e2c0628b2721a44735f4fc06e59b5c77.json new file mode 100644 index 0000000000..ee215b9615 --- /dev/null +++ b/backend/.sqlx/query-7c19d92d25b799e8872452c5c77ae110e2c0628b2721a44735f4fc06e59b5c77.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) FROM dbt_graph_snapshot WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "7c19d92d25b799e8872452c5c77ae110e2c0628b2721a44735f4fc06e59b5c77" +} diff --git a/backend/.sqlx/query-7cddbc01dbec7878e7a8e528f21fbb75b10af4af97f5ecbe094db8f8080bf21d.json b/backend/.sqlx/query-7cddbc01dbec7878e7a8e528f21fbb75b10af4af97f5ecbe094db8f8080bf21d.json new file mode 100644 index 0000000000..72093610da --- /dev/null +++ b/backend/.sqlx/query-7cddbc01dbec7878e7a8e528f21fbb75b10af4af97f5ecbe094db8f8080bf21d.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at)\n SELECT workspace_id, REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "7cddbc01dbec7878e7a8e528f21fbb75b10af4af97f5ecbe094db8f8080bf21d" +} diff --git a/backend/.sqlx/query-7ce4e7af3d3cd01e644b8f1cca15db49ab38ac4982c7c258b17bf9ef53047cb6.json b/backend/.sqlx/query-7ce4e7af3d3cd01e644b8f1cca15db49ab38ac4982c7c258b17bf9ef53047cb6.json new file mode 100644 index 0000000000..04f84cd931 --- /dev/null +++ b/backend/.sqlx/query-7ce4e7af3d3cd01e644b8f1cca15db49ab38ac4982c7c258b17bf9ef53047cb6.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE flow SET on_behalf_of = $1 WHERE on_behalf_of = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "7ce4e7af3d3cd01e644b8f1cca15db49ab38ac4982c7c258b17bf9ef53047cb6" +} diff --git a/backend/.sqlx/query-7e9e07f55ebb12d76efee5c92e98b85dbf7eebfdd64a605569b9b2f814f7fc77.json b/backend/.sqlx/query-7e9e07f55ebb12d76efee5c92e98b85dbf7eebfdd64a605569b9b2f814f7fc77.json new file mode 100644 index 0000000000..cc4dd630cf --- /dev/null +++ b/backend/.sqlx/query-7e9e07f55ebb12d76efee5c92e98b85dbf7eebfdd64a605569b9b2f814f7fc77.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs)\n SELECT $1, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs\n FROM flow WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "7e9e07f55ebb12d76efee5c92e98b85dbf7eebfdd64a605569b9b2f814f7fc77" +} diff --git a/backend/.sqlx/query-8058042d5564a934534517e9c0446c5fcc0c3654746c5e65994123ffb4d05f78.json b/backend/.sqlx/query-8058042d5564a934534517e9c0446c5fcc0c3654746c5e65994123ffb4d05f78.json deleted file mode 100644 index 5453ec6899..0000000000 --- a/backend/.sqlx/query-8058042d5564a934534517e9c0446c5fcc0c3654746c5e65994123ffb4d05f78.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO workspace_settings (workspace_id, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, ducklake, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts) SELECT $1, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, ducklake, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts FROM workspace_settings WHERE workspace_id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Text" - ] - }, - "nullable": [] - }, - "hash": "8058042d5564a934534517e9c0446c5fcc0c3654746c5e65994123ffb4d05f78" -} diff --git a/backend/.sqlx/query-80a3a4e8f35190352aa4e4c5716013ce4a6f7b3ba27c63c77c4c765bf9076624.json b/backend/.sqlx/query-80a3a4e8f35190352aa4e4c5716013ce4a6f7b3ba27c63c77c4c765bf9076624.json new file mode 100644 index 0000000000..a72789aa6b --- /dev/null +++ b/backend/.sqlx/query-80a3a4e8f35190352aa4e4c5716013ce4a6f7b3ba27c63c77c4c765bf9076624.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE app SET path = path WHERE path = $1 AND workspace_id = $2 RETURNING 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "80a3a4e8f35190352aa4e4c5716013ce4a6f7b3ba27c63c77c4c765bf9076624" +} diff --git a/backend/.sqlx/query-825b7e301307373ecfa6b59b4091e5fe53514913562ca93de44d9ad0a02a0fe8.json b/backend/.sqlx/query-825b7e301307373ecfa6b59b4091e5fe53514913562ca93de44d9ad0a02a0fe8.json new file mode 100644 index 0000000000..46ec0bde59 --- /dev/null +++ b/backend/.sqlx/query-825b7e301307373ecfa6b59b4091e5fe53514913562ca93de44d9ad0a02a0fe8.json @@ -0,0 +1,71 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT workspace_id, path, value, description, resource_type, extra_perms, created_by, edited_at, labels FROM resource WHERE workspace_id = $1 AND resource_type <> ALL($2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "value", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "description", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "resource_type", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "extra_perms", + "type_info": "Jsonb" + }, + { + "ordinal": 6, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 7, + "name": "edited_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 8, + "name": "labels", + "type_info": "TextArray" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false, + true, + true, + false, + false, + true, + true, + true + ] + }, + "hash": "825b7e301307373ecfa6b59b4091e5fe53514913562ca93de44d9ad0a02a0fe8" +} diff --git a/backend/.sqlx/query-8317932e7a9a5d927881bc9cc573895307eed9eb259862da07813b6daa37ba9e.json b/backend/.sqlx/query-8317932e7a9a5d927881bc9cc573895307eed9eb259862da07813b6daa37ba9e.json new file mode 100644 index 0000000000..660a01d710 --- /dev/null +++ b/backend/.sqlx/query-8317932e7a9a5d927881bc9cc573895307eed9eb259862da07813b6daa37ba9e.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM folder WHERE workspace_id = $1 AND name = $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "8317932e7a9a5d927881bc9cc573895307eed9eb259862da07813b6daa37ba9e" +} diff --git a/backend/.sqlx/query-836297c7d83185ac4f317e0bd0b2e73c55932927b05f963d45479103e3191ecd.json b/backend/.sqlx/query-836297c7d83185ac4f317e0bd0b2e73c55932927b05f963d45479103e3191ecd.json new file mode 100644 index 0000000000..a4026be742 --- /dev/null +++ b/backend/.sqlx/query-836297c7d83185ac4f317e0bd0b2e73c55932927b05f963d45479103e3191ecd.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO script\n (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n FROM script WHERE hash = $2 AND workspace_id = $3;\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Text" + ] + }, + "nullable": [] + }, + "hash": "836297c7d83185ac4f317e0bd0b2e73c55932927b05f963d45479103e3191ecd" +} diff --git a/backend/.sqlx/query-83c4fed5d1a05365c3f486543519963c1068ea26e11307acec28fd07d0ce6b90.json b/backend/.sqlx/query-83c4fed5d1a05365c3f486543519963c1068ea26e11307acec28fd07d0ce6b90.json new file mode 100644 index 0000000000..baf02539cf --- /dev/null +++ b/backend/.sqlx/query-83c4fed5d1a05365c3f486543519963c1068ea26e11307acec28fd07d0ce6b90.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH inserted AS (\n INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs)\n SELECT workspace_id, REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3\n RETURNING 1\n ) SELECT COUNT(*) FROM inserted", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "83c4fed5d1a05365c3f486543519963c1068ea26e11307acec28fd07d0ce6b90" +} diff --git a/backend/.sqlx/query-843ef7d1c619d2784306fa1dd06036c6967b5a2ebb88cf174f435a2ce692c5dc.json b/backend/.sqlx/query-843ef7d1c619d2784306fa1dd06036c6967b5a2ebb88cf174f435a2ce692c5dc.json new file mode 100644 index 0000000000..5c000ffd34 --- /dev/null +++ b/backend/.sqlx/query-843ef7d1c619d2784306fa1dd06036c6967b5a2ebb88cf174f435a2ce692c5dc.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) FROM dbt_graph_snapshot WHERE workspace_id = $1 AND job_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "843ef7d1c619d2784306fa1dd06036c6967b5a2ebb88cf174f435a2ce692c5dc" +} diff --git a/backend/.sqlx/query-84c88cd1c907317714dfe0e7acb3dd3ea34c1fc5c2700fdcccdd17fbff5dcee5.json b/backend/.sqlx/query-84c88cd1c907317714dfe0e7acb3dd3ea34c1fc5c2700fdcccdd17fbff5dcee5.json new file mode 100644 index 0000000000..d608d90996 --- /dev/null +++ b/backend/.sqlx/query-84c88cd1c907317714dfe0e7acb3dd3ea34c1fc5c2700fdcccdd17fbff5dcee5.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM script WHERE workspace_id = $1 AND path = $2 AND archived = false)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "84c88cd1c907317714dfe0e7acb3dd3ea34c1fc5c2700fdcccdd17fbff5dcee5" +} diff --git a/backend/.sqlx/query-850b5e3414851e53f9d5580b0ab1fdb97b053620ac14b8196ebf09208510fb1f.json b/backend/.sqlx/query-850b5e3414851e53f9d5580b0ab1fdb97b053620ac14b8196ebf09208510fb1f.json new file mode 100644 index 0000000000..4c72eca991 --- /dev/null +++ b/backend/.sqlx/query-850b5e3414851e53f9d5580b0ab1fdb97b053620ac14b8196ebf09208510fb1f.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE dbt_graph_snapshot SET relation_root_at_last_ingest = $4\n WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3\n AND job_id = '00000000-0000-0000-0000-000000000000'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Text" + ] + }, + "nullable": [] + }, + "hash": "850b5e3414851e53f9d5580b0ab1fdb97b053620ac14b8196ebf09208510fb1f" +} diff --git a/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json b/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json index 5aacf1a295..a535cf8aed 100644 --- a/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json +++ b/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json @@ -16,7 +16,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-86102dd1a05cb22bc20e153d33113dc5e56df70ec8743027fbde8d5c4155bfc5.json b/backend/.sqlx/query-86102dd1a05cb22bc20e153d33113dc5e56df70ec8743027fbde8d5c4155bfc5.json new file mode 100644 index 0000000000..b9f06081b4 --- /dev/null +++ b/backend/.sqlx/query-86102dd1a05cb22bc20e153d33113dc5e56df70ec8743027fbde8d5c4155bfc5.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, source_last_event_kind, source_last_event_origin)\n SELECT $1, unnest($2::varchar[]), $3, $4, 0, 1, NULL, $5, $6\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n behind = workspace_diff.behind + 1,\n has_changes = NULL,\n source_last_event_kind = CASE WHEN $6::varchar IS NULL THEN workspace_diff.source_last_event_kind ELSE $5 END,\n source_last_event_origin = COALESCE($6, workspace_diff.source_last_event_origin)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "VarcharArray", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "86102dd1a05cb22bc20e153d33113dc5e56df70ec8743027fbde8d5c4155bfc5" +} diff --git a/backend/.sqlx/query-86c1c8712587acbacc830fde1be608f7f9b84c13d8d906113467c00d440a8fa2.json b/backend/.sqlx/query-86c1c8712587acbacc830fde1be608f7f9b84c13d8d906113467c00d440a8fa2.json new file mode 100644 index 0000000000..3c4815172c --- /dev/null +++ b/backend/.sqlx/query-86c1c8712587acbacc830fde1be608f7f9b84c13d8d906113467c00d440a8fa2.json @@ -0,0 +1,42 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE native_trigger\n SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4,\n summary = $5, error = NULL, updated_at = NOW()\n WHERE\n workspace_id = $6\n AND service_name = $7\n AND external_id = $8\n AND script_path = $9\n AND is_flow = $10\n RETURNING 1 AS \"applied!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "applied!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Bool", + "Varchar", + "Jsonb", + "Varchar", + "Text", + { + "Custom": { + "name": "native_trigger_service", + "kind": { + "Enum": [ + "nextcloud", + "google", + "github" + ] + } + } + }, + "Text", + "Text", + "Bool" + ] + }, + "nullable": [ + null + ] + }, + "hash": "86c1c8712587acbacc830fde1be608f7f9b84c13d8d906113467c00d440a8fa2" +} diff --git a/backend/.sqlx/query-87afd5289e54b6accda8a399ec3c66df3d19217059f62903be3c74eadb9c15d5.json b/backend/.sqlx/query-87afd5289e54b6accda8a399ec3c66df3d19217059f62903be3c74eadb9c15d5.json new file mode 100644 index 0000000000..8cdb2d5907 --- /dev/null +++ b/backend/.sqlx/query-87afd5289e54b6accda8a399ec3c66df3d19217059f62903be3c74eadb9c15d5.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO draft (workspace_id, path, typ, value, created_at, email)\n SELECT $2, path, typ,\n CASE WHEN typ IN ('script', 'flow')\n THEN to_json(to_jsonb(value) - 'on_behalf_of')\n ELSE value END,\n created_at, email\n FROM draft\n WHERE workspace_id = $1 AND (email = $3 OR email IS NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "87afd5289e54b6accda8a399ec3c66df3d19217059f62903be3c74eadb9c15d5" +} diff --git a/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json index ffce4491e3..f26bca1232 100644 --- a/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json +++ b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json @@ -44,7 +44,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-8901dc69384685a7e05f39e1fae65471decea3d46ee35582c3b7c1ac333d8dd5.json b/backend/.sqlx/query-8901dc69384685a7e05f39e1fae65471decea3d46ee35582c3b7c1ac333d8dd5.json new file mode 100644 index 0000000000..e2b87fd182 --- /dev/null +++ b/backend/.sqlx/query-8901dc69384685a7e05f39e1fae65471decea3d46ee35582c3b7c1ac333d8dd5.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(\n SELECT 1 FROM script WHERE on_behalf_of = $1\n UNION ALL SELECT 1 FROM flow WHERE on_behalf_of = $1\n UNION ALL SELECT 1 FROM app WHERE policy->>'on_behalf_of' = $1\n UNION ALL SELECT 1 FROM schedule WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM http_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM websocket_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM postgres_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM mqtt_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM kafka_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM nats_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM sqs_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM gcp_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM email_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM amqp_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM azure_trigger WHERE permissioned_as = $1\n UNION ALL SELECT 1 FROM folder\n WHERE default_permissioned_as @> jsonb_build_array(\n jsonb_build_object('permissioned_as', $1::text)))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "8901dc69384685a7e05f39e1fae65471decea3d46ee35582c3b7c1ac333d8dd5" +} diff --git a/backend/.sqlx/query-8a40b3e7b9c59f1a0b50d3fe6b320748a912eff01dda22df9f489b6703435166.json b/backend/.sqlx/query-8a40b3e7b9c59f1a0b50d3fe6b320748a912eff01dda22df9f489b6703435166.json new file mode 100644 index 0000000000..0f743221dc --- /dev/null +++ b/backend/.sqlx/query-8a40b3e7b9c59f1a0b50d3fe6b320748a912eff01dda22df9f489b6703435166.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, hash, path, summary, description, content, created_by,\n language, lock)\n VALUES ($1, $2, $3, '', '', 'profile: {}', 'test-user', 'dbt', '')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Int8", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "8a40b3e7b9c59f1a0b50d3fe6b320748a912eff01dda22df9f489b6703435166" +} diff --git a/backend/.sqlx/query-8acc8c47123af84d2308042dcd34d04b941e928c7621e66d3bdc510c335f8c20.json b/backend/.sqlx/query-8acc8c47123af84d2308042dcd34d04b941e928c7621e66d3bdc510c335f8c20.json index 6f7468033a..44e6cc5c81 100644 --- a/backend/.sqlx/query-8acc8c47123af84d2308042dcd34d04b941e928c7621e66d3bdc510c335f8c20.json +++ b/backend/.sqlx/query-8acc8c47123af84d2308042dcd34d04b941e928c7621e66d3bdc510c335f8c20.json @@ -42,7 +42,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-8bad2a916f9afe1c037a01ffe69e136b562ace2d8265c9d26bcdf866f0f2c280.json b/backend/.sqlx/query-8bad2a916f9afe1c037a01ffe69e136b562ace2d8265c9d26bcdf866f0f2c280.json new file mode 100644 index 0000000000..a087c3619f --- /dev/null +++ b/backend/.sqlx/query-8bad2a916f9afe1c037a01ffe69e136b562ace2d8265c9d26bcdf866f0f2c280.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE dbt_run_progress SET workspace_id = $1\n WHERE workspace_id = $2 AND job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "8bad2a916f9afe1c037a01ffe69e136b562ace2d8265c9d26bcdf866f0f2c280" +} diff --git a/backend/.sqlx/query-8c33cbe3dc0a7539835fab61f981f5117da38c3410ad6e34ad67f5441deaf89e.json b/backend/.sqlx/query-8c33cbe3dc0a7539835fab61f981f5117da38c3410ad6e34ad67f5441deaf89e.json new file mode 100644 index 0000000000..ab72d4040a --- /dev/null +++ b/backend/.sqlx/query-8c33cbe3dc0a7539835fab61f981f5117da38c3410ad6e34ad67f5441deaf89e.json @@ -0,0 +1,71 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n workspace.id AS \"id!\",\n workspace.name AS \"name!\",\n workspace.owner AS \"owner!\",\n workspace.deleted AS \"deleted!\",\n workspace.premium AS \"premium!\",\n workspace_settings.color AS \"color\",\n workspace.parent_workspace_id AS \"parent_workspace_id\",\n workspace.is_dev_workspace AS \"is_dev_workspace!\",\n workspace.dev_workspace_label AS \"dev_workspace_label\"\n FROM workspace\n LEFT JOIN workspace_settings ON workspace.id = workspace_settings.workspace_id\n LIMIT $1 OFFSET $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "name!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "owner!", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "deleted!", + "type_info": "Bool" + }, + { + "ordinal": 4, + "name": "premium!", + "type_info": "Bool" + }, + { + "ordinal": 5, + "name": "color", + "type_info": "Varchar" + }, + { + "ordinal": 6, + "name": "parent_workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 7, + "name": "is_dev_workspace!", + "type_info": "Bool" + }, + { + "ordinal": 8, + "name": "dev_workspace_label", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + true, + true, + false, + true + ] + }, + "hash": "8c33cbe3dc0a7539835fab61f981f5117da38c3410ad6e34ad67f5441deaf89e" +} diff --git a/backend/.sqlx/query-8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b.json b/backend/.sqlx/query-8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b.json index 24d70efae1..09fc3c9ad1 100644 --- a/backend/.sqlx/query-8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b.json +++ b/backend/.sqlx/query-8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b.json @@ -36,7 +36,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-8d444380516faf56369cdbff645904e20477aa597e30891926af42eee56ab081.json b/backend/.sqlx/query-8d444380516faf56369cdbff645904e20477aa597e30891926af42eee56ab081.json index d7f59025e7..fbb3d579de 100644 --- a/backend/.sqlx/query-8d444380516faf56369cdbff645904e20477aa597e30891926af42eee56ab081.json +++ b/backend/.sqlx/query-8d444380516faf56369cdbff645904e20477aa597e30891926af42eee56ab081.json @@ -154,7 +154,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-8e4397a299c29687bec108899bd520d67718b2c60e717d69439ea13da272c81e.json b/backend/.sqlx/query-8e4397a299c29687bec108899bd520d67718b2c60e717d69439ea13da272c81e.json new file mode 100644 index 0000000000..04d2fe0a77 --- /dev/null +++ b/backend/.sqlx/query-8e4397a299c29687bec108899bd520d67718b2c60e717d69439ea13da272c81e.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) FROM dbt_graph_snapshot WHERE workspace_id = $1 AND script_hash = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "8e4397a299c29687bec108899bd520d67718b2c60e717d69439ea13da272c81e" +} diff --git a/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json index eac4547361..406917a99f 100644 --- a/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json +++ b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json @@ -16,7 +16,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-90489615820818332e40a46a3d720977b23f696e74735de1a56da58713e64b50.json b/backend/.sqlx/query-90489615820818332e40a46a3d720977b23f696e74735de1a56da58713e64b50.json new file mode 100644 index 0000000000..ebc76b2fb4 --- /dev/null +++ b/backend/.sqlx/query-90489615820818332e40a46a3d720977b23f696e74735de1a56da58713e64b50.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE script SET path = $1 WHERE workspace_id = 'test-workspace' AND path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "90489615820818332e40a46a3d720977b23f696e74735de1a56da58713e64b50" +} diff --git a/backend/.sqlx/query-916c6305639df0ecefd254901138b0663458209676179e8f64c3eb5c7d0262f8.json b/backend/.sqlx/query-916c6305639df0ecefd254901138b0663458209676179e8f64c3eb5c7d0262f8.json new file mode 100644 index 0000000000..8671ee1e77 --- /dev/null +++ b/backend/.sqlx/query-916c6305639df0ecefd254901138b0663458209676179e8f64c3eb5c7d0262f8.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, source_last_event_kind, source_last_event_origin)\n SELECT $1, unnest($2::varchar[]), $3, $4, 0, 1, NULL, $5, $6\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n behind = workspace_diff.behind + 1,\n has_changes = NULL,\n source_last_event_kind = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.source_last_event_origin IS NOT NULL) THEN workspace_diff.source_last_event_kind ELSE $5 END,\n source_last_event_origin = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.source_last_event_origin IS NOT NULL) THEN workspace_diff.source_last_event_origin ELSE $6 END", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "VarcharArray", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "916c6305639df0ecefd254901138b0663458209676179e8f64c3eb5c7d0262f8" +} diff --git a/backend/.sqlx/query-925359a7b0fe8f4ed17f072f7299c7d503eac045a6d5d13ac3c953a3fbdac0ff.json b/backend/.sqlx/query-925359a7b0fe8f4ed17f072f7299c7d503eac045a6d5d13ac3c953a3fbdac0ff.json new file mode 100644 index 0000000000..59472d78dd --- /dev/null +++ b/backend/.sqlx/query-925359a7b0fe8f4ed17f072f7299c7d503eac045a6d5d13ac3c953a3fbdac0ff.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT versions[array_upper(versions, 1)] FROM app\n WHERE path = $1 AND workspace_id = $2 FOR UPDATE", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "versions", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "925359a7b0fe8f4ed17f072f7299c7d503eac045a6d5d13ac3c953a3fbdac0ff" +} diff --git a/backend/.sqlx/query-92723ff15a596e97edc71d24a5b653deb9d0e44259b22f2c6c5f43579f687360.json b/backend/.sqlx/query-92723ff15a596e97edc71d24a5b653deb9d0e44259b22f2c6c5f43579f687360.json new file mode 100644 index 0000000000..860da63f0b --- /dev/null +++ b/backend/.sqlx/query-92723ff15a596e97edc71d24a5b653deb9d0e44259b22f2c6c5f43579f687360.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (\n workspace_id, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of, on_behalf_of_email, assets, modules\n )\n SELECT\n $1, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n -- Same three forms and the same prefix-first rule as permissioned_as_exists,\n -- superadmin fallback included: one acting outside their workspaces has no usr\n -- row but still authenticates.\n CASE WHEN on_behalf_of LIKE 'u/%' THEN\n (SELECT on_behalf_of WHERE EXISTS (\n SELECT 1 FROM usr u WHERE u.workspace_id = $1::varchar\n AND u.username = substring(on_behalf_of from 3)\n UNION ALL\n SELECT 1 FROM password p WHERE p.super_admin\n AND (p.username = substring(on_behalf_of from 3)\n OR p.email = substring(on_behalf_of from 3))))\n WHEN on_behalf_of LIKE 'g/%' THEN\n (SELECT on_behalf_of WHERE EXISTS (\n SELECT 1 FROM group_ g WHERE g.workspace_id = $1::varchar\n AND g.name = substring(on_behalf_of from 3)))\n ELSE\n (SELECT on_behalf_of WHERE EXISTS (\n SELECT 1 FROM usr u WHERE u.workspace_id = $1::varchar\n AND u.username = on_behalf_of\n UNION ALL\n SELECT 1 FROM password p WHERE p.email = on_behalf_of\n AND p.super_admin))\n END, on_behalf_of_email, assets, modules\n FROM script\n WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "92723ff15a596e97edc71d24a5b653deb9d0e44259b22f2c6c5f43579f687360" +} diff --git a/backend/.sqlx/query-9309262d8a37827e5ab0b3534d1595cb6d94118432fd1770f1f89f7cb52a4f12.json b/backend/.sqlx/query-9309262d8a37827e5ab0b3534d1595cb6d94118432fd1770f1f89f7cb52a4f12.json new file mode 100644 index 0000000000..da77dc1de5 --- /dev/null +++ b/backend/.sqlx/query-9309262d8a37827e5ab0b3534d1595cb6d94118432fd1770f1f89f7cb52a4f12.json @@ -0,0 +1,134 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH live AS (\n -- The graph is stored per deployed VERSION, so this endpoint — which\n -- describes the project as it is now — takes the newest live one per\n -- path. Resolved once here rather than per row: a correlated lookup\n -- on every node is what makes these queries fall over.\n SELECT * FROM (\n SELECT DISTINCT ON (s.path) s.path, s.hash\n FROM script s\n WHERE $5::text IS NULL AND s.workspace_id = $1 AND s.language = 'dbt'\n AND ($3::bigint IS NULL OR s.hash = $3)\n -- A pinned version may be archived by now; that is precisely\n -- the case a historical run needs, so the liveness filter\n -- applies only when picking the current one.\n AND ($3::bigint IS NOT NULL OR (s.deleted = false AND s.archived = false))\n ORDER BY s.path, s.created_at DESC\n ) cur\n UNION ALL\n -- A pinned run names its own version, so `script` is not consulted:\n -- under RLS it would answer for the CALLER's grants on the project,\n -- emptying the graph for a share-link viewer who is entitled to the\n -- run but not the script. A NULL hash here is a job that names no\n -- version at all — an editor buffer parse — and matches only the\n -- version-less rows that parse stored.\n SELECT $5::text, $3::bigint WHERE $5::text IS NOT NULL\n ),\n -- The run's own snapshot when it left one, the version's graph\n -- otherwise. A static descriptor never snapshots, so all of its runs\n -- fall through to the same rows. Existence comes from the marker, not\n -- from a node row: a dynamic run that disabled every model has a\n -- snapshot whose graph is legitimately empty.\n chosen AS (\n -- No visibility check on the job here: reaching this with a job at\n -- all means the caller passed `require_job_read_access` for it, and\n -- re-deciding it under plain RLS can only DISAGREE with that answer\n -- — silently, by falling back to the deployed graph rather than\n -- erroring. A share-link viewer is entitled to the run and would be\n -- shown a different run's model set. See `asset_graph_for`.\n SELECT CASE WHEN $4::uuid IS NOT NULL AND EXISTS (\n SELECT 1 FROM dbt_graph_snapshot g\n WHERE g.workspace_id = $1 AND g.job_id = $4)\n THEN $4::uuid\n ELSE '00000000-0000-0000-0000-000000000000'::uuid END AS job_id\n ),\n scoped AS (\n SELECT n.script_path, n.unique_id FROM dbt_node n\n -- `=` still, with the NULL-to-NULL case spelled out and gated on\n -- the pin: a version-less row's hash is NULL on both sides, which\n -- `=` never matches, but `IS NOT DISTINCT FROM` would cost the\n -- equality its index bound on the UNPINNED workspace graph — the\n -- hot path. Unpinned, `$5` is NULL and the second arm folds away.\n JOIN live l ON l.path = n.script_path\n AND (n.script_hash = l.hash\n OR ($5::text IS NOT NULL AND l.hash IS NULL\n AND n.script_hash IS NULL))\n JOIN chosen ch ON ch.job_id = n.job_id\n WHERE n.workspace_id = $1 AND n.asset_path IS NOT NULL\n -- Unpinned, the scope is the relations in view: `asset` says\n -- which of them this folder touches. Pinned, that table is the\n -- WRONG scope — it holds one row set per path, describing the\n -- current deploy, so a model this version had and the current one\n -- dropped would be filtered out of its own run's graph. The\n -- pinned graph's nodes are the scope. Keyed on the pin rather\n -- than on the hash: an editor parse pins without naming one, and\n -- its models are precisely the ones `asset` does not know yet.\n AND ($3::bigint IS NOT NULL OR $5::text IS NOT NULL\n OR n.asset_path IN (\n SELECT path FROM asset\n WHERE workspace_id = $1 AND kind = 'dbt'\n AND ($2::text IS NULL OR usage_path LIKE $2)))\n )\n SELECT n.script_path AS \"script_path!\", n.unique_id AS \"unique_id!\",\n n.resource_type AS \"resource_type!\", n.name AS \"name!\", n.asset_path,\n n.materialized, n.materialize_strategy, n.tags AS \"tags!\", n.description,\n n.test_kind, n.test_column, n.test_args, n.severity, n.attached_node,\n n.columns, n.freshness,\n n.raw_code, n.original_file_path,\n -- Whether the caller may read the project this row describes.\n -- The query deliberately reaches outside the requested folder\n -- so an in-scope consumer can explain the relation it reads,\n -- and `dbt_node` carries no RLS of its own; the relation's\n -- SHAPE is fine to answer that way, everything the project's\n -- author WROTE is not. Applied in Rust, over one predicate, so\n -- the fields it covers are named in one place. This runs in the\n -- authed transaction, so `script`'s RLS answers it. Matched on\n -- the HASH as well: `extra_perms` is per row, so a path\n -- recreated with narrower ones leaves the archived version\n -- readable, and a path-only probe would answer for THAT grant\n -- while returning this version's source.\n --\n -- A version-less row has no `script` row to ask, and needs\n -- none: it exists only because this caller's own parse job\n -- created it from a buffer they wrote, and the unpinned `live`\n -- branch — fed from `script` — can never join to one.\n (n.script_hash IS NULL OR EXISTS (\n SELECT 1 FROM script sc\n WHERE sc.workspace_id = n.workspace_id AND sc.path = n.script_path\n AND sc.hash = n.script_hash\n )) AS \"script_visible!\"\n FROM dbt_node n\n JOIN live l ON l.path = n.script_path\n AND (n.script_hash = l.hash\n OR ($5::text IS NOT NULL AND l.hash IS NULL\n AND n.script_hash IS NULL))\n -- Every join onto `dbt_node` needs this, not just the scoping CTE:\n -- `job_id` is part of the key, so without it each model comes back\n -- once per retained snapshot plus once for the version's graph.\n JOIN chosen ch ON ch.job_id = n.job_id\n WHERE n.workspace_id = $1\n -- Joined on BOTH columns: a dbt `unique_id` is project-local, so\n -- two projects with the same model name would otherwise pull each\n -- other's rows.\n AND (EXISTS (SELECT 1 FROM scoped s\n WHERE s.script_path = n.script_path\n AND s.unique_id = n.unique_id)\n OR EXISTS (SELECT 1 FROM scoped s\n WHERE s.script_path = n.script_path\n AND s.unique_id = n.attached_node))\n ORDER BY n.script_path, n.unique_id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "script_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "unique_id!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "resource_type!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "name!", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "asset_path", + "type_info": "Text" + }, + { + "ordinal": 5, + "name": "materialized", + "type_info": "Text" + }, + { + "ordinal": 6, + "name": "materialize_strategy", + "type_info": "Text" + }, + { + "ordinal": 7, + "name": "tags!", + "type_info": "TextArray" + }, + { + "ordinal": 8, + "name": "description", + "type_info": "Text" + }, + { + "ordinal": 9, + "name": "test_kind", + "type_info": "Text" + }, + { + "ordinal": 10, + "name": "test_column", + "type_info": "Text" + }, + { + "ordinal": 11, + "name": "test_args", + "type_info": "Jsonb" + }, + { + "ordinal": 12, + "name": "severity", + "type_info": "Text" + }, + { + "ordinal": 13, + "name": "attached_node", + "type_info": "Text" + }, + { + "ordinal": 14, + "name": "columns", + "type_info": "Jsonb" + }, + { + "ordinal": 15, + "name": "freshness", + "type_info": "Jsonb" + }, + { + "ordinal": 16, + "name": "raw_code", + "type_info": "Text" + }, + { + "ordinal": 17, + "name": "original_file_path", + "type_info": "Text" + }, + { + "ordinal": 18, + "name": "script_visible!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Uuid", + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + true, + true, + true, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + null + ] + }, + "hash": "9309262d8a37827e5ab0b3534d1595cb6d94118432fd1770f1f89f7cb52a4f12" +} diff --git a/backend/.sqlx/query-932f7d0df4ab3e200b9eaea424f51756433c03af6e7810c2a5f9421515df8aa3.json b/backend/.sqlx/query-932f7d0df4ab3e200b9eaea424f51756433c03af6e7810c2a5f9421515df8aa3.json new file mode 100644 index 0000000000..e2e3f374a6 --- /dev/null +++ b/backend/.sqlx/query-932f7d0df4ab3e200b9eaea424f51756433c03af6e7810c2a5f9421515df8aa3.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT created_by, on_behalf_of_email FROM script\n WHERE path = $1 AND workspace_id = $2 AND archived = false AND deleted = false\n ORDER BY created_at DESC LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "on_behalf_of_email", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + true + ] + }, + "hash": "932f7d0df4ab3e200b9eaea424f51756433c03af6e7810c2a5f9421515df8aa3" +} diff --git a/backend/.sqlx/query-94c0aae349172b9295b81b9e61e2f8f0ca36920ab66e8922815157a0c147bc47.json b/backend/.sqlx/query-94c0aae349172b9295b81b9e61e2f8f0ca36920ab66e8922815157a0c147bc47.json new file mode 100644 index 0000000000..823a65cda3 --- /dev/null +++ b/backend/.sqlx/query-94c0aae349172b9295b81b9e61e2f8f0ca36920ab66e8922815157a0c147bc47.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, materialized, materialize_strategy, unique_key,\n tags, description, test_kind, test_column, test_args, severity, attached_node,\n columns, freshness, raw_code, original_file_path, ingested_at)\n SELECT $2, script_path, script_hash, job_id, unique_id,\n resource_type, name, asset_path, materialized, materialize_strategy, unique_key,\n tags, description, test_kind, test_column, test_args, severity, attached_node,\n columns, freshness, raw_code, original_file_path, ingested_at\n FROM dbt_node\n WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "94c0aae349172b9295b81b9e61e2f8f0ca36920ab66e8922815157a0c147bc47" +} diff --git a/backend/.sqlx/query-95c184a8f7d9396ed124bc1e2c3fddaeed002fe148f5d0434b41fda5d1203f27.json b/backend/.sqlx/query-95c184a8f7d9396ed124bc1e2c3fddaeed002fe148f5d0434b41fda5d1203f27.json new file mode 100644 index 0000000000..0fc9445931 --- /dev/null +++ b/backend/.sqlx/query-95c184a8f7d9396ed124bc1e2c3fddaeed002fe148f5d0434b41fda5d1203f27.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of, on_behalf_of_email)\n VALUES\n ('test-workspace', 'u/test-user/obo_member', 91001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user', 'test@windmill.dev'),\n ('test-workspace', 'u/test-user/obo_stranger', 91002, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2', 'test2@windmill.dev'),\n ('test-workspace', 'u/test-user/obo_group', 91003, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'g/all', 'group-all@windmill.dev'),\n ('test-workspace', 'u/test-user/obo_superadmin', 91004, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/ext-sa', 'sa@windmill.dev')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "95c184a8f7d9396ed124bc1e2c3fddaeed002fe148f5d0434b41fda5d1203f27" +} diff --git a/backend/.sqlx/query-9621928b5736c56928368d17bd4111faeea6bfb54debbf8ee686db19364084fe.json b/backend/.sqlx/query-9621928b5736c56928368d17bd4111faeea6bfb54debbf8ee686db19364084fe.json new file mode 100644 index 0000000000..6c4bced8ce --- /dev/null +++ b/backend/.sqlx/query-9621928b5736c56928368d17bd4111faeea6bfb54debbf8ee686db19364084fe.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(\n SELECT 1 FROM usr WHERE workspace_id = $1 AND username = $2\n UNION ALL\n SELECT 1 FROM password WHERE email = $2 AND super_admin\n )", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "9621928b5736c56928368d17bd4111faeea6bfb54debbf8ee686db19364084fe" +} diff --git a/backend/.sqlx/query-96aa1db2859c80d37ffecc2885442ae6d6d4b6731c3f62e7b85ae7a4f1b666a7.json b/backend/.sqlx/query-96aa1db2859c80d37ffecc2885442ae6d6d4b6731c3f62e7b85ae7a4f1b666a7.json new file mode 100644 index 0000000000..733478e0e9 --- /dev/null +++ b/backend/.sqlx/query-96aa1db2859c80d37ffecc2885442ae6d6d4b6731c3f62e7b85ae7a4f1b666a7.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO script\n (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n FROM script WHERE hash = $2 AND workspace_id = $3;\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Text" + ] + }, + "nullable": [] + }, + "hash": "96aa1db2859c80d37ffecc2885442ae6d6d4b6731c3f62e7b85ae7a4f1b666a7" +} diff --git a/backend/.sqlx/query-96ba9c55e1f3b6ec20ce4c21a72b12a4c8aa3dda05cceb694234107eb324cbf9.json b/backend/.sqlx/query-96ba9c55e1f3b6ec20ce4c21a72b12a4c8aa3dda05cceb694234107eb324cbf9.json new file mode 100644 index 0000000000..5061c42c94 --- /dev/null +++ b/backend/.sqlx/query-96ba9c55e1f3b6ec20ce4c21a72b12a4c8aa3dda05cceb694234107eb324cbf9.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM materialized_partition\n WHERE workspace_id = $1 AND job_id = $2 AND status = 'running'\n AND asset_path = ANY($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Uuid", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "96ba9c55e1f3b6ec20ce4c21a72b12a4c8aa3dda05cceb694234107eb324cbf9" +} diff --git a/backend/.sqlx/query-972ee0dd58b98c1f0ff7242922e25b58b5ce504f2f4c4db952a796b3019a67d0.json b/backend/.sqlx/query-972ee0dd58b98c1f0ff7242922e25b58b5ce504f2f4c4db952a796b3019a67d0.json new file mode 100644 index 0000000000..ec34b5f46a --- /dev/null +++ b/backend/.sqlx/query-972ee0dd58b98c1f0ff7242922e25b58b5ce504f2f4c4db952a796b3019a67d0.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_run_state WHERE workspace_id = $1 AND script_path = $2\n AND NOT EXISTS (SELECT 1 FROM script\n WHERE workspace_id = $1 AND path = $2\n AND deleted = false AND archived = false)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "972ee0dd58b98c1f0ff7242922e25b58b5ce504f2f4c4db952a796b3019a67d0" +} diff --git a/backend/.sqlx/query-9809fad5552d306f8cf04dfd9059be14ec5518c6ba8572362f5a849657237294.json b/backend/.sqlx/query-9809fad5552d306f8cf04dfd9059be14ec5518c6ba8572362f5a849657237294.json new file mode 100644 index 0000000000..42b8a2b09c --- /dev/null +++ b/backend/.sqlx/query-9809fad5552d306f8cf04dfd9059be14ec5518c6ba8572362f5a849657237294.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id,\n resource_type, name, tags, test_kind, test_column, test_args,\n attached_node)\n VALUES ($1, $2, $3, $4, 'test.p.accepted_values_orders_status', 'test',\n 'accepted_values_orders_status', '{}', 'accepted_values', 'status',\n '{\"values\": [\"gold\", \"silver\"]}'::jsonb, 'model.p.orders')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "9809fad5552d306f8cf04dfd9059be14ec5518c6ba8572362f5a849657237294" +} diff --git a/backend/.sqlx/query-98206f64a4393791f679a609a5ee1a6bb6dae0bc091e2cbb2d583dd6a5b44941.json b/backend/.sqlx/query-98206f64a4393791f679a609a5ee1a6bb6dae0bc091e2cbb2d583dd6a5b44941.json new file mode 100644 index 0000000000..5d1b35b17d --- /dev/null +++ b/backend/.sqlx/query-98206f64a4393791f679a609a5ee1a6bb6dae0bc091e2cbb2d583dd6a5b44941.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT files FROM workspace_shared_ui WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "files", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "98206f64a4393791f679a609a5ee1a6bb6dae0bc091e2cbb2d583dd6a5b44941" +} diff --git a/backend/.sqlx/query-98500dd427fdc64e711090387e08543ae8dfa457c6db2d74ebdc0cd748d72ac3.json b/backend/.sqlx/query-98500dd427fdc64e711090387e08543ae8dfa457c6db2d74ebdc0cd748d72ac3.json new file mode 100644 index 0000000000..46f6c9765e --- /dev/null +++ b/backend/.sqlx/query-98500dd427fdc64e711090387e08543ae8dfa457c6db2d74ebdc0cd748d72ac3.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE draft SET value = to_json(jsonb_set(to_jsonb(value), ARRAY['on_behalf_of'], to_jsonb($1::text))) WHERE typ IN ('script', 'flow') AND value->>'on_behalf_of' = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "98500dd427fdc64e711090387e08543ae8dfa457c6db2d74ebdc0cd748d72ac3" +} diff --git a/backend/.sqlx/query-9ac8b60c53631de2c3dab765ddc23cc99f9154e4239d228c0ef8f3726b0d19fc.json b/backend/.sqlx/query-9ac8b60c53631de2c3dab765ddc23cc99f9154e4239d228c0ef8f3726b0d19fc.json new file mode 100644 index 0000000000..777aa0feb2 --- /dev/null +++ b/backend/.sqlx/query-9ac8b60c53631de2c3dab765ddc23cc99f9154e4239d228c0ef8f3726b0d19fc.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT g.job_id, g.ingested_at FROM dbt_graph_snapshot g\n WHERE g.workspace_id = $1 AND g.script_path = $2\n AND g.script_hash IS NOT DISTINCT FROM $3\n AND g.job_id = CASE WHEN EXISTS (\n SELECT 1 FROM dbt_graph_snapshot s\n WHERE s.workspace_id = $1 AND s.job_id = $4)\n THEN $4::uuid\n ELSE '00000000-0000-0000-0000-000000000000'::uuid END\n LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "job_id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "ingested_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Uuid" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "9ac8b60c53631de2c3dab765ddc23cc99f9154e4239d228c0ef8f3726b0d19fc" +} diff --git a/backend/.sqlx/query-9be99f3104f595b0c7e73b67b120c2789ded855c69d5cd6e8b07c88710252ac5.json b/backend/.sqlx/query-9be99f3104f595b0c7e73b67b120c2789ded855c69d5cd6e8b07c88710252ac5.json new file mode 100644 index 0000000000..fe90532de7 --- /dev/null +++ b/backend/.sqlx/query-9be99f3104f595b0c7e73b67b120c2789ded855c69d5cd6e8b07c88710252ac5.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO draft(workspace_id, path, typ, value, email)\n VALUES ('test-workspace', 'u/test-user/dg', 'script', '{\"on_behalf_of\": \"g/ops\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::json, 'test@windmill.dev'),\n ('test-workspace', 'u/test-user/du', 'script', '{\"on_behalf_of\": \"u/test-user-2\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::json, 'test@windmill.dev')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "9be99f3104f595b0c7e73b67b120c2789ded855c69d5cd6e8b07c88710252ac5" +} diff --git a/backend/.sqlx/query-9cd98de055f34b86e98f6f1c060837644c813f45572896b42a55b4996cde1be8.json b/backend/.sqlx/query-9cd98de055f34b86e98f6f1c060837644c813f45572896b42a55b4996cde1be8.json new file mode 100644 index 0000000000..6aa872bfd4 --- /dev/null +++ b/backend/.sqlx/query-9cd98de055f34b86e98f6f1c060837644c813f45572896b42a55b4996cde1be8.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE datatable_migrations SET datatable = $3 WHERE workspace_id = $1 AND datatable = $2 RETURNING timestamp, name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "timestamp", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Varchar" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "9cd98de055f34b86e98f6f1c060837644c813f45572896b42a55b4996cde1be8" +} diff --git a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json b/backend/.sqlx/query-9d331b63c901071a7e82c0a4a6859c77a37ef784fdac78ca1e7a971e141e780f.json similarity index 88% rename from backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json rename to backend/.sqlx/query-9d331b63c901071a7e82c0a4a6859c77a37ef784fdac78ca1e7a971e141e780f.json index 3bbd0f043e..b73ee34566 100644 --- a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json +++ b/backend/.sqlx/query-9d331b63c901071a7e82c0a4a6859c77a37ef784fdac78ca1e7a971e141e780f.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT\n id,\n q.runnable_settings_handle,\n q.workspace_id,\n j.runnable_id as \"runnable_id: ScriptHash\",\n scheduled_for,\n parent_job,\n flow_innermost_root_job,\n runnable_path,\n kind as \"kind: JobKind\",\n started_at,\n permissioned_as,\n created_by,\n script_lang as \"script_lang: ScriptLang\",\n permissioned_as_email,\n flow_step_id,\n trigger_kind as \"trigger_kind: JobTriggerKind\",\n trigger,\n q.priority,\n concurrent_limit,\n q.tag,\n cache_ttl,\n cache_ignore_s3_path,\n r.ping as last_ping,\n worker,\n memory_peak,\n running\n FROM v2_job_queue q\n JOIN v2_job j USING (id)\n LEFT JOIN v2_job_runtime r USING (id)\n LEFT JOIN v2_job_status s USING (id)\n WHERE j.id = $1", + "query": "SELECT\n id,\n q.runnable_settings_handle,\n q.workspace_id,\n j.runnable_id as \"runnable_id: ScriptHash\",\n scheduled_for,\n parent_job,\n flow_innermost_root_job,\n runnable_path,\n kind as \"kind: JobKind\",\n started_at,\n permissioned_as,\n created_by,\n script_lang as \"script_lang: ScriptLang\",\n permissioned_as_email,\n flow_step_id,\n trigger_kind as \"trigger_kind: TriggerKindLabel\",\n trigger,\n q.priority,\n concurrent_limit,\n q.tag,\n cache_ttl,\n cache_ignore_s3_path,\n r.ping as last_ping,\n worker,\n memory_peak,\n running\n FROM v2_job_queue q\n JOIN v2_job j USING (id)\n LEFT JOIN v2_job_runtime r USING (id)\n LEFT JOIN v2_job_status s USING (id)\n WHERE j.id = $1", "describe": { "columns": [ { @@ -124,7 +124,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } @@ -142,7 +143,7 @@ }, { "ordinal": 15, - "name": "trigger_kind: JobTriggerKind", + "name": "trigger_kind: TriggerKindLabel", "type_info": { "Custom": { "name": "job_trigger_kind", @@ -259,5 +260,5 @@ false ] }, - "hash": "b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc" + "hash": "9d331b63c901071a7e82c0a4a6859c77a37ef784fdac78ca1e7a971e141e780f" } diff --git a/backend/.sqlx/query-9ecd7eee162905fc55ae4f641c1a2095c02f020a0c8ee47e0b4dbc357ced6fd2.json b/backend/.sqlx/query-9ecd7eee162905fc55ae4f641c1a2095c02f020a0c8ee47e0b4dbc357ced6fd2.json new file mode 100644 index 0000000000..5555235da8 --- /dev/null +++ b/backend/.sqlx/query-9ecd7eee162905fc55ae4f641c1a2095c02f020a0c8ee47e0b4dbc357ced6fd2.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM resource_version rv\n USING (\n SELECT id FROM (\n SELECT id, row_number() OVER (\n PARTITION BY workspace_id, path ORDER BY id DESC\n ) AS rn\n FROM resource_version\n ) ranked WHERE rn > $1\n ) over_cap\n WHERE rv.id = over_cap.id\n RETURNING rv.id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Int8" + ] + }, + "nullable": [ + false + ] + }, + "hash": "9ecd7eee162905fc55ae4f641c1a2095c02f020a0c8ee47e0b4dbc357ced6fd2" +} diff --git a/backend/.sqlx/query-9f4a95c35fe740d7f47db99335fe8a43b1df30076d6b487bdb6f762892bbec76.json b/backend/.sqlx/query-9f4a95c35fe740d7f47db99335fe8a43b1df30076d6b487bdb6f762892bbec76.json new file mode 100644 index 0000000000..721eeee984 --- /dev/null +++ b/backend/.sqlx/query-9f4a95c35fe740d7f47db99335fe8a43b1df30076d6b487bdb6f762892bbec76.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE script SET on_behalf_of_email = $1 WHERE on_behalf_of_email = $2 AND (on_behalf_of IS NULL OR on_behalf_of NOT LIKE 'g/%')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "9f4a95c35fe740d7f47db99335fe8a43b1df30076d6b487bdb6f762892bbec76" +} diff --git a/backend/.sqlx/query-a0ef88bd0ec4823e4c8a226a790cf572fbf901326e80806bb6dcaa953f7b7b60.json b/backend/.sqlx/query-a0ef88bd0ec4823e4c8a226a790cf572fbf901326e80806bb6dcaa953f7b7b60.json new file mode 100644 index 0000000000..6097dab23e --- /dev/null +++ b/backend/.sqlx/query-a0ef88bd0ec4823e4c8a226a790cf572fbf901326e80806bb6dcaa953f7b7b60.json @@ -0,0 +1,53 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT service_config, webhook_token_hash, script_path, is_flow\n FROM native_trigger\n WHERE workspace_id = $1\n AND service_name = $2\n AND external_id = $3\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "service_config", + "type_info": "Jsonb" + }, + { + "ordinal": 1, + "name": "webhook_token_hash", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "script_path", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "is_flow", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "native_trigger_service", + "kind": { + "Enum": [ + "nextcloud", + "google", + "github" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + true, + false, + false, + false + ] + }, + "hash": "a0ef88bd0ec4823e4c8a226a790cf572fbf901326e80806bb6dcaa953f7b7b60" +} diff --git a/backend/.sqlx/query-a2b8356365a2e5e0efa5f851b2aa2d74a2f14a57ce8292f44b8fb482c7a07e2e.json b/backend/.sqlx/query-a2b8356365a2e5e0efa5f851b2aa2d74a2f14a57ce8292f44b8fb482c7a07e2e.json new file mode 100644 index 0000000000..b69ccb6cde --- /dev/null +++ b/backend/.sqlx/query-a2b8356365a2e5e0efa5f851b2aa2d74a2f14a57ce8292f44b8fb482c7a07e2e.json @@ -0,0 +1,78 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n flow_version.id AS version,\n flow_version.value->>'early_return' as early_return,\n flow_version.value->>'preprocessor_module' IS NOT NULL as has_preprocessor,\n flow_version.value->>'failure_module' IS NOT NULL as has_failure_module,\n (flow_version.value->>'chat_input_enabled')::boolean as chat_input_enabled,\n flow.tag,\n flow.dedicated_worker,\n flow.on_behalf_of,\n flow.edited_by,\n flow.labels\n FROM\n flow_version\n INNER JOIN flow\n ON flow.path = flow_version.path AND\n flow.workspace_id = flow_version.workspace_id\n WHERE\n flow_version.workspace_id = $1 AND\n flow_version.path = $2 AND\n flow_version.id = $3\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "early_return", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "has_preprocessor", + "type_info": "Bool" + }, + { + "ordinal": 3, + "name": "has_failure_module", + "type_info": "Bool" + }, + { + "ordinal": 4, + "name": "chat_input_enabled", + "type_info": "Bool" + }, + { + "ordinal": 5, + "name": "tag", + "type_info": "Varchar" + }, + { + "ordinal": 6, + "name": "dedicated_worker", + "type_info": "Bool" + }, + { + "ordinal": 7, + "name": "on_behalf_of", + "type_info": "Varchar" + }, + { + "ordinal": 8, + "name": "edited_by", + "type_info": "Varchar" + }, + { + "ordinal": 9, + "name": "labels", + "type_info": "TextArray" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + false, + null, + null, + null, + null, + true, + true, + true, + false, + true + ] + }, + "hash": "a2b8356365a2e5e0efa5f851b2aa2d74a2f14a57ce8292f44b8fb482c7a07e2e" +} diff --git a/backend/.sqlx/query-a4651f0ec5daaa0cbf09a71a7da9e16ea112e0a3855cc730d2fe99259c64eb41.json b/backend/.sqlx/query-a4651f0ec5daaa0cbf09a71a7da9e16ea112e0a3855cc730d2fe99259c64eb41.json new file mode 100644 index 0000000000..de91a548ad --- /dev/null +++ b/backend/.sqlx/query-a4651f0ec5daaa0cbf09a71a7da9e16ea112e0a3855cc730d2fe99259c64eb41.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT envs FROM script WHERE workspace_id = $1 AND hash = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "envs", + "type_info": "VarcharArray" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [ + true + ] + }, + "hash": "a4651f0ec5daaa0cbf09a71a7da9e16ea112e0a3855cc730d2fe99259c64eb41" +} diff --git a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json index 4824bba526..2b9ac8283b 100644 --- a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json +++ b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json @@ -134,7 +134,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-a606d8fc5876ef1e40278120d841994b8ec76d2008bcf749fcdcc607368bcdc3.json b/backend/.sqlx/query-a606d8fc5876ef1e40278120d841994b8ec76d2008bcf749fcdcc607368bcdc3.json new file mode 100644 index 0000000000..998563293a --- /dev/null +++ b/backend/.sqlx/query-a606d8fc5876ef1e40278120d841994b8ec76d2008bcf749fcdcc607368bcdc3.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at)\n SELECT workspace_id, REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "a606d8fc5876ef1e40278120d841994b8ec76d2008bcf749fcdcc607368bcdc3" +} diff --git a/backend/.sqlx/query-a6decdebcd9750691f20f874b66a9c6f2ede08c33605c6a3c3dfc213a3eda76a.json b/backend/.sqlx/query-a6decdebcd9750691f20f874b66a9c6f2ede08c33605c6a3c3dfc213a3eda76a.json new file mode 100644 index 0000000000..0532a5d3a0 --- /dev/null +++ b/backend/.sqlx/query-a6decdebcd9750691f20f874b66a9c6f2ede08c33605c6a3c3dfc213a3eda76a.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE workspace_settings\n SET\n ai_config = source_ws.ai_config,\n large_file_storage = source_ws.large_file_storage,\n ducklake = source_ws.ducklake,\n dbt_warehouses = source_ws.dbt_warehouses,\n datatable = source_ws.datatable,\n git_app_installations = source_ws.git_app_installations\n FROM workspace_settings source_ws\n WHERE source_ws.workspace_id = $1\n AND workspace_settings.workspace_id = $2\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "a6decdebcd9750691f20f874b66a9c6f2ede08c33605c6a3c3dfc213a3eda76a" +} diff --git a/backend/.sqlx/query-a6f59e8751b79d23fcd2a8204fbc38002fc6d0cf7e4a83f21d31fb28bd98efb8.json b/backend/.sqlx/query-a6f59e8751b79d23fcd2a8204fbc38002fc6d0cf7e4a83f21d31fb28bd98efb8.json new file mode 100644 index 0000000000..e7a544525f --- /dev/null +++ b/backend/.sqlx/query-a6f59e8751b79d23fcd2a8204fbc38002fc6d0cf7e4a83f21d31fb28bd98efb8.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO app(workspace_id, path, summary, policy, versions)\n VALUES ('test-workspace', 'u/test-user/g', '', '{\"on_behalf_of\": \"g/ops\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::jsonb, '{}'),\n ('test-workspace', 'u/test-user/u', '', '{\"on_behalf_of\": \"u/test-user-2\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::jsonb, '{}')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "a6f59e8751b79d23fcd2a8204fbc38002fc6d0cf7e4a83f21d31fb28bd98efb8" +} diff --git a/backend/.sqlx/query-a8bcae108af1eda6efe3a4b8c6f8807bc464a81c0883e68f5a69b89b94b0b34b.json b/backend/.sqlx/query-a8bcae108af1eda6efe3a4b8c6f8807bc464a81c0883e68f5a69b89b94b0b34b.json index 0d1bb6cb41..0faa9d0e33 100644 --- a/backend/.sqlx/query-a8bcae108af1eda6efe3a4b8c6f8807bc464a81c0883e68f5a69b89b94b0b34b.json +++ b/backend/.sqlx/query-a8bcae108af1eda6efe3a4b8c6f8807bc464a81c0883e68f5a69b89b94b0b34b.json @@ -39,7 +39,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-a8eb013bbbb4d8cdc0d4e5e3f46e805e039e57ef9f7be07d2e738d6628e585d5.json b/backend/.sqlx/query-a8eb013bbbb4d8cdc0d4e5e3f46e805e039e57ef9f7be07d2e738d6628e585d5.json new file mode 100644 index 0000000000..4419bf7a6d --- /dev/null +++ b/backend/.sqlx/query-a8eb013bbbb4d8cdc0d4e5e3f46e805e039e57ef9f7be07d2e738d6628e585d5.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE draft SET value = to_json(jsonb_set(jsonb_set(to_jsonb(value), ARRAY['on_behalf_of'], to_jsonb($1::text)), ARRAY['on_behalf_of_email'], to_jsonb($4::text))) WHERE typ IN ('script', 'flow') AND value->>'on_behalf_of' = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "a8eb013bbbb4d8cdc0d4e5e3f46e805e039e57ef9f7be07d2e738d6628e585d5" +} diff --git a/backend/.sqlx/query-aa463baab01d9530ae97ace8b5a296ce9855bce50ea8c5918941564a696c029c.json b/backend/.sqlx/query-aa463baab01d9530ae97ace8b5a296ce9855bce50ea8c5918941564a696c029c.json new file mode 100644 index 0000000000..e417d1e948 --- /dev/null +++ b/backend/.sqlx/query-aa463baab01d9530ae97ace8b5a296ce9855bce50ea8c5918941564a696c029c.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms)\n VALUES ($1, 'private', 'private', '{}', '{}')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "aa463baab01d9530ae97ace8b5a296ce9855bce50ea8c5918941564a696c029c" +} diff --git a/backend/.sqlx/query-ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2.json b/backend/.sqlx/query-ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2.json index 04131f8f88..0bba0710e8 100644 --- a/backend/.sqlx/query-ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2.json +++ b/backend/.sqlx/query-ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2.json @@ -16,7 +16,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-ab46ac8e14a05a79340511ad081f9f540139e1f593ac6f80221378ff856954cb.json b/backend/.sqlx/query-ab46ac8e14a05a79340511ad081f9f540139e1f593ac6f80221378ff856954cb.json new file mode 100644 index 0000000000..6ed6813681 --- /dev/null +++ b/backend/.sqlx/query-ab46ac8e14a05a79340511ad081f9f540139e1f593ac6f80221378ff856954cb.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels)\n SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels\n FROM flow\n WHERE path = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "ab46ac8e14a05a79340511ad081f9f540139e1f593ac6f80221378ff856954cb" +} diff --git a/backend/.sqlx/query-ab906821072ddaa6303c5ea632150b5db1bf5a11d7a162130ebb826178c4ea08.json b/backend/.sqlx/query-ab906821072ddaa6303c5ea632150b5db1bf5a11d7a162130ebb826178c4ea08.json new file mode 100644 index 0000000000..f6598ded42 --- /dev/null +++ b/backend/.sqlx/query-ab906821072ddaa6303c5ea632150b5db1bf5a11d7a162130ebb826178c4ea08.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id FROM workspace\n WHERE parent_workspace_id = $1 AND is_dev_workspace AND NOT deleted", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "ab906821072ddaa6303c5ea632150b5db1bf5a11d7a162130ebb826178c4ea08" +} diff --git a/backend/.sqlx/query-aba0caddc216e218225d41cc3e14db0d8013ca448ca469f1c8c90965a514a8fb.json b/backend/.sqlx/query-aba0caddc216e218225d41cc3e14db0d8013ca448ca469f1c8c90965a514a8fb.json new file mode 100644 index 0000000000..f344bf3795 --- /dev/null +++ b/backend/.sqlx/query-aba0caddc216e218225d41cc3e14db0d8013ca448ca469f1c8c90965a514a8fb.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff\n (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork)\n VALUES ('test-workspace', 'wm-fork-test-workspace', 'f/rt/parent_only', 'http_trigger', 1, 0, true, true, false)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "aba0caddc216e218225d41cc3e14db0d8013ca448ca469f1c8c90965a514a8fb" +} diff --git a/backend/.sqlx/query-ac113ee13ac2b6e8da5ebb6bf9f8207e1b46ba34fcb0f6957caef4be49bbbaa9.json b/backend/.sqlx/query-ac113ee13ac2b6e8da5ebb6bf9f8207e1b46ba34fcb0f6957caef4be49bbbaa9.json new file mode 100644 index 0000000000..52ebcb5422 --- /dev/null +++ b/backend/.sqlx/query-ac113ee13ac2b6e8da5ebb6bf9f8207e1b46ba34fcb0f6957caef4be49bbbaa9.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT hash FROM script WHERE workspace_id = $1 AND path = $2 AND deleted = false AND archived = false ORDER BY created_at DESC LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "hash", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "ac113ee13ac2b6e8da5ebb6bf9f8207e1b46ba34fcb0f6957caef4be49bbbaa9" +} diff --git a/backend/.sqlx/query-ad4a4e651ac84847554ce66bd6b04650df33cd4e57069a0194625cdc83f4c8eb.json b/backend/.sqlx/query-ad4a4e651ac84847554ce66bd6b04650df33cd4e57069a0194625cdc83f4c8eb.json new file mode 100644 index 0000000000..e660c17021 --- /dev/null +++ b/backend/.sqlx/query-ad4a4e651ac84847554ce66bd6b04650df33cd4e57069a0194625cdc83f4c8eb.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ahead, fork_last_event_kind, fork_last_event_origin FROM workspace_diff\n WHERE fork_workspace_id = 'wm-fork-deferred' AND kind = 'flow'\n AND path = 'u/admin/moved_by_user'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ahead", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "fork_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "fork_last_event_origin", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true, + true + ] + }, + "hash": "ad4a4e651ac84847554ce66bd6b04650df33cd4e57069a0194625cdc83f4c8eb" +} diff --git a/backend/.sqlx/query-ad943dcd77732868fb79fa7d6330fe3f297f5701e7ef8e3d6377a921d08360f9.json b/backend/.sqlx/query-ad943dcd77732868fb79fa7d6330fe3f297f5701e7ef8e3d6377a921d08360f9.json new file mode 100644 index 0000000000..6641a69ea2 --- /dev/null +++ b/backend/.sqlx/query-ad943dcd77732868fb79fa7d6330fe3f297f5701e7ef8e3d6377a921d08360f9.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT email, username FROM password\n WHERE disabled IS false\n AND (email ILIKE $2 OR username ILIKE $2)\n AND NOT EXISTS (SELECT 1 FROM usr WHERE usr.workspace_id = $1 AND usr.email = password.email)\n ORDER BY email\n LIMIT $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "email", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "username", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + false, + true + ] + }, + "hash": "ad943dcd77732868fb79fa7d6330fe3f297f5701e7ef8e3d6377a921d08360f9" +} diff --git a/backend/.sqlx/query-adbe6ce27d46ba0d2d3ce87d875f60c179e5bccc54a36c8d5dcf04fd84445309.json b/backend/.sqlx/query-adbe6ce27d46ba0d2d3ce87d875f60c179e5bccc54a36c8d5dcf04fd84445309.json new file mode 100644 index 0000000000..04ef8a7d24 --- /dev/null +++ b/backend/.sqlx/query-adbe6ce27d46ba0d2d3ce87d875f60c179e5bccc54a36c8d5dcf04fd84445309.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE workspace_protection_rule\n SET name = $1, rules = $2, bypass_groups = $3, bypass_users = $4\n WHERE workspace_id = $5 AND name = $6\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Int4", + "TextArray", + "TextArray", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "adbe6ce27d46ba0d2d3ce87d875f60c179e5bccc54a36c8d5dcf04fd84445309" +} diff --git a/backend/.sqlx/query-af2f8c70ad5a14d35048211631b250879d18751a53597ef262b63d1e29bcb6a7.json b/backend/.sqlx/query-af2f8c70ad5a14d35048211631b250879d18751a53597ef262b63d1e29bcb6a7.json new file mode 100644 index 0000000000..f8129440a8 --- /dev/null +++ b/backend/.sqlx/query-af2f8c70ad5a14d35048211631b250879d18751a53597ef262b63d1e29bcb6a7.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE dbt_run_state SET workspace_id = $1\n WHERE workspace_id = $2 AND job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "af2f8c70ad5a14d35048211631b250879d18751a53597ef262b63d1e29bcb6a7" +} diff --git a/backend/.sqlx/query-b085fff6f34149a53bff05c9aa75e3d30bd900eb2e1064c209846918c388a2a1.json b/backend/.sqlx/query-b085fff6f34149a53bff05c9aa75e3d30bd900eb2e1064c209846918c388a2a1.json new file mode 100644 index 0000000000..ebe190ebc1 --- /dev/null +++ b/backend/.sqlx/query-b085fff6f34149a53bff05c9aa75e3d30bd900eb2e1064c209846918c388a2a1.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b085fff6f34149a53bff05c9aa75e3d30bd900eb2e1064c209846918c388a2a1" +} diff --git a/backend/.sqlx/query-b0a7963de04faccae3262823dc98f2faa8ef6ead3f797a49be9f5b075bad1490.json b/backend/.sqlx/query-b0a7963de04faccae3262823dc98f2faa8ef6ead3f797a49be9f5b075bad1490.json new file mode 100644 index 0000000000..b825e2146e --- /dev/null +++ b/backend/.sqlx/query-b0a7963de04faccae3262823dc98f2faa8ef6ead3f797a49be9f5b075bad1490.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH RECURSIVE chain AS (\n SELECT id, parent_workspace_id, 0 AS depth FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, chain.depth + 1 FROM workspace w\n JOIN chain ON w.id = chain.parent_workspace_id\n WHERE chain.depth < 20\n )\n SELECT EXISTS(SELECT 1 FROM chain WHERE id = $2) AS \"cycle!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "cycle!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b0a7963de04faccae3262823dc98f2faa8ef6ead3f797a49be9f5b075bad1490" +} diff --git a/backend/.sqlx/query-b19f5156b3e36b5e831ce792637bf54a3f182c03cef0aef06a1c75b45992cd39.json b/backend/.sqlx/query-b19f5156b3e36b5e831ce792637bf54a3f182c03cef0aef06a1c75b45992cd39.json new file mode 100644 index 0000000000..10adc6e68d --- /dev/null +++ b/backend/.sqlx/query-b19f5156b3e36b5e831ce792637bf54a3f182c03cef0aef06a1c75b45992cd39.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) FROM dbt_edge WHERE workspace_id = $1 AND script_hash = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b19f5156b3e36b5e831ce792637bf54a3f182c03cef0aef06a1c75b45992cd39" +} diff --git a/backend/.sqlx/query-b23784ed2737c01aaf05efe42bcdcfc9579aa203005ead60322fb71fd33b15b0.json b/backend/.sqlx/query-b23784ed2737c01aaf05efe42bcdcfc9579aa203005ead60322fb71fd33b15b0.json index 1f21f33ce2..806b923acb 100644 --- a/backend/.sqlx/query-b23784ed2737c01aaf05efe42bcdcfc9579aa203005ead60322fb71fd33b15b0.json +++ b/backend/.sqlx/query-b23784ed2737c01aaf05efe42bcdcfc9579aa203005ead60322fb71fd33b15b0.json @@ -154,7 +154,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-b24c1ed9aa3b856eed3da73eadb99a11874d6680f584324827f0ae59bb9fe506.json b/backend/.sqlx/query-b24c1ed9aa3b856eed3da73eadb99a11874d6680f584324827f0ae59bb9fe506.json new file mode 100644 index 0000000000..e29c1c0dbb --- /dev/null +++ b/backend/.sqlx/query-b24c1ed9aa3b856eed3da73eadb99a11874d6680f584324827f0ae59bb9fe506.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_graph_snapshot\n WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "b24c1ed9aa3b856eed3da73eadb99a11874d6680f584324827f0ae59bb9fe506" +} diff --git a/backend/.sqlx/query-b2dd773eebe0005eb4220f19e6a01aa9d12e461a89243bd0b651d1e5b837b6f4.json b/backend/.sqlx/query-b2dd773eebe0005eb4220f19e6a01aa9d12e461a89243bd0b651d1e5b837b6f4.json new file mode 100644 index 0000000000..b865865c1d --- /dev/null +++ b/backend/.sqlx/query-b2dd773eebe0005eb4220f19e6a01aa9d12e461a89243bd0b651d1e5b837b6f4.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_run_state\n WHERE workspace_id = $1 AND script_path = $2 AND permissioned_as = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "b2dd773eebe0005eb4220f19e6a01aa9d12e461a89243bd0b651d1e5b837b6f4" +} diff --git a/backend/.sqlx/query-b44afcf1b9c047ac525638f0952c2cb01d65b1b46693331ac157dfca0dab6824.json b/backend/.sqlx/query-b44afcf1b9c047ac525638f0952c2cb01d65b1b46693331ac157dfca0dab6824.json index eab78c7de8..9187cc29b7 100644 --- a/backend/.sqlx/query-b44afcf1b9c047ac525638f0952c2cb01d65b1b46693331ac157dfca0dab6824.json +++ b/backend/.sqlx/query-b44afcf1b9c047ac525638f0952c2cb01d65b1b46693331ac157dfca0dab6824.json @@ -44,7 +44,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-b546a7a68143a3b0de1d900e5f635b8a1e4d1ba18298ace1602defacb9e67f0c.json b/backend/.sqlx/query-b546a7a68143a3b0de1d900e5f635b8a1e4d1ba18298ace1602defacb9e67f0c.json new file mode 100644 index 0000000000..3aaa58da40 --- /dev/null +++ b/backend/.sqlx/query-b546a7a68143a3b0de1d900e5f635b8a1e4d1ba18298ace1602defacb9e67f0c.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE dbt_run_state SET script_path = $3 WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "b546a7a68143a3b0de1d900e5f635b8a1e4d1ba18298ace1602defacb9e67f0c" +} diff --git a/backend/.sqlx/query-b6303603bebdcf6d5bad2006ce65fe7900e33a843711934799b7259d1c451da6.json b/backend/.sqlx/query-b6303603bebdcf6d5bad2006ce65fe7900e33a843711934799b7259d1c451da6.json new file mode 100644 index 0000000000..218cf8252f --- /dev/null +++ b/backend/.sqlx/query-b6303603bebdcf6d5bad2006ce65fe7900e33a843711934799b7259d1c451da6.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE script SET on_behalf_of = $1 WHERE on_behalf_of = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "b6303603bebdcf6d5bad2006ce65fe7900e33a843711934799b7259d1c451da6" +} diff --git a/backend/.sqlx/query-b6f0ebbe674e1290afda7b5bed3b463f1d73d66b5924734274272509c98573db.json b/backend/.sqlx/query-b6f0ebbe674e1290afda7b5bed3b463f1d73d66b5924734274272509c98573db.json new file mode 100644 index 0000000000..74d2a57de7 --- /dev/null +++ b/backend/.sqlx/query-b6f0ebbe674e1290afda7b5bed3b463f1d73d66b5924734274272509c98573db.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO app (workspace_id, path, summary, policy, versions)\n VALUES ($1, $2, '', '{}'::jsonb, '{}') RETURNING 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Varchar" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b6f0ebbe674e1290afda7b5bed3b463f1d73d66b5924734274272509c98573db" +} diff --git a/backend/.sqlx/query-b9686c115e83ad0725503e2c076960914f825eaa2b44281a4e147cea6e886e7a.json b/backend/.sqlx/query-b9686c115e83ad0725503e2c076960914f825eaa2b44281a4e147cea6e886e7a.json new file mode 100644 index 0000000000..ebb5cee72c --- /dev/null +++ b/backend/.sqlx/query-b9686c115e83ad0725503e2c076960914f825eaa2b44281a4e147cea6e886e7a.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM native_trigger WHERE workspace_id = $1 AND service_name = $2 RETURNING webhook_token_hash", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "webhook_token_hash", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "native_trigger_service", + "kind": { + "Enum": [ + "nextcloud", + "google", + "github" + ] + } + } + } + ] + }, + "nullable": [ + false + ] + }, + "hash": "b9686c115e83ad0725503e2c076960914f825eaa2b44281a4e147cea6e886e7a" +} diff --git a/backend/.sqlx/query-bc59b7d863fc9df10cddf7487d6c2a3bd3ca8334f6cb5eaf42e14265a32ae472.json b/backend/.sqlx/query-bc59b7d863fc9df10cddf7487d6c2a3bd3ca8334f6cb5eaf42e14265a32ae472.json new file mode 100644 index 0000000000..09367d07d3 --- /dev/null +++ b/backend/.sqlx/query-bc59b7d863fc9df10cddf7487d6c2a3bd3ca8334f6cb5eaf42e14265a32ae472.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE dbt_run_progress SET workspace_id = $1 WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "bc59b7d863fc9df10cddf7487d6c2a3bd3ca8334f6cb5eaf42e14265a32ae472" +} diff --git a/backend/.sqlx/query-bc75df084b33779868317fe5c1d09bc8f1e01228aeb60fb1cdc034b4422888e0.json b/backend/.sqlx/query-bc75df084b33779868317fe5c1d09bc8f1e01228aeb60fb1cdc034b4422888e0.json new file mode 100644 index 0000000000..a245f1868f --- /dev/null +++ b/backend/.sqlx/query-bc75df084b33779868317fe5c1d09bc8f1e01228aeb60fb1cdc034b4422888e0.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE flow SET on_behalf_of_email = $1 WHERE on_behalf_of_email = $2 AND (on_behalf_of IS NULL OR on_behalf_of NOT LIKE 'g/%')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "bc75df084b33779868317fe5c1d09bc8f1e01228aeb60fb1cdc034b4422888e0" +} diff --git a/backend/.sqlx/query-bd293b1e468f70dcc21ab4a91eb5225058fb14d29a8a4de04fe161af0ff2d167.json b/backend/.sqlx/query-bd293b1e468f70dcc21ab4a91eb5225058fb14d29a8a4de04fe161af0ff2d167.json new file mode 100644 index 0000000000..501a8460a1 --- /dev/null +++ b/backend/.sqlx/query-bd293b1e468f70dcc21ab4a91eb5225058fb14d29a8a4de04fe161af0ff2d167.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_run_state\n WHERE workspace_id = $1 AND script_path = $2 AND permissioned_as = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "bd293b1e468f70dcc21ab4a91eb5225058fb14d29a8a4de04fe161af0ff2d167" +} diff --git a/backend/.sqlx/query-bdb53068a223c12c2d317635993e8fda531e69a880d0ebc44fb5f13ac976b67d.json b/backend/.sqlx/query-bdb53068a223c12c2d317635993e8fda531e69a880d0ebc44fb5f13ac976b67d.json index 2e194c9580..be50a16b17 100644 --- a/backend/.sqlx/query-bdb53068a223c12c2d317635993e8fda531e69a880d0ebc44fb5f13ac976b67d.json +++ b/backend/.sqlx/query-bdb53068a223c12c2d317635993e8fda531e69a880d0ebc44fb5f13ac976b67d.json @@ -41,7 +41,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json index 809deb1a55..32e4c5404e 100644 --- a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json +++ b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json @@ -78,7 +78,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-bea1144ea4d50aff717654eb1c2fd3a98c72172fd907c5a710ae50abbd02fd92.json b/backend/.sqlx/query-bea1144ea4d50aff717654eb1c2fd3a98c72172fd907c5a710ae50abbd02fd92.json new file mode 100644 index 0000000000..ea3dbea9fb --- /dev/null +++ b/backend/.sqlx/query-bea1144ea4d50aff717654eb1c2fd3a98c72172fd907c5a710ae50abbd02fd92.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO group_(workspace_id, name, summary, extra_perms) VALUES ('test-workspace', 'ops', '', '{}')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "bea1144ea4d50aff717654eb1c2fd3a98c72172fd907c5a710ae50abbd02fd92" +} diff --git a/backend/.sqlx/query-bf45139e3ae5c5c16591afb26c51c01c1e2fd56a2dc2dbd1ed41f08be7e61e08.json b/backend/.sqlx/query-bf45139e3ae5c5c16591afb26c51c01c1e2fd56a2dc2dbd1ed41f08be7e61e08.json new file mode 100644 index 0000000000..d5763deb84 --- /dev/null +++ b/backend/.sqlx/query-bf45139e3ae5c5c16591afb26c51c01c1e2fd56a2dc2dbd1ed41f08be7e61e08.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, value->>'on_behalf_of_email' AS email, value->>'on_behalf_of' AS principal FROM draft WHERE workspace_id = 'test-workspace' ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "email", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "principal", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + null, + null + ] + }, + "hash": "bf45139e3ae5c5c16591afb26c51c01c1e2fd56a2dc2dbd1ed41f08be7e61e08" +} diff --git a/backend/.sqlx/query-bf63f1e3c2f06b1133a4cd6ced5a3f7752e79027a2a8636808bb41628b789ebc.json b/backend/.sqlx/query-bf63f1e3c2f06b1133a4cd6ced5a3f7752e79027a2a8636808bb41628b789ebc.json new file mode 100644 index 0000000000..a0ab56888b --- /dev/null +++ b/backend/.sqlx/query-bf63f1e3c2f06b1133a4cd6ced5a3f7752e79027a2a8636808bb41628b789ebc.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE usr SET email = 'group-ops@windmill.dev' WHERE workspace_id = 'test-workspace' AND username = 'test-user-2'", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "bf63f1e3c2f06b1133a4cd6ced5a3f7752e79027a2a8636808bb41628b789ebc" +} diff --git a/backend/.sqlx/query-bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1.json b/backend/.sqlx/query-bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1.json index a051b068a5..df982f3296 100644 --- a/backend/.sqlx/query-bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1.json +++ b/backend/.sqlx/query-bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1.json @@ -34,7 +34,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-c13f203478d6bc623c27bba9b6fc6e9c9d36bcf9f7138422414f31f0319e6cfb.json b/backend/.sqlx/query-c13f203478d6bc623c27bba9b6fc6e9c9d36bcf9f7138422414f31f0319e6cfb.json new file mode 100644 index 0000000000..a1bf434814 --- /dev/null +++ b/backend/.sqlx/query-c13f203478d6bc623c27bba9b6fc6e9c9d36bcf9f7138422414f31f0319e6cfb.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2 AND archived = false AND deleted = false)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "c13f203478d6bc623c27bba9b6fc6e9c9d36bcf9f7138422414f31f0319e6cfb" +} diff --git a/backend/.sqlx/query-c159975f7b9ad9c8eeef340ca03ed4919122d653e1235d0847bfad84f0a3e539.json b/backend/.sqlx/query-c159975f7b9ad9c8eeef340ca03ed4919122d653e1235d0847bfad84f0a3e539.json new file mode 100644 index 0000000000..4be222b94d --- /dev/null +++ b/backend/.sqlx/query-c159975f7b9ad9c8eeef340ca03ed4919122d653e1235d0847bfad84f0a3e539.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, policy->>'on_behalf_of_email' AS email FROM app WHERE workspace_id = 'test-workspace' ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "email", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + null + ] + }, + "hash": "c159975f7b9ad9c8eeef340ca03ed4919122d653e1235d0847bfad84f0a3e539" +} diff --git a/backend/.sqlx/query-c1a85cf2d1555ae33bbdaf85cf243d44a23f2fe2abf7a9ff8f271f9dfb1433e3.json b/backend/.sqlx/query-c1a85cf2d1555ae33bbdaf85cf243d44a23f2fe2abf7a9ff8f271f9dfb1433e3.json new file mode 100644 index 0000000000..5fb1a353e7 --- /dev/null +++ b/backend/.sqlx/query-c1a85cf2d1555ae33bbdaf85cf243d44a23f2fe2abf7a9ff8f271f9dfb1433e3.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_edge\n WHERE job_id <> '00000000-0000-0000-0000-000000000000'\n AND ingested_at < now() - make_interval(days => $1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int4" + ] + }, + "nullable": [] + }, + "hash": "c1a85cf2d1555ae33bbdaf85cf243d44a23f2fe2abf7a9ff8f271f9dfb1433e3" +} diff --git a/backend/.sqlx/query-c1fb844ac8fb9293510d08dfb1cec03751a661076ee3250446b5157c43defc97.json b/backend/.sqlx/query-c1fb844ac8fb9293510d08dfb1cec03751a661076ee3250446b5157c43defc97.json new file mode 100644 index 0000000000..cd2dead06d --- /dev/null +++ b/backend/.sqlx/query-c1fb844ac8fb9293510d08dfb1cec03751a661076ee3250446b5157c43defc97.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_graph_snapshot\n (workspace_id, script_path, script_hash, job_id, digest, ingested_at)\n VALUES ($1, $2, $3, $4, $5, now())\n ON CONFLICT (workspace_id, script_path, script_hash, job_id)\n WHERE script_hash IS NOT NULL\n DO UPDATE SET digest = EXCLUDED.digest, ingested_at = now()", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Uuid", + "Text" + ] + }, + "nullable": [] + }, + "hash": "c1fb844ac8fb9293510d08dfb1cec03751a661076ee3250446b5157c43defc97" +} diff --git a/backend/.sqlx/query-c237044a7e04a6f87bd90114a33672a181bf5399c5b17ef8c4f806163bf20f70.json b/backend/.sqlx/query-c237044a7e04a6f87bd90114a33672a181bf5399c5b17ef8c4f806163bf20f70.json new file mode 100644 index 0000000000..49745d0791 --- /dev/null +++ b/backend/.sqlx/query-c237044a7e04a6f87bd90114a33672a181bf5399c5b17ef8c4f806163bf20f70.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM flow WHERE workspace_id = $1 AND path = $2 AND archived = false)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "c237044a7e04a6f87bd90114a33672a181bf5399c5b17ef8c4f806163bf20f70" +} diff --git a/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json b/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json index 7981dbc983..9844615122 100644 --- a/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json +++ b/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json @@ -16,7 +16,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } @@ -87,7 +88,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-c2e5f233e134ccc70b05d62239aaf736b456dbdd7d358e58744aa565e5889241.json b/backend/.sqlx/query-c2e5f233e134ccc70b05d62239aaf736b456dbdd7d358e58744aa565e5889241.json new file mode 100644 index 0000000000..d8f9fbcee4 --- /dev/null +++ b/backend/.sqlx/query-c2e5f233e134ccc70b05d62239aaf736b456dbdd7d358e58744aa565e5889241.json @@ -0,0 +1,21 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_run_state (workspace_id, script_path, permissioned_as, identity, args, run_results, job_id, retryable, updated_at)\n SELECT $1::varchar, $2::varchar, $7::varchar, $3::text, $4::jsonb, $5::text, $6::uuid, $8::boolean, now()\n WHERE EXISTS (SELECT 1 FROM script\n WHERE workspace_id = $1 AND path = $2\n AND deleted = false AND archived = false\n AND language = 'dbt')\n ON CONFLICT (workspace_id, script_path, permissioned_as) DO UPDATE SET\n identity = EXCLUDED.identity, args = EXCLUDED.args,\n run_results = EXCLUDED.run_results, job_id = EXCLUDED.job_id,\n retryable = EXCLUDED.retryable, updated_at = now()", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Jsonb", + "Text", + "Uuid", + "Varchar", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "c2e5f233e134ccc70b05d62239aaf736b456dbdd7d358e58744aa565e5889241" +} diff --git a/backend/.sqlx/query-c6706971489603027edc678f9d6c42e8341d5d3c5e4f46a53b8dc637484dd760.json b/backend/.sqlx/query-c6706971489603027edc678f9d6c42e8341d5d3c5e4f46a53b8dc637484dd760.json new file mode 100644 index 0000000000..27087b4c4d --- /dev/null +++ b/backend/.sqlx/query-c6706971489603027edc678f9d6c42e8341d5d3c5e4f46a53b8dc637484dd760.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM script WHERE on_behalf_of = $1 AND NOT path LIKE $2 AND workspace_id = $3 AND NOT archived AND NOT deleted", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "c6706971489603027edc678f9d6c42e8341d5d3c5e4f46a53b8dc637484dd760" +} diff --git a/backend/.sqlx/query-c868e87f7ccc5ce19c421c857f67285177bd4eaa3a14af5bf08dbc32c7bda558.json b/backend/.sqlx/query-c868e87f7ccc5ce19c421c857f67285177bd4eaa3a14af5bf08dbc32c7bda558.json new file mode 100644 index 0000000000..8bc10809dc --- /dev/null +++ b/backend/.sqlx/query-c868e87f7ccc5ce19c421c857f67285177bd4eaa3a14af5bf08dbc32c7bda558.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT deleted FROM script WHERE workspace_id = $1 AND hash = $2 FOR UPDATE", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "deleted", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [ + false + ] + }, + "hash": "c868e87f7ccc5ce19c421c857f67285177bd4eaa3a14af5bf08dbc32c7bda558" +} diff --git a/backend/.sqlx/query-c894582ce2be367edd09aa82cf1261ce9fde9657bb4fbcb37e9c14b63ae909af.json b/backend/.sqlx/query-c894582ce2be367edd09aa82cf1261ce9fde9657bb4fbcb37e9c14b63ae909af.json new file mode 100644 index 0000000000..e55d8a1017 --- /dev/null +++ b/backend/.sqlx/query-c894582ce2be367edd09aa82cf1261ce9fde9657bb4fbcb37e9c14b63ae909af.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE resource SET value = $1, edited_at = now() WHERE path = $2 AND workspace_id = $3\n RETURNING resource_type", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "resource_type", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Jsonb", + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "c894582ce2be367edd09aa82cf1261ce9fde9657bb4fbcb37e9c14b63ae909af" +} diff --git a/backend/.sqlx/query-c8caa96a2003348768ed3dc9929f6d05234f25f8212ec61be5ed4cef386d2d14.json b/backend/.sqlx/query-c8caa96a2003348768ed3dc9929f6d05234f25f8212ec61be5ed4cef386d2d14.json new file mode 100644 index 0000000000..0c20b97d90 --- /dev/null +++ b/backend/.sqlx/query-c8caa96a2003348768ed3dc9929f6d05234f25f8212ec61be5ed4cef386d2d14.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM variable WHERE workspace_id = $1 AND path = ANY($2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false + ] + }, + "hash": "c8caa96a2003348768ed3dc9929f6d05234f25f8212ec61be5ed4cef386d2d14" +} diff --git a/backend/.sqlx/query-c8cdc44f76411a3a01b02eed6647e8319c368ca9ccdc095f94945cda082f8857.json b/backend/.sqlx/query-c8cdc44f76411a3a01b02eed6647e8319c368ca9ccdc095f94945cda082f8857.json new file mode 100644 index 0000000000..ac8ae1fa0b --- /dev/null +++ b/backend/.sqlx/query-c8cdc44f76411a3a01b02eed6647e8319c368ca9ccdc095f94945cda082f8857.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, permissioned_as, permissioned_as_email FROM v2_job WHERE id = ANY($1)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "permissioned_as", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "permissioned_as_email", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "c8cdc44f76411a3a01b02eed6647e8319c368ca9ccdc095f94945cda082f8857" +} diff --git a/backend/.sqlx/query-cbc68952e976e3daeba2dba398de11f81ef4d93f6f6d164269a5190af85512d9.json b/backend/.sqlx/query-cbc68952e976e3daeba2dba398de11f81ef4d93f6f6d164269a5190af85512d9.json new file mode 100644 index 0000000000..9ea2efa7ac --- /dev/null +++ b/backend/.sqlx/query-cbc68952e976e3daeba2dba398de11f81ef4d93f6f6d164269a5190af85512d9.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of FROM script WHERE path = 'u/test-user/s' AND workspace_id = 'test-workspace'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + true + ] + }, + "hash": "cbc68952e976e3daeba2dba398de11f81ef4d93f6f6d164269a5190af85512d9" +} diff --git a/backend/.sqlx/query-d01d85e7e5371591f0f2e5843448f464ed38bd4c314a19cc9a9e6f7d89b4a173.json b/backend/.sqlx/query-d01d85e7e5371591f0f2e5843448f464ed38bd4c314a19cc9a9e6f7d89b4a173.json new file mode 100644 index 0000000000..98550a98e9 --- /dev/null +++ b/backend/.sqlx/query-d01d85e7e5371591f0f2e5843448f464ed38bd4c314a19cc9a9e6f7d89b4a173.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of_email FROM script WHERE path = 'u/test-user/obo_recorded' AND workspace_id = 'test-workspace'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of_email", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + true + ] + }, + "hash": "d01d85e7e5371591f0f2e5843448f464ed38bd4c314a19cc9a9e6f7d89b4a173" +} diff --git a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json b/backend/.sqlx/query-d0df9d1fda20f0366505bf6abf67446a5e8e23d87eebc46bbb90bdbdf128935b.json similarity index 94% rename from backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json rename to backend/.sqlx/query-d0df9d1fda20f0366505bf6abf67446a5e8e23d87eebc46bbb90bdbdf128935b.json index 0bd066afd1..0da783f056 100644 --- a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json +++ b/backend/.sqlx/query-d0df9d1fda20f0366505bf6abf67446a5e8e23d87eebc46bbb90bdbdf128935b.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT\n v2_job_queue.workspace_id,\n v2_job_queue.id,\n v2_job.args as \"args: sqlx::types::Json>>\",\n v2_job.parent_job,\n v2_job.created_by,\n v2_job_queue.started_at,\n v2_job_queue.runnable_settings_handle,\n scheduled_for,\n runnable_path,\n kind as \"kind: JobKind\",\n runnable_id as \"runnable_id: ScriptHash\",\n canceled_reason,\n canceled_by,\n permissioned_as,\n permissioned_as_email,\n flow_status as \"flow_status: sqlx::types::Json>\",\n v2_job.tag,\n script_lang as \"script_lang: ScriptLang\",\n same_worker,\n pre_run_error,\n concurrent_limit,\n concurrency_time_window_s,\n flow_innermost_root_job,\n root_job,\n timeout,\n flow_step_id,\n cache_ttl,\n cache_ignore_s3_path,\n v2_job_queue.priority,\n preprocessed,\n script_entrypoint_override,\n trigger,\n trigger_kind as \"trigger_kind: JobTriggerKind\",\n visible_to_owner,\n NULL as permissioned_as_end_user_email\n FROM v2_job_queue INNER JOIN v2_job ON v2_job.id = v2_job_queue.id LEFT JOIN v2_job_status ON v2_job_status.id = v2_job_queue.id WHERE v2_job_queue.id = $1", + "query": "SELECT\n v2_job_queue.workspace_id,\n v2_job_queue.id,\n v2_job.args as \"args: sqlx::types::Json>>\",\n v2_job.parent_job,\n v2_job.created_by,\n v2_job_queue.started_at,\n v2_job_queue.runnable_settings_handle,\n scheduled_for,\n runnable_path,\n kind as \"kind: JobKind\",\n runnable_id as \"runnable_id: ScriptHash\",\n canceled_reason,\n canceled_by,\n permissioned_as,\n permissioned_as_email,\n flow_status as \"flow_status: sqlx::types::Json>\",\n v2_job.tag,\n script_lang as \"script_lang: ScriptLang\",\n same_worker,\n pre_run_error,\n concurrent_limit,\n concurrency_time_window_s,\n flow_innermost_root_job,\n root_job,\n timeout,\n flow_step_id,\n cache_ttl,\n cache_ignore_s3_path,\n v2_job_queue.priority,\n preprocessed,\n script_entrypoint_override,\n trigger,\n trigger_kind as \"trigger_kind: TriggerKindLabel\",\n visible_to_owner,\n NULL as permissioned_as_end_user_email\n FROM v2_job_queue INNER JOIN v2_job ON v2_job.id = v2_job_queue.id LEFT JOIN v2_job_status ON v2_job_status.id = v2_job_queue.id WHERE v2_job_queue.id = $1", "describe": { "columns": [ { @@ -149,7 +149,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } @@ -227,7 +228,7 @@ }, { "ordinal": 32, - "name": "trigger_kind: JobTriggerKind", + "name": "trigger_kind: TriggerKindLabel", "type_info": { "Custom": { "name": "job_trigger_kind", @@ -313,5 +314,5 @@ null ] }, - "hash": "d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9" + "hash": "d0df9d1fda20f0366505bf6abf67446a5e8e23d87eebc46bbb90bdbdf128935b" } diff --git a/backend/.sqlx/query-d22f81df67d55b2d8f1d15404194a7b4604171d12c6a4620514c6d030086936d.json b/backend/.sqlx/query-d22f81df67d55b2d8f1d15404194a7b4604171d12c6a4620514c6d030086936d.json index c4526daccc..64b4056596 100644 --- a/backend/.sqlx/query-d22f81df67d55b2d8f1d15404194a7b4604171d12c6a4620514c6d030086936d.json +++ b/backend/.sqlx/query-d22f81df67d55b2d8f1d15404194a7b4604171d12c6a4620514c6d030086936d.json @@ -69,7 +69,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-d29aedfdf8ac4220085c9a6ee2f1d9bc795e24291d4e1f83d3314d56a65587bd.json b/backend/.sqlx/query-d29aedfdf8ac4220085c9a6ee2f1d9bc795e24291d4e1f83d3314d56a65587bd.json new file mode 100644 index 0000000000..df4b4bcc1a --- /dev/null +++ b/backend/.sqlx/query-d29aedfdf8ac4220085c9a6ee2f1d9bc795e24291d4e1f83d3314d56a65587bd.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, source_last_event_kind, source_last_event_origin)\n SELECT $1, unnest($2::varchar[]), $3, $4, 0, 1, NULL, $5, $6\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n behind = workspace_diff.behind + 1,\n has_changes = NULL,\n source_last_event_kind = EXCLUDED.source_last_event_kind,\n source_last_event_origin = EXCLUDED.source_last_event_origin", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "VarcharArray", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "d29aedfdf8ac4220085c9a6ee2f1d9bc795e24291d4e1f83d3314d56a65587bd" +} diff --git a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json index 2660ca05ef..cd1e0c4c39 100644 --- a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json +++ b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json @@ -78,7 +78,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-d735f355d9eb2612b563b606c6c897c5ad764a882b2c9054a7665dd68b7a528b.json b/backend/.sqlx/query-d735f355d9eb2612b563b606c6c897c5ad764a882b2c9054a7665dd68b7a528b.json new file mode 100644 index 0000000000..edd7f2f97a --- /dev/null +++ b/backend/.sqlx/query-d735f355d9eb2612b563b606c6c897c5ad764a882b2c9054a7665dd68b7a528b.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_graph_snapshot\n WHERE job_id <> '00000000-0000-0000-0000-000000000000'\n AND ingested_at < now() - make_interval(days => $1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int4" + ] + }, + "nullable": [] + }, + "hash": "d735f355d9eb2612b563b606c6c897c5ad764a882b2c9054a7665dd68b7a528b" +} diff --git a/backend/.sqlx/query-d75bbac607ef4439d8aef2b33d1b2a035d2ecb8169fd26e6a1cb2b49ead6c4ea.json b/backend/.sqlx/query-d75bbac607ef4439d8aef2b33d1b2a035d2ecb8169fd26e6a1cb2b49ead6c4ea.json new file mode 100644 index 0000000000..02845d173a --- /dev/null +++ b/backend/.sqlx/query-d75bbac607ef4439d8aef2b33d1b2a035d2ecb8169fd26e6a1cb2b49ead6c4ea.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO password(email, password_hash, login_type, super_admin, verified, name)\n VALUES ('ext@windmill.dev', 'not-a-real-hash', 'password', true, true, 'Ext')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "d75bbac607ef4439d8aef2b33d1b2a035d2ecb8169fd26e6a1cb2b49ead6c4ea" +} diff --git a/backend/.sqlx/query-da10117644334b74bdb68e3f9d7ef37cd4aa442e68f80d8ce5f2b74a20d5146f.json b/backend/.sqlx/query-da10117644334b74bdb68e3f9d7ef37cd4aa442e68f80d8ce5f2b74a20d5146f.json new file mode 100644 index 0000000000..55b950ab4f --- /dev/null +++ b/backend/.sqlx/query-da10117644334b74bdb68e3f9d7ef37cd4aa442e68f80d8ce5f2b74a20d5146f.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of, on_behalf_of_email FROM script WHERE path = 'f/shared/obo' AND workspace_id = 'test-workspace'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "on_behalf_of_email", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + true, + true + ] + }, + "hash": "da10117644334b74bdb68e3f9d7ef37cd4aa442e68f80d8ce5f2b74a20d5146f" +} diff --git a/backend/.sqlx/query-daccf6fd7d626e99cdd8795e204344f36c27d8757e78fa1990a86c2dfcddf808.json b/backend/.sqlx/query-daccf6fd7d626e99cdd8795e204344f36c27d8757e78fa1990a86c2dfcddf808.json new file mode 100644 index 0000000000..001995f90b --- /dev/null +++ b/backend/.sqlx/query-daccf6fd7d626e99cdd8795e204344f36c27d8757e78fa1990a86c2dfcddf808.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, source_last_event_kind, source_last_event_origin)\n SELECT $1, unnest($2::varchar[]), $3, $4, 0, 1, NULL, $5, $6\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n behind = workspace_diff.behind + 1,\n has_changes = NULL,\n source_last_event_kind = EXCLUDED.source_last_event_kind,\n source_last_event_origin = EXCLUDED.source_last_event_origin", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "VarcharArray", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "daccf6fd7d626e99cdd8795e204344f36c27d8757e78fa1990a86c2dfcddf808" +} diff --git a/backend/.sqlx/query-dbe64a8367fad74866b15f96565ef1ae7c58848714974a8c803085a0c0e655d6.json b/backend/.sqlx/query-dbe64a8367fad74866b15f96565ef1ae7c58848714974a8c803085a0c0e655d6.json new file mode 100644 index 0000000000..520d04c762 --- /dev/null +++ b/backend/.sqlx/query-dbe64a8367fad74866b15f96565ef1ae7c58848714974a8c803085a0c0e655d6.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_edge WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "dbe64a8367fad74866b15f96565ef1ae7c58848714974a8c803085a0c0e655d6" +} diff --git a/backend/.sqlx/query-dcbf4630616d214c510dd8bda9cb77215453c094cc1caab16ae4b4e5148b0c58.json b/backend/.sqlx/query-dcbf4630616d214c510dd8bda9cb77215453c094cc1caab16ae4b4e5148b0c58.json new file mode 100644 index 0000000000..363c96f694 --- /dev/null +++ b/backend/.sqlx/query-dcbf4630616d214c510dd8bda9cb77215453c094cc1caab16ae4b4e5148b0c58.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs)\n SELECT $1, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs\n FROM flow WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "dcbf4630616d214c510dd8bda9cb77215453c094cc1caab16ae4b4e5148b0c58" +} diff --git a/backend/.sqlx/query-df278f1115680387ee1a5385c4ac5d112a798a8debb0b37bbbc5cebbe9af0087.json b/backend/.sqlx/query-df278f1115680387ee1a5385c4ac5d112a798a8debb0b37bbbc5cebbe9af0087.json new file mode 100644 index 0000000000..b455cc1b5c --- /dev/null +++ b/backend/.sqlx/query-df278f1115680387ee1a5385c4ac5d112a798a8debb0b37bbbc5cebbe9af0087.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM resource WHERE workspace_id = $1 AND path = $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "df278f1115680387ee1a5385c4ac5d112a798a8debb0b37bbbc5cebbe9af0087" +} diff --git a/backend/.sqlx/query-e013b906ba3e95350ebcda6a8b8cc3ee04f6782587bc34d44338d2b2cad03165.json b/backend/.sqlx/query-e013b906ba3e95350ebcda6a8b8cc3ee04f6782587bc34d44338d2b2cad03165.json new file mode 100644 index 0000000000..36161cb99c --- /dev/null +++ b/backend/.sqlx/query-e013b906ba3e95350ebcda6a8b8cc3ee04f6782587bc34d44338d2b2cad03165.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dbt_node\n WHERE job_id <> '00000000-0000-0000-0000-000000000000'\n AND ingested_at < now() - make_interval(days => $1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int4" + ] + }, + "nullable": [] + }, + "hash": "e013b906ba3e95350ebcda6a8b8cc3ee04f6782587bc34d44338d2b2cad03165" +} diff --git a/backend/.sqlx/query-e01e9040f2336d9db41638842c5a4a07af79130e1dbfd91f1601c04721186609.json b/backend/.sqlx/query-e01e9040f2336d9db41638842c5a4a07af79130e1dbfd91f1601c04721186609.json new file mode 100644 index 0000000000..4967ac0212 --- /dev/null +++ b/backend/.sqlx/query-e01e9040f2336d9db41638842c5a4a07af79130e1dbfd91f1601c04721186609.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE dbt_run_progress\n SET status = 'failed',\n error = COALESCE(error, 'the run ended before this model finished')\n WHERE workspace_id = $1 AND job_id = $2 AND status = 'running'\n AND NOT (asset_path = ANY($3))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Uuid", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "e01e9040f2336d9db41638842c5a4a07af79130e1dbfd91f1601c04721186609" +} diff --git a/backend/.sqlx/query-e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82.json b/backend/.sqlx/query-e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82.json index 037c796fc7..0470656ba6 100644 --- a/backend/.sqlx/query-e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82.json +++ b/backend/.sqlx/query-e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82.json @@ -17,7 +17,8 @@ "variable", "ducklake", "datatable", - "volume" + "volume", + "dbt" ] } } diff --git a/backend/.sqlx/query-e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5.json b/backend/.sqlx/query-e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5.json new file mode 100644 index 0000000000..68825a19c7 --- /dev/null +++ b/backend/.sqlx/query-e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, created_at, created_by FROM resource_version\n WHERE workspace_id = $1 AND path = $2 ORDER BY id DESC LIMIT $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "created_by", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5" +} diff --git a/backend/.sqlx/query-e3fd8cfde2ae3e374b7c4ffcab91fcce10865992242cd053779760cc68909ff7.json b/backend/.sqlx/query-e3fd8cfde2ae3e374b7c4ffcab91fcce10865992242cd053779760cc68909ff7.json new file mode 100644 index 0000000000..9509a76604 --- /dev/null +++ b/backend/.sqlx/query-e3fd8cfde2ae3e374b7c4ffcab91fcce10865992242cd053779760cc68909ff7.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM script_trigger\n WHERE workspace_id = $1 AND runnable_kind = 'script' AND runnable_path = $2\n AND trigger_kind = 'asset' AND trigger_ref LIKE 'dbt://%'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "e3fd8cfde2ae3e374b7c4ffcab91fcce10865992242cd053779760cc68909ff7" +} diff --git a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json index ad34773e36..29d3eede53 100644 --- a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json +++ b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json @@ -134,7 +134,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-704d873d4585e00aa0a88cc9949d3a6cc806b33a844b8e0e600cb8bfae428e84.json b/backend/.sqlx/query-e5775b1417d506e90953dc8e2b769ba049d314d93e4436f8f14f4812eb4990a0.json similarity index 92% rename from backend/.sqlx/query-704d873d4585e00aa0a88cc9949d3a6cc806b33a844b8e0e600cb8bfae428e84.json rename to backend/.sqlx/query-e5775b1417d506e90953dc8e2b769ba049d314d93e4436f8f14f4812eb4990a0.json index 9f80167752..3fa97c8ae5 100644 --- a/backend/.sqlx/query-704d873d4585e00aa0a88cc9949d3a6cc806b33a844b8e0e600cb8bfae428e84.json +++ b/backend/.sqlx/query-e5775b1417d506e90953dc8e2b769ba049d314d93e4436f8f14f4812eb4990a0.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT workspace_id, username, email, is_admin, created_at, operator, disabled, role, added_via, is_service_account\n FROM usr\n WHERE workspace_id = $1\n ", + "query": "\n SELECT workspace_id, username, email, is_admin, created_at, operator, disabled, role, added_via, is_service_account\n FROM usr\n WHERE workspace_id = $1\n ORDER BY email\n ", "describe": { "columns": [ { @@ -72,5 +72,5 @@ false ] }, - "hash": "704d873d4585e00aa0a88cc9949d3a6cc806b33a844b8e0e600cb8bfae428e84" + "hash": "e5775b1417d506e90953dc8e2b769ba049d314d93e4436f8f14f4812eb4990a0" } diff --git a/backend/.sqlx/query-e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e.json b/backend/.sqlx/query-e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e.json new file mode 100644 index 0000000000..77399cdf62 --- /dev/null +++ b/backend/.sqlx/query-e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, value FROM resource_version WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "value", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [ + false, + true + ] + }, + "hash": "e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e" +} diff --git a/backend/.sqlx/query-e782878f409796f1fe2a773f054f0b3f63185a6e6a9f4e0dfe961dd466a41808.json b/backend/.sqlx/query-e782878f409796f1fe2a773f054f0b3f63185a6e6a9f4e0dfe961dd466a41808.json new file mode 100644 index 0000000000..73f5c2e9f7 --- /dev/null +++ b/backend/.sqlx/query-e782878f409796f1fe2a773f054f0b3f63185a6e6a9f4e0dfe961dd466a41808.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH inserted AS (\n INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs)\n SELECT workspace_id, REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3\n RETURNING 1\n ) SELECT COUNT(*) FROM inserted", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "e782878f409796f1fe2a773f054f0b3f63185a6e6a9f4e0dfe961dd466a41808" +} diff --git a/backend/.sqlx/query-e7f617f9abc523708567ee415f95150048be5d7d0e494fbd3aaf550f4c4d8125.json b/backend/.sqlx/query-e7f617f9abc523708567ee415f95150048be5d7d0e494fbd3aaf550f4c4d8125.json new file mode 100644 index 0000000000..a96671ab04 --- /dev/null +++ b/backend/.sqlx/query-e7f617f9abc523708567ee415f95150048be5d7d0e494fbd3aaf550f4c4d8125.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO group_ (workspace_id, name, summary, extra_perms) VALUES ('test-workspace', 'ops', '', '{}') ON CONFLICT DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "e7f617f9abc523708567ee415f95150048be5d7d0e494fbd3aaf550f4c4d8125" +} diff --git a/backend/.sqlx/query-e9ea69ee2a7927e18a2bae9721407e05b1ab61512e00034982dfd9b4ac3257aa.json b/backend/.sqlx/query-e9ea69ee2a7927e18a2bae9721407e05b1ab61512e00034982dfd9b4ac3257aa.json new file mode 100644 index 0000000000..a698761d8a --- /dev/null +++ b/backend/.sqlx/query-e9ea69ee2a7927e18a2bae9721407e05b1ab61512e00034982dfd9b4ac3257aa.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (parent_workspace_id IS NOT NULL) AS \"is_fork!\", is_dev_workspace, deleted\n FROM workspace WHERE id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "is_fork!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "is_dev_workspace", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "deleted", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null, + false, + false + ] + }, + "hash": "e9ea69ee2a7927e18a2bae9721407e05b1ab61512e00034982dfd9b4ac3257aa" +} diff --git a/backend/.sqlx/query-ea95c0684cd436673ddc5f242ffaede3b52ebab41c74c57a4748ee912f45f6b1.json b/backend/.sqlx/query-ea95c0684cd436673ddc5f242ffaede3b52ebab41c74c57a4748ee912f45f6b1.json new file mode 100644 index 0000000000..8591604eb8 --- /dev/null +++ b/backend/.sqlx/query-ea95c0684cd436673ddc5f242ffaede3b52ebab41c74c57a4748ee912f45f6b1.json @@ -0,0 +1,83 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork,\n fork_last_event_kind, fork_last_event_origin, source_last_event_kind, source_last_event_origin\n FROM workspace_diff\n WHERE source_workspace_id = $1 AND fork_workspace_id = $2\n AND NOT EXISTS (\n SELECT 1 FROM ws_specific ws\n WHERE ws.path = workspace_diff.path\n AND ws.item_kind = workspace_diff.kind\n AND ws.workspace_id IN (workspace_diff.source_workspace_id, workspace_diff.fork_workspace_id)\n )", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "kind", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "ahead", + "type_info": "Int4" + }, + { + "ordinal": 3, + "name": "behind", + "type_info": "Int4" + }, + { + "ordinal": 4, + "name": "has_changes", + "type_info": "Bool" + }, + { + "ordinal": 5, + "name": "exists_in_source", + "type_info": "Bool" + }, + { + "ordinal": 6, + "name": "exists_in_fork", + "type_info": "Bool" + }, + { + "ordinal": 7, + "name": "fork_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 8, + "name": "fork_last_event_origin", + "type_info": "Varchar" + }, + { + "ordinal": 9, + "name": "source_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 10, + "name": "source_last_event_origin", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true + ] + }, + "hash": "ea95c0684cd436673ddc5f242ffaede3b52ebab41c74c57a4748ee912f45f6b1" +} diff --git a/backend/.sqlx/query-ad25201d0eea65972234cade87a95d8cd99fc26e5bd466942423cbd09efcebe4.json b/backend/.sqlx/query-eaad94dd1b419f814af4cb54f2843e8d12a191ca97954c499ae5dc022410a240.json similarity index 74% rename from backend/.sqlx/query-ad25201d0eea65972234cade87a95d8cd99fc26e5bd466942423cbd09efcebe4.json rename to backend/.sqlx/query-eaad94dd1b419f814af4cb54f2843e8d12a191ca97954c499ae5dc022410a240.json index 7b989b68f5..29aa952292 100644 --- a/backend/.sqlx/query-ad25201d0eea65972234cade87a95d8cd99fc26e5bd466942423cbd09efcebe4.json +++ b/backend/.sqlx/query-eaad94dd1b419f814af4cb54f2843e8d12a191ca97954c499ae5dc022410a240.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n INSERT INTO azure_trigger (\n azure_resource_path, azure_mode, scope_resource_id, topic_name,\n subscription_name, event_type_filters,\n push_auth_config, workspace_id, path, script_path, is_flow,\n permissioned_as, mode, edited_by, email,\n error_handler_path, error_handler_args, retry\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7,\n $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18\n )\n ", + "query": "\n INSERT INTO azure_trigger (\n azure_resource_path, azure_mode, scope_resource_id, topic_name,\n subscription_name, event_type_filters,\n push_auth_config, workspace_id, path, script_path, is_flow,\n permissioned_as, mode, edited_by,\n error_handler_path, error_handler_args, retry\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7,\n $8, $9, $10, $11, $12, $13, $14, $15, $16, $17\n )\n ", "describe": { "columns": [], "parameters": { @@ -42,12 +42,11 @@ }, "Varchar", "Varchar", - "Varchar", "Jsonb", "Jsonb" ] }, "nullable": [] }, - "hash": "ad25201d0eea65972234cade87a95d8cd99fc26e5bd466942423cbd09efcebe4" + "hash": "eaad94dd1b419f814af4cb54f2843e8d12a191ca97954c499ae5dc022410a240" } diff --git a/backend/.sqlx/query-ebe7822c6fbdd8afea833e96cfd65e3f5fe96ff6dee2c6d41798328a3852fe76.json b/backend/.sqlx/query-ebe7822c6fbdd8afea833e96cfd65e3f5fe96ff6dee2c6d41798328a3852fe76.json new file mode 100644 index 0000000000..ac9b645b01 --- /dev/null +++ b/backend/.sqlx/query-ebe7822c6fbdd8afea833e96cfd65e3f5fe96ff6dee2c6d41798328a3852fe76.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH RECURSIVE seeded AS (SELECT unnest($1::text[]) AS id),\n up AS (\n SELECT w.id, w.parent_workspace_id, 0 AS depth\n FROM workspace w JOIN seeded s ON w.id = s.id\n UNION ALL\n SELECT w.id, w.parent_workspace_id, up.depth + 1\n FROM workspace w JOIN up ON w.id = up.parent_workspace_id\n WHERE up.depth < 20\n ),\n down AS (\n SELECT w.id, 0 AS depth FROM workspace w JOIN seeded s ON w.id = s.id\n UNION ALL\n SELECT w.id, down.depth + 1\n FROM workspace w JOIN down ON w.parent_workspace_id = down.id\n WHERE down.depth < 20 AND w.is_dev_workspace\n )\n SELECT id AS \"id!\" FROM (\n SELECT id FROM seeded UNION SELECT id FROM up UNION SELECT id FROM down\n ) n ORDER BY id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "TextArray" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ebe7822c6fbdd8afea833e96cfd65e3f5fe96ff6dee2c6d41798328a3852fe76" +} diff --git a/backend/.sqlx/query-ec2ed5203effe26c9584b266c5ccbc2a399e3a0e126c9ba7d0558d37c16926fa.json b/backend/.sqlx/query-ec2ed5203effe26c9584b266c5ccbc2a399e3a0e126c9ba7d0558d37c16926fa.json new file mode 100644 index 0000000000..d6e1a1bb57 --- /dev/null +++ b/backend/.sqlx/query-ec2ed5203effe26c9584b266c5ccbc2a399e3a0e126c9ba7d0558d37c16926fa.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_run_progress\n (workspace_id, job_id, asset_kind, asset_path, status, row_count, error, updated_at)\n VALUES ($1, $2, 'dbt', $3, $4, $5, $6, now())\n ON CONFLICT (workspace_id, job_id, asset_kind, asset_path)\n DO UPDATE SET status = EXCLUDED.status, row_count = EXCLUDED.row_count,\n error = EXCLUDED.error, updated_at = now()", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Uuid", + "Varchar", + { + "Custom": { + "name": "materialization_status", + "kind": { + "Enum": [ + "running", + "materialized", + "failed" + ] + } + } + }, + "Int8", + "Text" + ] + }, + "nullable": [] + }, + "hash": "ec2ed5203effe26c9584b266c5ccbc2a399e3a0e126c9ba7d0558d37c16926fa" +} diff --git a/backend/.sqlx/query-ed421d15d2f9883196619418b3ebd7c687900b66cd8c3a173f2e3410e4ed86d9.json b/backend/.sqlx/query-ed421d15d2f9883196619418b3ebd7c687900b66cd8c3a173f2e3410e4ed86d9.json new file mode 100644 index 0000000000..90e4ba1e36 --- /dev/null +++ b/backend/.sqlx/query-ed421d15d2f9883196619418b3ebd7c687900b66cd8c3a173f2e3410e4ed86d9.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM v2_job_queue q JOIN v2_job j ON j.id = q.id\n WHERE q.id = $1 AND j.workspace_id = $2 AND q.tag = ANY($3))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text", + "TextArray" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ed421d15d2f9883196619418b3ebd7c687900b66cd8c3a173f2e3410e4ed86d9" +} diff --git a/backend/.sqlx/query-edd6c09b7f012588788fd3c572d20eb439a80d52ae75ebd25128ffce759cd313.json b/backend/.sqlx/query-edd6c09b7f012588788fd3c572d20eb439a80d52ae75ebd25128ffce759cd313.json index 33b0b1f144..2357865ebc 100644 --- a/backend/.sqlx/query-edd6c09b7f012588788fd3c572d20eb439a80d52ae75ebd25128ffce759cd313.json +++ b/backend/.sqlx/query-edd6c09b7f012588788fd3c572d20eb439a80d52ae75ebd25128ffce759cd313.json @@ -44,7 +44,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-ee85a02a8e3abe3385524c9e549a9f1685387f527eb8739125f6a75ecfb991ac.json b/backend/.sqlx/query-ee85a02a8e3abe3385524c9e549a9f1685387f527eb8739125f6a75ecfb991ac.json new file mode 100644 index 0000000000..fe4797b43b --- /dev/null +++ b/backend/.sqlx/query-ee85a02a8e3abe3385524c9e549a9f1685387f527eb8739125f6a75ecfb991ac.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE\n flow\n SET\n path = $1,\n summary = $2,\n description = $3,\n dependency_job = NULL,\n lock_error_logs = '',\n tag = $4,\n dedicated_worker = $5,\n visible_to_runner_only = $6,\n ws_error_handler_muted = $7,\n value = $8,\n schema = $9::text::json,\n edited_by = $10,\n edited_at = now(),\n labels = COALESCE($13, labels),\n on_behalf_of = $14,\n on_behalf_of_email = $15\n WHERE\n path = $11 AND workspace_id = $12", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text", + "Varchar", + "Bool", + "Bool", + "Bool", + "Jsonb", + "Text", + "Varchar", + "Text", + "Text", + "TextArray", + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "ee85a02a8e3abe3385524c9e549a9f1685387f527eb8739125f6a75ecfb991ac" +} diff --git a/backend/.sqlx/query-eed8122fdcd007543dc6ca28b216845c6249c4598c4e16aa50067fdc6136451c.json b/backend/.sqlx/query-eed8122fdcd007543dc6ca28b216845c6249c4598c4e16aa50067fdc6136451c.json new file mode 100644 index 0000000000..63babb575e --- /dev/null +++ b/backend/.sqlx/query-eed8122fdcd007543dc6ca28b216845c6249c4598c4e16aa50067fdc6136451c.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id,\n digest, relation_root_at_last_ingest, ingested_at)\n SELECT $2, script_path, script_hash, job_id, digest, relation_root_at_last_ingest,\n ingested_at\n FROM dbt_graph_snapshot\n WHERE workspace_id = $1 AND job_id = '00000000-0000-0000-0000-000000000000'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "eed8122fdcd007543dc6ca28b216845c6249c4598c4e16aa50067fdc6136451c" +} diff --git a/backend/.sqlx/query-ef50e495c86edf69817a0850f2b8001814ac69e890f8cfdd57a345711194150e.json b/backend/.sqlx/query-ef50e495c86edf69817a0850f2b8001814ac69e890f8cfdd57a345711194150e.json new file mode 100644 index 0000000000..e8ad9d0335 --- /dev/null +++ b/backend/.sqlx/query-ef50e495c86edf69817a0850f2b8001814ac69e890f8cfdd57a345711194150e.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, fork_last_event_kind, fork_last_event_origin)\n VALUES ($1, $2, $3, $4, 1, 0, NULL, $5, $6)\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n ahead = workspace_diff.ahead + 1,\n has_changes = NULL,\n fork_last_event_kind = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.fork_last_event_origin IS NOT NULL) THEN workspace_diff.fork_last_event_kind ELSE $5 END,\n fork_last_event_origin = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.fork_last_event_origin IS NOT NULL) THEN workspace_diff.fork_last_event_origin ELSE $6 END", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "ef50e495c86edf69817a0850f2b8001814ac69e890f8cfdd57a345711194150e" +} diff --git a/backend/.sqlx/query-ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35.json b/backend/.sqlx/query-ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35.json new file mode 100644 index 0000000000..2a9c7cac4f --- /dev/null +++ b/backend/.sqlx/query-ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35.json @@ -0,0 +1,47 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, path, created_at, created_by, value FROM resource_version\n WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 3, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 4, + "name": "value", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [ + false, + false, + false, + true, + true + ] + }, + "hash": "ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35" +} diff --git a/backend/.sqlx/query-f0858450021df721d8a48b1b5dc887c5424562acd9769c80c5899193ef16b56b.json b/backend/.sqlx/query-f0858450021df721d8a48b1b5dc887c5424562acd9769c80c5899193ef16b56b.json index ad6028c14f..1bd565d00c 100644 --- a/backend/.sqlx/query-f0858450021df721d8a48b1b5dc887c5424562acd9769c80c5899193ef16b56b.json +++ b/backend/.sqlx/query-f0858450021df721d8a48b1b5dc887c5424562acd9769c80c5899193ef16b56b.json @@ -44,7 +44,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-f0d8ebbf62a1d92f39c2f86a70236cece6d6c9e28cce9aff7fe5496bba569807.json b/backend/.sqlx/query-f0d8ebbf62a1d92f39c2f86a70236cece6d6c9e28cce9aff7fe5496bba569807.json new file mode 100644 index 0000000000..912db1a932 --- /dev/null +++ b/backend/.sqlx/query-f0d8ebbf62a1d92f39c2f86a70236cece6d6c9e28cce9aff7fe5496bba569807.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT fork_last_event_kind, fork_last_event_origin FROM workspace_diff\n WHERE fork_workspace_id = 'wm-fork-renamed' AND kind = 'script'\n AND path = 'u/admin/inline' AND fork_last_event_kind IS NOT NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "fork_last_event_kind", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "fork_last_event_origin", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + true, + true + ] + }, + "hash": "f0d8ebbf62a1d92f39c2f86a70236cece6d6c9e28cce9aff7fe5496bba569807" +} diff --git a/backend/.sqlx/query-88a6a86285fcfaa778e9716d5072f555dff175feb704c3d825d6100defe91bce.json b/backend/.sqlx/query-f1a34561f208c05d75b2f981cd55465e579dd71a9439a2182ef05d9d3cfe462e.json similarity index 83% rename from backend/.sqlx/query-88a6a86285fcfaa778e9716d5072f555dff175feb704c3d825d6100defe91bce.json rename to backend/.sqlx/query-f1a34561f208c05d75b2f981cd55465e579dd71a9439a2182ef05d9d3cfe462e.json index 9097eb8584..81034a3861 100644 --- a/backend/.sqlx/query-88a6a86285fcfaa778e9716d5072f555dff175feb704c3d825d6100defe91bce.json +++ b/backend/.sqlx/query-f1a34561f208c05d75b2f981cd55465e579dd71a9439a2182ef05d9d3cfe462e.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT\n workspace_id,\n slack_team_id,\n teams_team_id,\n teams_team_name,\n teams_team_guid,\n slack_name,\n slack_command_script,\n teams_command_script,\n slack_email,\n slack_oauth_client_id,\n slack_oauth_client_secret,\n customer_id,\n plan,\n webhook,\n ai_config,\n large_file_storage,\n datatable,\n ducklake,\n git_sync,\n deploy_ui,\n default_app,\n default_scripts,\n mute_critical_alerts,\n color,\n operator_settings,\n git_app_installations,\n auto_invite,\n error_handler,\n success_handler,\n public_app_execution_limit_per_minute,\n error_handler_fallback_to_instance_alerts\n FROM\n workspace_settings\n WHERE\n workspace_id = $1\n ", + "query": "\n SELECT\n workspace_id,\n slack_team_id,\n teams_team_id,\n teams_team_name,\n teams_team_guid,\n slack_name,\n slack_command_script,\n teams_command_script,\n slack_email,\n slack_oauth_client_id,\n slack_oauth_client_secret,\n customer_id,\n plan,\n webhook,\n ai_config,\n dbt_warehouses,\n large_file_storage,\n datatable,\n ducklake,\n git_sync,\n deploy_ui,\n default_app,\n default_scripts,\n mute_critical_alerts,\n color,\n operator_settings,\n git_app_installations,\n auto_invite,\n error_handler,\n success_handler,\n public_app_execution_limit_per_minute,\n error_handler_fallback_to_instance_alerts\n FROM\n workspace_settings\n WHERE\n workspace_id = $1\n ", "describe": { "columns": [ { @@ -80,81 +80,86 @@ }, { "ordinal": 15, - "name": "large_file_storage", + "name": "dbt_warehouses", "type_info": "Jsonb" }, { "ordinal": 16, - "name": "datatable", + "name": "large_file_storage", "type_info": "Jsonb" }, { "ordinal": 17, - "name": "ducklake", + "name": "datatable", "type_info": "Jsonb" }, { "ordinal": 18, - "name": "git_sync", + "name": "ducklake", "type_info": "Jsonb" }, { "ordinal": 19, - "name": "deploy_ui", + "name": "git_sync", "type_info": "Jsonb" }, { "ordinal": 20, + "name": "deploy_ui", + "type_info": "Jsonb" + }, + { + "ordinal": 21, "name": "default_app", "type_info": "Varchar" }, { - "ordinal": 21, + "ordinal": 22, "name": "default_scripts", "type_info": "Jsonb" }, { - "ordinal": 22, + "ordinal": 23, "name": "mute_critical_alerts", "type_info": "Bool" }, { - "ordinal": 23, + "ordinal": 24, "name": "color", "type_info": "Varchar" }, { - "ordinal": 24, + "ordinal": 25, "name": "operator_settings", "type_info": "Jsonb" }, { - "ordinal": 25, + "ordinal": 26, "name": "git_app_installations", "type_info": "Jsonb" }, { - "ordinal": 26, + "ordinal": 27, "name": "auto_invite", "type_info": "Jsonb" }, { - "ordinal": 27, + "ordinal": 28, "name": "error_handler", "type_info": "Jsonb" }, { - "ordinal": 28, + "ordinal": 29, "name": "success_handler", "type_info": "Jsonb" }, { - "ordinal": 29, + "ordinal": 30, "name": "public_app_execution_limit_per_minute", "type_info": "Int4" }, { - "ordinal": 30, + "ordinal": 31, "name": "error_handler_fallback_to_instance_alerts", "type_info": "Bool" } @@ -190,6 +195,7 @@ true, true, true, + true, false, true, true, @@ -198,5 +204,5 @@ false ] }, - "hash": "88a6a86285fcfaa778e9716d5072f555dff175feb704c3d825d6100defe91bce" + "hash": "f1a34561f208c05d75b2f981cd55465e579dd71a9439a2182ef05d9d3cfe462e" } diff --git a/backend/.sqlx/query-f1b13ed0a8b834b0a55bfc7ef6470c2a3aaef44f19fbff2673d7bbd9d9a9aca2.json b/backend/.sqlx/query-f1b13ed0a8b834b0a55bfc7ef6470c2a3aaef44f19fbff2673d7bbd9d9a9aca2.json new file mode 100644 index 0000000000..43f0740aad --- /dev/null +++ b/backend/.sqlx/query-f1b13ed0a8b834b0a55bfc7ef6470c2a3aaef44f19fbff2673d7bbd9d9a9aca2.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT created_by, runnable_path,\n CASE WHEN kind = 'script' THEN runnable_id END AS script_hash,\n -- Whether this job PARSED a graph of its own with no version\n -- behind it: the dbt editor refreshing its buffer. That graph is\n -- reachable no other way, so the job pins to it; every other\n -- versionless job keeps answering with the workspace graph.\n EXISTS (SELECT 1 FROM dbt_graph_snapshot g\n WHERE g.workspace_id = $2 AND g.job_id = $1\n AND g.script_hash IS NULL) AS \"editor_graph!\"\n FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "runnable_path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "script_hash", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "editor_graph!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + false, + true, + null, + null + ] + }, + "hash": "f1b13ed0a8b834b0a55bfc7ef6470c2a3aaef44f19fbff2673d7bbd9d9a9aca2" +} diff --git a/backend/.sqlx/query-f22cb6c5443352cfc826c122ff971df2b3725f8551baf29223b22f00f65495d4.json b/backend/.sqlx/query-f22cb6c5443352cfc826c122ff971df2b3725f8551baf29223b22f00f65495d4.json new file mode 100644 index 0000000000..1c764c4987 --- /dev/null +++ b/backend/.sqlx/query-f22cb6c5443352cfc826c122ff971df2b3725f8551baf29223b22f00f65495d4.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, fork_last_event_kind, fork_last_event_origin)\n VALUES ($1, $2, $3, $4, 1, 0, NULL, $5, $6)\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n ahead = workspace_diff.ahead + 1,\n has_changes = NULL,\n fork_last_event_kind = CASE WHEN $6::varchar IS NULL THEN workspace_diff.fork_last_event_kind ELSE $5 END,\n fork_last_event_origin = COALESCE($6, workspace_diff.fork_last_event_origin)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "f22cb6c5443352cfc826c122ff971df2b3725f8551baf29223b22f00f65495d4" +} diff --git a/backend/.sqlx/query-f2bd875385052618533c868bee309de8863a5d10a482f012ee790eda64e88211.json b/backend/.sqlx/query-f2bd875385052618533c868bee309de8863a5d10a482f012ee790eda64e88211.json index 07191c20b8..197229bb4c 100644 --- a/backend/.sqlx/query-f2bd875385052618533c868bee309de8863a5d10a482f012ee790eda64e88211.json +++ b/backend/.sqlx/query-f2bd875385052618533c868bee309de8863a5d10a482f012ee790eda64e88211.json @@ -41,7 +41,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6.json b/backend/.sqlx/query-f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6.json index 12eceb0110..57d80d6aaa 100644 --- a/backend/.sqlx/query-f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6.json +++ b/backend/.sqlx/query-f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6.json @@ -39,7 +39,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } @@ -86,7 +87,8 @@ "java", "duckdb", "ruby", - "rlang" + "rlang", + "dbt" ] } } diff --git a/backend/.sqlx/query-f45962e947ae236266f8eb9b08ca6420ac7e0d548fb59473a2f7ac5be2ee5194.json b/backend/.sqlx/query-f45962e947ae236266f8eb9b08ca6420ac7e0d548fb59473a2f7ac5be2ee5194.json new file mode 100644 index 0000000000..8cae250396 --- /dev/null +++ b/backend/.sqlx/query-f45962e947ae236266f8eb9b08ca6420ac7e0d548fb59473a2f7ac5be2ee5194.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT job_id FROM dbt_run_state\n WHERE workspace_id = $1 AND script_path = $2 AND permissioned_as = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "job_id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "f45962e947ae236266f8eb9b08ca6420ac7e0d548fb59473a2f7ac5be2ee5194" +} diff --git a/backend/.sqlx/query-f5eedc14b082c88a1096e61279faa73793c99a2ab5793bebc5924146f60d19a8.json b/backend/.sqlx/query-f5eedc14b082c88a1096e61279faa73793c99a2ab5793bebc5924146f60d19a8.json new file mode 100644 index 0000000000..0c2078f793 --- /dev/null +++ b/backend/.sqlx/query-f5eedc14b082c88a1096e61279faa73793c99a2ab5793bebc5924146f60d19a8.json @@ -0,0 +1,62 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT asset_kind AS \"asset_kind: windmill_common::assets::AssetKind\", asset_path,\n status::text AS \"status!\", row_count, error\n FROM materialized_partition\n WHERE workspace_id = $1 AND job_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_kind: windmill_common::assets::AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume", + "dbt" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "asset_path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "status!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "row_count", + "type_info": "Int8" + }, + { + "ordinal": 4, + "name": "error", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [ + false, + false, + null, + true, + true + ] + }, + "hash": "f5eedc14b082c88a1096e61279faa73793c99a2ab5793bebc5924146f60d19a8" +} diff --git a/backend/.sqlx/query-f60339bdf63e72c63df45530be4e13380720237a1a310d8acf5e975189981422.json b/backend/.sqlx/query-f60339bdf63e72c63df45530be4e13380720237a1a310d8acf5e975189981422.json new file mode 100644 index 0000000000..ee7d298c23 --- /dev/null +++ b/backend/.sqlx/query-f60339bdf63e72c63df45530be4e13380720237a1a310d8acf5e975189981422.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "on_behalf_of", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "f60339bdf63e72c63df45530be4e13380720237a1a310d8acf5e975189981422" +} diff --git a/backend/.sqlx/query-f6516385c2ff4691aa5b41ad72034b58f7f1deb9526d982ad7e91c2ac93c1f70.json b/backend/.sqlx/query-f6516385c2ff4691aa5b41ad72034b58f7f1deb9526d982ad7e91c2ac93c1f70.json new file mode 100644 index 0000000000..4e3a7e9b7c --- /dev/null +++ b/backend/.sqlx/query-f6516385c2ff4691aa5b41ad72034b58f7f1deb9526d982ad7e91c2ac93c1f70.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, source_last_event_kind, source_last_event_origin)\n SELECT $1, unnest($2::varchar[]), $3, $4, 0, 1, NULL, $5, $6\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n behind = workspace_diff.behind + 1,\n has_changes = NULL,\n source_last_event_kind = CASE WHEN $6::varchar IS NULL THEN workspace_diff.source_last_event_kind ELSE $5 END,\n source_last_event_origin = COALESCE($6, workspace_diff.source_last_event_origin)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "VarcharArray", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "f6516385c2ff4691aa5b41ad72034b58f7f1deb9526d982ad7e91c2ac93c1f70" +} diff --git a/backend/.sqlx/query-f720dc3321c8a351655fbb774cc954c655e4368f7e496e7d05b6beeaf091db95.json b/backend/.sqlx/query-f720dc3321c8a351655fbb774cc954c655e4368f7e496e7d05b6beeaf091db95.json new file mode 100644 index 0000000000..d5e8ddcdd5 --- /dev/null +++ b/backend/.sqlx/query-f720dc3321c8a351655fbb774cc954c655e4368f7e496e7d05b6beeaf091db95.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) FROM dbt_node\n WHERE workspace_id = $1 AND job_id = $2 AND script_hash IS NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "f720dc3321c8a351655fbb774cc954c655e4368f7e496e7d05b6beeaf091db95" +} diff --git a/backend/.sqlx/query-fa160bec1a7eb688f95a5176f64983f5e92cb1e330a5f1611cd5c9ba0681dece.json b/backend/.sqlx/query-fa160bec1a7eb688f95a5176f64983f5e92cb1e330a5f1611cd5c9ba0681dece.json new file mode 100644 index 0000000000..3c009edaff --- /dev/null +++ b/backend/.sqlx/query-fa160bec1a7eb688f95a5176f64983f5e92cb1e330a5f1611cd5c9ba0681dece.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, fork_last_event_kind, fork_last_event_origin)\n VALUES ($1, $2, $3, $4, 1, 0, NULL, $5, $6)\n ON CONFLICT (source_workspace_id, fork_workspace_id, path, kind)\n DO UPDATE SET\n ahead = workspace_diff.ahead + 1,\n has_changes = NULL,\n fork_last_event_kind = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.fork_last_event_origin IS NOT NULL) THEN workspace_diff.fork_last_event_kind ELSE $5 END,\n fork_last_event_origin = CASE WHEN $6::varchar IS NULL OR (NOT $7 AND workspace_diff.fork_last_event_origin IS NOT NULL) THEN workspace_diff.fork_last_event_origin ELSE $6 END", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "fa160bec1a7eb688f95a5176f64983f5e92cb1e330a5f1611cd5c9ba0681dece" +} diff --git a/backend/.sqlx/query-fad10d08b6a7fb4c6974a6dce79028f9bae1b9a9806c9efb2844f7e7e8196f5b.json b/backend/.sqlx/query-fad10d08b6a7fb4c6974a6dce79028f9bae1b9a9806c9efb2844f7e7e8196f5b.json new file mode 100644 index 0000000000..6d0fdd1bd8 --- /dev/null +++ b/backend/.sqlx/query-fad10d08b6a7fb4c6974a6dce79028f9bae1b9a9806c9efb2844f7e7e8196f5b.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH live AS (\n SELECT * FROM (\n SELECT DISTINCT ON (s.path) s.path, s.hash\n FROM script s\n WHERE $5::text IS NULL AND s.workspace_id = $1 AND s.language = 'dbt'\n AND ($3::bigint IS NULL OR s.hash = $3)\n AND ($3::bigint IS NOT NULL OR (s.deleted = false AND s.archived = false))\n ORDER BY s.path, s.created_at DESC\n ) cur\n UNION ALL\n SELECT $5::text, $3::bigint WHERE $5::text IS NOT NULL\n ),\n -- The run's own snapshot when it left one, the version's graph\n -- otherwise. A static descriptor never snapshots, so all of its runs\n -- fall through to the same rows. Existence comes from the marker, not\n -- from a node row: a dynamic run that disabled every model has a\n -- snapshot whose graph is legitimately empty.\n chosen AS (\n -- No visibility check on the job here: reaching this with a job at\n -- all means the caller passed `require_job_read_access` for it, and\n -- re-deciding it under plain RLS can only DISAGREE with that answer\n -- — silently, by falling back to the deployed graph rather than\n -- erroring. A share-link viewer is entitled to the run and would be\n -- shown a different run's model set. See `asset_graph_for`.\n SELECT CASE WHEN $4::uuid IS NOT NULL AND EXISTS (\n SELECT 1 FROM dbt_graph_snapshot g\n WHERE g.workspace_id = $1 AND g.job_id = $4)\n THEN $4::uuid\n ELSE '00000000-0000-0000-0000-000000000000'::uuid END AS job_id\n )\n SELECT p.asset_path AS \"from_path!\", c.asset_path AS \"to_path!\"\n FROM dbt_edge e\n JOIN live l ON l.path = e.script_path\n AND (e.script_hash = l.hash\n OR ($5::text IS NOT NULL AND l.hash IS NULL\n AND e.script_hash IS NULL))\n JOIN chosen ch ON ch.job_id = e.job_id\n -- Gated the same way as the `live` joins above, and for the same\n -- reason twice over: unpinned this folds to a plain equality the\n -- versioned key can bound, and pinned it is the only way an editor\n -- graph's NULL-to-NULL hashes meet at all.\n JOIN dbt_node p ON p.workspace_id = e.workspace_id\n AND p.script_path = e.script_path\n AND (p.script_hash = e.script_hash\n OR ($5::text IS NOT NULL AND e.script_hash IS NULL\n AND p.script_hash IS NULL))\n AND p.job_id = ch.job_id\n AND p.unique_id = e.parent_unique_id\n JOIN dbt_node c ON c.workspace_id = e.workspace_id\n AND c.script_path = e.script_path\n AND (c.script_hash = e.script_hash\n OR ($5::text IS NOT NULL AND e.script_hash IS NULL\n AND c.script_hash IS NULL))\n AND c.job_id = ch.job_id\n AND c.unique_id = e.child_unique_id\n WHERE e.workspace_id = $1\n AND p.asset_path IS NOT NULL AND c.asset_path IS NOT NULL\n -- Tests attach to their model as a badge, not as a lineage edge.\n AND c.resource_type <> 'test'\n -- Scoped by the RELATIONS, like the node provenance above, not by\n -- the producing script's folder: two tables consumed in this\n -- folder but produced by a dbt project outside it would otherwise\n -- both render with their `ref()` edge missing.\n -- Same as the node scope: pinned, `asset` describes the CURRENT\n -- deploy, so gating on it drops the edges of models this version\n -- had and a later one removed. The pinned graph's own edges are\n -- the answer.\n AND ($3::bigint IS NOT NULL OR $5::text IS NOT NULL OR EXISTS (\n SELECT 1 FROM asset a\n WHERE a.workspace_id = $1 AND a.kind = 'dbt'\n AND a.path = c.asset_path\n AND ($2::text IS NULL OR a.usage_path LIKE $2)))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "from_path!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "to_path!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Uuid", + "Text" + ] + }, + "nullable": [ + true, + true + ] + }, + "hash": "fad10d08b6a7fb4c6974a6dce79028f9bae1b9a9806c9efb2844f7e7e8196f5b" +} diff --git a/backend/.sqlx/query-fb2a7f9b797a45351439746e1478545c969d995dc02aaf6673183a73cab67179.json b/backend/.sqlx/query-fb2a7f9b797a45351439746e1478545c969d995dc02aaf6673183a73cab67179.json new file mode 100644 index 0000000000..d74c17a1ed --- /dev/null +++ b/backend/.sqlx/query-fb2a7f9b797a45351439746e1478545c969d995dc02aaf6673183a73cab67179.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT job_id FROM dbt_run_state\n WHERE workspace_id = $1 AND script_path = $2 AND permissioned_as = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "job_id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "fb2a7f9b797a45351439746e1478545c969d995dc02aaf6673183a73cab67179" +} diff --git a/backend/.sqlx/query-fb3f9cfbe874a1f19ccf54660496532ad995ba641f0833babe5d5d597fedd55a.json b/backend/.sqlx/query-fb3f9cfbe874a1f19ccf54660496532ad995ba641f0833babe5d5d597fedd55a.json new file mode 100644 index 0000000000..d2d36be4a5 --- /dev/null +++ b/backend/.sqlx/query-fb3f9cfbe874a1f19ccf54660496532ad995ba641f0833babe5d5d597fedd55a.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COALESCE(\n (SELECT username FROM usr WHERE workspace_id = $1 AND email = $2),\n (SELECT COALESCE(username, email) FROM password WHERE email = $2 AND super_admin = true)\n )", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "coalesce", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "fb3f9cfbe874a1f19ccf54660496532ad995ba641f0833babe5d5d597fedd55a" +} diff --git a/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json deleted file mode 100644 index 484dd23e2a..0000000000 --- a/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT req.id AS \"id!\",\n (CASE\n WHEN usr.email IS NULL THEN 'deleted'\n WHEN workspace.deleted THEN 'archived'\n ELSE 'active'\n END) AS \"status!\"\n FROM unnest($1::text[]) AS req(id)\n LEFT JOIN workspace ON workspace.id = req.id\n LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id!", - "type_info": "Text" - }, - { - "ordinal": 1, - "name": "status!", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "TextArray", - "Text" - ] - }, - "nullable": [ - null, - null - ] - }, - "hash": "fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441" -} diff --git a/backend/.sqlx/query-fcca85a3f80bac6c7c1e6e53bc514c050960c7f8f515c0e601ef486c68c7fcfd.json b/backend/.sqlx/query-fcca85a3f80bac6c7c1e6e53bc514c050960c7f8f515c0e601ef486c68c7fcfd.json new file mode 100644 index 0000000000..777b03cc41 --- /dev/null +++ b/backend/.sqlx/query-fcca85a3f80bac6c7c1e6e53bc514c050960c7f8f515c0e601ef486c68c7fcfd.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of)\n VALUES\n ('test-workspace', 'u/test-user/obo_member', 91001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user'),\n ('test-workspace', 'u/test-user/obo_stranger', 91002, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2'),\n ('test-workspace', 'u/test-user/obo_group', 91003, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'g/all'),\n ('test-workspace', 'u/test-user/obo_superadmin', 91004, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/ext-sa')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "fcca85a3f80bac6c7c1e6e53bc514c050960c7f8f515c0e601ef486c68c7fcfd" +} diff --git a/backend/.sqlx/query-fd790067c02d384aead5ff8f94efb37cc20d2f7a3749d21a1de5642e888b5f42.json b/backend/.sqlx/query-fd790067c02d384aead5ff8f94efb37cc20d2f7a3749d21a1de5642e888b5f42.json deleted file mode 100644 index a39ccd7796..0000000000 --- a/backend/.sqlx/query-fd790067c02d384aead5ff8f94efb37cc20d2f7a3749d21a1de5642e888b5f42.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE resource SET value = $1, edited_at = now() WHERE path = $2 AND workspace_id = $3", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Jsonb", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "fd790067c02d384aead5ff8f94efb37cc20d2f7a3749d21a1de5642e888b5f42" -} diff --git a/backend/.sqlx/query-fddab529cbb59c1dd621113a1018457546eeeaba57f5dd3abac948d60ecd952f.json b/backend/.sqlx/query-fddab529cbb59c1dd621113a1018457546eeeaba57f5dd3abac948d60ecd952f.json new file mode 100644 index 0000000000..35c401fdef --- /dev/null +++ b/backend/.sqlx/query-fddab529cbb59c1dd621113a1018457546eeeaba57f5dd3abac948d60ecd952f.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id, digest)\n VALUES ($1, $2, NULL, $3, 'd')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "fddab529cbb59c1dd621113a1018457546eeeaba57f5dd3abac948d60ecd952f" +} diff --git a/backend/.sqlx/query-ff244d58f154fc3132205fbb8374f01ceb707d9ae0b02bf382d4f5dca73829e2.json b/backend/.sqlx/query-ff244d58f154fc3132205fbb8374f01ceb707d9ae0b02bf382d4f5dca73829e2.json new file mode 100644 index 0000000000..13ed472958 --- /dev/null +++ b/backend/.sqlx/query-ff244d58f154fc3132205fbb8374f01ceb707d9ae0b02bf382d4f5dca73829e2.json @@ -0,0 +1,25 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT digest FROM dbt_graph_snapshot\n WHERE workspace_id = $1 AND script_path = $2 AND script_hash = $3\n AND job_id = $4", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "digest", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Uuid" + ] + }, + "nullable": [ + false + ] + }, + "hash": "ff244d58f154fc3132205fbb8374f01ceb707d9ae0b02bf382d4f5dca73829e2" +} diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 7dd3c795aa..d406dc6dcb 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -100,9 +100,9 @@ dependencies = [ [[package]] name = "aho-corasick" -version = "1.1.4" +version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" dependencies = [ "memchr", ] @@ -178,9 +178,9 @@ dependencies = [ [[package]] name = "android_system_properties" -version = "0.1.5" +version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc" dependencies = [ "libc", ] @@ -243,11 +243,11 @@ checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" [[package]] name = "ar_archive_writer" -version = "0.5.2" +version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4087686b4b0a3427190bae57a1d9a478dbb2d40c5dc1bd6e2b6d797913bdd348" +checksum = "73cd58deff2140a0a8eae87e417bd01db68a33e148aa93d1e8cd837e55e312b6" dependencies = [ - "object", + "object 0.39.1", ] [[package]] @@ -803,7 +803,7 @@ checksum = "850b60ddcc664dcd848f8a2fa8436ab9336e051d6dd2b3f21f897dd8e9c24703" dependencies = [ "base64 0.22.1", "bytes", - "http 1.4.2", + "http 1.5.0", "rand 0.8.5", "reqwest 0.12.28", "serde", @@ -953,7 +953,7 @@ dependencies = [ "bytes", "fastrand 2.5.0", "hex", - "http 1.4.2", + "http 1.5.0", "ring 0.17.14", "time", "tokio", @@ -1016,7 +1016,7 @@ dependencies = [ "bytes", "bytes-utils", "fastrand 2.5.0", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "percent-encoding", "pin-project-lite", @@ -1043,7 +1043,7 @@ dependencies = [ "bytes", "fastrand 2.5.0", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "regex-lite", "tracing", ] @@ -1118,7 +1118,7 @@ dependencies = [ "aws-types", "fastrand 2.5.0", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "regex-lite", "tracing", "url", @@ -1143,7 +1143,7 @@ dependencies = [ "bytes", "fastrand 2.5.0", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "regex-lite", "tracing", ] @@ -1189,7 +1189,7 @@ dependencies = [ "bytes", "fastrand 2.5.0", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "regex-lite", "tracing", ] @@ -1213,7 +1213,7 @@ dependencies = [ "bytes", "fastrand 2.5.0", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "regex-lite", "tracing", ] @@ -1257,7 +1257,7 @@ dependencies = [ "hex", "hmac", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "percent-encoding", "sha2 0.10.9", "time", @@ -1300,7 +1300,7 @@ dependencies = [ "futures-core", "futures-util", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "http-body 0.4.6", "percent-encoding", "pin-project-lite", @@ -1320,7 +1320,7 @@ dependencies = [ "bytes-utils", "futures-core", "futures-util", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "percent-encoding", @@ -1341,7 +1341,7 @@ dependencies = [ "h2 0.3.27", "h2 0.4.15", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "http-body 0.4.6", "hyper 0.14.32", "hyper 1.11.0", @@ -1411,7 +1411,7 @@ dependencies = [ "bytes", "fastrand 2.5.0", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "http-body 0.4.6", "http-body 1.1.0", "http-body-util", @@ -1431,7 +1431,7 @@ dependencies = [ "aws-smithy-types", "bytes", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "pin-project-lite", "tokio", "tracing", @@ -1449,7 +1449,7 @@ dependencies = [ "bytes-utils", "futures-core", "http 0.2.12", - "http 1.4.2", + "http 1.5.0", "http-body 0.4.6", "http-body 1.1.0", "http-body-util", @@ -1507,7 +1507,7 @@ dependencies = [ "axum-core 0.4.5", "bytes", "futures-util", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "itoa", @@ -1535,7 +1535,7 @@ dependencies = [ "bytes", "form_urlencoded", "futures-util", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "hyper 1.11.0", @@ -1568,7 +1568,7 @@ dependencies = [ "async-trait", "bytes", "futures-util", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "mime", @@ -1587,7 +1587,7 @@ checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1" dependencies = [ "bytes", "futures-core", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "mime", @@ -1636,7 +1636,7 @@ dependencies = [ "cfg-if", "libc", "miniz_oxide", - "object", + "object 0.37.3", "rustc-demangle", "windows-link 0.2.1", ] @@ -1671,6 +1671,12 @@ version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" +[[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + [[package]] name = "base64-simd" version = "0.8.0" @@ -1826,9 +1832,9 @@ dependencies = [ [[package]] name = "blake3" -version = "1.8.5" +version = "1.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" +checksum = "76ae7bad254120e9e4c63bafc385310756f90c484eac0e36b8317cf09cb92a77" dependencies = [ "arrayref", "arrayvec", @@ -1907,7 +1913,7 @@ dependencies = [ "futures-core", "futures-util", "hex", - "http 1.4.2", + "http 1.5.0", "http-body-util", "hyper 1.11.0", "hyper-named-pipe", @@ -2123,13 +2129,13 @@ dependencies = [ [[package]] name = "bytemuck_derive" -version = "1.11.0" +version = "1.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f65693059b6b9c588b9f62fed1cedbf0a8b805631457ea162d68f0de186f3de5" +checksum = "fc0e56a716f1e132ff6bf4bdac1c944a3fcdc1cae65f70a4a2a1ac3b401d2d1f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.119", + "syn 3.0.3", ] [[package]] @@ -2270,6 +2276,12 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "cast" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" + [[package]] name = "castaway" version = "0.2.4" @@ -2306,12 +2318,6 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4f4c707c6a209cbe82d10abd08e1ea8995e9ea937d2550646e02798948992be0" -[[package]] -name = "cesu8" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d43a04d8753f35258c91f8ec639f792891f748a1edbd759cf1dcea3382ad83c" - [[package]] name = "cexpr" version = "0.6.0" @@ -2399,9 +2405,9 @@ dependencies = [ [[package]] name = "clang-sys" -version = "1.8.1" +version = "1.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" +checksum = "157a8ba7b480713b56f4c09fd13fc3e0a22a5dfab8097ba61cbc5feef950788a" dependencies = [ "glob", "libc", @@ -2410,9 +2416,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.6.4" +version = "4.6.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91e0c145792ef73a6ad36d27c75ac09f1832222a3c209689d90f534685ee5b7" +checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" dependencies = [ "clap_builder", "clap_derive", @@ -2420,9 +2426,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.6.2" +version = "4.6.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f09628afdcc538b57f3c6341e9c8e9970f18e4a481690a64974d7023bd33548b" +checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" dependencies = [ "anstream", "anstyle", @@ -2933,6 +2939,16 @@ dependencies = [ "darling_macro 0.23.0", ] +[[package]] +name = "darling" +version = "0.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88490bf1b990d87eaaa7ac8aa887f629a08e7359765b4911faf63c3763347d23" +dependencies = [ + "darling_core 0.24.0", + "darling_macro 0.24.0", +] + [[package]] name = "darling_core" version = "0.13.4" @@ -2988,6 +3004,19 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "darling_core" +version = "0.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "084e274f91c482280130e1e34e0b8d6e66776a060d7b6de7b84289ca778868c4" +dependencies = [ + "ident_case", + "proc-macro2", + "quote", + "strsim 0.11.1", + "syn 3.0.3", +] + [[package]] name = "darling_macro" version = "0.13.4" @@ -3032,6 +3061,17 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "darling_macro" +version = "0.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68f5792fa0d41cd2325ce0ffa64f0a340eaebd4971a3a0c5e1ffd2cc488a355e" +dependencies = [ + "darling_core 0.24.0", + "quote", + "syn 3.0.3", +] + [[package]] name = "dashmap" version = "6.2.1" @@ -3048,9 +3088,9 @@ dependencies = [ [[package]] name = "data-encoding" -version = "2.11.0" +version = "2.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" +checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06" [[package]] name = "data-url" @@ -3889,7 +3929,7 @@ dependencies = [ "error_reporter", "h2 0.4.15", "hickory-resolver", - "http 1.4.2", + "http 1.5.0", "http-body-util", "hyper 1.11.0", "hyper-rustls 0.27.9", @@ -4720,22 +4760,22 @@ dependencies = [ [[package]] name = "enum-ordinalize" -version = "4.4.1" +version = "4.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07f808d588c10e464ea6f7d3eaed500049eff30aaac103460f61828c2d65b3eb" +checksum = "89dd01549b09589510cf0647475075d12071456586d70f5c75c98ae2a5537677" dependencies = [ "enum-ordinalize-derive", ] [[package]] name = "enum-ordinalize-derive" -version = "4.4.1" +version = "4.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42e528e2d34ba8a67a1a650b86beae8ef69fc5fdb638016f386b973226590432" +checksum = "a65863d15a4ce2888bd2f0f543cc963d3879c3a022c8ee43f6141d479a3ac815" dependencies = [ "proc-macro2", "quote", - "syn 2.0.119", + "syn 3.0.3", ] [[package]] @@ -5515,9 +5555,9 @@ checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b" [[package]] name = "globset" -version = "0.4.19" +version = "0.4.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e47d37d2ae4464254884b60ab7071be2b876a9c35b696bd018ddcc76847309cd" +checksum = "07c34a9410465b45bd9787443bc7370f37735bad04b0f0cd57ff1a3186c98988" dependencies = [ "aho-corasick", "bstr", @@ -5567,7 +5607,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "de13e62d7e0ffc3eb40a0113ddf753cf6ec741be739164442b08893db4f9bfca" dependencies = [ "google-cloud-token", - "http 1.4.2", + "http 1.5.0", "thiserror 1.0.69", "tokio", "tokio-retry2", @@ -5688,7 +5728,7 @@ dependencies = [ "fnv", "futures-core", "futures-sink", - "http 1.4.2", + "http 1.5.0", "indexmap 2.14.0", "slab", "tokio", @@ -5795,7 +5835,7 @@ dependencies = [ "base64 0.22.1", "bytes", "headers-core", - "http 1.4.2", + "http 1.5.0", "httpdate", "mime", "sha1", @@ -5807,7 +5847,7 @@ version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "54b4a22553d4242c49fddb9ba998a99962b5cc6f22cb5a3482bec22522403ce4" dependencies = [ - "http 1.4.2", + "http 1.5.0", ] [[package]] @@ -5842,7 +5882,7 @@ checksum = "629d8f3bbeda9d148036d6b0de0a3ab947abd08ce90626327fc3547a49d59d97" dependencies = [ "dirs 6.0.0", "futures", - "http 1.4.2", + "http 1.5.0", "indicatif", "libc", "log", @@ -5976,9 +6016,9 @@ dependencies = [ [[package]] name = "http" -version = "1.4.2" +version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" dependencies = [ "bytes", "itoa", @@ -6002,7 +6042,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" dependencies = [ "bytes", - "http 1.4.2", + "http 1.5.0", ] [[package]] @@ -6013,7 +6053,7 @@ checksum = "e9f41fd6a08e4d4ec69df65976da761afd5ad5e58a9d4acb46bd1c953a9e3ff2" dependencies = [ "bytes", "futures-core", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "pin-project-lite", ] @@ -6039,7 +6079,7 @@ dependencies = [ "async-compression", "bstr", "futures", - "http 1.4.2", + "http 1.5.0", "http-body-util", "hyper 1.11.0", "hyper-rustls 0.26.0", @@ -6100,7 +6140,7 @@ dependencies = [ "futures-channel", "futures-core", "h2 0.4.15", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "httparse", "httpdate", @@ -6120,7 +6160,7 @@ dependencies = [ "bytes", "futures-util", "headers", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "hyper-rustls 0.27.9", "hyper-util", @@ -6167,7 +6207,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a0bea761b46ae2b24eb4aef630d8d1c398157b6fc29e6350ecf090a0b70c952c" dependencies = [ "futures-util", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "hyper-util", "log", @@ -6185,7 +6225,7 @@ version = "0.27.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" dependencies = [ - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "hyper-util", "log", @@ -6251,7 +6291,7 @@ dependencies = [ "bytes", "futures-channel", "futures-util", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "hyper 1.11.0", "ipnet", @@ -6527,9 +6567,9 @@ dependencies = [ [[package]] name = "ipnet" -version = "2.12.0" +version = "2.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" +checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78" [[package]] name = "ipnetwork" @@ -6608,27 +6648,32 @@ checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" [[package]] name = "jni" -version = "0.21.1" +version = "0.22.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" dependencies = [ - "cesu8", "cfg-if", "combine", - "jni-sys 0.3.1", + "jni-macros", + "jni-sys", "log", - "thiserror 1.0.69", + "simd_cesu8", + "thiserror 2.0.19", "walkdir", - "windows-sys 0.45.0", + "windows-link 0.2.1", ] [[package]] -name = "jni-sys" -version = "0.3.1" +name = "jni-macros" +version = "0.22.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41a652e1f9b6e0275df1f15b32661cf0d4b78d4d87ddec5e0c3c20f097433258" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" dependencies = [ - "jni-sys 0.4.1", + "proc-macro2", + "quote", + "rustc_version 0.4.1", + "simd_cesu8", + "syn 2.0.119", ] [[package]] @@ -6662,11 +6707,12 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.80" +version = "0.3.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "852f13bec5eba4ba9afbeb93fd7c13fe56147f055939ae21c43a29a0ecb2702e" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" dependencies = [ - "once_cell", + "cfg-if", + "futures-util", "wasm-bindgen", ] @@ -6782,9 +6828,9 @@ checksum = "a4933f3f57a8e9d9da04db23fb153356ecaf00cbd14aee46279c33dc80925c37" [[package]] name = "kstat-rs" -version = "0.2.4" +version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27964e4632377753acb0898ce6f28770d50cbca1339200ae63d700cff97b5c2b" +checksum = "52d0786643a0b49f595bd1fc81e1c1aa7bad8555bc820c6892f4d28cb20cf210" dependencies = [ "libc", "thiserror 1.0.69", @@ -6815,7 +6861,7 @@ dependencies = [ "either", "futures", "home", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "hyper 1.11.0", @@ -6849,7 +6895,7 @@ dependencies = [ "chrono", "derive_more 2.1.1", "form_urlencoded", - "http 1.4.2", + "http 1.5.0", "json-patch", "k8s-openapi", "schemars 0.8.22", @@ -7066,9 +7112,9 @@ dependencies = [ [[package]] name = "libredox" -version = "0.1.18" +version = "0.1.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c943259e342f1e06ff2da7a83eabdfe7f92ce10262688dbf1895ff0b3e6e4652" +checksum = "2026a5056764a10b2bf5d56488cba40da507f5493a6a429340e2004d9ed085fa" dependencies = [ "bitflags 2.13.1", "libc", @@ -7200,9 +7246,9 @@ dependencies = [ [[package]] name = "lru" -version = "0.18.1" +version = "0.18.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b6180140927ee907000b0aa540091f6ea512ead4447c92b8fc35bc72788a5a6" +checksum = "5d2f2f9b4ba7e6b24d95e7e899329d35be83bcded72c8540cdd5368932d1d90a" dependencies = [ "hashbrown 0.17.1", ] @@ -7265,19 +7311,19 @@ checksum = "dae608c151f68243f2b000364e1f7b186d9c29845f7d2d85bd31b9ad77ad552b" [[package]] name = "macro_rules_attribute" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65049d7923698040cd0b1ddcced9b0eb14dd22c5f86ae59c3740eab64a676520" +checksum = "b3ae8f6d608c795738406608304d30a2dfbdc8e58e44f7ba43236da5208ded3c" dependencies = [ "macro_rules_attribute-proc_macro", - "paste", + "pastey", ] [[package]] name = "macro_rules_attribute-proc_macro" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "670fdfda89751bc4a84ac13eaa63e205cf0fd22b4c9a5fbfa085b63c1f1d3a30" +checksum = "fc04a4c58212d57930a24bf47d3fa87485264a3a054e9c10e042eb373573ad3c" [[package]] name = "magic-crypt" @@ -7640,7 +7686,7 @@ dependencies = [ "bytes", "encoding_rs", "futures-util", - "http 1.4.2", + "http 1.5.0", "httparse", "memchr", "mime", @@ -7686,7 +7732,7 @@ dependencies = [ "futures-sink", "futures-util", "keyed_priority_queue", - "lru 0.18.1", + "lru 0.18.2", "mysql_common", "native-tls", "pem 3.0.6", @@ -8147,7 +8193,7 @@ dependencies = [ "base64 0.22.1", "chrono", "getrandom 0.2.17", - "http 1.4.2", + "http 1.5.0", "rand 0.8.5", "reqwest 0.12.28", "serde", @@ -8167,6 +8213,15 @@ dependencies = [ "memchr", ] +[[package]] +name = "object" +version = "0.39.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b" +dependencies = [ + "memchr", +] + [[package]] name = "object_store" version = "0.12.0" @@ -8178,7 +8233,7 @@ dependencies = [ "chrono", "form_urlencoded", "futures", - "http 1.4.2", + "http 1.5.0", "http-body-util", "httparse", "humantime", @@ -8277,6 +8332,12 @@ dependencies = [ "pkg-config", ] +[[package]] +name = "oorandom" +version = "11.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e" + [[package]] name = "opaque-debug" version = "0.3.1" @@ -8294,7 +8355,7 @@ dependencies = [ "dyn-clone", "ed25519-dalek", "hmac", - "http 1.4.2", + "http 1.5.0", "itertools 0.10.5", "log", "oauth2", @@ -8421,7 +8482,7 @@ checksum = "10a8a7f5f6ba7c1b286c2fbca0454eaba116f63bbe69ed250b642d36fbb04d80" dependencies = [ "async-trait", "bytes", - "http 1.4.2", + "http 1.5.0", "opentelemetry 0.27.1", ] @@ -8433,7 +8494,7 @@ checksum = "50f6639e842a97dbea8886e3439710ae463120091e2e064518ba8e716e6ac36d" dependencies = [ "async-trait", "bytes", - "http 1.4.2", + "http 1.5.0", "opentelemetry 0.30.0", "reqwest 0.12.28", ] @@ -8446,7 +8507,7 @@ checksum = "91cf61a1868dacc576bf2b2a1c3e9ab150af7272909e80085c3173384fe11f76" dependencies = [ "async-trait", "futures-core", - "http 1.4.2", + "http 1.5.0", "opentelemetry 0.27.1", "opentelemetry-http 0.27.0", "opentelemetry-proto 0.27.0", @@ -8465,7 +8526,7 @@ version = "0.30.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dbee664a43e07615731afc539ca60c6d9f1a9425e25ca09c57bc36c87c55852b" dependencies = [ - "http 1.4.2", + "http 1.5.0", "opentelemetry 0.30.0", "opentelemetry-http 0.30.0", "opentelemetry-proto 0.30.0", @@ -9453,9 +9514,9 @@ dependencies = [ [[package]] name = "psm" -version = "0.1.31" +version = "0.1.32" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea" +checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622" dependencies = [ "ar_archive_writer", "cc", @@ -9983,9 +10044,9 @@ dependencies = [ [[package]] name = "regex-automata" -version = "0.4.16" +version = "0.4.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad" +checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" dependencies = [ "aho-corasick", "memchr", @@ -10041,7 +10102,7 @@ dependencies = [ "futures-core", "futures-util", "h2 0.4.15", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "hyper 1.11.0", @@ -10072,16 +10133,16 @@ dependencies = [ "url", "wasm-bindgen", "wasm-bindgen-futures", - "wasm-streams", + "wasm-streams 0.4.2", "web-sys", "webpki-roots 1.0.9", ] [[package]] name = "reqwest" -version = "0.13.1" +version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04e9018c9d814e5f30cc16a0f03271aeab3571e609612d9fe78c1aa8d11c2f62" +checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" dependencies = [ "base64 0.22.1", "bytes", @@ -10089,7 +10150,7 @@ dependencies = [ "futures-core", "futures-util", "h2 0.4.15", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "hyper 1.11.0", @@ -10118,7 +10179,7 @@ dependencies = [ "url", "wasm-bindgen", "wasm-bindgen-futures", - "wasm-streams", + "wasm-streams 0.5.0", "web-sys", ] @@ -10130,8 +10191,8 @@ checksum = "07bc3f1384cffa4f274dad2d4ddd73aed32fed8f786d96c6be8aa4e5fd3c3b58" dependencies = [ "anyhow", "async-trait", - "http 1.4.2", - "reqwest 0.13.1", + "http 1.5.0", + "reqwest 0.13.4", "serde", "thiserror 2.0.19", "tower-service", @@ -10147,9 +10208,9 @@ dependencies = [ "async-trait", "futures", "getrandom 0.2.17", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", - "reqwest 0.13.1", + "reqwest 0.13.4", "reqwest-middleware", "retry-policies", "thiserror 2.0.19", @@ -10243,24 +10304,23 @@ dependencies = [ [[package]] name = "rmcp" -version = "0.15.0" +version = "3.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bef41ebc9ebed2c1b1d90203e9d1756091e8a00bbc3107676151f39868ca0ee" +checksum = "ad26b216c966e987e80e86daf784a455c039c43d98575ceed57b8faa259e5695" dependencies = [ "async-trait", - "axum 0.8.9", - "base64 0.22.1", + "base64 0.23.1", "bytes", "chrono", "futures", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "oauth2", "pastey", "pin-project-lite", - "rand 0.9.0", - "reqwest 0.12.28", + "rand 0.10.2", + "reqwest 0.13.4", "rmcp-macros", "schemars 1.2.2", "serde", @@ -10278,15 +10338,15 @@ dependencies = [ [[package]] name = "rmcp-macros" -version = "0.15.0" +version = "3.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e88ad84b8b6237a934534a62b379a5be6388915663c0cc598ceb9b3292bbbfe" +checksum = "737d947bcfd946fae6a179a4ef6487be6dcf25c930c2393856b820f1386e52a6" dependencies = [ - "darling 0.23.0", + "darling 0.24.0", "proc-macro2", "quote", "serde_json", - "syn 2.0.119", + "syn 3.0.3", ] [[package]] @@ -10647,9 +10707,9 @@ dependencies = [ [[package]] name = "rustls-platform-verifier" -version = "0.6.2" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d99feebc72bae7ab76ba994bb5e121b8d83d910ca40b36e0921f53becc41784" +checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0" dependencies = [ "core-foundation 0.10.1", "core-foundation-sys", @@ -11454,6 +11514,16 @@ version = "0.3.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea" +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version 0.4.1", + "simdutf8", +] + [[package]] name = "simdutf8" version = "0.1.5" @@ -11492,9 +11562,9 @@ checksum = "1b6709c7b6754dca1311b3c73e79fcce40dd414c782c66d88e8823030093b02b" [[package]] name = "sketches-ddsketch" -version = "0.4.0" +version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05e40b6cf54d988dc1a2223531b969c9a9e30906ad90ef64890c27b4bfbb46ea" +checksum = "513c3f5f732bfd6fbb187619c2dfe9d2f25f1a2976f01d575f0fd329d565df56" dependencies = [ "serde", ] @@ -11899,9 +11969,9 @@ checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" [[package]] name = "stacker" -version = "0.1.24" +version = "0.1.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190" +checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967" dependencies = [ "cc", "cfg-if", @@ -12912,9 +12982,9 @@ dependencies = [ [[package]] name = "time" -version = "0.3.54" +version = "0.3.55" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3e1d5e639ff6bab73cb6885cc7e7b1de96c3f32c68ec55f3952614bec1092244" +checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134" dependencies = [ "deranged", "num-conv", @@ -13262,7 +13332,7 @@ dependencies = [ "bytes", "futures-core", "futures-sink", - "http 1.4.2", + "http 1.5.0", "httparse", "rand 0.8.5", "ring 0.17.14", @@ -13350,7 +13420,7 @@ dependencies = [ "bytes", "flate2", "h2 0.4.15", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "hyper 1.11.0", @@ -13382,7 +13452,7 @@ dependencies = [ "base64 0.22.1", "bytes", "h2 0.4.15", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "hyper 1.11.0", @@ -13450,7 +13520,7 @@ dependencies = [ "axum-core 0.5.6", "cookie", "futures-util", - "http 1.4.2", + "http 1.5.0", "parking_lot", "pin-project-lite", "tower-layer", @@ -13469,7 +13539,7 @@ dependencies = [ "bytes", "futures-core", "futures-util", - "http 1.4.2", + "http 1.5.0", "http-body 1.1.0", "http-body-util", "mime", @@ -13704,7 +13774,7 @@ dependencies = [ "byteorder", "bytes", "data-encoding", - "http 1.4.2", + "http 1.5.0", "httparse", "log", "native-tls", @@ -13726,7 +13796,7 @@ dependencies = [ "byteorder", "bytes", "data-encoding", - "http 1.4.2", + "http 1.5.0", "httparse", "log", "native-tls", @@ -14211,9 +14281,9 @@ checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" [[package]] name = "wasm-bindgen" -version = "0.2.103" +version = "0.2.126" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab10a69fbd0a177f5f649ad4d8d3305499c42bab9aef2f7ff592d0ec8f833819" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" dependencies = [ "cfg-if", "once_cell", @@ -14223,38 +14293,21 @@ dependencies = [ "wasm-bindgen-shared", ] -[[package]] -name = "wasm-bindgen-backend" -version = "0.2.103" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0bb702423545a6007bbc368fde243ba47ca275e549c8a28617f56f6ba53b1d1c" -dependencies = [ - "bumpalo", - "log", - "proc-macro2", - "quote", - "syn 2.0.119", - "wasm-bindgen-shared", -] - [[package]] name = "wasm-bindgen-futures" -version = "0.4.53" +version = "0.4.76" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0b221ff421256839509adbb55998214a70d829d3a28c69b4a6672e9d2a42f67" +checksum = "c62df1340f32221cb9c54d6a27b030e3dba64361d4a95bed55f9aacb44da291d" dependencies = [ - "cfg-if", "js-sys", - "once_cell", "wasm-bindgen", - "web-sys", ] [[package]] name = "wasm-bindgen-macro" -version = "0.2.103" +version = "0.2.126" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc65f4f411d91494355917b605e1480033152658d71f722a90647f56a70c88a0" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -14262,50 +14315,65 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.103" +version = "0.2.126" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffc003a991398a8ee604a401e194b6b3a39677b3173d6e74495eb51b82e99a32" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" dependencies = [ + "bumpalo", "proc-macro2", "quote", "syn 2.0.119", - "wasm-bindgen-backend", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" -version = "0.2.103" +version = "0.2.126" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "293c37f4efa430ca14db3721dfbe48d8c33308096bd44d80ebaa775ab71ba1cf" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" dependencies = [ "unicode-ident", ] [[package]] name = "wasm-bindgen-test" -version = "0.3.53" +version = "0.3.76" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee0a0f5343de9221a0d233b04520ed8dc2e6728dce180b1dcd9288ec9d9fa3c" +checksum = "2a0d555ca874445df8d314f94f5c948a4e74e5418f332c89f660a3d8310a96f4" dependencies = [ + "async-trait", + "cast", "js-sys", + "libm", "minicov", + "nu-ansi-term", + "num-traits", + "oorandom", + "serde", + "serde_json", "wasm-bindgen", "wasm-bindgen-futures", "wasm-bindgen-test-macro", + "wasm-bindgen-test-shared", ] [[package]] name = "wasm-bindgen-test-macro" -version = "0.3.53" +version = "0.3.76" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a369369e4360c2884c3168d22bded735c43cccae97bbc147586d4b480edd138d" +checksum = "94eb68555b95bcea5e8cf4abe280b529049479fa995bfc23734af96a6aedc120" dependencies = [ "proc-macro2", "quote", "syn 2.0.119", ] +[[package]] +name = "wasm-bindgen-test-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c31d56021e873866c968588ed85ccdf56db5c426e44afdb4618c39895104b920" + [[package]] name = "wasm-streams" version = "0.4.2" @@ -14319,6 +14387,19 @@ dependencies = [ "web-sys", ] +[[package]] +name = "wasm-streams" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + [[package]] name = "wasm_dep_analyzer" version = "0.3.0" @@ -14345,9 +14426,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.80" +version = "0.3.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fbe734895e869dc429d78c4b433f8d17d95f8d05317440b4fad5ab2d33e596dc" +checksum = "8622dcb61c0bcc9fffa6938bed81210af2da9a7e4a1a834b2e37a59b6dfb6141" dependencies = [ "js-sys", "wasm-bindgen", @@ -14370,7 +14451,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "974fa1e325e6cc5327de8887f189a441fcff4f8eedcd31ec87f0ef0cc5283fbc" dependencies = [ "bytes", - "http 1.4.2", + "http 1.5.0", "thiserror 2.0.19", "url", ] @@ -14489,7 +14570,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-nats", @@ -14513,7 +14594,7 @@ dependencies = [ "prometheus", "rand 0.9.0", "rdkafka", - "reqwest 0.13.1", + "reqwest 0.13.4", "rumqttc", "rustls 0.23.35", "serde", @@ -14574,7 +14655,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.775.2" +version = "1.783.0" dependencies = [ "async-stream", "async-trait", @@ -14587,10 +14668,10 @@ dependencies = [ "bytes", "eventsource-stream", "futures", - "http 1.4.2", + "http 1.5.0", "lazy_static", "mime_guess", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sqlx", @@ -14607,7 +14688,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14620,7 +14701,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "argon2", @@ -14649,7 +14730,7 @@ dependencies = [ "git-version", "hex", "hmac", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "indexmap 2.14.0", "itertools 0.14.0", @@ -14666,10 +14747,11 @@ dependencies = [ "pin-project", "postgres-native-tls 0.5.3", "prometheus", + "quick-xml", "quick_cache", "rand 0.9.0", "regex", - "reqwest 0.13.1", + "reqwest 0.13.4", "rsa", "rust-embed", "rustls 0.23.35", @@ -14759,11 +14841,11 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "lazy_static", "quick_cache", @@ -14782,14 +14864,16 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", "serde", "serde_json", "sqlx", + "tokio", "tracing", + "uuid", "windmill-api-auth", "windmill-common", "windmill-parser-sql-asset", @@ -14797,17 +14881,17 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "axum 0.8.9", "chrono", - "http 1.4.2", + "http 1.5.0", "itertools 0.14.0", "jsonwebtoken 8.3.0", "lazy_static", "quick_cache", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sqlx", @@ -14823,7 +14907,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.775.2" +version = "1.783.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14833,7 +14917,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14850,7 +14934,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14872,7 +14956,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14881,7 +14965,7 @@ dependencies = [ "candle-transformers", "hf-hub", "lazy_static", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sqlx", @@ -14895,7 +14979,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14911,7 +14995,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14927,12 +15011,13 @@ dependencies = [ "windmill-common", "windmill-dep-map", "windmill-git-sync", + "windmill-native-triggers", "windmill-queue", ] [[package]] name = "windmill-api-groups" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14953,7 +15038,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14967,7 +15052,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-nats", @@ -14981,7 +15066,7 @@ dependencies = [ "hmac", "rand 0.9.0", "rdkafka", - "reqwest 0.13.1", + "reqwest 0.13.4", "rmcp", "rumqttc", "serde", @@ -15002,13 +15087,13 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "axum 0.8.9", "base64 0.22.1", "chrono", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "lazy_static", "serde", @@ -15027,11 +15112,11 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "flate2", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sqlx", @@ -15045,11 +15130,11 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "axum 0.8.9", - "http 1.4.2", + "http 1.5.0", "indexmap 2.14.0", "itertools 0.14.0", "lazy_static", @@ -15067,7 +15152,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -15087,18 +15172,18 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", "futures", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "itertools 0.14.0", "lazy_static", "prometheus", "quick_cache", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sql-builder", @@ -15110,6 +15195,7 @@ dependencies = [ "windmill-common", "windmill-dep-map", "windmill-git-sync", + "windmill-native-triggers", "windmill-object-store", "windmill-parser", "windmill-parser-py", @@ -15124,7 +15210,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -15152,7 +15238,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.775.2" +version = "1.783.0" dependencies = [ "lazy_static", "serde", @@ -15164,13 +15250,13 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.775.2" +version = "1.783.0" dependencies = [ "argon2", "axum 0.8.9", "chrono", "dashmap", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "lazy_static", "serde", @@ -15189,7 +15275,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", @@ -15203,13 +15289,13 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.775.2" +version = "1.783.0" dependencies = [ "axum 0.8.9", "chrono", "futures", "hex", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "lazy_static", "magic-crypt", @@ -15238,7 +15324,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.775.2" +version = "1.783.0" dependencies = [ "chrono", "lazy_static", @@ -15252,7 +15338,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -15271,7 +15357,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.775.2" +version = "1.783.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -15332,7 +15418,7 @@ dependencies = [ "quick_cache", "rand 0.9.0", "regex", - "reqwest 0.13.1", + "reqwest 0.13.4", "reqwest-middleware", "reqwest-retry", "rsa", @@ -15375,7 +15461,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.775.2" +version = "1.783.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -15394,7 +15480,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.775.2" +version = "1.783.0" dependencies = [ "regex", "serde", @@ -15409,7 +15495,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -15433,7 +15519,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "futures", @@ -15450,7 +15536,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.775.2" +version = "1.783.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -15466,15 +15552,15 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", "chrono", "futures", - "http 1.4.2", + "http 1.5.0", "oauth2", - "reqwest 0.12.28", + "reqwest 0.13.4", "rmcp", "serde", "serde_json", @@ -15487,7 +15573,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -15496,10 +15582,10 @@ dependencies = [ "base64 0.22.1", "chrono", "hmac", - "http 1.4.2", + "http 1.5.0", "itertools 0.14.0", "lazy_static", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sha2 0.10.9", @@ -15518,7 +15604,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "arc-swap", @@ -15543,7 +15629,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-stream", @@ -15561,7 +15647,7 @@ dependencies = [ "lazy_static", "object_store", "quick_cache", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sqlx", @@ -15577,7 +15663,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "futures", @@ -15595,7 +15681,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.775.2" +version = "1.783.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -15604,7 +15690,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -15616,7 +15702,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde_json", @@ -15628,7 +15714,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "gosyn", @@ -15640,7 +15726,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -15652,7 +15738,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde_json", @@ -15664,7 +15750,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "nu-parser", @@ -15675,7 +15761,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -15686,7 +15772,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -15698,7 +15784,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15709,7 +15795,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-recursion", @@ -15731,7 +15817,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde_json", @@ -15743,7 +15829,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -15757,7 +15843,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15774,7 +15860,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -15787,7 +15873,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde", @@ -15799,7 +15885,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -15817,7 +15903,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15833,7 +15919,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15849,18 +15935,21 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", + "lazy_static", + "regex", "serde", "serde_json", + "serde_yml", "windmill-parser", "yaml-rust", ] [[package]] name = "windmill-queue" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-recursion", @@ -15881,7 +15970,7 @@ dependencies = [ "quick_cache", "rand 0.9.0", "regex", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "serde_urlencoded", @@ -15899,7 +15988,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "const_format", @@ -15923,7 +16012,7 @@ dependencies = [ "lazy_static", "rcgen", "regex", - "reqwest 0.13.1", + "reqwest 0.13.4", "rustls 0.23.35", "serde", "serde_json", @@ -15939,7 +16028,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.775.2" +version = "1.783.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15950,21 +16039,22 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-recursion", "axum 0.8.9", "base64 0.22.1", "chrono", + "dashmap", "futures", "hex", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "lazy_static", "magic-crypt", "quick_cache", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sha2 0.10.9", @@ -15984,7 +16074,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16008,18 +16098,18 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", "axum 0.8.9", "chrono", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "itertools 0.14.0", "lazy_static", "rand 0.9.0", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sql-builder", @@ -16041,7 +16131,7 @@ dependencies = [ [[package]] name = "windmill-trigger-amqp" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16068,7 +16158,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16078,12 +16168,12 @@ dependencies = [ "chrono", "constant_time_eq 0.3.1", "hex", - "http 1.4.2", + "http 1.5.0", "itertools 0.14.0", "lazy_static", "quick_cache", "rand 0.9.0", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sha2 0.10.9", @@ -16101,7 +16191,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16121,7 +16211,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16131,12 +16221,12 @@ dependencies = [ "chrono", "google-cloud-googleapis", "google-cloud-pubsub", - "http 1.4.2", + "http 1.5.0", "itertools 0.14.0", "jsonwebtoken 8.3.0", "lazy_static", "quick_cache", - "reqwest 0.13.1", + "reqwest 0.13.4", "serde", "serde_json", "sqlx", @@ -16155,7 +16245,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16166,7 +16256,7 @@ dependencies = [ "futures", "hex", "hmac", - "http 1.4.2", + "http 1.5.0", "hyper 1.11.0", "itertools 0.14.0", "lazy_static", @@ -16191,7 +16281,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16214,7 +16304,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16238,7 +16328,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-nats", @@ -16262,7 +16352,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16297,7 +16387,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", @@ -16325,14 +16415,14 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-trait", "axum 0.8.9", "base64 0.22.1", "futures", - "http 1.4.2", + "http 1.5.0", "itertools 0.14.0", "serde", "serde_json", @@ -16350,7 +16440,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "bitflags 2.13.1", @@ -16369,7 +16459,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-once-cell", @@ -16423,7 +16513,7 @@ dependencies = [ "rand 0.9.0", "rcgen", "regex", - "reqwest 0.13.1", + "reqwest 0.13.4", "reqwest-middleware", "rsa", "rust_decimal", @@ -16432,6 +16522,7 @@ dependencies = [ "rustls-pemfile 2.2.0", "serde", "serde_json", + "serde_yml", "sha2 0.10.9", "sqlx", "tar", @@ -16484,7 +16575,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.775.2" +version = "1.783.0" dependencies = [ "bytes", "futures", @@ -16758,15 +16849,6 @@ dependencies = [ "windows-link 0.2.1", ] -[[package]] -name = "windows-sys" -version = "0.45.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0" -dependencies = [ - "windows-targets 0.42.2", -] - [[package]] name = "windows-sys" version = "0.48.0" @@ -16812,21 +16894,6 @@ dependencies = [ "windows-link 0.2.1", ] -[[package]] -name = "windows-targets" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071" -dependencies = [ - "windows_aarch64_gnullvm 0.42.2", - "windows_aarch64_msvc 0.42.2", - "windows_i686_gnu 0.42.2", - "windows_i686_msvc 0.42.2", - "windows_x86_64_gnu 0.42.2", - "windows_x86_64_gnullvm 0.42.2", - "windows_x86_64_msvc 0.42.2", -] - [[package]] name = "windows-targets" version = "0.48.5" @@ -16884,12 +16951,6 @@ dependencies = [ "windows-link 0.1.3", ] -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8" - [[package]] name = "windows_aarch64_gnullvm" version = "0.48.5" @@ -16908,12 +16969,6 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" -[[package]] -name = "windows_aarch64_msvc" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43" - [[package]] name = "windows_aarch64_msvc" version = "0.48.5" @@ -16932,12 +16987,6 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" -[[package]] -name = "windows_i686_gnu" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f" - [[package]] name = "windows_i686_gnu" version = "0.48.5" @@ -16968,12 +17017,6 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" -[[package]] -name = "windows_i686_msvc" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060" - [[package]] name = "windows_i686_msvc" version = "0.48.5" @@ -16992,12 +17035,6 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" -[[package]] -name = "windows_x86_64_gnu" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36" - [[package]] name = "windows_x86_64_gnu" version = "0.48.5" @@ -17016,12 +17053,6 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3" - [[package]] name = "windows_x86_64_gnullvm" version = "0.48.5" @@ -17040,12 +17071,6 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" -[[package]] -name = "windows_x86_64_msvc" -version = "0.42.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" - [[package]] name = "windows_x86_64_msvc" version = "0.48.5" @@ -17250,18 +17275,18 @@ dependencies = [ [[package]] name = "zerocopy" -version = "0.8.55" +version = "0.8.56" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5a105cd7b140f6eeec8acff2ea38135d3cab283ada58540f629fe51e46696eb" +checksum = "556764e583adb45a9f8d413c2a147fa7e8d821e48e12b14fd560b607998b75eb" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.55" +version = "0.8.56" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fe976fb70c78cd64cccfe3a6fc142244e8a77b70959b30faf9d0ac37ee228eb" +checksum = "f2ab42fc20575779bd240faa45f94a74256f755c0fa9e89f0ede20d91d0cdfc1" dependencies = [ "proc-macro2", "quote", @@ -17356,9 +17381,9 @@ dependencies = [ [[package]] name = "zlib-rs" -version = "0.6.6" +version = "0.6.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b142a20ec14a91d5bc708c1dc21b080c550113d8aa77afa29635673a65dd02c5" +checksum = "34b31d188d9d685a4f9c7b46d6e36631b07058d2cfe190267adce54dc230bf12" [[package]] name = "zmij" diff --git a/backend/Cargo.toml b/backend/Cargo.toml index a6fffe33f4..9163904548 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.775.2" +version = "1.783.0" authors.workspace = true edition.workspace = true @@ -88,7 +88,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.775.2" +version = "1.783.0" authors = ["Ruben Fiszel "] edition = "2021" @@ -467,6 +467,7 @@ php-parser-rs = { git = "https://github.com/php-rust-tools/parser", rev = "ec4cb cron = "^0" mail-send = { version = "0.4.0", features = ["builder"], default-features=false } urlencoding = "^2" +quick-xml = { version = "^0.37", features = ["serialize"] } url = { version = "^2" , features = ["serde"]} async-oauth2 = "0.5.1" reqwest = { version = "^0.13", features = ["json", "stream", "gzip", "multipart", "query", "form"] } @@ -567,7 +568,12 @@ dashmap = "6.1.0" gosyn = "0.2.6" bytes = "1.4.0" gethostname = "0.4.3" -wasm-bindgen = "=0.2.103" +# Not pinned exactly: the excluded `parsers/windmill-parser-wasm` workspace pins +# =0.2.103 to match its vendored `cli/wasm/*` artifacts, yet path-depends on +# sibling parser crates that inherit this requirement from here. Two exact pins +# on the same semver range cannot both resolve, so keep this a range and let each +# workspace's lockfile settle it (here, js-sys forces 0.2.108). +wasm-bindgen = "0.2" serde-wasm-bindgen = "^0" wasm-bindgen-test = "^0" convert_case = "0.6.0" @@ -611,7 +617,7 @@ nkeys = "0.4.4" nu-parser = { version = "0.101.0", default-features = false } globset = "0.4.16" croner = "2.2.0" -rmcp = { version = "=0.15.0", features = ["client", "transport-streamable-http-client", "transport-streamable-http-client-reqwest"] } +rmcp = { version = "=3.1.0", features = ["client", "transport-streamable-http-client", "transport-streamable-http-client-reqwest"] } rquickjs = { version = "0.11", features = ["futures", "parallel", "macro"] } process-wrap = { version = "8.2.1", features = ["tokio1"] } diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index f008a12d4c..f6c4f387b4 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -aa05ca8e97fc8265cd724753a80db37f83243254 +f0df8b82c4c089d384423ed64b8504506084820d diff --git a/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py b/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py index 20ea1a0a6c..3ed9cc259d 100644 --- a/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py +++ b/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py @@ -398,7 +398,10 @@ def schema_to_rust_value(schema: Optional[Dict[str, Any]]) -> str: """Convert a schema dict to a Rust serde_json::json! expression.""" if schema is None: return "None" - return f"Some(serde_json::json!({json.dumps(schema, indent=8)}))" + # ensure_ascii=False: json.dumps would escape a non-ASCII character in a + # description as \uXXXX, which Rust rejects — its string literals spell it + # \u{XXXX}. Emitting the character itself is valid in both. + return f"Some(serde_json::json!({json.dumps(schema, indent=8, ensure_ascii=False)}))" def build_tool_description(operation: Dict[str, Any], method: str, path: str) -> str: """Build the MCP tool description from OpenAPI summary and description.""" @@ -423,7 +426,11 @@ def find_mcp_tools(spec: Dict[str, Any]) -> List[Dict[str, Any]]: if isinstance(operation, dict) and operation.get('x-mcp-tool') is True: # Extract tool information tool = { - 'name': operation.get('operationId', f"{method}_{path.replace('/', '_').replace('{', '').replace('}', '')}"), + # The name an agent sees. Defaults to the operationId, which + # every generated client is keyed on, so an endpoint whose + # tool should read differently overrides it here rather than + # by renaming the operation. + 'name': operation.get('x-mcp-tool-name') or operation.get('operationId', f"{method}_{path.replace('/', '_').replace('{', '').replace('}', '')}"), 'description': build_tool_description(operation, method, path), 'instructions': operation.get('x-mcp-instructions', ''), 'path': path, @@ -436,7 +443,16 @@ def find_mcp_tools(spec: Dict[str, Any]) -> List[Dict[str, Any]]: 'include_query_params': operation.get('x-mcp-tool-include-query-params'), } tools.append(tool) - + + # A tool name used to be an operationId, which OpenAPI already keeps unique. + # `x-mcp-tool-name` gives that up, and a duplicate would be silent: a token's + # `mcp:endpoints:` resolves by first match, so which endpoint it reaches + # would depend on the order of this file. + names = [t['name'] for t in tools] + duplicates = sorted({n for n in names if names.count(n) > 1}) + if duplicates: + raise ValueError(f"duplicate MCP tool name(s): {', '.join(duplicates)}") + return tools def generate_typescript_code(tools: List[Dict[str, Any]], spec: Dict[str, Any], base_path: str = "") -> str: diff --git a/backend/migrations/20260725084224_add_dbt_lang.down.sql b/backend/migrations/20260725084224_add_dbt_lang.down.sql new file mode 100644 index 0000000000..d2f607c5b8 --- /dev/null +++ b/backend/migrations/20260725084224_add_dbt_lang.down.sql @@ -0,0 +1 @@ +-- Add down migration script here diff --git a/backend/migrations/20260725084224_add_dbt_lang.up.sql b/backend/migrations/20260725084224_add_dbt_lang.up.sql new file mode 100644 index 0000000000..ec99678460 --- /dev/null +++ b/backend/migrations/20260725084224_add_dbt_lang.up.sql @@ -0,0 +1,2 @@ +ALTER TYPE SCRIPT_LANG ADD VALUE IF NOT EXISTS 'dbt'; +UPDATE config SET config = jsonb_set(config, '{worker_tags}', config->'worker_tags' || '["dbt"]'::jsonb) WHERE name = 'worker__default' AND config @> '{"worker_tags": ["deno", "python3", "go", "bash", "powershell", "dependency", "flow", "hub", "other", "bun", "php", "rust", "ansible", "csharp", "nu", "java", "duckdb", "ruby", "rlang"]}'::jsonb AND NOT config->'worker_tags' @> '"dbt"'::jsonb; diff --git a/backend/migrations/20260725084314_dbt_runtime.down.sql b/backend/migrations/20260725084314_dbt_runtime.down.sql new file mode 100644 index 0000000000..93fd7ef4b5 --- /dev/null +++ b/backend/migrations/20260725084314_dbt_runtime.down.sql @@ -0,0 +1,7 @@ +DROP TABLE IF EXISTS dbt_run_progress; +DROP TABLE IF EXISTS dbt_run_state; +DROP TABLE IF EXISTS dbt_graph_snapshot; +DROP TABLE IF EXISTS dbt_edge; +DROP TABLE IF EXISTS dbt_node; + +ALTER TABLE workspace_settings DROP COLUMN IF EXISTS dbt_warehouses; diff --git a/backend/migrations/20260725084314_dbt_runtime.up.sql b/backend/migrations/20260725084314_dbt_runtime.up.sql new file mode 100644 index 0000000000..c40441933d --- /dev/null +++ b/backend/migrations/20260725084314_dbt_runtime.up.sql @@ -0,0 +1,218 @@ +-- The dbt runtime's tables: the parsed manifest that becomes the asset graph, +-- the retry state a `dbt retry` resumes from, and a run's live per-model +-- progress. + +-- Physical-relation asset kind. Identity is the relation itself +-- (`//`, the warehouse named as the workspace +-- configures it), never the producing tool, so a dbt mart and a native script +-- reading the same table resolve to one node and the lineage is one graph +-- across the boundary (docs/dbt-runtime.md, decision 11). +ALTER TYPE ASSET_KIND ADD VALUE IF NOT EXISTS 'dbt'; + +-- Warehouses configured once for the workspace, so a dbt project needs no +-- connection knowledge to run: the descriptor names one by NAME (or nothing, for +-- `main`) and the value here points at the RESOURCE that holds the credentials, +-- exactly as `large_file_storage` does for buckets. A flat map keyed by name, +-- which is also what asset identity keys on. +-- +-- {"main": {"resource_path": "$res:u/admin/wh", "target": "prod"}, +-- "eu": {"resource_path": "$res:u/admin/wh_eu"}} +ALTER TABLE workspace_settings ADD COLUMN IF NOT EXISTS dbt_warehouses JSONB; + +-- Parsed dbt manifest, one row per dbt node. The full manifest.json is not +-- stored -- only the fields the asset graph renders. +-- +-- Keyed by (path, version, job). The VERSION because each deployed version of a +-- script keeps its own graph, so a finished run can be shown the project as it +-- was rather than whatever is deployed today. The JOB because a dynamic +-- descriptor -- a `{{ }}` placeholder in `vars` -- can resolve to a different +-- set of models per run, and those runs re-ingest; the zero UUID means "the +-- version's own graph, as deployed", which is what a static descriptor writes +-- once and every one of its runs reads. A sentinel rather than NULL because +-- `job_id` is part of the key and Postgres does not treat two NULLs as the same +-- key, so each re-ingest would add a row set instead of replacing one. +CREATE TABLE IF NOT EXISTS dbt_node ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + script_path VARCHAR(255) NOT NULL, + script_hash BIGINT NOT NULL, + job_id UUID NOT NULL DEFAULT '00000000-0000-0000-0000-000000000000', + -- dbt's own node id, e.g. `model.jaffle_shop.customers`. Stable across runs + -- and the join key for dbt_edge and for run_results ingestion. + unique_id TEXT NOT NULL, + resource_type TEXT NOT NULL, + name TEXT NOT NULL, + -- The physical relation this node produces, spelled as the matching `asset` + -- row's path (`//`). NULL for nodes with no + -- relation: ephemeral models, tests. + asset_path TEXT, + materialized TEXT, + -- Windmill's equivalent write strategy (replace | append | merge | scd2), + -- NULL when dbt's materialization has no analogue (view, ephemeral). + materialize_strategy TEXT, + unique_key TEXT, + tags TEXT[] NOT NULL DEFAULT '{}', + description TEXT, + -- Test nodes only: the generic test name (`unique`, `not_null`, + -- `accepted_values`, `relationships`, or a package/custom test's own name), + -- the column it is attached to, its rendered kwargs and dbt's severity. + -- Severity is stored as dbt spells it; readers compare case-insensitively + -- because dbt-core 1.x echoes the author's casing while 2.x uppercases it. + test_kind TEXT, + test_column TEXT, + test_args JSONB, + severity TEXT, + attached_node TEXT, + -- Declared per-column metadata (name -> description), the column sets the + -- asset graph shows. dbt's manifest carries no column-to-column lineage, so + -- this is not a column lineage graph. + columns JSONB, + freshness JSONB, + -- The model's SQL as written, for the graph to render. `dbt parse` fills it; + -- `compiled_code` would need a `dbt compile`, which no phase runs. + raw_code TEXT, + -- Its path inside the dbt project, e.g. `models/staging/stg_orders.sql`. + original_file_path TEXT, + ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, script_path, script_hash, job_id, unique_id), + -- Composite because `script`'s key is (workspace_id, hash). A version's graph + -- dies with the version and nothing has to sweep it. + CONSTRAINT dbt_node_script_fkey FOREIGN KEY (workspace_id, script_hash) + REFERENCES script (workspace_id, hash) ON DELETE CASCADE ON UPDATE CASCADE +); + +-- `ref()` / `source()` lineage, straight from the manifest's parent_map. +CREATE TABLE IF NOT EXISTS dbt_edge ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + script_path VARCHAR(255) NOT NULL, + script_hash BIGINT NOT NULL, + job_id UUID NOT NULL DEFAULT '00000000-0000-0000-0000-000000000000', + parent_unique_id TEXT NOT NULL, + child_unique_id TEXT NOT NULL, + ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, script_path, script_hash, job_id, parent_unique_id, child_unique_id), + CONSTRAINT dbt_edge_script_fkey FOREIGN KEY (workspace_id, script_hash) + REFERENCES script (workspace_id, hash) ON DELETE CASCADE ON UPDATE CASCADE +); + +-- One row per stored graph, so a snapshot's EXISTENCE does not depend on it +-- having any nodes: a dynamic run can disable every model, and that empty graph +-- has to be distinguishable from a run that stored nothing -- otherwise the run +-- page falls back to the deployed models and shows what the run never built. +-- +-- It is also where the content digest lives, once. A run whose digest matches +-- the version's writes no snapshot at all: marking a descriptor dynamic is +-- conservative (a `{{ }}` in `vars` says the arguments reach dbt, not that they +-- change which models exist), so the usual dynamic run resolves to exactly the +-- graph the deploy stored. +CREATE TABLE IF NOT EXISTS dbt_graph_snapshot ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + script_path VARCHAR(255) NOT NULL, + script_hash BIGINT NOT NULL, + job_id UUID NOT NULL, + digest TEXT NOT NULL, + -- On the DEPLOYED row only: the relation root the last ingest to write this + -- row resolved. Written by every ingest that is not a run's own snapshot, + -- including one that publishes no ownership — a version that cannot claim the + -- path would otherwise record nothing and compare against a stale root. + -- + -- The drift check needs "where do the current usages point", and no other + -- row answers it: the deploy's own root goes stale the moment a run at a + -- moved profile republishes, and "the newest ingest" is wrong because a run + -- whose graph matches the deploy's stores nothing at all. + relation_root_at_last_ingest TEXT, + ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, script_path, script_hash, job_id), + -- Same cascade as the rows it stands for. A marker outliving its nodes is + -- read as a snapshot that has none, and its digest still answers the + -- suppression check. + CONSTRAINT dbt_graph_snapshot_script_fkey FOREIGN KEY (workspace_id, script_hash) + REFERENCES script (workspace_id, hash) ON DELETE CASCADE ON UPDATE CASCADE +); + +-- What a `dbt retry` resumes from. Keyed by path, not by version: there is one +-- saved run per script, and `identity` -- the project digest, warehouse and +-- engine -- is what refuses a resume that no longer describes the same run. +CREATE TABLE IF NOT EXISTS dbt_run_state ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + script_path VARCHAR(255) NOT NULL, + -- Part of the key: a retry replaces the caller's arguments with the saved + -- ones, so state written by one principal must not be restorable by another + -- — that would hand them the literal `select` and `vars` of a run they were + -- never entitled to see. + permissioned_as VARCHAR(255) NOT NULL, + identity TEXT NOT NULL, + -- The invocation's job arguments. `dbt retry` reuses the original selection + -- and vars, so the graph refresh and the build must agree with them rather + -- than with the retry request's. + args JSONB NOT NULL DEFAULT '{}'::jsonb, + -- Text rather than jsonb: nothing queries inside it, and this is highly + -- repetitive JSON that TOAST's compression handles well (about nine to one). + run_results TEXT NOT NULL, + job_id UUID, + -- Whether those results hold a node `dbt retry` would rebuild. A run that + -- failed before building anything, or succeeded outright, is still saved — + -- restoring it is how a retry can say the run succeeded rather than that no + -- state exists — but nothing may offer it as resumable. + retryable BOOLEAN NOT NULL DEFAULT false, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, script_path, permissioned_as) +); + +-- Live per-model progress for one RUN. +-- +-- `materialized_partition` cannot answer this: its key is the relation, one row +-- per table, and its `job_id` names only the last writer. Two runs of one +-- project building the same models take that row from each other, so a progress +-- read filtered by job loses nodes and flickers between states. That table is +-- left as it is -- the current state of a relation is what the pipeline canvas +-- and fork defer read, and one row per relation is right for them. +CREATE TABLE IF NOT EXISTS dbt_run_progress ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + job_id UUID NOT NULL, + asset_kind ASSET_KIND NOT NULL, + asset_path VARCHAR(255) NOT NULL, + status MATERIALIZATION_STATUS NOT NULL, + row_count BIGINT, + error TEXT, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, job_id, asset_kind, asset_path) +); + +-- Resolve a physical relation back to the dbt node that produces it: the asset +-- graph renders from `asset` rows and needs the dbt provenance per node. +CREATE INDEX IF NOT EXISTS idx_dbt_node_asset_path + ON dbt_node (workspace_id, asset_path) WHERE asset_path IS NOT NULL; + +-- No foreign key from the per-run rows to `v2_job`: three were deliberately +-- dropped from it for the write amplification they cost on the hottest table in +-- the system. Growth is bounded by age instead and pruned by the dbt runs +-- themselves, so no background sweep has to learn about these tables -- hence +-- an index per age sweep, and one per "does this job have a snapshot" lookup. +CREATE INDEX IF NOT EXISTS idx_dbt_node_job ON dbt_node (job_id) + WHERE job_id <> '00000000-0000-0000-0000-000000000000'; +CREATE INDEX IF NOT EXISTS idx_dbt_node_run_age ON dbt_node (ingested_at) + WHERE job_id <> '00000000-0000-0000-0000-000000000000'; +CREATE INDEX IF NOT EXISTS idx_dbt_edge_run_age ON dbt_edge (ingested_at) + WHERE job_id <> '00000000-0000-0000-0000-000000000000'; +CREATE INDEX IF NOT EXISTS idx_dbt_graph_snapshot_job + ON dbt_graph_snapshot (workspace_id, job_id); +CREATE INDEX IF NOT EXISTS idx_dbt_graph_snapshot_age + ON dbt_graph_snapshot (ingested_at) + WHERE job_id <> '00000000-0000-0000-0000-000000000000'; +CREATE INDEX IF NOT EXISTS idx_dbt_run_progress_updated_at + ON dbt_run_progress (updated_at); + +-- All of these are written on a user_db transaction (SET LOCAL ROLE +-- windmill_user/windmill_admin), so they must be granted explicitly rather than +-- relying on ALTER DEFAULT PRIVILEGES, which only covers objects created by the +-- role that set them (see 20260720131744). +GRANT ALL ON dbt_node TO windmill_user; +GRANT ALL ON dbt_node TO windmill_admin; +GRANT ALL ON dbt_edge TO windmill_user; +GRANT ALL ON dbt_edge TO windmill_admin; +GRANT ALL ON dbt_graph_snapshot TO windmill_user; +GRANT ALL ON dbt_graph_snapshot TO windmill_admin; +GRANT ALL ON dbt_run_state TO windmill_user; +GRANT ALL ON dbt_run_state TO windmill_admin; +GRANT ALL ON dbt_run_progress TO windmill_user; +GRANT ALL ON dbt_run_progress TO windmill_admin; diff --git a/backend/migrations/20260727145314_materialized_partition_job_id_index.down.sql b/backend/migrations/20260727145314_materialized_partition_job_id_index.down.sql new file mode 100644 index 0000000000..9b33bff978 --- /dev/null +++ b/backend/migrations/20260727145314_materialized_partition_job_id_index.down.sql @@ -0,0 +1 @@ +DROP INDEX IF EXISTS idx_materialized_partition_job; diff --git a/backend/migrations/20260727145314_materialized_partition_job_id_index.up.sql b/backend/migrations/20260727145314_materialized_partition_job_id_index.up.sql new file mode 100644 index 0000000000..6bcdc246ae --- /dev/null +++ b/backend/migrations/20260727145314_materialized_partition_job_id_index.up.sql @@ -0,0 +1,9 @@ +-- `job_id` leads neither the primary key nor the asset/status index, both shaped +-- for the per-relation lookups that came first — but the closing sweep a dbt run +-- makes (`terminalize_running_relations`) reads and writes by it, settling the +-- models a cancelled or timed-out run left `running`. +-- Small table until a partitioned asset is backfilled, then one row per slice +-- per asset. +CREATE INDEX IF NOT EXISTS idx_materialized_partition_job + ON materialized_partition (workspace_id, job_id) + WHERE job_id IS NOT NULL; diff --git a/backend/migrations/20260731161812_backfill_datatablemigration_git_sync_include_type.down.sql b/backend/migrations/20260731161812_backfill_datatablemigration_git_sync_include_type.down.sql new file mode 100644 index 0000000000..e1b90875dc --- /dev/null +++ b/backend/migrations/20260731161812_backfill_datatablemigration_git_sync_include_type.down.sql @@ -0,0 +1,37 @@ +-- Strip `datatablemigration` back out of every include_type, at both levels. +-- Lossy in the same way the up is: a config that had opted in deliberately is +-- indistinguishable from one the backfill touched. + +UPDATE workspace_settings +SET git_sync = jsonb_set( + git_sync, + '{include_type}', + (git_sync->'include_type') - 'datatablemigration' + ) +WHERE jsonb_typeof(git_sync->'include_type') = 'array' + AND git_sync->'include_type' @> '"datatablemigration"'::jsonb; + +UPDATE workspace_settings ws +SET git_sync = jsonb_set(ws.git_sync, '{repositories}', updated.repositories) +FROM ( + SELECT + s.workspace_id, + jsonb_agg( + CASE + WHEN repo->'settings'->'include_type' @> '"datatablemigration"'::jsonb + THEN jsonb_set( + repo, + '{settings,include_type}', + (repo->'settings'->'include_type') - 'datatablemigration' + ) + ELSE repo + END + ORDER BY idx + ) AS repositories + FROM workspace_settings s, + LATERAL jsonb_array_elements(s.git_sync->'repositories') WITH ORDINALITY AS t(repo, idx) + WHERE jsonb_typeof(s.git_sync->'repositories') = 'array' + GROUP BY s.workspace_id +) AS updated +WHERE ws.workspace_id = updated.workspace_id + AND ws.git_sync->'repositories' IS DISTINCT FROM updated.repositories; diff --git a/backend/migrations/20260731161812_backfill_datatablemigration_git_sync_include_type.up.sql b/backend/migrations/20260731161812_backfill_datatablemigration_git_sync_include_type.up.sql new file mode 100644 index 0000000000..3ec18bf607 --- /dev/null +++ b/backend/migrations/20260731161812_backfill_datatablemigration_git_sync_include_type.up.sql @@ -0,0 +1,45 @@ +-- Data table SQL migrations became a git-sync object type (`datatablemigration`). +-- Existing settings were written before it existed, so every workspace would read +-- as "migrations opted out" until someone re-saved the form. Opt in the configs +-- that already sync something, at both levels an include_type can live: the +-- workspace-level default and each repository's own settings. +-- +-- An absent or empty include_type means "inherit / not configured yet" and is left +-- alone; the defaults the UI and CLI now write already carry the new type. + +UPDATE workspace_settings +SET git_sync = jsonb_set( + git_sync, + '{include_type}', + (git_sync->'include_type') || '"datatablemigration"'::jsonb + ) +WHERE jsonb_typeof(git_sync->'include_type') = 'array' + AND jsonb_array_length(git_sync->'include_type') > 0 + AND NOT (git_sync->'include_type' @> '"datatablemigration"'::jsonb); + +UPDATE workspace_settings ws +SET git_sync = jsonb_set(ws.git_sync, '{repositories}', updated.repositories) +FROM ( + SELECT + s.workspace_id, + jsonb_agg( + CASE + WHEN jsonb_typeof(repo->'settings'->'include_type') = 'array' + AND jsonb_array_length(repo->'settings'->'include_type') > 0 + AND NOT (repo->'settings'->'include_type' @> '"datatablemigration"'::jsonb) + THEN jsonb_set( + repo, + '{settings,include_type}', + (repo->'settings'->'include_type') || '"datatablemigration"'::jsonb + ) + ELSE repo + END + ORDER BY idx + ) AS repositories + FROM workspace_settings s, + LATERAL jsonb_array_elements(s.git_sync->'repositories') WITH ORDINALITY AS t(repo, idx) + WHERE jsonb_typeof(s.git_sync->'repositories') = 'array' + GROUP BY s.workspace_id +) AS updated +WHERE ws.workspace_id = updated.workspace_id + AND ws.git_sync->'repositories' IS DISTINCT FROM updated.repositories; diff --git a/backend/migrations/20260731220637_on_behalf_of.down.sql b/backend/migrations/20260731220637_on_behalf_of.down.sql new file mode 100644 index 0000000000..f4947b27bf --- /dev/null +++ b/backend/migrations/20260731220637_on_behalf_of.down.sql @@ -0,0 +1,3 @@ +-- Add down migration script here +ALTER TABLE script DROP COLUMN IF EXISTS on_behalf_of; +ALTER TABLE flow DROP COLUMN IF EXISTS on_behalf_of; diff --git a/backend/migrations/20260731220637_on_behalf_of.up.sql b/backend/migrations/20260731220637_on_behalf_of.up.sql new file mode 100644 index 0000000000..ca2c6fd884 --- /dev/null +++ b/backend/migrations/20260731220637_on_behalf_of.up.sql @@ -0,0 +1,6 @@ +-- Add up migration script here +-- Authorization identity a script/flow runs as. NULL means "not recorded"; the migration that +-- follows this one backfills every row that had an on_behalf_of_email, so by the end of this +-- release a NULL principal means the runnable runs as its caller. +ALTER TABLE script ADD COLUMN IF NOT EXISTS on_behalf_of VARCHAR(255); +ALTER TABLE flow ADD COLUMN IF NOT EXISTS on_behalf_of VARCHAR(255); diff --git a/backend/migrations/20260801043001_backfill_on_behalf_of.down.sql b/backend/migrations/20260801043001_backfill_on_behalf_of.down.sql new file mode 100644 index 0000000000..bcd8d2cbd5 --- /dev/null +++ b/backend/migrations/20260801043001_backfill_on_behalf_of.down.sql @@ -0,0 +1,33 @@ +-- Add down migration script here +-- The column was never dropped, so this only re-derives it from the principal for the rows the +-- up migration backfilled. One naming a group gets that group's synthetic address, which is what +-- the column held before. + +CREATE FUNCTION pg_temp.email_from_permissioned_as(w_id VARCHAR, permissioned_as VARCHAR) +RETURNS VARCHAR AS $$ + -- Prefix first, as every reader decides: a group name or an email-shaped username may + -- itself contain '@', so only an unprefixed value is an address. + SELECT CASE + WHEN $2 LIKE 'g/%' + THEN 'group-' || substring($2 from 3) || '@windmill.dev' + -- Mirrors the up migration's `u/` arm, superadmin fallback included: one acting + -- outside their workspaces has no usr row, and losing their address here would + -- leave the previous runtime unable to authenticate the runnable. + WHEN $2 LIKE 'u/%' + THEN COALESCE( + (SELECT u.email FROM usr u WHERE u.workspace_id = $1 AND u.username = substring($2 from 3)), + (SELECT p.email FROM password p WHERE p.super_admin + AND (p.username = substring($2 from 3) OR p.email = substring($2 from 3)) + ORDER BY p.email LIMIT 1), + substring($2 from 3) || '@unknown.windmill.dev') + ELSE $2 + END; +$$ LANGUAGE SQL STABLE; + +UPDATE script SET on_behalf_of_email = + pg_temp.email_from_permissioned_as(workspace_id, on_behalf_of) + WHERE on_behalf_of IS NOT NULL; + +UPDATE flow SET on_behalf_of_email = + pg_temp.email_from_permissioned_as(workspace_id, on_behalf_of) + WHERE on_behalf_of IS NOT NULL; diff --git a/backend/migrations/20260801043001_backfill_on_behalf_of.up.sql b/backend/migrations/20260801043001_backfill_on_behalf_of.up.sql new file mode 100644 index 0000000000..fc27c23999 --- /dev/null +++ b/backend/migrations/20260801043001_backfill_on_behalf_of.up.sql @@ -0,0 +1,63 @@ +-- Add up migration script here +-- The permissioned_as becomes the identity this release reads; the email is derived from it at +-- read time, as triggers have always done. Backfilling it also retires the fallback to +-- created_by / edited_by, so a runnable deployed before the column existed starts running +-- as the user its on_behalf_of_email always named rather than as whoever last deployed it. +-- +-- `on_behalf_of_email` stays for now: a worker predating this release resolves a script or flow +-- through `get_script_info_for_hash` / `get_latest_hash_for_path`, which select that column, and +-- workers are expected to lag the server. Deploys keep writing it until every live worker is new +-- (MIN_VERSION_SUPPORTS_ON_BEHALF_OF_PRINCIPAL); a later release stops writing it and drops it. +-- +-- That later migration MUST re-run this backfill before dropping: server pods are mixed for the +-- minute or two a rollout takes, and one still on the previous release writes only the address — +-- leaving a runnable deployed in that window with no principal, which reads as no identity at +-- all and runs it as its caller. Re-deriving picks those up; dropping without it makes them +-- permanent. + +-- Mirrors `users::permissioned_as_from_email`: a real account wins over the synthetic group +-- namespace, which is not reserved and may be a user's own address. +CREATE FUNCTION pg_temp.permissioned_as_from_email(w_id VARCHAR, email VARCHAR) +RETURNS VARCHAR AS $$ + SELECT COALESCE( + -- `username_to_permissioned_as`: an email-shaped username is its own principal unless + -- it contains a slash, which a reader would split on. + (SELECT CASE WHEN u.username LIKE '%@%' AND u.username NOT LIKE '%/%' THEN u.username + ELSE 'u/' || u.username END + FROM usr u WHERE u.workspace_id = $1 AND u.email = $2), + -- A superadmin acting outside their workspaces has no usr row. + (SELECT CASE WHEN COALESCE(p.username, p.email) LIKE '%@%' + AND COALESCE(p.username, p.email) NOT LIKE '%/%' + THEN COALESCE(p.username, p.email) + ELSE 'u/' || COALESCE(p.username, p.email) END + FROM password p WHERE p.email = $2 AND p.super_admin), + (SELECT 'g/' || g.name FROM group_ g + WHERE g.workspace_id = $1 + AND $2 = 'group-' || g.name || '@windmill.dev') + ); +$$ LANGUAGE SQL STABLE; + +-- A principal wider than `v2_job.permissioned_as` could not be enqueued, so it is not recorded +-- at all — the runnable falls back to running as its caller until someone picks an identity the +-- deploy path accepts. Only an address-derived principal can reach that width; a username is +-- capped at 50. +UPDATE script SET on_behalf_of = + pg_temp.permissioned_as_from_email(workspace_id, on_behalf_of_email) + WHERE on_behalf_of_email IS NOT NULL AND on_behalf_of IS NULL + AND length(pg_temp.permissioned_as_from_email(workspace_id, on_behalf_of_email)) <= 55; + +UPDATE flow SET on_behalf_of = + pg_temp.permissioned_as_from_email(workspace_id, on_behalf_of_email) + WHERE on_behalf_of_email IS NOT NULL AND on_behalf_of IS NULL + AND length(pg_temp.permissioned_as_from_email(workspace_id, on_behalf_of_email)) <= 55; + +-- Drafts carry the same pair in their value. +UPDATE draft SET value = to_json(jsonb_set( + to_jsonb(value), + ARRAY['on_behalf_of'], + to_jsonb(pg_temp.permissioned_as_from_email(workspace_id, value->>'on_behalf_of_email')))) + WHERE typ IN ('script', 'flow') + AND value->>'on_behalf_of_email' IS NOT NULL + AND value->>'on_behalf_of' IS NULL + AND length(pg_temp.permissioned_as_from_email(workspace_id, value->>'on_behalf_of_email')) <= 55; + diff --git a/backend/migrations/20260801060114_drop_azure_trigger_email.down.sql b/backend/migrations/20260801060114_drop_azure_trigger_email.down.sql new file mode 100644 index 0000000000..cf51c37487 --- /dev/null +++ b/backend/migrations/20260801060114_drop_azure_trigger_email.down.sql @@ -0,0 +1,23 @@ +-- Rebuild email from permissioned_as, mirroring +-- windmill_common::users::get_email_from_permissioned_as. + +ALTER TABLE azure_trigger ADD COLUMN email VARCHAR(255) NOT NULL DEFAULT ''; + +UPDATE azure_trigger t SET email = CASE + WHEN t.permissioned_as LIKE 'u/%' THEN COALESCE( + (SELECT u.email FROM usr u + WHERE u.workspace_id = t.workspace_id + AND u.username = SUBSTRING(t.permissioned_as FROM 3)), + (SELECT p.email FROM password p + WHERE p.super_admin + AND (p.username = SUBSTRING(t.permissioned_as FROM 3) + OR p.email = SUBSTRING(t.permissioned_as FROM 3)) + LIMIT 1), + SUBSTRING(t.permissioned_as FROM 3) || '@unknown.windmill.dev' + ) + WHEN t.permissioned_as LIKE 'g/%' + THEN 'group-' || SUBSTRING(t.permissioned_as FROM 3) || '@windmill.dev' + ELSE t.permissioned_as +END; + +ALTER TABLE azure_trigger ALTER COLUMN email DROP DEFAULT; diff --git a/backend/migrations/20260801060114_drop_azure_trigger_email.up.sql b/backend/migrations/20260801060114_drop_azure_trigger_email.up.sql new file mode 100644 index 0000000000..293822e25c --- /dev/null +++ b/backend/migrations/20260801060114_drop_azure_trigger_email.up.sql @@ -0,0 +1,4 @@ +-- azure_trigger.email duplicated an address that permissioned_as already +-- determines, and which every other trigger table derives at fire time. + +ALTER TABLE azure_trigger DROP COLUMN email; diff --git a/backend/migrations/20260801121717_dbt_editor_preview_graph.down.sql b/backend/migrations/20260801121717_dbt_editor_preview_graph.down.sql new file mode 100644 index 0000000000..ada8fa319c --- /dev/null +++ b/backend/migrations/20260801121717_dbt_editor_preview_graph.down.sql @@ -0,0 +1,26 @@ +-- The editor graphs go first: they are exactly the rows the restored NOT NULL +-- would reject, and they are throwaway by construction. +DELETE FROM dbt_node WHERE script_hash IS NULL; +DELETE FROM dbt_edge WHERE script_hash IS NULL; +DELETE FROM dbt_graph_snapshot WHERE script_hash IS NULL; + +DROP INDEX IF EXISTS idx_dbt_graph_snapshot_editor_path; +ALTER TABLE dbt_graph_snapshot DROP COLUMN IF EXISTS permissioned_as; + +DROP INDEX IF EXISTS dbt_node_editor_key; +DROP INDEX IF EXISTS dbt_node_versioned_key; +ALTER TABLE dbt_node ALTER COLUMN script_hash SET NOT NULL; +ALTER TABLE dbt_node ADD CONSTRAINT dbt_node_pkey + PRIMARY KEY (workspace_id, script_path, script_hash, job_id, unique_id); + +DROP INDEX IF EXISTS dbt_edge_editor_key; +DROP INDEX IF EXISTS dbt_edge_versioned_key; +ALTER TABLE dbt_edge ALTER COLUMN script_hash SET NOT NULL; +ALTER TABLE dbt_edge ADD CONSTRAINT dbt_edge_pkey + PRIMARY KEY (workspace_id, script_path, script_hash, job_id, parent_unique_id, child_unique_id); + +DROP INDEX IF EXISTS dbt_graph_snapshot_editor_key; +DROP INDEX IF EXISTS dbt_graph_snapshot_versioned_key; +ALTER TABLE dbt_graph_snapshot ALTER COLUMN script_hash SET NOT NULL; +ALTER TABLE dbt_graph_snapshot ADD CONSTRAINT dbt_graph_snapshot_pkey + PRIMARY KEY (workspace_id, script_path, script_hash, job_id); diff --git a/backend/migrations/20260801121717_dbt_editor_preview_graph.up.sql b/backend/migrations/20260801121717_dbt_editor_preview_graph.up.sql new file mode 100644 index 0000000000..b43950517b --- /dev/null +++ b/backend/migrations/20260801121717_dbt_editor_preview_graph.up.sql @@ -0,0 +1,62 @@ +-- A graph parsed from the EDITOR's buffer, which names no deployed version. +-- +-- The two provenances that existed keyed on one: a version's own graph (the +-- zero-UUID `job_id`) and a run's snapshot of a version. A buffer refresh is +-- neither. It cannot borrow the deployed version's hash — the buffer differs +-- from it, which is the point, and a project being written has no deployed +-- version at all — so `script_hash` becomes NULL and the graph is keyed to the +-- preview job that parsed it, readable only back through that job and never +-- through the path. +-- +-- The foreign key to `script` is left exactly as written: it is MATCH SIMPLE, +-- so a NULL in either column satisfies it. A versioned row still dies with its +-- version and a version-less one is simply outside its reach. +-- +-- Two partial unique indexes rather than one with NULLS NOT DISTINCT, which +-- needs Postgres 15: a versioned graph is keyed by its version, a buffer parse +-- by its job alone (a preview job id is unique on its own). +ALTER TABLE dbt_node DROP CONSTRAINT dbt_node_pkey; +ALTER TABLE dbt_node ALTER COLUMN script_hash DROP NOT NULL; +CREATE UNIQUE INDEX IF NOT EXISTS dbt_node_versioned_key + ON dbt_node (workspace_id, script_path, script_hash, job_id, unique_id) + WHERE script_hash IS NOT NULL; +CREATE UNIQUE INDEX IF NOT EXISTS dbt_node_editor_key + ON dbt_node (workspace_id, job_id, unique_id) + WHERE script_hash IS NULL; + +ALTER TABLE dbt_edge DROP CONSTRAINT dbt_edge_pkey; +ALTER TABLE dbt_edge ALTER COLUMN script_hash DROP NOT NULL; +CREATE UNIQUE INDEX IF NOT EXISTS dbt_edge_versioned_key + ON dbt_edge (workspace_id, script_path, script_hash, job_id, parent_unique_id, child_unique_id) + WHERE script_hash IS NOT NULL; +CREATE UNIQUE INDEX IF NOT EXISTS dbt_edge_editor_key + ON dbt_edge (workspace_id, job_id, parent_unique_id, child_unique_id) + WHERE script_hash IS NULL; + +ALTER TABLE dbt_graph_snapshot DROP CONSTRAINT dbt_graph_snapshot_pkey; +ALTER TABLE dbt_graph_snapshot ALTER COLUMN script_hash DROP NOT NULL; +CREATE UNIQUE INDEX IF NOT EXISTS dbt_graph_snapshot_versioned_key + ON dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id) + WHERE script_hash IS NOT NULL; +CREATE UNIQUE INDEX IF NOT EXISTS dbt_graph_snapshot_editor_key + ON dbt_graph_snapshot (workspace_id, job_id) + WHERE script_hash IS NULL; + +-- On an EDITOR graph only: the principal whose parse wrote it. +-- +-- Retention for these is a count per script rather than a clock, and the scope +-- has to be something the caller cannot choose. A preview's path IS the +-- caller's to choose and needs only `jobs:run`, so bounding by path alone would +-- let one caller's parses evict the graphs of whoever is actually editing that +-- script. `permissioned_as` is the execution principal the queue derived, which +-- is the same reason `dbt_run_state` keys on it. +ALTER TABLE dbt_graph_snapshot ADD COLUMN IF NOT EXISTS permissioned_as VARCHAR(255); + +-- An editor graph is bounded per (path, principal) rather than by age: the +-- newest few are kept and the rest dropped as each refresh lands, so this is the +-- index that ordering reads. The instance-wide age sweep every dbt run already +-- performs (`job_id <> DEPLOYED_GRAPH`) still catches one refreshed once and +-- left. +CREATE INDEX IF NOT EXISTS idx_dbt_graph_snapshot_editor_path + ON dbt_graph_snapshot (workspace_id, script_path, permissioned_as, ingested_at) + WHERE script_hash IS NULL; diff --git a/backend/migrations/20260803182436_workspace_diff_last_event.down.sql b/backend/migrations/20260803182436_workspace_diff_last_event.down.sql new file mode 100644 index 0000000000..8c09aacf27 --- /dev/null +++ b/backend/migrations/20260803182436_workspace_diff_last_event.down.sql @@ -0,0 +1,5 @@ +ALTER TABLE workspace_diff + DROP COLUMN fork_last_event_kind, + DROP COLUMN fork_last_event_origin, + DROP COLUMN source_last_event_kind, + DROP COLUMN source_last_event_origin; diff --git a/backend/migrations/20260803182436_workspace_diff_last_event.up.sql b/backend/migrations/20260803182436_workspace_diff_last_event.up.sql new file mode 100644 index 0000000000..c2c0257c19 --- /dev/null +++ b/backend/migrations/20260803182436_workspace_diff_last_event.up.sql @@ -0,0 +1,21 @@ +-- `ahead`/`behind` count writes on a side without recording what kind of write it +-- was or who made it, which leaves one row shape undecidable: an item the parent +-- has and the fork does not can mean the parent added it, the fork deleted it, or +-- a sync reverted it in the fork. Record the last event per side so the merge +-- direction can offer a removal on evidence. +-- +-- `kind` is 'write' | 'delete' | 'rename_from' (the path was vacated by a rename +-- to another path); create and update are not split because nothing at the tally +-- point tells them apart for every item kind. `origin` is 'authored' | 'sync' +-- (applied by a git-sync pull or a workspace-to-workspace deploy rather than +-- authored in that workspace) and is the load-bearing half: a sync-origin delete +-- is a revert, not a fork deletion. +-- +-- NULL on every row written before this migration, and on a side that has not +-- been written since: such a row has no history and must keep the conservative +-- rule (no removal offered). +ALTER TABLE workspace_diff + ADD COLUMN fork_last_event_kind VARCHAR(20), + ADD COLUMN fork_last_event_origin VARCHAR(20), + ADD COLUMN source_last_event_kind VARCHAR(20), + ADD COLUMN source_last_event_origin VARCHAR(20); diff --git a/backend/migrations/20260807103416_resource_version.down.sql b/backend/migrations/20260807103416_resource_version.down.sql new file mode 100644 index 0000000000..f2407acf80 --- /dev/null +++ b/backend/migrations/20260807103416_resource_version.down.sql @@ -0,0 +1,6 @@ +-- Add down migration script here + +DROP TRIGGER IF EXISTS record_resource_version_insert_trigger ON resource; +DROP TRIGGER IF EXISTS record_resource_version_update_trigger ON resource; +DROP FUNCTION IF EXISTS record_resource_version(); +DROP TABLE resource_version; diff --git a/backend/migrations/20260807103416_resource_version.up.sql b/backend/migrations/20260807103416_resource_version.up.sql new file mode 100644 index 0000000000..f4eef9ae7c --- /dev/null +++ b/backend/migrations/20260807103416_resource_version.up.sql @@ -0,0 +1,108 @@ +-- Add up migration script here + +-- Append-only value history for resources. Keyed by (workspace_id, path) with a +-- cascading FK so a rename carries the history along and a delete takes it with it; +-- without ON UPDATE CASCADE a rename would strand the history at the old path. +CREATE TABLE resource_version ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + workspace_id VARCHAR(50) NOT NULL, + path VARCHAR(255) NOT NULL, + resource_type VARCHAR(50) NOT NULL, + value JSONB, + created_by VARCHAR(500), + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(), + FOREIGN KEY (workspace_id, path) REFERENCES resource (workspace_id, path) + ON DELETE CASCADE ON UPDATE CASCADE +); + +-- Serves both "latest version of this path" and the paginated history listing. +CREATE INDEX index_resource_version_path ON resource_version (workspace_id, path, id DESC); + +GRANT ALL ON resource_version TO windmill_admin; +GRANT ALL ON resource_version TO windmill_user; + +ALTER TABLE resource_version ENABLE ROW LEVEL SECURITY; + +CREATE POLICY admin_policy ON resource_version FOR ALL TO windmill_admin USING (true); + +-- Read visibility follows the parent resource: the subquery is itself subject to resource's +-- own path/extra_perms policies, so the several rules there stay stated once instead of being +-- mirrored (and left to drift) here. +-- +-- SELECT only, deliberately. `FOR ALL ... USING` would be reused as the INSERT/UPDATE/DELETE +-- check expression, and since the subquery is a SELECT it applies resource's *read* policies — +-- which would let a user with read-only access to a resource write its history. Nothing but the +-- trigger below writes this table, and it is SECURITY DEFINER so it does not need a policy. +CREATE POLICY see_parent_resource ON resource_version FOR SELECT TO windmill_user +USING ( + EXISTS ( + SELECT 1 FROM resource r + WHERE r.workspace_id = resource_version.workspace_id + AND r.path = resource_version.path + ) +); + +-- Seed the value every existing resource currently holds, so the first edit after +-- this migration has something to diff against. `state` and `cache` are machine +-- churn (setState, job result caching) and are excluded here and in the trigger, the +-- same way workspace export excludes them (INTERNAL_RESOURCE_TYPES in +-- windmill-store/src/resources.rs — keep the two lists in step). +INSERT INTO resource_version (workspace_id, path, resource_type, value, created_by, created_at) +SELECT workspace_id, path, resource_type, value, created_by, COALESCE(edited_at, now()) +FROM resource +WHERE resource_type NOT IN ('state', 'cache'); + +-- Recording lives in a trigger rather than the API handlers because resource values are +-- written from well outside them: a variable rename rewrites its linked resource +-- (windmill-store/src/variables.rs), workspace forks clone resources wholesale +-- (windmill-api-workspaces/src/workspaces.rs), native integrations upsert them +-- (windmill-native-triggers/src/workspace_integrations.rs), and trashbin restores +-- reinsert them (windmill-api/src/trash.rs). Handler-level hooks would miss all of those +-- and every path added later, leaving a history that silently disagrees with the value it +-- claims to describe. +-- +-- Deleting a resource hard-deletes it, so the FK cascade takes its whole history with it: a +-- trashbin restore lands on an empty history and starts a new one at v1. +CREATE OR REPLACE FUNCTION record_resource_version() RETURNS trigger AS $$ +BEGIN + -- `session.user` is set by UserDB::begin for authed requests; worker and system writes fall + -- back to the row's own author. NULLIF because a transaction-local set_config resets the + -- placeholder to the empty string rather than unsetting it, so a pooled connection that + -- previously served an authed request reports '' here, not NULL. + INSERT INTO resource_version (workspace_id, path, resource_type, value, created_by) + VALUES ( + NEW.workspace_id, NEW.path, NEW.resource_type, NEW.value, + COALESCE(NULLIF(current_setting('session.user', true), ''), NEW.created_by) + ); + + -- The per-path cap is enforced by trim_resource_versions in the monitor, not here: trimming + -- on every write would tax a path `setResource` can drive in a loop, to keep a bound that + -- does not need to hold instantaneously. + + RETURN NEW; +END; +-- SECURITY DEFINER so history is written on behalf of every writer, including the read-only +-- policy above, without granting anyone direct write access to the table. `SET search_path FROM +-- CURRENT` is the injection hardening that goes with it, captured rather than hardcoded so +-- installs running a non-public PG_SCHEMA still resolve (see +-- 20260624103600_repair_folder_labels_search_path.up.sql). +$$ LANGUAGE plpgsql SECURITY DEFINER SET search_path FROM CURRENT; + +-- Split in two, and gated in WHEN rather than in the function body, so the rows that never +-- produce a version never enter plpgsql at all: `state`/`cache` writes (one per setState and +-- per cached job result — by far the hottest writers of this table) and updates that leave the +-- value alone. Separate triggers because a WHEN on INSERT cannot reference OLD. +CREATE TRIGGER record_resource_version_insert_trigger +AFTER INSERT ON resource +FOR EACH ROW +WHEN (NEW.resource_type NOT IN ('state', 'cache')) +EXECUTE FUNCTION record_resource_version(); + +CREATE TRIGGER record_resource_version_update_trigger +AFTER UPDATE ON resource +FOR EACH ROW +WHEN ( + NEW.resource_type NOT IN ('state', 'cache') + AND NEW.value IS DISTINCT FROM OLD.value +) +EXECUTE FUNCTION record_resource_version(); diff --git a/backend/parsers/windmill-parser-py-asset/src/lib.rs b/backend/parsers/windmill-parser-py-asset/src/lib.rs index b8a12fb476..9c70b93d64 100644 --- a/backend/parsers/windmill-parser-py-asset/src/lib.rs +++ b/backend/parsers/windmill-parser-py-asset/src/lib.rs @@ -273,7 +273,7 @@ impl AssetsFinder { Some(Expr::Constant(ExprConstant { value: Constant::Str(value), .. })) => { let path = parse_asset_syntax(&value, false) .map(|(_, p)| p) - .unwrap_or(&value); + .unwrap_or(std::borrow::Cow::Borrowed(value)); self.assets.push(ParseAssetsResult { kind, path: path.to_string(), diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 0e4e12eb89..8bd1d5bb1b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.775.2" +version = "1.783.0" dependencies = [ "aho-corasick", "anyhow", @@ -6227,6 +6227,7 @@ dependencies = [ "jsonwebtoken", "lazy_static", "magic-crypt", + "memchr", "native-tls", "once_cell", "pep440_rs", @@ -6266,13 +6267,14 @@ dependencies = [ "windmill-macros", "windmill-parser", "windmill-parser-sql", + "windmill-parser-sql-asset", "windmill-parser-ts", "windmill-types", ] [[package]] name = "windmill-macros" -version = "1.775.2" +version = "1.783.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6286,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.775.2" +version = "1.783.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6295,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6307,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6319,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6331,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6343,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6355,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6366,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6377,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6389,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6400,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6422,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6434,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6448,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6465,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6478,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6490,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6508,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6524,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6540,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,18 +6572,21 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", + "lazy_static", + "regex", "serde", "serde_json", + "serde_yml", "windmill-parser", "yaml-rust", ] [[package]] name = "windmill-types" -version = "1.775.2" +version = "1.783.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 1cb5004022..2e527cef86 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.775.2" +version = "1.783.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/parsers/windmill-parser-wasm/src/lib.rs b/backend/parsers/windmill-parser-wasm/src/lib.rs index c843652e26..59e963d52c 100644 --- a/backend/parsers/windmill-parser-wasm/src/lib.rs +++ b/backend/parsers/windmill-parser-wasm/src/lib.rs @@ -164,6 +164,14 @@ pub fn parse_ansible(code: &str) -> String { wrap_sig(windmill_parser_yaml::parse_ansible_sig(code)) } +/// Shares the ansible parser's feature: both live in windmill-parser-yaml, and +/// a dbt descriptor is YAML too. +#[cfg(feature = "ansible-parser")] +#[wasm_bindgen] +pub fn parse_dbt(code: &str) -> String { + wrap_sig(windmill_parser_yaml::parse_dbt_sig(code)) +} + #[cfg(feature = "ansible-parser")] #[wasm_bindgen] pub fn parse_ansible_delegate(code: &str) -> String { diff --git a/backend/parsers/windmill-parser-yaml/Cargo.toml b/backend/parsers/windmill-parser-yaml/Cargo.toml index eb01ba1936..eb2f16fccc 100644 --- a/backend/parsers/windmill-parser-yaml/Cargo.toml +++ b/backend/parsers/windmill-parser-yaml/Cargo.toml @@ -14,3 +14,6 @@ windmill-parser.workspace = true anyhow.workspace = true serde_json.workspace = true serde.workspace = true +serde_yml.workspace = true +regex.workspace = true +lazy_static.workspace = true diff --git a/backend/parsers/windmill-parser-yaml/src/dbt.rs b/backend/parsers/windmill-parser-yaml/src/dbt.rs new file mode 100644 index 0000000000..24f1e2c39d --- /dev/null +++ b/backend/parsers/windmill-parser-yaml/src/dbt.rs @@ -0,0 +1,781 @@ +//! The `ScriptLang::Dbt` script artifact: a YAML descriptor holding a dbt +//! project's run configuration. The project itself is the script's module +//! bundle, so the descriptor names no source for it. +//! +//! Field names track dbt's and astronomer-cosmos's vocabulary so the mental +//! model ports without translation (docs/dbt-runtime.md, decision 22). +//! `select` / `exclude` / `selector` are passed to dbt **verbatim**: the +//! selector grammar is dbt's, and reimplementing it is a standing source of +//! divergence. + +use std::collections::BTreeMap; + +use serde::{Deserialize, Serialize}; +use windmill_parser::{Arg, MainArgSignature, Typ}; + +/// Which dbt to run. The shipped default is `dbt-core-1x` because it runs +/// today's projects untouched; `fusion` is never bundled and is fetched from +/// dbt Labs at runtime (docs/dbt-runtime.md, decision 1). +#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, Default)] +#[serde(rename_all = "kebab-case")] +pub enum DbtEngine { + #[default] + #[serde(rename = "dbt-core-1x")] + DbtCore1x, + #[serde(rename = "dbt-core-2x")] + DbtCore2x, + Fusion, +} + +impl DbtEngine { + pub fn as_str(&self) -> &'static str { + match self { + DbtEngine::DbtCore1x => "dbt-core-1x", + DbtEngine::DbtCore2x => "dbt-core-2x", + DbtEngine::Fusion => "fusion", + } + } + + /// The level the machine-readable file log is written at. Separate from the + /// console log, which always stays human-readable at the default level and + /// is what reaches the job log. + pub fn progress_log_level(&self) -> &'static str { + match self { + DbtEngine::DbtCore1x => "info", + DbtEngine::DbtCore2x | DbtEngine::Fusion => "debug", + } + } + + /// Whether the engine writes per-node events to its JSON **file** log — the + /// only source of *live* per-model status. + /// + /// Not a claim that the others produce none: dbt-core 2.x and Fusion emit + /// the same events, on the console, and ignore `--log-format-file json` + /// though both accept it. Reading them would mean taking over the console + /// and re-rendering the job log. Flip this the moment either honours the + /// flag — nothing else has to change (docs/dbt-runtime.md, "Live per-model + /// progress"). + pub fn emits_node_events(&self) -> bool { + matches!(self, DbtEngine::DbtCore1x) + } +} + +/// How the warehouse connection is supplied. Both paths are supported +/// (decision 8): render `profiles.yml` from a Windmill resource, or keep the +/// project's own file and inject Windmill secrets as env vars for +/// `{{ env_var() }}`. +#[derive(Serialize, Deserialize, Debug, Clone, Default)] +#[serde(deny_unknown_fields)] +pub struct DbtProfile { + /// A warehouse configured on the workspace, by name; omitted takes the + /// default one. This is the ONLY way a project names a warehouse — there is + /// no per-descriptor resource — so a dbt project carries no connection at + /// all, the same bargain `s3://` and `ducklake://` make with workspace + /// storage, and asset identity has exactly one spelling to key on. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub warehouse: Option, + + /// dbt target name. It selects which output of the profile runs; the + /// `` component of asset identity comes from `warehouse` above. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub target: Option, + /// Path (relative to `project`) of the project's own `profiles.yml`, used + /// instead of rendering one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub profiles_yml: Option, + /// Target schema (BigQuery calls it the dataset). Required for adapters + /// whose Windmill resource carries none — a BigQuery resource is a raw + /// service-account JSON, which has no dataset in it. Overrides the + /// resource's own value where there is one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub schema: Option, + /// dbt adapter (`postgres` | `snowflake` | `bigquery` | `databricks`), + /// spelled as dbt's own `type:`. Optional: the worker infers it from the + /// resource's shape, and this pins it when the inference is wrong or the + /// resource is a custom type. + #[serde(default, rename = "type", skip_serializing_if = "Option::is_none")] + pub adapter: Option, +} + +#[derive(Serialize, Deserialize, Debug, Clone, Default)] +#[serde(rename_all = "snake_case")] +pub enum DbtTestBehavior { + /// `dbt build` — models and their tests interleaved, a model's tests gating + /// its children. dbt's own default and the only behavior that stops bad + /// data propagating mid-run. + #[default] + Build, + /// `dbt run` then `dbt test`. + AfterAll, + /// `dbt run` only. + None, +} + +#[derive(Serialize, Deserialize, Debug, Clone, Default)] +#[serde(deny_unknown_fields)] +pub struct DbtDescriptor { + #[serde(default)] + pub engine: Option, + #[serde(default)] + pub profile: DbtProfile, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub select: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub exclude: Vec, + /// A named selector from the project's `selectors.yml`, passed verbatim. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub selector: Option, + #[serde(default)] + pub test_behavior: DbtTestBehavior, + /// `--vars`. dbt vars are typed — numbers, booleans, lists and objects are + /// all normal — so values keep their YAML type; only string leaves carry + /// `{{ arg }}` placeholders the worker substitutes from job args. Coercing + /// everything to a string would make a `false` var truthy in Jinja. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub vars: BTreeMap, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub threads: Option, + #[serde(default)] + pub full_refresh: bool, + /// Automatic in-job retry of the nodes a build failed on. + /// + /// dbt already confines a failure to its own subtree, and `dbt retry` + /// rebuilds exactly the failed and skipped set, so a transient warehouse + /// error costs those nodes rather than the whole project. Doing it inside + /// the same job is what keeps the state question out of it: the previous + /// attempt's `run_results.json` is still in the job directory. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub retry_failed_nodes: Option, + /// Extra environment for the dbt process, for the project's own + /// `{{ env_var() }}` lookups and for engine flags such as + /// `DBT_ALLOW_EXPERIMENTAL_ADAPTERS`. A `$var:` value is resolved to + /// that Windmill variable's value by the worker, so a password never has to + /// sit in the descriptor — which is versioned script content. + /// + /// This is the map to use for anything the GRAPH depends on — an + /// `env_var()` driving a schema, alias or `enabled` — because it applies at + /// deploy as well as at run. Script-level environment variables reach the + /// run only, so a graph parsed without them would disagree with what the + /// build writes. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub env: BTreeMap, +} + +/// How many times, and how far apart, to retry a build's failed nodes. +#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct DbtNodeRetry { + /// Extra `dbt retry` attempts for the job, spent across whichever phases + /// fail — a model phase that uses them all leaves none for the tests. + pub attempts: u32, + /// Seconds between attempts. A transient warehouse error is usually a lock + /// or a restart, so a pause is the point. + #[serde(default)] + pub delay_seconds: u64, +} + +impl DbtNodeRetry { + /// Bounded because each attempt is a real dbt invocation inside a job that + /// already holds a worker slot, and the job's own deadline still applies. + pub const MAX_ATTEMPTS: u32 = 10; + + pub fn attempts(&self) -> u32 { + self.attempts.min(Self::MAX_ATTEMPTS) + } +} + +/// The dbt subcommands a run may ask for. Kept here so the signature and the +/// worker's validation cannot drift apart. +/// +/// An allowlist rather than a passthrough: the value becomes the dbt subcommand, +/// and running a script needs weaker permission than editing it, so an unchecked +/// arg would let a runner invoke `clean` or `source freshness` against the +/// descriptor's warehouse. +/// +/// `run` is absent because it covers models only: a project with seeds or +/// snapshots would build a subset of what its graph claims. Narrowing what a run +/// touches is `select`/`exclude`, which scope the graph too. Tests run as part +/// of `build`, or as the second phase of `test_behavior: after_all`. +/// +/// `show` writes nothing — it SELECTs from a model and returns rows. That is +/// only admissible because a dbt run no longer dispatches: while it did, any +/// successful job fired the script's whole deploy-time write set, so a command +/// that built none of them woke every consumer for relations nothing touched. +/// +/// `parse` reads no relation either: it resolves the project into a manifest and +/// stores the graph, which is what the editor's model panel is refreshed by. It +/// still needs a resolvable warehouse, because the profile is rendered before +/// any dbt invocation — a misconfigured project therefore fails a refresh the way +/// it would fail a run, which is the early feedback worth having. +pub const DBT_COMMANDS: &[&str] = &["build", "retry", "show", "parse"]; + +/// Rows a `show` returns unless the run asks for fewer. dbt enforces it, so the +/// bound is not us splicing a `LIMIT` into someone's SQL. +pub const DBT_SHOW_DEFAULT_LIMIT: u32 = 100; + +/// Hard ceiling on that limit. The worker buffers `dbt show`'s whole stdout to +/// read the rows out of it, so the argument decides how much memory a caller can +/// make it hold — and running a script needs only run permission. A preview is +/// for looking at a few rows; anything larger is a query, which is what a SQL +/// script is for. +pub const DBT_SHOW_MAX_LIMIT: u32 = 1_000; + +/// Whether the command only reads. Such a run publishes no graph, records no +/// materializations and runs no test phase — there is nothing it could have +/// changed. +pub fn is_read_only_command(command: &str) -> bool { + command == "show" +} + +/// The command a run uses when it does not name one. Public because the worker +/// must choose exactly what the run form's default advertises. +pub fn default_command(d: &DbtDescriptor) -> &'static str { + match d.test_behavior { + DbtTestBehavior::Build => "build", + // Also `build`, with the tests excluded rather than the command + // narrowed to `run`: `run` covers models only, so a selection that + // includes a seed or a snapshot would silently not build it and the + // models reading it would fail — or worse, read a stale table. + DbtTestBehavior::AfterAll | DbtTestBehavior::None => "build", + } +} + +impl DbtDescriptor { + pub fn engine(&self) -> DbtEngine { + self.engine.unwrap_or_default() + } +} + +/// Names a `{{ placeholder }}` may not take. `command` is the argument holding +/// the run's command block; the rest are the fields inside it, which the worker +/// spreads over the run's arguments to read them — a placeholder of the same +/// name would be shadowed there, and the script could never be run (`select` is +/// an array, and interpolating one into a string is not something any invocation +/// can satisfy). +pub const RESERVED_ARG_NAMES: &[&str] = &[ + "command", + "select", + "exclude", + "vars", + "full_refresh", + "dbt_command", + "dbt_retry_job", + "model", + "limit", +]; + +pub fn parse_dbt_descriptor(inner_content: &str) -> anyhow::Result { + let d = serde_yml::from_str::(inner_content) + .map_err(|e| anyhow::anyhow!("Failed to parse dbt descriptor: {e}"))?; + if let Some(name) = placeholders(&d) + .into_iter() + .find(|n| RESERVED_ARG_NAMES.contains(&n.as_str())) + { + return Err(anyhow::anyhow!( + "`{{{{ {name} }}}}` collides with the run argument `{name}` this runtime already \ + defines ({}); rename the placeholder", + RESERVED_ARG_NAMES.join(", ") + )); + } + Ok(d) +} + +/// The workspace warehouse a descriptor gets when it names none — spelled like +/// the default lake (`ducklake://main.orders`), so one workspace concept reads +/// the same across kinds. +pub const DBT_DEFAULT_WAREHOUSE: &str = "main"; + +/// The single argument holding the command and the overrides it takes. Its +/// variant IS the command, so a run cannot carry an override the command +/// ignores: `retry` rebuilds the failed run's nodes with the arguments that run +/// had, and `full_refresh` means nothing to a `show`. +pub const DBT_COMMAND_ARG: &str = "command"; + +/// The variant discriminator, which is the key Windmill's run form tags a +/// `oneOf` value with. +pub const DBT_COMMAND_LABEL: &str = "label"; + +fn list_arg(name: &str, default: &[String]) -> Arg { + Arg { + name: name.to_string(), + otyp: None, + typ: Typ::List(Box::new(Typ::Str(None))), + has_default: true, + default: Some(serde_json::json!(default)), + oidx: None, + otyp_inferred: false, + } +} + +/// An OVERRIDE map, so its default is empty rather than a copy of the +/// descriptor's vars. Seeding it with the descriptor would make the run form +/// post the raw `{{ placeholder }}` text back and clobber the value the worker +/// just interpolated for it. +fn vars_arg() -> Arg { + Arg { + name: "vars".to_string(), + otyp: None, + typ: Typ::Object(windmill_parser::ObjectType::new(None, Some(vec![]))), + has_default: true, + default: Some(serde_json::json!({})), + oidx: None, + otyp_inferred: false, + } +} + +/// What each command takes, in form order. +/// +/// `show` and `parse` are absent on purpose, for the same reason: each is a thing +/// to do to the project you are looking at rather than a job to fill a form in +/// for. `show` previews ONE model's rows, offered by the run page's graph and the +/// assets list where the tables are; `parse` refreshes the model graph, offered by +/// the dbt editor over the buffer being edited. The worker still accepts +/// `{label: show, model, limit}` and `{label: parse, vars}` from a flow, the CLI or +/// the API (`DBT_COMMANDS`, docs/dbt-runtime.md). +fn command_variants(d: &DbtDescriptor) -> Vec<(&'static str, Vec)> { + let selection = || { + vec![ + list_arg("select", &d.select), + list_arg("exclude", &d.exclude), + vars_arg(), + ] + }; + vec![ + ( + "build", + selection() + .into_iter() + .chain([Arg { + name: "full_refresh".to_string(), + otyp: None, + typ: Typ::Bool, + has_default: true, + default: Some(serde_json::json!(d.full_refresh)), + oidx: None, + otyp_inferred: false, + }]) + .collect(), + ), + ( + "retry", + // The run it resumes, named rather than implied: only the latest + // failure of this script is kept, so "resume the last one" would + // silently aim somewhere else the moment another run failed. No + // default, so this variant requires it. + vec![Arg { + name: "dbt_retry_job".to_string(), + otyp: None, + typ: Typ::Str(None), + has_default: false, + default: None, + oidx: None, + otyp_inferred: false, + }], + ), + ] +} + +/// The command block a run gets when it overrides nothing: the descriptor's own +/// selection under the descriptor's default command, so an untouched run +/// reproduces it exactly. +fn default_command_value(d: &DbtDescriptor) -> serde_json::Value { + let label = default_command(d); + let mut obj = serde_json::Map::new(); + obj.insert(DBT_COMMAND_LABEL.to_string(), serde_json::json!(label)); + for arg in command_variants(d) + .into_iter() + .find(|(l, _)| *l == label) + .map(|(_, args)| args) + .unwrap_or_default() + { + if let Some(default) = arg.default { + obj.insert(arg.name, default); + } + } + serde_json::Value::Object(obj) +} + +/// Run-time arguments of a dbt script: the command block above, plus one +/// argument per `{{ placeholder }}` the descriptor interpolates. Placeholders +/// stay top-level — they are the project's own inputs, not a command's. +pub fn parse_dbt_sig(inner_content: &str) -> anyhow::Result { + let d = parse_dbt_descriptor(inner_content)?; + let mut args = vec![Arg { + name: DBT_COMMAND_ARG.to_string(), + otyp: None, + typ: Typ::Object(windmill_parser::ObjectType::new(None, Some(vec![]))), + has_default: true, + default: Some(default_command_value(&d)), + oidx: None, + otyp_inferred: false, + }]; + + for name in placeholders(&d) { + if args.iter().any(|a| a.name == name) { + continue; + } + args.push(Arg { + name, + otyp: None, + // Untyped, not `Str`: a placeholder standing alone in a var takes + // the argument's own JSON type, and declaring it a string makes the + // run form post `"false"` — truthy in Jinja — for a boolean. + typ: Typ::Unknown, + has_default: false, + default: None, + oidx: None, + otyp_inferred: false, + }); + } + + Ok(MainArgSignature { + star_args: false, + star_kwargs: false, + args, + auto_kind: None, + has_preprocessor: None, + ..Default::default() + }) +} + +/// The run form's JSON schema for a dbt descriptor. +/// +/// Derived here rather than in the browser or the CLI: both infer a script's +/// schema client-side through `windmill-parser-wasm`, whose published package +/// has no dbt arm, so they leave the schema untouched. Without this a dbt +/// script deploys with an empty schema and its run form offers none of the +/// overrides — and an edited descriptor keeps the previous one's arguments. +/// The command is a `oneOf` rather than an enum beside the overrides it selects: +/// a variant carries exactly the arguments its command takes, so `dbt_retry_job` +/// is required where it means something and absent everywhere else, and no run +/// can submit a `full_refresh` to a command that ignores it. The run form renders +/// it as a toggle over the variants and tags the value with `label`. +pub fn dbt_arg_schema(inner_content: &str) -> anyhow::Result { + let d = parse_dbt_descriptor(inner_content)?; + let sig = parse_dbt_sig(inner_content)?; + let mut properties = serde_json::Map::new(); + let mut required: Vec = vec![]; + let mut order: Vec = vec![]; + + let variants: Vec = command_variants(&d) + .into_iter() + .map(|(label, args)| { + let mut props = serde_json::Map::new(); + let mut var_required: Vec = vec![]; + let mut var_order: Vec = vec![serde_json::json!(DBT_COMMAND_LABEL)]; + // The discriminator. Single-valued, so the form's toggle is what sets + // it and a submitted block cannot claim one command while carrying + // another's fields. + props.insert( + DBT_COMMAND_LABEL.to_string(), + serde_json::json!({"type": "string", "enum": [label]}), + ); + for arg in args { + if !arg.has_default { + var_required.push(serde_json::json!(arg.name)); + } + var_order.push(serde_json::json!(arg.name)); + props.insert(arg.name.clone(), property_of(&arg)); + } + serde_json::json!({ + "title": label, + "type": "object", + "properties": props, + "order": var_order, + "required": var_required, + }) + }) + .collect(); + + for arg in &sig.args { + let mut prop = property_of(arg); + if arg.name == DBT_COMMAND_ARG { + if let Some(obj) = prop.as_object_mut() { + obj.insert( + "oneOf".to_string(), + serde_json::Value::Array(variants.clone()), + ); + obj.insert( + "description".to_string(), + serde_json::json!( + "`build` runs the project. `retry` resumes a failed run, rebuilding only \ + its failed and skipped nodes with the arguments it ran with." + ), + ); + } + } + if !arg.has_default { + required.push(serde_json::json!(arg.name)); + } + order.push(serde_json::json!(arg.name)); + properties.insert(arg.name.clone(), prop); + } + Ok(serde_json::json!({ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": properties, + "required": required, + "order": order, + })) +} + +/// One argument as a JSON-schema property, with the description the run form +/// shows under its label. +fn property_of(arg: &Arg) -> serde_json::Value { + let mut prop = match &arg.typ { + Typ::Bool => serde_json::json!({"type": "boolean"}), + // `limit` is an integer to the worker, which clamps it; without this the + // generated clients and the run form offer no numeric control. + Typ::Int => serde_json::json!({"type": "integer"}), + Typ::List(_) => serde_json::json!({"type": "array", "items": {"type": "string"}}), + Typ::Object(_) => serde_json::json!({"type": "object"}), + Typ::Str(Some(variants)) => serde_json::json!({"type": "string", "enum": variants}), + Typ::Str(None) => serde_json::json!({"type": "string"}), + // A placeholder takes the JSON type of whatever is passed, so it is + // deliberately left untyped rather than guessed at. + _ => serde_json::json!({}), + }; + if let Some(obj) = prop.as_object_mut() { + if let Some(default) = arg.default.as_ref() { + obj.insert("default".to_string(), default.clone()); + } + let description = match arg.name.as_str() { + "dbt_retry_job" => Some( + "The failed run to resume, by run id. Its failed and skipped nodes are rebuilt \ + with the arguments it ran with. Resuming from that run's page fills this in.", + ), + "select" => Some( + "dbt selection syntax, e.g. `tag:nightly`, `stg_orders+`, \ + `config.materialized:incremental`. Empty runs the descriptor's own selection.", + ), + "exclude" => Some("Nodes to leave out of the selection above, same syntax."), + "vars" => Some( + "dbt `--vars`, merged over the descriptor's. A var that changes which models \ + exist makes this run store its own graph rather than the deployed one.", + ), + "full_refresh" => Some("Rebuild incremental models from scratch instead of appending."), + "model" => Some( + "The model to preview, by name — `stg_orders`, or `my_package.stg_orders` when \ + two packages share a name. Any dbt selector resolving to ONE node works.", + ), + "limit" => Some("Rows the preview returns."), + // A placeholder the descriptor interpolates: its meaning is the + // project's, so there is nothing generic to say about it. + _ => None, + }; + if let Some(d) = description { + obj.insert("description".to_string(), serde_json::json!(d)); + } + } + prop +} + +/// `{{ name }}` placeholders in the interpolated descriptor fields, in a stable +/// order. Must stay in sync with the fields the worker actually interpolates. +fn placeholders(d: &DbtDescriptor) -> Vec { + let mut out: Vec = vec![]; + let mut push_from = |s: &str| { + for caps in PLACEHOLDER_RE.captures_iter(s) { + let name = caps[1].to_string(); + if !out.contains(&name) { + out.push(name); + } + } + }; + for v in d.vars.values() { + for leaf in string_leaves(v) { + push_from(leaf); + } + } + out +} + +/// Every string inside a var's value, at any depth — the only places a +/// `{{ arg }}` placeholder can appear. +pub fn string_leaves(v: &serde_json::Value) -> Vec<&str> { + match v { + serde_json::Value::String(s) => vec![s.as_str()], + serde_json::Value::Array(a) => a.iter().flat_map(string_leaves).collect(), + serde_json::Value::Object(o) => o.values().flat_map(string_leaves).collect(), + _ => vec![], + } +} + +lazy_static::lazy_static! { + /// Same spelling as the Ansible executor's `interpolate_template`, which is + /// what actually performs the substitution at run time. + static ref PLACEHOLDER_RE: regex::Regex = + regex::Regex::new(r"\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}").unwrap(); +} + +#[cfg(test)] +mod tests { + use super::*; + + const DESCRIPTOR: &str = r#" +engine: dbt-core-2x +profile: + warehouse: main + target: prod +select: ["tag:nightly+"] +test_behavior: after_all +vars: + run_date: "{{ day }}" + strict: false +threads: 8 +full_refresh: true +"#; + + // A descriptor drives warehouse writes, so a field it does not recognise is + // an error rather than a default. `selcet:` would otherwise leave `select` + // empty and build the entire project; a misspelled `target` would silently + // fall back to the profile's own default. + #[test] + fn an_unknown_descriptor_field_is_refused() { + let err = parse_dbt_descriptor("profile:\n warehouse: main\nselcet: [a]\n") + .unwrap_err() + .to_string(); + assert!(err.contains("selcet"), "{err}"); + let err = parse_dbt_descriptor("profile:\n warehouse: main\n targt: prod\n") + .unwrap_err() + .to_string(); + assert!(err.contains("targt"), "{err}"); + } + + // A placeholder that takes a run argument's name would be shadowed by it, + // leaving a descriptor no invocation can satisfy: `select` is an array and + // the interpolation needs a scalar. Refused at parse, so the deploy says so + // rather than the script becoming unrunnable after it. + #[test] + fn a_placeholder_may_not_take_a_run_argument_name() { + for name in RESERVED_ARG_NAMES { + let d = format!("profile:\n warehouse: main\nvars:\n v: \"{{{{ {name} }}}}\"\n"); + let err = parse_dbt_descriptor(&d).unwrap_err().to_string(); + assert!(err.contains(name), "{name}: {err}"); + } + // A name of its own is fine. + assert!( + parse_dbt_descriptor("profile:\n warehouse: main\nvars:\n v: \"{{ day }}\"\n") + .is_ok() + ); + } + + // The run form is built from this schema, and nothing else can build it: + // the browser and the CLI both infer through a wasm parser that has no dbt + // arm, so a missing property here is an override the user cannot reach. + // The variants are what make the form show a command's own fields and only + // those, so each is asserted by the arguments it carries. + #[test] + fn the_schema_carries_every_run_override_and_placeholder() { + let schema = dbt_arg_schema(DESCRIPTOR).unwrap(); + let props = schema["properties"].as_object().unwrap(); + let variants = props[DBT_COMMAND_ARG]["oneOf"].as_array().unwrap(); + let of = |label: &str| { + let v = variants + .iter() + .find(|v| v["title"] == label) + .unwrap_or_else(|| panic!("no `{label}` variant: {schema}")); + let mut names: Vec = v["properties"] + .as_object() + .unwrap() + .keys() + .filter(|k| k.as_str() != DBT_COMMAND_LABEL) + .cloned() + .collect(); + names.sort(); + (v.clone(), names) + }; + + let (build, build_args) = of("build"); + assert_eq!(build_args, ["exclude", "full_refresh", "select", "vars"]); + assert_eq!(build["properties"]["full_refresh"]["type"], "boolean"); + // Defaults come from the descriptor, so an untouched run reproduces it. + assert_eq!( + build["properties"]["select"]["default"], + serde_json::json!(["tag:nightly+"]) + ); + // The discriminator takes one value per variant: the toggle sets it, and + // a block cannot claim one command while carrying another's fields. + assert_eq!( + build["properties"][DBT_COMMAND_LABEL]["enum"], + serde_json::json!(["build"]) + ); + + let (retry, retry_args) = of("retry"); + assert_eq!(retry_args, ["dbt_retry_job"]); + // Required where it means something, rather than required everywhere and + // hidden: a retry names the run it resumes. + assert_eq!(retry["required"], serde_json::json!(["dbt_retry_job"])); + + // Neither `show` nor `parse` is a form variant: previewing one model's + // rows and refreshing the model graph are both things you do to the + // project in front of you, and the graph, the assets list and the dbt + // editor are where those live. The worker still accepts both + // programmatically, which is what makes them scriptable. + for hidden in ["show", "parse"] { + assert!( + !variants.iter().any(|v| v["title"] == hidden), + "{hidden} must not be offered in the run form: {schema}" + ); + assert!( + DBT_COMMANDS.contains(&hidden), + "but the worker still takes {hidden}" + ); + } + + // Every `{{ placeholder }}` the descriptor interpolates is an argument a + // run must supply. The command block is not one: it defaults to the + // descriptor's own selection under the descriptor's command. + assert_eq!(schema["required"], serde_json::json!(["day"])); + assert_eq!(schema["order"], serde_json::json!([DBT_COMMAND_ARG, "day"])); + } + + #[test] + fn parses_descriptor() { + let d = parse_dbt_descriptor(DESCRIPTOR).unwrap(); + assert_eq!(d.engine(), DbtEngine::DbtCore2x); + assert_eq!(d.profile.target.as_deref(), Some("prod")); + assert_eq!(d.select, vec!["tag:nightly+"]); + assert_eq!(d.threads, Some(8)); + assert!(d.full_refresh); + // dbt vars keep their YAML type: a `false` coerced to "false" is truthy + // in Jinja and would silently invert the condition it gates. + assert_eq!(d.vars["strict"], serde_json::json!(false)); + assert_eq!(d.vars["run_date"], serde_json::json!("{{ day }}")); + } + + #[test] + fn an_empty_descriptor_defaults_to_the_bundled_engine() { + let d = parse_dbt_descriptor("").unwrap(); + assert_eq!(d.engine(), DbtEngine::DbtCore1x); + } + + // The ORDER is asserted, not just the set: the schema's `order` is built from + // this vec and the run form follows it, so the command block leading is what + // puts the choice of what the run does above the project's own inputs. + #[test] + fn signature_exposes_the_command_block_and_placeholders() { + let sig = parse_dbt_sig(DESCRIPTOR).unwrap(); + let names: Vec<&str> = sig.args.iter().map(|a| a.name.as_str()).collect(); + assert_eq!(names, vec![DBT_COMMAND_ARG, "day"]); + // An untouched run reproduces the descriptor: its default is the whole + // block — the descriptor's command, carrying the descriptor's selection. + let cmd = sig.args.iter().find(|a| a.name == DBT_COMMAND_ARG).unwrap(); + let default = cmd.default.clone().unwrap(); + // `build`, not `run`: `run` covers models only, so a selection naming a + // seed or a snapshot would silently not build it. + assert_eq!(default[DBT_COMMAND_LABEL], serde_json::json!("build")); + assert_eq!(default["select"], serde_json::json!(["tag:nightly+"])); + assert_eq!(default["full_refresh"], serde_json::json!(true)); + // `vars` is the exception: it overrides, so its default must be empty. + // Seeded with the descriptor, the run form would post `{{ day }}` back + // and overwrite the value the worker interpolated for it. + assert_eq!(default["vars"], serde_json::json!({})); + // Placeholders are required (the descriptor names no value for them) + // and untyped, so a `{{ }}` var can carry a boolean or a number rather + // than the string "false", which Jinja treats as truthy. + let day = sig.args.iter().find(|a| a.name == "day").unwrap(); + assert!(!day.has_default); + assert_eq!(day.typ, Typ::Unknown); + } +} diff --git a/backend/parsers/windmill-parser-yaml/src/lib.rs b/backend/parsers/windmill-parser-yaml/src/lib.rs index f2a1a2b824..e025849769 100644 --- a/backend/parsers/windmill-parser-yaml/src/lib.rs +++ b/backend/parsers/windmill-parser-yaml/src/lib.rs @@ -9,6 +9,13 @@ use yaml_rust::{Yaml, YamlEmitter, YamlLoader}; pub mod asset_parser; pub use asset_parser::parse_assets; +pub mod dbt; +pub use dbt::{ + dbt_arg_schema, default_command as default_dbt_command, parse_dbt_descriptor, parse_dbt_sig, + DbtDescriptor, DbtEngine, DbtTestBehavior, DBT_COMMANDS, DBT_COMMAND_ARG, DBT_COMMAND_LABEL, + DBT_DEFAULT_WAREHOUSE, +}; + pub fn parse_ansible_sig(inner_content: &str) -> anyhow::Result { let docs = YamlLoader::load_from_str(inner_content) .map_err(|e| anyhow!("Failed to parse yaml: {}", e))?; diff --git a/backend/parsers/windmill-parser/src/asset_parser.rs b/backend/parsers/windmill-parser/src/asset_parser.rs index de35ccdd6e..843a83a310 100644 --- a/backend/parsers/windmill-parser/src/asset_parser.rs +++ b/backend/parsers/windmill-parser/src/asset_parser.rs @@ -1,4 +1,5 @@ use serde::Serialize; +use std::borrow::Cow; use std::collections::BTreeMap; // Token recognized inside declared asset URIs that the runtime substitutes @@ -28,6 +29,11 @@ pub enum AssetKind { Ducklake, DataTable, Volume, + /// A warehouse relation a dbt project builds or reads, + /// `dbt:////`, the warehouse named as the + /// workspace configures it. The scheme names the producer, the path stays + /// the relation — see `windmill_types::AssetKind::Dbt`. + Dbt, } #[derive(Serialize, Debug, PartialEq, Clone)] @@ -714,21 +720,32 @@ pub fn asset_was_used(assets: &Vec, (kind, path): (AssetKind, }) } -pub fn parse_asset_syntax(s: &str, enable_default_syntax: bool) -> Option<(AssetKind, &str)> { +/// Split an asset URI into `(kind, path)`. The single point where user-written +/// asset URIs become graph keys, and therefore where `dbt://` paths get +/// canonicalized (see `canonicalize_table_asset_path`) — every other kind's +/// suffix is kept verbatim and stays borrowed. +pub fn parse_asset_syntax( + s: &str, + enable_default_syntax: bool, +) -> Option<(AssetKind, Cow<'_, str>)> { if enable_default_syntax && s == "datatable" { - return Some((AssetKind::DataTable, "main")); + return Some((AssetKind::DataTable, Cow::Borrowed("main"))); } else if enable_default_syntax && s == "ducklake" { - return Some((AssetKind::Ducklake, "main")); + return Some((AssetKind::Ducklake, Cow::Borrowed("main"))); } for (prefix, kind) in ASSET_KINDS.iter() { if s.starts_with(prefix) { + let suffix = &s[prefix.len()..]; + if *kind == AssetKind::Dbt { + return Some((*kind, Cow::Owned(canonicalize_table_asset_path(suffix)))); + } // The suffix is kept verbatim. For S3 the path encodes the storage: // `s3:///`, with an EMPTY storage segment for the // workspace default — so `s3:///key` yields `/key` (leading slash // significant, default storage) while `s3://secondary/key` yields // `secondary/key`. Stripping leading slashes here would conflate a // default-storage object with a named-storage one. - return Some((*kind, &s[prefix.len()..])); + return Some((*kind, Cow::Borrowed(suffix))); } } None @@ -741,8 +758,123 @@ pub const ASSET_KINDS: &[(&str, AssetKind)] = &[ ("ducklake://", AssetKind::Ducklake), ("datatable://", AssetKind::DataTable), ("volume://", AssetKind::Volume), + ("dbt://", AssetKind::Dbt), ]; +/// Canonical spelling of a `dbt://` path, `//`. +/// +/// The whole point of keying warehouse relations on the relation rather than on +/// the producing tool is that a dbt mart and a native script reading the same +/// table land on one graph node. That only holds if both sides spell the key +/// identically, and they will not by default: dbt's `manifest.json` gives +/// `relation_name` pre-quoted (`"db"."Schema"."Tbl"`) while an annotation is +/// written by hand, and the warehouses disagree on case (Snowflake folds +/// unquoted identifiers up, Postgres folds them down, DuckDB compares them +/// case-insensitively). Two spellings of one table produce two nodes, no edge, +/// and nothing looks broken in isolation — so the rule is applied once, here, +/// on every path that becomes a `dbt://` asset. +/// +/// The rule: strip the quote characters the warehouses use (`"`, backtick, +/// `[`/`]`) from the schema and name, then ASCII-lowercase them. This matches +/// the case-insensitive identifier comparison the DuckDB paths already use +/// (`schema_contracts::CapturedSchema::find`). The warehouse-name prefix is +/// spelled as the workspace configures it, which is case-sensitive, and is left +/// untouched. +/// +/// Consequence to accept: a relation deliberately created under a quoted +/// mixed-case identifier collides with its lowercase spelling. That is rarer +/// than the case-fold mismatch it prevents, and it errs toward unifying nodes +/// rather than splitting them. +pub fn canonicalize_table_asset_path(path: &str) -> String { + let Some((resource, rest)) = path.rsplit_once('/').and_then(|(head, name)| { + head.rsplit_once('/') + .map(|(resource, schema)| (resource, (schema, name))) + }) else { + // Fewer than three segments: not a well-formed relation path. Leave it + // alone so the malformed value stays visible instead of being reshaped + // into something that looks valid. + return path.to_string(); + }; + let (schema, name) = rest; + format!( + "{}/{}/{}", + resource, + unquote_schema_segment(schema).to_ascii_lowercase(), + unquote_identifier(name).to_ascii_lowercase() + ) +} + +/// A doubled delimiter inside a quoted identifier is that delimiter, literally — +/// the same rule the worker's `split_relation` applies to `relation_name`. Both +/// have to decode it or one spelling of a table becomes two graph nodes: the dbt +/// model under `sales"east`, its native consumer under `sales""east`. +fn unquote_identifier(s: &str) -> String { + let s = s.trim(); + for (open, close) in [('"', '"'), ('`', '`'), ('[', ']')] { + if s.len() >= 2 && s.starts_with(open) && s.ends_with(close) { + let inner = &s[1..s.len() - 1]; + return inner.replace(&format!("{close}{close}"), &close.to_string()); + } + } + s.to_string() +} + +/// Unquote a schema segment, which carries a `.` pair when a +/// relation overrode its database. Each half is separately quotable, so +/// stripping only the outer pair leaves `"Archive"."Sales"` as +/// `Archive"."Sales` — a different key from the ingest's `archive.sales`, which +/// silently splits the model and its native consumer into two nodes. +/// +/// The halves go through `unquote_identifier`, which requires the quote at both +/// ends, because this function sees both a warehouse SPELLING (from a `dbt://` +/// annotation) and an already-DECODED identifier (from `table_asset_path`, whose +/// callers ran `split_relation` or read the manifest) and cannot tell them apart. +/// A lone `"` in a decoded name — `sa"les` — must survive: treated as opening a +/// quote it would be dropped, filing the ingest's `sa"les` under `sales` while +/// the annotation's `"sa""les"` decodes to `sa"les`, which is the split this +/// canonicalization exists to prevent. +fn unquote_schema_segment(s: &str) -> String { + let s = s.trim(); + let mut parts: Vec<&str> = Vec::new(); + let mut start = 0usize; + let mut quote: Option = None; + let bytes: Vec<(usize, char)> = s.char_indices().collect(); + let mut k = 0usize; + while k < bytes.len() { + let (i, c) = bytes[k]; + match quote { + Some(q) => { + let close = if q == '[' { ']' } else { q }; + if c == close { + // Doubled: a literal delimiter, so the quote stays open. + if bytes.get(k + 1).map(|(_, n)| *n) == Some(close) { + k += 1; + } else { + quote = None; + } + } + } + // Only a quote that OPENS the part can be a delimiter; one in the + // middle of a decoded identifier is part of the name. + None if (c == '"' || c == '`' || c == '[') && i == start => quote = Some(c), + // Only an UNQUOTED period separates the database from the schema; + // one inside an identifier is part of the name. + None if c == '.' => { + parts.push(&s[start..i]); + start = i + c.len_utf8(); + } + None => {} + } + k += 1; + } + parts.push(&s[start..]); + parts + .into_iter() + .map(unquote_identifier) + .collect::>() + .join(".") +} + // Tokenize a `key=value [key="quoted value"] ...` option string. Bare // values run until the next whitespace; quoted values consume until the // matching quote. Malformed pairs (missing `=` or empty key) are skipped @@ -1555,11 +1687,11 @@ mod pipeline_annotation_tests { // objects and must never collapse to one identity. assert_eq!( parse_asset_syntax("s3:///exports/x", false), - Some((AssetKind::S3Object, "/exports/x")) + Some((AssetKind::S3Object, Cow::Borrowed("/exports/x"))) ); assert_eq!( parse_asset_syntax("s3://exports/x", false), - Some((AssetKind::S3Object, "exports/x")) + Some((AssetKind::S3Object, Cow::Borrowed("exports/x"))) ); assert_ne!( parse_asset_syntax("s3:///exports/x", false), @@ -1569,28 +1701,181 @@ mod pipeline_annotation_tests { // The `// on` trigger annotation goes through the same function. assert_eq!( parse_asset_syntax("s3:///exports/x", true), - Some((AssetKind::S3Object, "/exports/x")) + Some((AssetKind::S3Object, Cow::Borrowed("/exports/x"))) ); assert_eq!( parse_asset_syntax("s3://secondary_storage/path/to/file.csv", false), - Some((AssetKind::S3Object, "secondary_storage/path/to/file.csv")) + Some(( + AssetKind::S3Object, + Cow::Borrowed("secondary_storage/path/to/file.csv") + )) ); // Hive-partition keys are preserved verbatim. assert_eq!( parse_asset_syntax("s3:///t/year=2024/month=01/f.parquet", false), - Some((AssetKind::S3Object, "/t/year=2024/month=01/f.parquet")) + Some(( + AssetKind::S3Object, + Cow::Borrowed("/t/year=2024/month=01/f.parquet") + )) ); // Non-S3 kinds also keep their suffix verbatim. assert_eq!( parse_asset_syntax("res://f/foo", false), - Some((AssetKind::Resource, "f/foo")) + Some((AssetKind::Resource, Cow::Borrowed("f/foo"))) ); assert_eq!( parse_asset_syntax("ducklake://analytics/orders", false), - Some((AssetKind::Ducklake, "analytics/orders")) + Some((AssetKind::Ducklake, Cow::Borrowed("analytics/orders"))) + ); + } + + // A dbt mart and a native script reading the same warehouse table must land + // on ONE graph node. They only do if every spelling of the relation + // canonicalizes identically — dbt's manifest gives it pre-quoted, the + // annotation is hand-written, and the warehouses fold case in opposite + // directions. A regression here is invisible: both nodes still render, they + // just stop being the same node and the cross-boundary cascade never fires. + #[test] + fn table_paths_from_every_spelling_canonicalize_to_one_key() { + let canonical = Some(( + AssetKind::Dbt, + Cow::Owned("main/analytics/orders".into()), + )); + for spelling in [ + // Hand-written annotation. + "dbt://main/analytics/orders", + // dbt manifest `relation_name`, quoted (database segment already + // dropped by the ingest, which keys on the warehouse name instead). + "dbt://main/\"analytics\"/\"orders\"", + // Snowflake folds unquoted identifiers up, Postgres folds down. + "dbt://main/ANALYTICS/ORDERS", + "dbt://main/Analytics/Orders", + // BigQuery / Databricks backticks, SQL Server brackets. + "dbt://main/`analytics`/`orders`", + "dbt://main/[Analytics]/[Orders]", + ] { + assert_eq!( + parse_asset_syntax(spelling, false), + canonical, + "spelling {spelling} did not canonicalize" + ); + } + } + + // A relation that overrode its database carries `.` in + // one segment, and each half can be quoted independently. Stripping only + // the outer pair leaves a key the manifest ingest never produces, so the + // model and a native script reading it become separate nodes. + #[test] + fn qualified_schema_segments_unquote_each_half() { + let canonical = Some(( + AssetKind::Dbt, + Cow::Owned("main/archive.sales/orders".into()), + )); + for spelling in [ + "dbt://main/archive.sales/orders", + "dbt://main/\"Archive\".\"Sales\"/\"Orders\"", + "dbt://main/`Archive`.`Sales`/`Orders`", + "dbt://main/[Archive].[Sales]/[Orders]", + ] { + assert_eq!( + parse_asset_syntax(spelling, false), + canonical, + "spelling {spelling} did not canonicalize" + ); + } + // A period INSIDE one quoted identifier is part of the name, not a + // database qualifier. + assert_eq!( + parse_asset_syntax("dbt://main/\"sales.v2\"/orders", false), + Some(( + AssetKind::Dbt, + Cow::Owned("main/sales.v2/orders".into()) + )) + ); + } + + // Every dialect here escapes its own delimiter by doubling it, and the + // worker's `split_relation` decodes it — so a canonicalizer that did not + // would file the dbt model under `sales"east` and the native script reading + // it under `sales""east`, which is the split this key exists to prevent. + #[test] + fn a_doubled_delimiter_canonicalizes_like_the_relation_it_names() { + for (spelling, expected) in [ + ( + "dbt://main/\"sales\"\"east\"/\"orders\"", + "main/sales\"east/orders", + ), + ( + "dbt://main/analytics/\"order\"\"s\"", + "main/analytics/order\"s", + ), + ( + "dbt://main/`da``ta`/`orders`", + "main/da`ta/orders", + ), + ( + "dbt://main/[my]]schema]/[orders]", + "main/my]schema/orders", + ), + // And in one half of a database-qualified segment. + ( + "dbt://main/\"arch\"\"ive\".\"sales\"/orders", + "main/arch\"ive.sales/orders", + ), + ] { + assert_eq!( + parse_asset_syntax(spelling, false), + Some((AssetKind::Dbt, Cow::Owned(expected.into()))), + "spelling {spelling} did not canonicalize" + ); + } + // The DECODED form has to land on the same key, in both halves: this + // function sees the warehouse's spelling from an annotation and an + // already-decoded identifier from the ingest, and cannot tell them + // apart. A lone delimiter treated as opening a quote would be dropped — + // `sa"les` filed as `sales` — and the two derivations would split. + for (decoded, spelled) in [ + ("dbt://main/sa\"les/orders", "dbt://main/\"sa\"\"les\"/orders"), + ("dbt://main/analytics/order\"s", "dbt://main/analytics/\"order\"\"s\""), + ( + "dbt://main/arch\"ive.sales/orders", + "dbt://main/\"arch\"\"ive\".\"sales\"/orders", + ), + ] { + assert_eq!( + parse_asset_syntax(decoded, false), + parse_asset_syntax(spelled, false), + "the decoded and quoted spellings of {decoded} disagree" + ); + } + } + + #[test] + fn table_canonicalization_leaves_the_warehouse_name_case_alone() { + // The warehouse-name prefix is spelled as the workspace configures it + // and is case-sensitive; only the two identifier segments are folded. + assert_eq!( + parse_asset_syntax("dbt://MyWarehouse/Sales/Orders", false), + Some(( + AssetKind::Dbt, + Cow::Owned("MyWarehouse/sales/orders".into()) + )) + ); + // A database-qualified schema keeps its own case rules: the whole + // segment folds, dot included. + assert_eq!( + parse_asset_syntax("dbt://main/PROD.Sales/Orders", false), + Some((AssetKind::Dbt, Cow::Owned("main/prod.sales/orders".into()))) + ); + // Too few segments to be a relation: left alone rather than reshaped + // into something that looks well-formed. + assert_eq!( + parse_asset_syntax("dbt://Orders", false), + Some((AssetKind::Dbt, Cow::Owned("Orders".into()))) ); } diff --git a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs index 1936f7ff70..9493a78606 100644 --- a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs +++ b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs @@ -110,6 +110,7 @@ fn kind_str(k: AssetKind) -> &'static str { AssetKind::Ducklake => "ducklake", AssetKind::DataTable => "datatable", AssetKind::Volume => "volume", + AssetKind::Dbt => "dbt", } } diff --git a/backend/src/main.rs b/backend/src/main.rs index 969fd4e94d..51febad6f0 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -932,46 +932,62 @@ async fn windmill_main() -> anyhow::Result<()> { } // Lower the worker's oom_score_adj so the OOM killer strongly prefers killing - // job subprocesses (oom_score_adj=1000) over the worker itself. + // job subprocesses (oom_score_adj=JOB_OOM_SCORE_ADJ) over the worker itself. // Kubernetes sets it high for burstable QoS (e.g. 937), leaving a tiny gap vs jobs. // Requires CAP_SYS_RESOURCE to lower it; if missing, we just warn. #[cfg(any(target_os = "linux"))] - match std::fs::read_to_string("/proc/self/oom_score_adj") { - Ok(current) => { - let current = current.trim().to_string(); - let current_val = match current.parse::() { - Ok(v) => v, - Err(e) => { - tracing::warn!("Could not parse oom_score_adj '{current}': {e}"); - 0 - } - }; - if current_val > 0 { - match std::fs::write("/proc/self/oom_score_adj", "0") { - Ok(_) => { - tracing::info!( - "Lowered worker oom_score_adj from {current} to 0 \ - (jobs get 1000, gap=1000)" - ); + { + // Badness is (memory used, in permille of host RAM) + oom_score_adj, so the gap + // must exceed the worker's own footprint in permille to actually steer the kill. + // 100 covers a worker holding up to ~10% of host RAM. + const MIN_OOM_SCORE_GAP: i32 = 100; + + let job_adj = *windmill_common::worker::JOB_OOM_SCORE_ADJ; + match std::fs::read_to_string("/proc/self/oom_score_adj") { + Ok(current) => { + let current = current.trim().to_string(); + match current.parse::() { + Ok(mut worker_adj) => { + if worker_adj > 0 { + match std::fs::write("/proc/self/oom_score_adj", "0") { + Ok(_) => { + tracing::info!( + "Lowered worker oom_score_adj from {worker_adj} to 0" + ); + worker_adj = 0; + } + Err(e) => { + tracing::warn!( + "Could not lower worker oom_score_adj from {worker_adj} to 0: {e}. \ + Add CAP_SYS_RESOURCE to the container to fix this" + ); + } + } + } + let gap = job_adj - worker_adj; + if gap >= MIN_OOM_SCORE_GAP { + tracing::info!( + "Worker oom_score_adj={worker_adj}, jobs get {job_adj} (gap={gap})" + ); + } else { + tracing::warn!( + "Worker oom_score_adj={worker_adj}, jobs get {job_adj} (gap={gap}): \ + too small to reliably steer the OOM killer to the job. \ + Raise JOB_OOM_SCORE_ADJ or lower the worker's own score" + ); + } } Err(e) => { tracing::warn!( - "Could not lower worker oom_score_adj from {current} to 0: {e}. \ - Gap to jobs is only {} — OOM killer may target the worker instead. \ - Add CAP_SYS_RESOURCE to the container to fix this", - 1000 - current_val + "Could not parse worker oom_score_adj '{current}': {e}. \ + Cannot tell whether jobs (oom_score_adj={job_adj}) outrank the worker" ); } } - } else { - tracing::info!( - "Worker oom_score_adj={current} (jobs get 1000, gap={})", - 1000 - current_val - ); } - } - Err(e) => { - tracing::warn!("Could not read worker oom_score_adj: {e}"); + Err(e) => { + tracing::warn!("Could not read worker oom_score_adj: {e}"); + } } } } diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index ccff5c2c7b..a9f747cfe5 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -46,7 +46,7 @@ use windmill_common::otel_oss::{ use windmill_common::{ agent_workers::DECODED_AGENT_TOKEN, apps::APP_WORKSPACED_ROUTE, - auth::create_token_for_owner, + auth::{create_token_for_owner, ephemeral_script_token_label}, ee_oss::CriticalErrorChannel, email_oss::send_email_if_possible, error, @@ -117,17 +117,18 @@ use windmill_queue::{ schedule::{find_unarmed_schedules, rearm_schedule, RearmOutcome}, SameWorkerPayload, }; +use windmill_store::resources::MAX_RESOURCE_VERSIONS; use windmill_worker::{ result_processor::handle_job_error, JobCompletedSender, JobIsolationLevel, - OtelTracingProxySettings, SameWorkerSender, WorkspaceRegistryMap, BUNFIG_INSTALL_SCOPES, - BUN_INSTALL_MIN_RELEASE_AGE, CARGO_REGISTRIES, INSTANCE_PYTHON_VERSION, JAVA_HOME_DIR, - JOB_DEFAULT_TIMEOUT, JOB_ISOLATION, KEEP_JOB_DIR, MAVEN_REPOS, MAVEN_SETTINGS_XML, - NO_DEFAULT_MAVEN, NPMRC, NPM_CONFIG_REGISTRY, NSJAIL_AVAILABLE, NSJAIL_TMPFS_SIZE_MB, - NSJAIL_TMP_BACKING, NUGET_CONFIG, OTEL_TRACING_PROXY_SETTINGS, PIP_EXTRA_INDEX_URL, - PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, SANDBOX_IMAGE_CACHE_MAX_MB, - SANDBOX_IMAGE_DEFAULT_REGISTRY, SANDBOX_IMAGE_MAX_SIZE_MB, SANDBOX_IMAGE_PULL_POLICY, - SANDBOX_REGISTRY_AUTH, UNSHARE_PATH, UV_EXCLUDE_NEWER, UV_INDEX_STRATEGY, - UV_PYTHON_INSTALL_MIRROR, WORKSPACE_REGISTRIES, + OtelTracingProxySettings, SameWorkerSender, StepFailureKind, WorkspaceRegistryMap, + BUNFIG_INSTALL_SCOPES, BUN_INSTALL_MIN_RELEASE_AGE, CARGO_REGISTRIES, INSTANCE_PYTHON_VERSION, + JAVA_HOME_DIR, JOB_DEFAULT_TIMEOUT, JOB_ISOLATION, KEEP_JOB_DIR, MAVEN_REPOS, + MAVEN_SETTINGS_XML, NO_DEFAULT_MAVEN, NPMRC, NPM_CONFIG_REGISTRY, NSJAIL_AVAILABLE, + NSJAIL_TMPFS_SIZE_MB, NSJAIL_TMP_BACKING, NUGET_CONFIG, OTEL_TRACING_PROXY_SETTINGS, + PIP_EXTRA_INDEX_URL, PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, + SANDBOX_IMAGE_CACHE_MAX_MB, SANDBOX_IMAGE_DEFAULT_REGISTRY, SANDBOX_IMAGE_MAX_SIZE_MB, + SANDBOX_IMAGE_PULL_POLICY, SANDBOX_REGISTRY_AUTH, UNSHARE_PATH, UV_EXCLUDE_NEWER, + UV_INDEX_STRATEGY, UV_PYTHON_INSTALL_MIRROR, WORKSPACE_REGISTRIES, }; #[cfg(feature = "parquet")] @@ -1258,6 +1259,39 @@ async fn report_token_expiration(db: &DB, token: &TokenRow, expired: bool) { } } +/// Trim resource version history down to the per-path cap. +/// +/// Deliberately not part of `delete_expired_items` (which runs every monitor tick): this is a +/// full pass over `resource_version`, and the cap only has to hold eventually, so the call site +/// gates it to the same rare cadence as the other heavy sweeps. +pub async fn trim_resource_versions(db: &DB) -> () { + let trimmed = sqlx::query_scalar!( + "DELETE FROM resource_version rv + USING ( + SELECT id FROM ( + SELECT id, row_number() OVER ( + PARTITION BY workspace_id, path ORDER BY id DESC + ) AS rn + FROM resource_version + ) ranked WHERE rn > $1 + ) over_cap + WHERE rv.id = over_cap.id + RETURNING rv.id", + MAX_RESOURCE_VERSIONS, + ) + .fetch_all(db) + .await; + + match trimmed { + Ok(ids) => { + if ids.len() > 0 { + tracing::info!("trimmed {} resource versions past the cap", ids.len()) + } + } + Err(e) => tracing::error!("Error trimming resource versions: {}", e.to_string()), + } +} + pub async fn delete_expired_items(db: &DB) -> () { let expired_tokens_r = sqlx::query_as!( TokenRow, @@ -3177,6 +3211,15 @@ pub async fn monitor_db( } }; + // run every 120 iterations (~20min at the default LISTEN_NEW_EVENTS_INTERVAL_SEC) + let trim_resource_versions_f = async { + if server_mode && iteration.is_some() && iteration.as_ref().unwrap().should_run(120) { + if let Some(db) = conn.as_sql() { + trim_resource_versions(&db).await; + } + } + }; + // run every hour let vacuum_queue_f = async { if server_mode && iteration.is_some() && iteration.as_ref().unwrap().should_run(60) { @@ -3450,6 +3493,7 @@ pub async fn monitor_db( expired_items_f, zombie_jobs_f, stale_jobs_f, + trim_resource_versions_f, vacuum_queue_f, expose_queue_metrics_f, verify_license_key_f, @@ -4662,13 +4706,7 @@ async fn handle_zombie_jobs(db: &Pool, base_internal_url: &str, node_n continue; } if let Some(job) = job.unwrap() { - let label = if job.permissioned_as != format!("u/{}", job.created_by) - && job.permissioned_as != job.created_by - { - format!("ephemeral-script-end-user-{}", job.created_by) - } else { - "ephemeral-script".to_string() - }; + let label = ephemeral_script_token_label(&job.permissioned_as, &job.created_by); let token = create_token_for_owner( &db, &job.workspace_id, @@ -4706,7 +4744,12 @@ async fn handle_zombie_jobs(db: &Pool, base_internal_url: &str, node_n memory_peak, None, error::Error::ExecutionErr(error_message.clone()), - matches!(error_kind, ErrorMessage::SameWorker), // unrecoverable if the job is a same worker zombie + // a same worker zombie means the worker itself is gone + if matches!(error_kind, ErrorMessage::SameWorker) { + StepFailureKind::Unrecoverable + } else { + StepFailureKind::Normal + }, Some(&same_worker_tx_never_used), "", node_name, diff --git a/backend/summarized_schema.txt b/backend/summarized_schema.txt index b47a6556cd..44b62ca51e 100644 --- a/backend/summarized_schema.txt +++ b/backend/summarized_schema.txt @@ -3,7 +3,7 @@ ## ENUMs action_kind: create, update, delete, execute asset_access_type: r, w, rw -asset_kind: s3object, resource, variable, ducklake, datatable +asset_kind: s3object, resource, variable, ducklake, datatable, table asset_usage_kind: script, flow, job authentication_method: none, windmill, api_key, basic_http, custom_script, signature autoscaling_event_type: full_scaleout, scalein, scaleout @@ -19,6 +19,7 @@ job_status: success, failure, canceled, skipped job_trigger_kind: webhook, http, websocket, kafka, email, nats, schedule, app, ui, postgres, sqs, gcp, mqtt, nextcloud, ci_test log_mode: standalone, server, worker, agent, indexer, mcp login_type: password, github +materialization_status: running, materialized, failed message_type: user, assistant, tool metric_kind: scalar_int, scalar_float, timeseries_int, timeseries_float mqtt_client_version: v3, v5 @@ -26,7 +27,7 @@ native_trigger_service: nextcloud request_type: sync, async, sync_sse runnable_type: ScriptHash, ScriptPath, FlowPath script_kind: script, trigger, failure, command, approval, preprocessor -script_lang: python3, deno, go, bash, postgresql, nativets, bun, mysql, bigquery, snowflake, graphql, powershell, mssql, php, bunnative, rust, ansible, csharp, oracledb, nu, java, duckdb, ruby, rlang +script_lang: python3, deno, go, bash, postgresql, nativets, bun, mysql, bigquery, snowflake, graphql, powershell, mssql, php, bunnative, rust, ansible, csharp, oracledb, nu, java, duckdb, ruby, rlang, dbt trigger_kind: webhook, http, websocket, kafka, email, nats, postgres, sqs, mqtt, gcp, default_email, nextcloud trigger_mode: enabled, disabled, suspended workspace_key_kind: cloud @@ -65,6 +66,16 @@ ci_test_reference: workspace_id(char), test_script_path(char), test_script_hash( concurrency_settings: hash(bigint), concurrency_key(char), concurrent_limit(int), concurrency_time_window_s(int) config: name(char), config(jsonb) custom_concurrency_key_ended: key(char), ended_at(ts) +dbt_edge: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), parent_unique_id(text), child_unique_id(text), ingested_at(ts) + FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash) +dbt_graph_snapshot: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), digest(text), relation_root_at_last_ingest(text), ingested_at(ts), permissioned_as(char) + FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash) +dbt_node: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), unique_id(text), resource_type(text), name(text), asset_path(text), materialized(text), materialize_strategy(text), unique_key(text), tags(text[]), description(text), test_kind(text), test_column(text), test_args(jsonb), severity(text), attached_node(text), columns(jsonb), freshness(jsonb), raw_code(text), original_file_path(text), ingested_at(ts) + FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash) +dbt_run_progress: workspace_id(char), job_id(uuid), asset_kind(asset_kind), asset_path(char), status(materialization_status), row_count(bigint), error(text), updated_at(ts) + FK: (workspace_id) -> workspace(id) +dbt_run_state: workspace_id(char), script_path(char), permissioned_as(char), identity(text), args(jsonb), run_results(text), job_id(uuid), retryable(bool), updated_at(ts) + FK: (workspace_id) -> workspace(id) debounce_key: key(char), job_id(uuid), previous_job_id(uuid), first_started_at(ts), debounced_times(int) debounce_stale_data: job_id(uuid), to_relock(text[]) debouncing_settings: hash(bigint), debounce_key(char), debounce_delay_s(int), max_total_debouncing_time(int), max_total_debounces_amount(int), debounce_args_to_accumulate(text[]) @@ -76,7 +87,7 @@ draft: workspace_id(char), path(char), typ(draft_type), value(json), created_at( email_to_igroup: email(char), igroup(char) email_trigger: path(char), local_part(char), workspaced_local_part(bool), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode), labels(text[]) favorite: usr(char), workspace_id(char), path(char), favorite_kind(favorite_kind) -flow: workspace_id(char), path(char), summary(text), description(text), value(jsonb), edited_by(char), edited_at(ts), archived(bool), schema(json), extra_perms(jsonb), dependency_job(uuid), draft_only(bool), tag(char), ws_error_handler_muted(bool), dedicated_worker(bool), timeout(int), visible_to_runner_only(bool), concurrency_key(char), versions(bigint[]), on_behalf_of_email(text), lock_error_logs(text), labels(text[]) +flow: workspace_id(char), path(char), summary(text), description(text), value(jsonb), edited_by(char), edited_at(ts), archived(bool), schema(json), extra_perms(jsonb), dependency_job(uuid), draft_only(bool), tag(char), ws_error_handler_muted(bool), dedicated_worker(bool), timeout(int), visible_to_runner_only(bool), concurrency_key(char), versions(bigint[]), on_behalf_of(varchar), on_behalf_of_email(text), lock_error_logs(text), labels(text[]) FK: (workspace_id) -> workspace(id) flow_conversation: id(uuid), workspace_id(char), flow_path(char), title(char), created_at(ts), updated_at(ts), created_by(char) FK: (workspace_id) -> workspace(id) @@ -153,7 +164,7 @@ resume_job: id(uuid), job(uuid), flow(uuid), created_at(ts), value(jsonb), appro runnable_settings: hash(bigint), debouncing_settings(bigint), concurrency_settings(bigint) schedule: workspace_id(char), path(char), edited_by(char), edited_at(ts), schedule(char), enabled(bool), script_path(char), args(jsonb), extra_perms(jsonb), is_flow(bool), email(char), error(text), timezone(char), on_failure(char), on_recovery(char), on_failure_times(int), on_failure_exact(bool), on_failure_extra_args(jsonb), on_recovery_times(int), on_recovery_extra_args(jsonb), ws_error_handler_muted(bool), retry(jsonb), summary(char), no_flow_overlap(bool), tag(char), paused_until(ts), on_success(char), on_success_extra_args(jsonb), cron_version(text), description(text), dynamic_skip(char), labels(text[]) FK: (workspace_id) -> workspace(id) -script: workspace_id(char), hash(bigint), path(char), parent_hashes(bigint[]), summary(text), description(text), content(text), created_by(char), created_at(ts), archived(bool), schema(json), deleted(bool), is_template(bool), extra_perms(jsonb), lock(text), lock_error_logs(text), language(script_lang), kind(script_kind), tag(char), draft_only(bool), envs(char), concurrent_limit(int), concurrency_time_window_s(int), cache_ttl(int), dedicated_worker(bool), ws_error_handler_muted(bool), priority(smallint), timeout(int), delete_after_use(bool), restart_unless_cancelled(bool), concurrency_key(char), visible_to_runner_only(bool), auto_kind(varchar), codebase(char), has_preprocessor(bool), on_behalf_of_email(text), schema_validation(bool), assets(jsonb), debounce_key(char), debounce_delay_s(int), cache_ignore_s3_path(bool), runnable_settings_handle(bigint), labels(text[]) +script: workspace_id(char), hash(bigint), path(char), parent_hashes(bigint[]), summary(text), description(text), content(text), created_by(char), created_at(ts), archived(bool), schema(json), deleted(bool), is_template(bool), extra_perms(jsonb), lock(text), lock_error_logs(text), language(script_lang), kind(script_kind), tag(char), draft_only(bool), envs(char), concurrent_limit(int), concurrency_time_window_s(int), cache_ttl(int), dedicated_worker(bool), ws_error_handler_muted(bool), priority(smallint), timeout(int), delete_after_use(bool), restart_unless_cancelled(bool), concurrency_key(char), visible_to_runner_only(bool), auto_kind(varchar), codebase(char), has_preprocessor(bool), schema_validation(bool), assets(jsonb), debounce_key(char), debounce_delay_s(int), cache_ignore_s3_path(bool), runnable_settings_handle(bigint), labels(text[]), on_behalf_of(varchar), on_behalf_of_email(text) FK: (workspace_id) -> workspace(id) skip_workspace_diff_tally: workspace_id(char), added_at(ts) sqs_trigger: path(char), queue_url(char), aws_resource_path(char), message_attributes(text[]), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), error(text), server_id(char), last_server_ping(ts), aws_auth_resource_type(aws_auth_resource_type), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode), labels(text[]) @@ -204,7 +215,7 @@ workspace_protection_rule: workspace_id(char), name(char), rules(int), bypass_gr FK: (workspace_id) -> workspace(id) workspace_runnable_dependencies: flow_path(char), runnable_path(char), script_hash(bigint), runnable_is_flow(bool), workspace_id(char), app_path(char) FK: (app_path, workspace_id) -> app(path, workspace_id) | (flow_path, workspace_id) -> flow(path, workspace_id) -workspace_settings: workspace_id(char), slack_team_id(char), slack_name(char), slack_command_script(char), slack_email(char), customer_id(char), plan(char), webhook(text), ai_config(jsonb), large_file_storage(jsonb), git_sync(jsonb), default_app(char), default_scripts(jsonb), deploy_ui(jsonb), mute_critical_alerts(bool), color(char), operator_settings(jsonb), teams_command_script(text), teams_team_id(text), teams_team_name(text), git_app_installations(jsonb), ducklake(jsonb), slack_oauth_client_id(char), slack_oauth_client_secret(char), datatable(jsonb), teams_team_guid(text), auto_invite(jsonb), error_handler(jsonb), success_handler(jsonb), public_app_execution_limit_per_minute(int) +workspace_settings: workspace_id(char), slack_team_id(char), slack_name(char), slack_command_script(char), slack_email(char), customer_id(char), plan(char), webhook(text), ai_config(jsonb), large_file_storage(jsonb), git_sync(jsonb), default_app(char), default_scripts(jsonb), deploy_ui(jsonb), mute_critical_alerts(bool), color(char), operator_settings(jsonb), teams_command_script(text), teams_team_id(text), teams_team_name(text), git_app_installations(jsonb), ducklake(jsonb), slack_oauth_client_id(char), slack_oauth_client_secret(char), datatable(jsonb), teams_team_guid(text), auto_invite(jsonb), error_handler(jsonb), success_handler(jsonb), public_app_execution_limit_per_minute(int), dbt_warehouses(jsonb) FK: (workspace_id) -> workspace(id) zombie_job_counter: job_id(uuid), counter(int) FK: (job_id) -> v2_job(id) diff --git a/backend/tests/app_anonymous_execution_mode.rs b/backend/tests/app_anonymous_execution_mode.rs index a40d880f1b..286581c042 100644 --- a/backend/tests/app_anonymous_execution_mode.rs +++ b/backend/tests/app_anonymous_execution_mode.rs @@ -282,8 +282,8 @@ async fn test_restrict_anonymous_app_deployment_rule(db: Pool) -> anyh async fn seed_script(db: &Pool, path: &str, content: &str) -> anyhow::Result<()> { sqlx::query( r#"INSERT INTO script (workspace_id, hash, path, summary, description, content, - created_by, on_behalf_of_email, language, tag, lock) - VALUES ('test-workspace', hashtext($2)::bigint, $1, '', '', $2, 'test-user', 'test@windmill.dev', + created_by, on_behalf_of, language, tag, lock) + VALUES ('test-workspace', hashtext($2)::bigint, $1, '', '', $2, 'test-user', 'u/test-user', 'deno'::script_lang, 'deno', '')"#, ) .bind(path) diff --git a/backend/tests/app_preview_auth.rs b/backend/tests/app_preview_auth.rs index 2a69004a7b..cd84e6f278 100644 --- a/backend/tests/app_preview_auth.rs +++ b/backend/tests/app_preview_auth.rs @@ -22,7 +22,9 @@ //! - run mode against a deployed Viewer app rejects caller-supplied inline //! `raw_code` whose sha is not publisher-pinned (CVE-2026-22683 residual: //! the Viewer default-triggerable fallback let any caller / an operator run -//! arbitrary code as themselves, bypassing the content-hash pin). +//! arbitrary code as themselves, bypassing the content-hash pin), and +//! - a path-qualified `apps:run|write:` token reaches only that app +//! (the route layer is resource-blind, so the handler must path-check). use serde_json::json; use sqlx::{Pool, Postgres}; @@ -344,5 +346,47 @@ async fn test_app_preview_authorization(db: Pool) -> anyhow::Result<() "rejection must be the content-hash pin (unpinned app_script sha), got: {body}" ); + // 12. Path confinement: the scope picker mints `apps:run:` / + // `apps:write:`, but the route layer matches domain + action only, so the + // handler is the only place the path is enforced. A token scoped to `vapp` must + // not execute another app's components — those run under that app's identity. + for token in ["APPS_RUN_VAPP_TOKEN", "APPS_WRITE_VAPP_TOKEN"] { + let resp = authed(client().post(format!("{base}/u/test-user/private")), token) + .json(&json!({ "args": {}, "component": "comp", "path": "script/u/test-user/private" })) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!( + status, 403, + "{token} must not execute an app outside its scope path (got {status}): {body}" + ); + assert!( + body.contains("apps:run:u/test-user/private"), + "rejection must be the app path scope gate, got: {body}" + ); + } + + // 13. ...and must not over-block the app it IS scoped to: both tokens clear the + // scope gate for `vapp` and reach the policy (which then rejects the unpinned + // code, as in step 9). `apps:write` covering run is what keeps an app-editor + // token working without it also holding `apps:run`. + for token in ["APPS_RUN_VAPP_TOKEN", "APPS_WRITE_VAPP_TOKEN"] { + let resp = authed(client().post(format!("{base}/u/test-user/vapp")), token) + .json(&run_mode_raw_code) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!( + status, 400, + "{token} must clear the scope gate for its own app and reach the policy (got {status}): {body}" + ); + assert!( + body.contains("forbidden by policy"), + "{token} must be stopped by the policy, not the scope gate, got: {body}" + ); + } + Ok(()) } diff --git a/backend/tests/app_s3_onbehalf.rs b/backend/tests/app_s3_onbehalf.rs index 7b233b505d..c11c94be1e 100644 --- a/backend/tests/app_s3_onbehalf.rs +++ b/backend/tests/app_s3_onbehalf.rs @@ -505,8 +505,8 @@ async fn seed_script(db: &Pool, path: &str, content: &str) -> anyhow:: } sqlx::query( r#"INSERT INTO script (workspace_id, hash, path, summary, description, content, - created_by, on_behalf_of_email, language, tag, lock) - VALUES ('test-workspace', $1, $2, '', '', $3, 'test-user-2', 'test2@windmill.dev', + created_by, on_behalf_of, language, tag, lock) + VALUES ('test-workspace', $1, $2, '', '', $3, 'test-user-2', 'u/test-user-2', 'deno'::script_lang, 'deno', '') ON CONFLICT DO NOTHING"#, ) diff --git a/backend/tests/dependency_map.rs b/backend/tests/dependency_map.rs index 0d58615be0..4520cae848 100644 --- a/backend/tests/dependency_map.rs +++ b/backend/tests/dependency_map.rs @@ -447,6 +447,7 @@ def main(): deployment_message: None, visible_to_runner_only: None, on_behalf_of_email: None, + on_behalf_of: None, preserve_on_behalf_of: None, ws_error_handler_muted: None, labels: None, @@ -514,6 +515,7 @@ def main(): preserve_on_behalf_of: None, labels: None, skip_draft_deletion: None, + allow_kind_change: None, }) .send() .await diff --git a/backend/tests/end_user_email.rs b/backend/tests/end_user_email.rs index 539eeaaa6e..366c220c62 100644 --- a/backend/tests/end_user_email.rs +++ b/backend/tests/end_user_email.rs @@ -153,6 +153,72 @@ async fn create_raw_app_with_inline_script(port: u16, path: &str) -> anyhow::Res Ok(()) } +/// Create an app whose policy allows triggering the fixture flow +async fn create_app_triggering_flow(port: u16, path: &str, flow_path: &str) -> anyhow::Result<()> { + let url = format!("http://localhost:{}/api/w/test-workspace/apps/create", port); + let resp = authed(client().post(&url), SAME_WS_TOKEN) + .json(&json!({ + "path": path, + "summary": "Test app running a flow for WM_END_USER_EMAIL", + "value": { + "type": "app", + "grid": [], + "subgrids": {}, + "hiddenInlineScripts": [] + }, + "policy": { + "execution_mode": "anonymous", + "on_behalf_of": null, + "on_behalf_of_email": null, + "triggerables_v2": { + flow_path: { + "static_inputs": {}, + "one_of_inputs": {} + } + } + } + })) + .send() + .await?; + if !resp.status().is_success() { + anyhow::bail!( + "create app failed: {} - {}", + resp.status(), + resp.text().await? + ); + } + Ok(()) +} + +async fn run_app_flow( + port: u16, + token: &str, + app_path: &str, + flow_path: &str, +) -> anyhow::Result { + let url = format!( + "http://localhost:{}/api/w/test-workspace/apps_u/execute_component/{}", + port, app_path + ); + let resp = authed(client().post(&url), token) + .json(&json!({ + "args": {}, + "component": "run_flow", + "path": flow_path + })) + .send() + .await?; + if !resp.status().is_success() { + anyhow::bail!( + "app flow run failed: {} - {}", + resp.status(), + resp.text().await? + ); + } + let job_id = resp.text().await?; + wait_for_job_result(port, token, &job_id).await +} + async fn run_app_inline_script( port: u16, token: &str, @@ -328,6 +394,42 @@ async fn test_app_wm_end_user_email(db: Pool) -> anyhow::Result<()> { Ok(()) } +/// Every flow step must see the same end user as the flow itself: `end_user_email` is only +/// stamped on the job the app pushes, so it has to be forwarded down to each step. The +/// fixture flow has two steps on purpose - the first one is pushed from the freshly pulled +/// flow job, every later one from a flow job re-fetched without its `job_perms`. +#[cfg(feature = "deno_core")] +#[sqlx::test(fixtures("base", "end_user_email"))] +async fn test_app_flow_step_wm_end_user_email(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + set_jwt_secret().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let app_path = "f/test/email_flow_app"; + let flow_path = "flow/f/test/get_end_user_email_flow"; + + in_test_worker( + Connection::Sql(db.clone()), + async move { + create_app_triggering_flow(port, app_path, flow_path).await?; + + // The flow result is its last step's result. + let result = run_app_flow(port, OTHER_WS_TOKEN, app_path, flow_path).await?; + assert_eq!( + result, OTHER_WS_EMAIL, + "every flow step should see the end user email, not the app publisher's" + ); + + Ok::<(), anyhow::Error>(()) + }, + port, + ) + .await?; + + Ok(()) +} + #[cfg(feature = "deno_core")] #[sqlx::test(fixtures("base", "end_user_email"))] async fn test_raw_app_wm_end_user_email(db: Pool) -> anyhow::Result<()> { diff --git a/backend/tests/fixtures/app_preview_auth.sql b/backend/tests/fixtures/app_preview_auth.sql index b94e08f3fd..62173d687c 100644 --- a/backend/tests/fixtures/app_preview_auth.sql +++ b/backend/tests/fixtures/app_preview_auth.sql @@ -19,6 +19,13 @@ INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin) VA INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, scopes) VALUES (encode(sha256('APPS_RUN_TOKEN'::bytea), 'hex'), 'APPS_RUN_T', 'APPS_RUN_TOKEN', 'test2@windmill.dev', 'apps:run scoped token', false, '{apps:run}'); +-- Path-qualified app scopes, as the token scope picker mints them. Both must be +-- confined to `u/test-user/vapp` on the execution route (`apps:write` covers run +-- for the same app, never for another one). +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, scopes) VALUES + (encode(sha256('APPS_RUN_VAPP_TOKEN'::bytea), 'hex'), 'APPS_RUN_V', 'APPS_RUN_VAPP_TOKEN', 'test2@windmill.dev', 'apps:run path-scoped token', false, '{apps:run:u/test-user/vapp}'), + (encode(sha256('APPS_WRITE_VAPP_TOKEN'::bytea), 'hex'), 'APPS_WRIT_', 'APPS_WRITE_VAPP_TOKEN', 'test2@windmill.dev', 'apps:write path-scoped token', false, '{apps:write:u/test-user/vapp}'); + -- A private app owned by `test-user` with a persisted inline script. Used to -- assert that `test-user-2` cannot preview-execute another app's app_script id. INSERT INTO app (id, workspace_id, path, summary, policy, versions) VALUES diff --git a/backend/tests/fixtures/end_user_email.sql b/backend/tests/fixtures/end_user_email.sql index 17d459f7de..596d34e3e3 100644 --- a/backend/tests/fixtures/end_user_email.sql +++ b/backend/tests/fixtures/end_user_email.sql @@ -51,7 +51,7 @@ INSERT INTO flow (workspace_id, summary, description, path, versions, schema, va VALUES ( 'test-workspace', 'Returns WM_END_USER_EMAIL', '', 'f/test/get_end_user_email_flow', '{900002}', '{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"required":[],"type":"object"}', - '{"modules": [{"id": "a", "value": {"type": "rawscript", "language": "deno", "content": "export function main() { return Deno.env.get(\"WM_END_USER_EMAIL\") || \"\"; }", "input_transforms": {}}}]}', + '{"modules": [{"id": "a", "value": {"type": "rawscript", "language": "deno", "content": "export function main() { return Deno.env.get(\"WM_END_USER_EMAIL\") || \"\"; }", "input_transforms": {}}}, {"id": "b", "value": {"type": "rawscript", "language": "deno", "content": "export function main() { return Deno.env.get(\"WM_END_USER_EMAIL\") || \"\"; }", "input_transforms": {}}}]}', 'test-user', '{"g/all": true}' ); @@ -60,6 +60,6 @@ INSERT INTO flow_version (id, workspace_id, path, schema, value, created_by) VALUES ( 900002, 'test-workspace', 'f/test/get_end_user_email_flow', '{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"required":[],"type":"object"}', - '{"modules": [{"id": "a", "value": {"type": "rawscript", "language": "deno", "content": "export function main() { return Deno.env.get(\"WM_END_USER_EMAIL\") || \"\"; }", "input_transforms": {}}}]}', + '{"modules": [{"id": "a", "value": {"type": "rawscript", "language": "deno", "content": "export function main() { return Deno.env.get(\"WM_END_USER_EMAIL\") || \"\"; }", "input_transforms": {}}}, {"id": "b", "value": {"type": "rawscript", "language": "deno", "content": "export function main() { return Deno.env.get(\"WM_END_USER_EMAIL\") || \"\"; }", "input_transforms": {}}}]}', 'test-user' ); diff --git a/backend/tests/fixtures/resource_versioning.sql b/backend/tests/fixtures/resource_versioning.sql new file mode 100644 index 0000000000..81c1dc4549 --- /dev/null +++ b/backend/tests/fixtures/resource_versioning.sql @@ -0,0 +1,24 @@ +-- Fixture for the resource version-history regression test. +-- Just a workspace + admin user + token; the test drives the real resource +-- endpoints and asserts what does and does not land in resource_version. + +INSERT INTO workspace (id, name, owner) VALUES + ('rver-ws', 'RVER WS', 'rver-admin'); + +INSERT INTO workspace_key (workspace_id, kind, key) VALUES + ('rver-ws', 'cloud', 'rver-key'); + +INSERT INTO workspace_settings (workspace_id) VALUES + ('rver-ws'); + +INSERT INTO group_ (workspace_id, name, summary, extra_perms) VALUES + ('rver-ws', 'all', 'All users', '{}'); + +INSERT INTO password(email, password_hash, login_type, super_admin, verified, name, username) + VALUES ('rver-admin@windmill.dev', 'x', 'password', true, true, 'RVER Admin', 'rver-admin'); + +INSERT INTO usr(workspace_id, email, username, is_admin, role) VALUES + ('rver-ws', 'rver-admin@windmill.dev', 'rver-admin', true, 'Admin'); + +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin) + VALUES (encode(sha256('RVER_ADMIN_TOKEN'::bytea), 'hex'), 'RVER_ADMIN', 'RVER_ADMIN_TOKEN', 'rver-admin@windmill.dev', 't', true); diff --git a/backend/tests/fixtures/schedule_push.sql b/backend/tests/fixtures/schedule_push.sql index 9312d0bbd9..5dcc9e1528 100644 --- a/backend/tests/fixtures/schedule_push.sql +++ b/backend/tests/fixtures/schedule_push.sql @@ -22,13 +22,13 @@ VALUES ( 'Test script', '', 'f/system/test_script', 100001, 'deno', '', 'script' ); --- A script with on_behalf_of_email -INSERT INTO script (workspace_id, created_by, content, schema, summary, description, path, hash, language, lock, kind, on_behalf_of_email) +-- A script with an on-behalf-of identity +INSERT INTO script (workspace_id, created_by, content, schema, summary, description, path, hash, language, lock, kind, on_behalf_of) VALUES ( 'test-workspace', 'test-user', 'export async function main() { return "obo"; }', '{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"required":[],"type":"object"}', - 'OBO script', '', 'f/system/obo_script', 100002, 'deno', '', 'script', 'obo@windmill.dev' + 'OBO script', '', 'f/system/obo_script', 100002, 'deno', '', 'script', 'u/obo-user' ); -- A script with a tag @@ -66,13 +66,13 @@ VALUES ( 'test-user' ); --- A flow with on_behalf_of_email -INSERT INTO flow (workspace_id, summary, description, path, versions, schema, value, edited_by, on_behalf_of_email) +-- A flow with an on-behalf-of identity +INSERT INTO flow (workspace_id, summary, description, path, versions, schema, value, edited_by, on_behalf_of) VALUES ( 'test-workspace', 'OBO flow', '', 'f/system/obo_flow', '{200002}', '{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"required":[],"type":"object"}', '{"modules": [{"id": "a", "value": {"path": "f/system/test_script", "type": "script", "input_transforms": {}}}]}', - 'test-user', 'obo@windmill.dev' + 'test-user', 'u/obo-user' ); INSERT INTO flow_version (id, workspace_id, path, schema, value, created_by) diff --git a/backend/tests/fixtures/session_workspace_status.sql b/backend/tests/fixtures/session_workspace_status.sql new file mode 100644 index 0000000000..1d6dbee738 --- /dev/null +++ b/backend/tests/fixtures/session_workspace_status.sql @@ -0,0 +1,15 @@ +-- Extends base.sql with two workspaces `test@windmill.dev` (the instance superadmin) has no +-- `usr` row in: one live, one soft-deleted. The third no-membership case, `admins`, is +-- created by migration with no `usr` rows at all and needs no fixture. + +INSERT INTO workspace (id, name, owner, deleted) VALUES + ('foreign-workspace', 'foreign-workspace', 'someone-else', false), + ('archived-workspace', 'archived-workspace', 'someone-else', true); + +INSERT INTO workspace_key(workspace_id, kind, key) VALUES + ('foreign-workspace', 'cloud', 'test-key'), + ('archived-workspace', 'cloud', 'test-key'); + +INSERT INTO workspace_settings (workspace_id) VALUES + ('foreign-workspace'), + ('archived-workspace'); diff --git a/backend/tests/folder_default_permissioned_as.rs b/backend/tests/folder_default_permissioned_as.rs index 1e71202595..7aa4f0a003 100644 --- a/backend/tests/folder_default_permissioned_as.rs +++ b/backend/tests/folder_default_permissioned_as.rs @@ -475,17 +475,24 @@ async fn test_folder_default_permissioned_as(db: Pool) -> anyhow::Resu resp.text().await? ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "f/prodfolder/jobs/flow_admin_match", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("group-wm_deployers@windmill.dev"), + flow.on_behalf_of.as_deref(), + Some("g/wm_deployers"), "flow should resolve folder default to group email" ); + // The group email is synthetic and resolves back to nobody, so the group can + // only survive as the stored permissioned_as. + assert_eq!( + flow.on_behalf_of.as_deref(), + Some("g/wm_deployers"), + "flow should run as the group the folder rule names" + ); // 5b. Admin, reports/* rule (email directly as permissioned_as) let resp = authed( @@ -502,16 +509,16 @@ async fn test_folder_default_permissioned_as(db: Pool) -> anyhow::Resu resp.text().await? ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "f/prodfolder/reports/weekly", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), - "email rule should pass through as-is" + flow.on_behalf_of.as_deref(), + Some("u/original-user"), + "a rule naming an address stores the principal it names, which is what carries that user's groups and folders" ); // 5c. Non-matching flow path — no default @@ -524,14 +531,14 @@ async fn test_folder_default_permissioned_as(db: Pool) -> anyhow::Resu .await?; assert_eq!(resp.status(), 201); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "f/prodfolder/dev/flow_no_match", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email, None, + flow.on_behalf_of, None, "no folder default ⇒ no on_behalf_of_email written" ); @@ -545,14 +552,14 @@ async fn test_folder_default_permissioned_as(db: Pool) -> anyhow::Resu .await?; assert_eq!(resp.status(), 201); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "u/test-user/outside_flow", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email, None, + flow.on_behalf_of, None, "paths outside folders are never touched" ); @@ -575,15 +582,15 @@ async fn test_folder_default_permissioned_as(db: Pool) -> anyhow::Resu resp.text().await? ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", "f/prodfolder/jobs/new_script", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("group-wm_deployers@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("g/wm_deployers"), "new script at matching path gets folder default" ); diff --git a/backend/tests/jobs_read_auth.rs b/backend/tests/jobs_read_auth.rs index e95748d8de..317b1c6866 100644 --- a/backend/tests/jobs_read_auth.rs +++ b/backend/tests/jobs_read_auth.rs @@ -83,6 +83,32 @@ async fn post(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCo (status, body) } +async fn post_json( + url: &str, + token: Option<&str>, + body: serde_json::Value, +) -> (reqwest::StatusCode, String) { + let mut req = client().post(url).json(&body); + if let Some(token) = token { + req = req.header("Authorization", format!("Bearer {token}")); + } + let resp = req.send().await.expect("request"); + let status = resp.status(); + let body = resp.text().await.expect("body"); + (status, body) +} + +async fn delete(url: &str, token: Option<&str>) -> (reqwest::StatusCode, String) { + let mut req = client().delete(url); + if let Some(token) = token { + req = req.header("Authorization", format!("Bearer {token}")); + } + let resp = req.send().await.expect("request"); + let status = resp.status(); + let body = resp.text().await.expect("body"); + (status, body) +} + #[sqlx::test(fixtures("base", "jobs_read_auth"))] async fn test_single_job_read_authorization(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; @@ -92,6 +118,8 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul let base = format!("http://localhost:{port}/api/w/test-workspace/jobs_u"); // result_by_id / get_otel_traces live on the authed `/jobs` service, not `/jobs_u`. let authed_base = format!("http://localhost:{port}/api/w/test-workspace/jobs"); + let rules_url = + format!("http://localhost:{port}/api/w/test-workspace/workspaces/protection_rules"); // The endpoints that return the victim job's sensitive data by UUID. let endpoints = [ @@ -507,6 +535,161 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul "a non-reader must not be able to mint a share token (got {status})" ); + // ---- PUBLIC SHARE READ LINK (public view_token) ---- + // A member-audience token must never turn into a public one: links already handed + // out stay confined to logged-in members. + let (status, body) = get( + &base, + &format!("completed/get_result/{VICTIM}?view_token={token}"), + None, + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::BAD_REQUEST, + "a member view_token must not grant unauthenticated read (got {status}): {body}" + ); + assert!( + !body.contains(RESULT_SECRET), + "unauth body must not leak with a member token: {body}" + ); + + // The owner mints the public flavor for the top flow. + let (status, mint_body) = get( + &authed_base, + &format!("job_public_view_token/{TOP_SECRET_FLOW}"), + Some("SECRET_TOKEN_2"), + ) + .await; + assert!( + status.is_success(), + "owner must be able to mint a public share token (got {status}): {mint_body}" + ); + let public_token = mint_body.trim().trim_matches('"').to_string(); + + // It grants a logged-out visitor the shared job and its whole flow subtree. + for path in [ + format!("get/{TOP_SECRET_FLOW}?view_token={public_token}"), + format!("get_args/{TOP_SECRET_FLOW}?view_token={public_token}"), + format!("getupdate/{TOP_SECRET_FLOW}?view_token={public_token}"), + format!("completed/get_result/{DEEP_LEAF_JOB}?view_token={public_token}"), + format!("get_logs/{DEEP_LEAF_JOB}?view_token={public_token}"), + ] { + let (status, body) = get(&base, &path, None).await; + assert!( + status.is_success(), + "a public view_token must grant unauthenticated read of {path} (got {status}): {body}" + ); + } + + // Same scoping as the member flavor: another job, and a tampered signature, are refused. + for path in [ + format!("get/{VICTIM}?view_token={public_token}"), + format!("get/{TOP_SECRET_FLOW}?view_token={TOP_SECRET_FLOW}.deadbeef"), + ] { + let (status, body) = get(&base, &path, None).await; + assert_eq!( + status, + reqwest::StatusCode::BAD_REQUEST, + "an out-of-scope or forged public token must not grant read of {path} (got {status}): {body}" + ); + } + + // The public audience is a superset of the member one: it must also satisfy the + // authenticated ACL check, so a viewer handed a public link is not worse off. + let (status, body) = get( + &base, + &format!("completed/get_result/{DEEP_LEAF_JOB}?view_token={public_token}"), + Some("SECRET_TOKEN_3"), + ) + .await; + assert!( + status.is_success(), + "a public view_token must also grant an authenticated member read (got {status}): {body}" + ); + + // Tag-scoped caller, job inside its scope: the member flavor is allowed (asserted + // further down), the public one must not be — i.e. strictly stricter. + let (status, body) = get( + &authed_base, + &format!("job_public_view_token/{VICTIM}"), + Some("SCOPED_DENO_TOKEN"), + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::FORBIDDEN, + "a tag-scoped token must NOT mint a public link, even for an in-scope job (got {status}): {body}" + ); + + // Minting the public flavor takes the same read access as the member one. + let (status, _) = get( + &authed_base, + &format!("job_public_view_token/{TOP_SECRET_FLOW}"), + Some("SECRET_TOKEN_3"), + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::FORBIDDEN, + "a non-reader must not be able to mint a public share token (got {status})" + ); + + // Publishing a run is gated by `RestrictPublicRunSharing`, so a wrong bitflag or + // rule-kind arm would silently let a restricted member expose one. Admins bypass, and + // the member flavor is untouched by the rule. + let (status, body) = post_json( + &rules_url, + Some("SECRET_TOKEN"), + serde_json::json!({ + "name": "no-public-runs", + "rules": ["RestrictPublicRunSharing"], + "bypass_users": [], + "bypass_groups": [] + }), + ) + .await; + assert!( + status.is_success(), + "admin should create the protection rule (got {status}): {body}" + ); + let (status, body) = get( + &authed_base, + &format!("job_public_view_token/{TOP_SECRET_FLOW}"), + Some("SECRET_TOKEN_2"), + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::FORBIDDEN, + "the rule must block a non-admin owner from minting a public link (got {status}): {body}" + ); + let (status, body) = get( + &authed_base, + &format!("job_view_token/{TOP_SECRET_FLOW}"), + Some("SECRET_TOKEN_2"), + ) + .await; + assert!( + status.is_success(), + "the rule must NOT affect the member flavor (got {status}): {body}" + ); + let (status, body) = get( + &authed_base, + &format!("job_public_view_token/{TOP_SECRET_FLOW}"), + Some("SECRET_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "an admin must bypass the rule (got {status}): {body}" + ); + + // Later assertions in this test mint public tokens; drop the rule so they see the + // same workspace state as before. + let (status, _) = delete(&format!("{rules_url}/no-public-runs"), Some("SECRET_TOKEN")).await; + assert!(status.is_success(), "admin should delete the rule"); + // ---- TAG-SCOPED token must not mint a token outside its allowed tags ---- // SCOPED_DENO_TOKEN (test-user-2, scope `if_jobs:filter_tags:deno`) can read both // VICTIM (tag deno) and FLOW_JOB (tag flow) by RLS, but minting must honor the @@ -618,6 +801,36 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul "viewer must not cancel another user's job ({path}, got {status}): {body}" ); } + // A public share link grants read, never the right to kill the run: cancelling stays + // confined to anonymously-created jobs for a logged-out caller. + let (status, mint_body) = get( + &authed_base, + &format!("job_public_view_token/{RUNNING_JOB}"), + Some("SECRET_TOKEN_2"), + ) + .await; + assert!(status.is_success(), "owner mints for the running job"); + let running_public = mint_body.trim().trim_matches('"').to_string(); + let (status, body) = get( + &base, + &format!("get/{RUNNING_JOB}?view_token={running_public}"), + None, + ) + .await; + assert!( + status.is_success(), + "the public token must grant the anonymous read it is for (got {status}): {body}" + ); + let (status, body) = post( + &base, + &format!("queue/cancel/{RUNNING_JOB}?view_token={running_public}"), + None, + ) + .await; + assert!( + !status.is_success(), + "a public token must not let an anonymous caller cancel the run (got {status}): {body}" + ); // The owner still cancels their own job (no over-blocking). Keep this last: it // takes RUNNING_JOB out of the queue. let (status, body) = post( diff --git a/backend/tests/nativets_stress.rs b/backend/tests/nativets_stress.rs index 1a8827b486..32057b41d2 100644 --- a/backend/tests/nativets_stress.rs +++ b/backend/tests/nativets_stress.rs @@ -171,6 +171,7 @@ async fn push_job(db: &Pool, content: &str, args: &serde_json::Value) /* email */ "test@windmill.dev", /* permissioned_as */ "u/test-user".to_string(), /* token_prefix */ None, + /* audit_end_user */ None, /* scheduled_for */ None, /* schedule_path */ None, /* parent_job */ None, diff --git a/backend/tests/on_behalf_of.rs b/backend/tests/on_behalf_of.rs new file mode 100644 index 0000000000..8576bcdfe8 --- /dev/null +++ b/backend/tests/on_behalf_of.rs @@ -0,0 +1,391 @@ +//! A script's on-behalf-of identity must drive the permissions of the jobs it produces, not +//! just their address. The principal is the only stored half; the address is derived from it, +//! so a request naming one, the other, or a mismatched pair all resolve to one account. + +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder { + builder.header("Authorization", format!("Bearer {}", token)) +} + +fn script_body(path: &str, on_behalf_of: Option<&str>) -> serde_json::Value { + let mut body = json!({ + "path": path, + "summary": "", + "description": "", + "content": "export async function main() { return 42; }", + "language": "deno", + "on_behalf_of_email": "original@windmill.dev", + "preserve_on_behalf_of": true, + "auto_parent": true, + }); + if let Some(permissioned_as) = on_behalf_of { + body["on_behalf_of"] = json!(permissioned_as); + } + body +} + +/// Deploys as `test-user` (admin) so the recorded identity is nobody's default: neither +/// the caller's nor the deployer's. Returns the hex hash the run-by-hash route parses. +async fn create_script( + base: &str, + path: &str, + on_behalf_of: Option<&str>, +) -> anyhow::Result { + let resp = authed( + client().post(format!("{base}/scripts/create")), + "SECRET_TOKEN", + ) + .json(&script_body(path, on_behalf_of)) + .send() + .await?; + let status = resp.status(); + let hash = resp.text().await?; + assert_eq!(status, 201, "creating {path}: {hash}"); + Ok(hash.trim().trim_matches('"').to_string()) +} + +async fn run_by_hash(base: &str, hash: &str) -> anyhow::Result { + let resp = authed( + client().post(format!("{base}/jobs/run/h/{hash}")), + "SECRET_TOKEN", + ) + .json(&json!({})) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!(status, 201, "running {hash}: {body}"); + Ok(uuid::Uuid::parse_str(body.trim().trim_matches('"'))?) +} + +async fn stored_permissioned_as( + db: &Pool, + table: &str, + path: &str, +) -> anyhow::Result> { + // `table` is a literal from this test, never caller input. + Ok(sqlx::query_scalar(&format!( + "SELECT on_behalf_of FROM {table} \ + WHERE path = $1 AND workspace_id = 'test-workspace' AND NOT archived" + )) + .bind(path) + .fetch_one(db) + .await?) +} + +#[sqlx::test(fixtures("preserve_on_behalf_of"))] +async fn test_on_behalf_of_drives_job_identity( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let base = format!( + "http://localhost:{}/api/w/test-workspace", + server.addr.port() + ); + + let recorded = + create_script(&base, "u/test-user/obo_recorded", Some("u/original-user")).await?; + assert_eq!( + stored_permissioned_as(&db, "script", "u/test-user/obo_recorded") + .await? + .as_deref(), + Some("u/original-user") + ); + + // Workers predating this release read only the address, and they are expected to lag the + // server, so a deploy keeps filling it in until every live worker is new. + assert_eq!( + sqlx::query_scalar!( + "SELECT on_behalf_of_email FROM script WHERE path = 'u/test-user/obo_recorded' AND workspace_id = 'test-workspace'" + ) + .fetch_one(&db) + .await? + .as_deref(), + Some("original@windmill.dev"), + ); + + // A client that predates the field names only the email; deriving the principal from + // it is what stops a routine redeploy from handing the script to whoever deploys it. + let derived = create_script(&base, "u/test-user/obo_derived", None).await?; + assert_eq!( + stored_permissioned_as(&db, "script", "u/test-user/obo_derived") + .await? + .as_deref(), + Some("u/original-user") + ); + + let recorded_job = run_by_hash(&base, &recorded).await?; + let derived_job = run_by_hash(&base, &derived).await?; + + let jobs = sqlx::query!( + "SELECT id, permissioned_as, permissioned_as_email FROM v2_job WHERE id = ANY($1)", + &[recorded_job, derived_job][..] + ) + .fetch_all(&db) + .await?; + let identity = |id: uuid::Uuid| { + let job = jobs.iter().find(|j| j.id == id).expect("job was pushed"); + ( + job.permissioned_as.clone(), + job.permissioned_as_email.clone(), + ) + }; + + assert_eq!( + identity(recorded_job), + ( + "u/original-user".to_string(), + "original@windmill.dev".to_string() + ), + "a recorded permissioned_as must be what the job runs as" + ); + assert_eq!( + identity(derived_job), + ( + "u/original-user".to_string(), + "original@windmill.dev".to_string() + ), + "a principal derived from the address drives the job the same way an explicit one does" + ); + + // A superadmin acting outside their workspaces has no `usr` row. Dropping them on an + // email-only redeploy would keep their superadmin email next to the deployer's + // permissions — the hybrid identity this whole change exists to remove. + let resp = authed( + client().post(format!("{base}/scripts/create")), + "SECRET_TOKEN", + ) + .json(&json!({ + "path": "u/test-user/obo_superadmin", + "summary": "", + "description": "", + "content": "export async function main() { return 42; }", + "language": "deno", + "on_behalf_of_email": "superadmin-external@windmill.dev", + "preserve_on_behalf_of": true, + })) + .send() + .await?; + assert_eq!(resp.status(), 201, "creating: {}", resp.text().await?); + assert_eq!( + stored_permissioned_as(&db, "script", "u/test-user/obo_superadmin") + .await? + .as_deref(), + Some("u/superadmin-external") + ); + + // The synthetic group namespace is not reserved, so a real account holding such an + // address must win over the like-named group — otherwise an email-only deploy would + // hand the runnable that group's folder access. + sqlx::query!( + "INSERT INTO group_ (workspace_id, name, summary, extra_perms) \ + VALUES ('test-workspace', 'ops', '', '{}') ON CONFLICT DO NOTHING" + ) + .execute(&db) + .await?; + sqlx::query!( + "UPDATE usr SET email = 'group-ops@windmill.dev' WHERE workspace_id = 'test-workspace' \ + AND username = 'test-user-2'" + ) + .execute(&db) + .await?; + let resp = authed( + client().post(format!("{base}/scripts/create")), + "SECRET_TOKEN", + ) + .json(&json!({ + "path": "u/test-user/obo_group_collision", + "summary": "", + "description": "", + "content": "export async function main() { return 42; }", + "language": "deno", + "on_behalf_of_email": "group-ops@windmill.dev", + "preserve_on_behalf_of": true, + })) + .send() + .await?; + assert_eq!(resp.status(), 201, "creating: {}", resp.text().await?); + assert_eq!( + stored_permissioned_as(&db, "script", "u/test-user/obo_group_collision") + .await? + .as_deref(), + Some("u/test-user-2"), + "a real account must win over the like-named group" + ); + + // An address is the principal only for an account whose username is that address; for + // anybody else it is canonicalized, since the bare form carries neither their groups nor + // their folders. + let resp = authed( + client().post(format!("{base}/scripts/create")), + "SECRET_TOKEN", + ) + .json(&script_body( + "u/test-user/obo_bare_address", + Some("original@windmill.dev"), + )) + .send() + .await?; + assert_eq!(resp.status(), 201, "creating: {}", resp.text().await?); + assert_eq!( + stored_permissioned_as(&db, "script", "u/test-user/obo_bare_address") + .await? + .as_deref(), + Some("u/original-user") + ); + + // A job row carries a narrower identity column than the runnable it comes from, so an + // address too long to be enqueued is refused at deploy rather than at the first run — + // whether it is preserved from someone else or is the deployer's own. + const LONG_ADDRESS: &str = "a-very-long-superadmin-address-for-this-test@windmill.dev"; + sqlx::query!( + "INSERT INTO password(email, password_hash, login_type, super_admin, verified, name) + VALUES ($1, '', 'password', true, true, '')", + LONG_ADDRESS + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin) + VALUES (encode(sha256('LONG_TOKEN'::bytea), 'hex'), 'LONG_TOKEN', 'LONG_TOKEN', $1, 'long', true)", + LONG_ADDRESS + ) + .execute(&db) + .await?; + + let too_long = |path: &str| { + json!({ + "path": path, + "summary": "", + "description": "", + "content": "export async function main() { return 42; }", + "language": "deno", + "on_behalf_of_email": LONG_ADDRESS, + "preserve_on_behalf_of": true, + }) + }; + let resp = authed( + client().post(format!("{base}/scripts/create")), + "SECRET_TOKEN", + ) + .json(&too_long("u/test-user/obo_too_long")) + .send() + .await?; + assert_eq!(resp.status(), 400, "an unenqueueable identity must be rejected"); + + // Picking "me" does not preserve anyone, so it takes the branch that stores the caller's + // own principal — which for an account acting without a `usr` row is their address. + let mut own = too_long("u/test-user/obo_too_long_self"); + own["preserve_on_behalf_of"] = json!(false); + let resp = authed( + client().post(format!("{base}/scripts/create")), + "LONG_TOKEN", + ) + .json(&own) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!(status, 400, "{body}"); + assert!( + body.contains("characters a job can carry"), + "the caller's own identity has to be refused by the same check, not by a column \ + overflow further down: {body}" + ); + + // A pair naming two different principals would run as a composite of both. + let resp = authed( + client().post(format!("{base}/scripts/create")), + "SECRET_TOKEN", + ) + .json(&script_body( + "u/test-user/obo_mismatch", + Some("u/test-user-2"), + )) + .send() + .await?; + assert_eq!(resp.status(), 400, "a mismatched pair must be rejected"); + + // The identity a no-op push is compared against is the stored principal, so a push that + // names it by address alone still has to read as unchanged — otherwise every idempotent + // CLI push of a configured script would cut a version and a phantom git-sync commit. + async fn push_noop_guarded(base: &str, body: serde_json::Value) -> anyhow::Result { + let resp = authed( + client().post(format!("{base}/scripts/create?skip_if_noop=true")), + "SECRET_TOKEN", + ) + .json(&body) + .send() + .await?; + let status = resp.status(); + let hash = resp.text().await?; + assert_eq!(status, 201, "creating: {hash}"); + Ok(hash.trim().trim_matches('"').to_string()) + } + // The no-op check compares every field, so the body has to carry the values a deploy + // fills in by itself, or it would be rejected before reaching the identity comparison. + let noop_body = |permissioned_as| { + let mut body = script_body("u/test-user/obo_noop", permissioned_as); + body["ws_error_handler_muted"] = json!(false); + body["assets"] = json!([]); + body + }; + let first = push_noop_guarded(&base, noop_body(Some("u/original-user"))).await?; + let again = push_noop_guarded(&base, noop_body(None)).await?; + assert_eq!( + first, again, + "an identical push naming the same identity by address must not cut a new version" + ); + + // Flows resolve the same way, but through their own UPDATE — which must not drop the + // principal when the body names only the email. + let flow = json!({ + "path": "u/test-user/obo_flow", + "summary": "", + "value": { "modules": [] }, + "on_behalf_of_email": "original@windmill.dev", + "on_behalf_of": "u/original-user", + "preserve_on_behalf_of": true, + }); + let resp = authed( + client().post(format!("{base}/flows/create")), + "SECRET_TOKEN", + ) + .json(&flow) + .send() + .await?; + assert_eq!(resp.status(), 201, "creating flow: {}", resp.text().await?); + + let mut update = flow.clone(); + update["summary"] = json!("edited"); + update + .as_object_mut() + .unwrap() + .remove("on_behalf_of"); + let resp = authed( + client().post(format!("{base}/flows/update/u/test-user/obo_flow")), + "SECRET_TOKEN", + ) + .json(&update) + .send() + .await?; + assert_eq!(resp.status(), 200, "updating flow: {}", resp.text().await?); + assert_eq!( + stored_permissioned_as(&db, "flow", "u/test-user/obo_flow") + .await? + .as_deref(), + Some("u/original-user"), + "an update that names only the email must not drop the flow's principal" + ); + + Ok(()) +} diff --git a/backend/tests/postgres_trigger_scope.rs b/backend/tests/postgres_trigger_scope.rs index 1172e03ab6..cf98a1e6de 100644 --- a/backend/tests/postgres_trigger_scope.rs +++ b/backend/tests/postgres_trigger_scope.rs @@ -20,6 +20,8 @@ fn scoped_authed(scopes: Vec<&str>) -> ApiAuthed { folders: vec![], scopes: Some(scopes.into_iter().map(str::to_string).collect()), username_override: None, + username_override_is_token_label: false, + is_session_token: false, token_prefix: None, read_only: false, } diff --git a/backend/tests/preserve_on_behalf_of.rs b/backend/tests/preserve_on_behalf_of.rs index d808348937..e696de44ea 100644 --- a/backend/tests/preserve_on_behalf_of.rs +++ b/backend/tests/preserve_on_behalf_of.rs @@ -214,15 +214,15 @@ async fn test_preserve_on_behalf_of(db: Pool) -> anyhow::Result<()> { ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2", "u/test-user/script_admin_preserve", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("u/original-user"), "Admin should preserve on_behalf_of_email" ); @@ -249,15 +249,15 @@ async fn test_preserve_on_behalf_of(db: Pool) -> anyhow::Result<()> { ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2", "u/deployer-user/script_deployer_preserve", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("u/original-user"), "Deployer should preserve on_behalf_of_email" ); @@ -284,15 +284,15 @@ async fn test_preserve_on_behalf_of(db: Pool) -> anyhow::Result<()> { ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2", "u/test-user-2/script_no_preserve", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("test2@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("u/test-user-2"), "Non-admin should have their own email as on_behalf_of_email" ); @@ -319,15 +319,15 @@ async fn test_preserve_on_behalf_of(db: Pool) -> anyhow::Result<()> { ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "u/test-user/flow_admin_preserve", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + flow.on_behalf_of.as_deref(), + Some("u/original-user"), "Admin should preserve flow on_behalf_of_email" ); @@ -354,15 +354,15 @@ async fn test_preserve_on_behalf_of(db: Pool) -> anyhow::Result<()> { ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "u/deployer-user/flow_deployer_preserve", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + flow.on_behalf_of.as_deref(), + Some("u/original-user"), "Deployer should preserve flow on_behalf_of_email" ); @@ -389,15 +389,15 @@ async fn test_preserve_on_behalf_of(db: Pool) -> anyhow::Result<()> { ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "u/test-user-2/flow_no_preserve", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("test2@windmill.dev"), + flow.on_behalf_of.as_deref(), + Some("u/test-user-2"), "Non-admin should have their own email as flow on_behalf_of_email" ); @@ -745,15 +745,15 @@ async fn test_preserve_on_behalf_of(db: Pool) -> anyhow::Result<()> { ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2", "u/test-user/script_no_flag", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("test@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("u/test-user"), "Without preserve flag, admin's email should be used" ); @@ -841,15 +841,15 @@ async fn test_script_update_preserves_on_behalf_of(db: Pool) -> anyhow ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", "u/original-user/script_to_update", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("u/original-user"), "Admin update should preserve script on_behalf_of_email" ); @@ -896,15 +896,15 @@ async fn test_script_update_preserves_on_behalf_of(db: Pool) -> anyhow ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", "u/deployer-user/script_deploy_update", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("u/original-user"), "Deployer update should preserve script on_behalf_of_email" ); @@ -951,15 +951,15 @@ async fn test_script_update_preserves_on_behalf_of(db: Pool) -> anyhow ); let script = sqlx::query!( - "SELECT on_behalf_of_email FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", + "SELECT on_behalf_of FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", "u/test-user-2/script_nonadmin_update", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - script.on_behalf_of_email.as_deref(), - Some("test2@windmill.dev"), + script.on_behalf_of.as_deref(), + Some("u/test-user-2"), "Non-admin update should overwrite script on_behalf_of_email with their own" ); @@ -1025,15 +1025,15 @@ async fn test_flow_update_preserves_on_behalf_of(db: Pool) -> anyhow:: ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "u/original-user/flow_to_update", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + flow.on_behalf_of.as_deref(), + Some("u/original-user"), "Admin update should preserve flow on_behalf_of_email" ); @@ -1091,15 +1091,15 @@ async fn test_flow_update_preserves_on_behalf_of(db: Pool) -> anyhow:: ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "u/deployer-user/flow_deploy_update", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("original@windmill.dev"), + flow.on_behalf_of.as_deref(), + Some("u/original-user"), "Deployer update should preserve flow on_behalf_of_email" ); @@ -1157,15 +1157,15 @@ async fn test_flow_update_preserves_on_behalf_of(db: Pool) -> anyhow:: ); let flow = sqlx::query!( - "SELECT on_behalf_of_email FROM flow WHERE path = $1 AND workspace_id = $2", + "SELECT on_behalf_of FROM flow WHERE path = $1 AND workspace_id = $2", "u/test-user-2/flow_nonadmin_update", "test-workspace" ) .fetch_one(&db) .await?; assert_eq!( - flow.on_behalf_of_email.as_deref(), - Some("test2@windmill.dev"), + flow.on_behalf_of.as_deref(), + Some("u/test-user-2"), "Non-admin update should overwrite flow on_behalf_of_email with their own" ); diff --git a/backend/tests/preview_native_tag.rs b/backend/tests/preview_native_tag.rs index 29aefa588a..eba686396a 100644 --- a/backend/tests/preview_native_tag.rs +++ b/backend/tests/preview_native_tag.rs @@ -56,6 +56,7 @@ async fn push_preview_and_get_row( /* email */ "test@windmill.dev", /* permissioned_as */ "u/test-user".to_string(), /* token_prefix */ None, + /* audit_end_user */ None, /* scheduled_for */ None, /* schedule_path */ None, /* parent_job */ None, diff --git a/backend/tests/protection_rules.rs b/backend/tests/protection_rules.rs index 69873bf158..6714b24801 100644 --- a/backend/tests/protection_rules.rs +++ b/backend/tests/protection_rules.rs @@ -421,3 +421,268 @@ async fn test_restrict_deploy_to_deployers(db: Pool) -> anyhow::Result Ok(()) } + +/// The dev-workspace pairing owns `dev_workspace_lock` by name: attaching creates it, detaching +/// deletes it. Only those two ends of the name are reserved. Updating the rule has to stay open, +/// since relaxing a restriction from the rulesets UI is the only way an admin can loosen a pairing's +/// lock without detaching the dev workspace outright. +/// +/// Reads the row directly rather than through `list_protection_rules`, which goes via the +/// process-wide PROTECTION_RULES_CACHE that other tests in this file share. +#[sqlx::test(fixtures("base"))] +async fn test_dev_workspace_lock_rule_reservation(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace"); + let name = windmill_common::workspaces::DEV_WORKSPACE_LOCK_RULE_NAME; + + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules")), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": name, + "rules": ["DisableDirectDeployment"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!( + resp.status(), + 400, + "creating the reserved rule by hand should be refused" + ); + + // Stands in for an attach, which is what really creates the rule. 3 = DisableDirectDeployment | + // DisableWorkspaceForking, the pairing's default lock. + sqlx::query( + "INSERT INTO workspace_protection_rule (workspace_id, name, rules, bypass_groups, bypass_users) + VALUES ('test-workspace', $1, 3, '{}', '{}')", + ) + .bind(name) + .execute(&db) + .await?; + + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules/{name}")), + "SECRET_TOKEN", + ) + .json(&json!({ + "rules": ["DisableWorkspaceForking"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!( + resp.status(), + 200, + "an admin must be able to drop a restriction from the reserved rule: {}", + resp.text().await? + ); + + let rules: i32 = sqlx::query_scalar( + "SELECT rules FROM workspace_protection_rule WHERE workspace_id = 'test-workspace' AND name = $1", + ) + .bind(name) + .fetch_one(&db) + .await?; + assert_eq!(rules, 2, "only DisableWorkspaceForking should remain"); + + // Renaming it away would strand the pairing's lock, which is located by name. + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules/{name}")), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": "renamed-lock", + "rules": ["DisableWorkspaceForking"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!( + resp.status(), + 400, + "renaming the reserved rule should be refused" + ); + + let resp = authed( + client().delete(format!("{base}/workspaces/protection_rules/{name}")), + "SECRET_TOKEN", + ) + .send() + .await?; + assert_eq!( + resp.status(), + 400, + "deleting the reserved rule should stay refused: detaching the dev workspace removes it" + ); + + Ok(()) +} + +/// The name is half the row's primary key, so a rename has to move the row: the update applies the +/// name from the body, not just the one in the path. Collisions and the reserved name are refused, +/// and a name submitted unchanged is left exactly as stored. +#[sqlx::test(fixtures("base"))] +async fn test_protection_rule_rename(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace"); + + for name in ["before-rename", "occupied"] { + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules")), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": name, + "rules": ["DisableDirectDeployment"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!(resp.status(), 200, "setup: create '{}'", name); + } + + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules/before-rename")), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": "after-rename", + "rules": ["DisableWorkspaceForking"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!( + resp.status(), + 200, + "rename should succeed: {}", + resp.text().await? + ); + + let names: Vec = sqlx::query_scalar( + "SELECT name FROM workspace_protection_rule WHERE workspace_id = 'test-workspace' ORDER BY name", + ) + .fetch_all(&db) + .await?; + assert_eq!( + names, + vec!["after-rename".to_string(), "occupied".to_string()], + "the row should have moved to the new name, not been duplicated" + ); + + for (target, why) in [ + ( + "occupied", + "renaming onto an existing rule should be refused", + ), + ( + windmill_common::workspaces::DEV_WORKSPACE_LOCK_RULE_NAME, + "renaming onto the reserved name should be refused", + ), + ] { + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules/after-rename")), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": target, + "rules": ["DisableWorkspaceForking"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!(resp.status(), 400, "{}", why); + } + + // Names are stored verbatim and the editor submits the current name on every save, so a padded + // name has to survive a restrictions-only edit rather than being trimmed into a rename. + let padded = " padded-name "; + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules")), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": padded, + "rules": ["DisableDirectDeployment"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!(resp.status(), 200, "setup: create the padded rule"); + + let resp = authed( + client().post(format!( + "{base}/workspaces/protection_rules/%20padded-name%20" + )), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": padded, + "rules": ["DisableWorkspaceForking"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!( + resp.status(), + 200, + "restrictions-only edit of a padded name should succeed: {}", + resp.text().await? + ); + + let still_there: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM workspace_protection_rule WHERE workspace_id = 'test-workspace' AND name = $1)", + ) + .bind(padded) + .fetch_one(&db) + .await?; + assert!( + still_there, + "the padded name must not have been trimmed into a rename" + ); + + // The mirror case: a name differing only in surrounding whitespace is a real rename, applied + // verbatim rather than collapsing into a success that changed nothing. + let resp = authed( + client().post(format!("{base}/workspaces/protection_rules/after-rename")), + "SECRET_TOKEN", + ) + .json(&json!({ + "name": " after-rename ", + "rules": ["DisableWorkspaceForking"], + "bypass_users": [], + "bypass_groups": [] + })) + .send() + .await?; + assert_eq!( + resp.status(), + 200, + "whitespace-only rename should apply: {}", + resp.text().await? + ); + let renamed: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM workspace_protection_rule WHERE workspace_id = 'test-workspace' AND name = ' after-rename ')", + ) + .fetch_one(&db) + .await?; + assert!(renamed, "the padded form should now be the stored name"); + + Ok(()) +} diff --git a/backend/tests/resource_versioning.rs b/backend/tests/resource_versioning.rs new file mode 100644 index 0000000000..c4b0bcdc02 --- /dev/null +++ b/backend/tests/resource_versioning.rs @@ -0,0 +1,215 @@ +//! Regression tests for resource value history. +//! +//! Four properties that a later refactor could plausibly undo, each with a cost +//! that is invisible until it bites: +//! - `state` and `cache` resources are excluded. They are rewritten by every job +//! that calls `setState` or caches a result, so versioning them would grow the +//! table without bound and without anyone asking for it. +//! - an unchanged value mints nothing, which is what keeps no-op saves, renames and +//! description edits out of the history. +//! - restore appends the old value as a new version rather than rewinding, so the +//! history stays append-only and the restore is itself attributable. +//! - writes that never touch the resource handlers are still recorded. Variable +//! renames, workspace forks and native integrations all write `resource` directly, +//! and a rename that changes path and value in one statement used to leave the +//! newest version holding the pre-rename value while the UI labelled it "Current". +//! This is why recording lives in a database trigger rather than the handlers. + +use serde_json::{json, Value}; +use sqlx::{Pool, Postgres}; + +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(b: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + b.header("Authorization", "Bearer RVER_ADMIN_TOKEN") +} + +async fn history(base: &str, path: &str) -> anyhow::Result> { + let body: Value = authed(client().get(format!("{base}/resources/history/p/{path}"))) + .send() + .await? + .json() + .await?; + Ok(body["versions"].as_array().cloned().unwrap_or_default()) +} + +#[sqlx::test(fixtures("resource_versioning"))] +async fn test_resource_version_history(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let base = format!("http://localhost:{}/api/w/rver-ws", server.addr.port()); + let path = "u/rver-admin/db"; + + let resp = authed(client().post(format!("{base}/resources/create"))) + .json(&json!({ + "path": path, + "value": {"host": "one"}, + "resource_type": "postgresql" + })) + .send() + .await?; + assert_eq!(resp.status(), 201, "create should succeed"); + assert_eq!(history(&base, path).await?.len(), 1, "create mints v1"); + + authed(client().post(format!("{base}/resources/update_value/{path}"))) + .json(&json!({"value": {"host": "two"}})) + .send() + .await?; + let versions = history(&base, path).await?; + assert_eq!(versions.len(), 2, "a changed value mints a version"); + + authed(client().post(format!("{base}/resources/update_value/{path}"))) + .json(&json!({"value": {"host": "two"}})) + .send() + .await?; + assert_eq!( + history(&base, path).await?.len(), + 2, + "re-saving an identical value must not mint a version" + ); + + // Restore the oldest version: the value goes back, the history grows. + let oldest = versions.last().unwrap()["id"].as_i64().unwrap(); + let resp = authed(client().post(format!("{base}/resources/history/restore/v/{oldest}"))) + .send() + .await?; + assert_eq!(resp.status(), 200, "restore should succeed"); + + let restored: Value = authed(client().get(format!("{base}/resources/get_value/{path}"))) + .send() + .await? + .json() + .await?; + assert_eq!(restored["host"], "one", "restore brings the old value back"); + assert_eq!( + history(&base, path).await?.len(), + 3, + "restore appends rather than rewinding" + ); + + // Machine-written resource types stay out of the history entirely. + for internal in ["state", "cache"] { + let internal_path = format!("u/rver-admin/{internal}_item"); + authed(client().post(format!("{base}/resources/create"))) + .json(&json!({ + "path": internal_path, + "value": {"a": 1}, + "resource_type": internal + })) + .send() + .await?; + assert_eq!( + history(&base, &internal_path).await?.len(), + 0, + "{internal} resources must not be versioned" + ); + } + + Ok(()) +} + +/// A version's reported dangling references. Pins that `$jsonvar:` is matched on its own prefix +/// rather than colliding with `$var:` — the two share a suffix, so a check written with a +/// substring test instead of a prefix test would report every `$jsonvar:` as missing. +#[sqlx::test(fixtures("resource_versioning"))] +async fn test_missing_references(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let base = format!("http://localhost:{}/api/w/rver-ws", server.addr.port()); + let path = "u/rver-admin/refs"; + + authed(client().post(format!("{base}/variables/create"))) + .json(&json!({ + "path": "u/rver-admin/present", + "value": "v", + "is_secret": false, + "description": "" + })) + .send() + .await?; + + authed(client().post(format!("{base}/resources/create"))) + .json(&json!({ + "path": path, + "value": { + "a": "$jsonvar:u/rver-admin/present", + "b": "$jsonvar:u/rver-admin/absent", + "c": "$var:u/rver-admin/present", + "d": "$res:u/rver-admin/absent" + }, + "resource_type": "postgresql" + })) + .send() + .await?; + + let id = history(&base, path).await?[0]["id"].as_i64().unwrap(); + let version: Value = authed(client().get(format!("{base}/resources/history/v/{id}"))) + .send() + .await? + .json() + .await?; + let mut missing: Vec<&str> = version["missing_references"] + .as_array() + .unwrap() + .iter() + .map(|v| v.as_str().unwrap()) + .collect(); + missing.sort(); + + assert_eq!( + missing, + vec!["$jsonvar:u/rver-admin/absent", "$res:u/rver-admin/absent"], + "only the references that do not resolve should be reported" + ); + + Ok(()) +} + +/// Writes that bypass the resource handlers entirely. A variable rename changes a linked +/// resource's path and value in one statement; the cascading FK moves the history to the new +/// path, so without trigger-level recording the newest row would keep the pre-rename value and +/// disagree with the resource it describes. +#[sqlx::test(fixtures("resource_versioning"))] +async fn test_direct_writes_are_recorded(db: Pool) -> anyhow::Result<()> { + let path = "u/rver-admin/direct"; + sqlx::query( + "INSERT INTO resource (workspace_id, path, value, resource_type, created_by, edited_at) + VALUES ('rver-ws', $1, '{\"h\":\"one\"}', 'postgresql', 'rver-admin', now())", + ) + .bind(path) + .execute(&db) + .await?; + + let renamed = "u/rver-admin/direct_renamed"; + sqlx::query( + "UPDATE resource SET path = $1, value = '{\"h\":\"two\"}', edited_at = now() + WHERE workspace_id = 'rver-ws' AND path = $2", + ) + .bind(renamed) + .bind(path) + .execute(&db) + .await?; + + let (count, newest): (i64, Option) = sqlx::query_as( + "SELECT count(*), (SELECT value->>'h' FROM resource_version + WHERE workspace_id = 'rver-ws' AND path = $1 + ORDER BY id DESC LIMIT 1) + FROM resource_version WHERE workspace_id = 'rver-ws' AND path = $1", + ) + .bind(renamed) + .fetch_one(&db) + .await?; + + assert_eq!(count, 2, "the direct insert and the rename both record"); + assert_eq!( + newest.as_deref(), + Some("two"), + "newest version must match the live value after a rename that also changed it" + ); + + Ok(()) +} diff --git a/backend/tests/session_workspace_status.rs b/backend/tests/session_workspace_status.rs new file mode 100644 index 0000000000..b4fc98da5e --- /dev/null +++ b/backend/tests/session_workspace_status.rs @@ -0,0 +1,62 @@ +//! `POST /workspaces/session_workspace_status` is what the client uses to decide whether to +//! keep or destroy an AI session, so its notion of "reachable" must match what the authed +//! extractor actually grants. Membership is not the only path: a superadmin is authed into +//! any existing workspace without a `usr` row, and `admins` has no `usr` rows at all, so +//! answering from `usr` alone reports live workspaces as unresolvable and the client deletes +//! sessions that still work. + +use serde_json::json; +use sqlx::{Pool, Postgres}; +use std::collections::HashMap; +use windmill_test_utils::*; + +async fn status(port: u16, token: &str, ids: &[&str]) -> anyhow::Result> { + let resp = reqwest::Client::new() + .post(format!( + "http://localhost:{port}/api/workspaces/session_workspace_status" + )) + .header("Authorization", format!("Bearer {token}")) + .json(&json!({ "workspace_ids": ids })) + .send() + .await?; + assert_eq!(resp.status(), 200, "{}", resp.text().await?); + Ok(resp.json().await?) +} + +#[sqlx::test(fixtures("base", "session_workspace_status"))] +async fn test_superadmin_reaches_workspaces_without_a_usr_row( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let ids = [ + "admins", + "foreign-workspace", + "archived-workspace", + "test-workspace", + "no-such-workspace", + ]; + + // test@windmill.dev is an instance superadmin, and a member of test-workspace only. + let sa = status(port, "SECRET_TOKEN", &ids).await?; + assert_eq!(sa["admins"], "active"); + assert_eq!(sa["foreign-workspace"], "active"); + // Reachable, but soft-deleted: superadmins must not bypass the archived state. + assert_eq!(sa["archived-workspace"], "archived"); + assert_eq!(sa["test-workspace"], "active"); + // A workspace that never existed stays unresolvable — the superadmin arm must not + // swallow the hard-deleted case, or those sessions would linger forever. + assert_eq!(sa["no-such-workspace"], "deleted"); + + // test2@windmill.dev is not a superadmin, and a member of test-workspace only. + let usr = status(port, "SECRET_TOKEN_2", &ids).await?; + assert_eq!(usr["admins"], "deleted"); + assert_eq!(usr["foreign-workspace"], "deleted"); + assert_eq!(usr["archived-workspace"], "deleted"); + assert_eq!(usr["test-workspace"], "active"); + assert_eq!(usr["no-such-workspace"], "deleted"); + + Ok(()) +} diff --git a/backend/tests/suspend_resume.rs b/backend/tests/suspend_resume.rs index 29fe38d2c3..d112ca0b96 100644 --- a/backend/tests/suspend_resume.rs +++ b/backend/tests/suspend_resume.rs @@ -235,6 +235,88 @@ mod suspend_resume { Ok(()) } + /// A suspend gate that ends without approval leaves the worker that ran the approval step + /// alive, so the error handler it routes to must stay pinned to that worker rather than + /// being unpinned and routed by tag — which would break the `./shared` contract of a + /// `same_worker` flow. + #[cfg(feature = "deno_core")] + #[sqlx::test(fixtures("base"))] + async fn disapproved_suspend_keeps_same_worker_pin(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let value: FlowValue = serde_json::from_value(json!({ + "same_worker": true, + "modules": [{ + "id": "a", + "value": { + "input_transforms": { + "port": { "type": "javascript", "expr": "flow_input.port" }, + }, + "type": "rawscript", + "language": "deno", + "content": "\ + export async function main(port) {\ + const job = Deno.env.get('WM_JOB_ID');\ + const token = Deno.env.get('WM_TOKEN');\ + const secret = await (await fetch(\ + `http://localhost:${port}/api/w/test-workspace/jobs/job_signature/${job}/0?token=${token}&approver=ruben`,\ + { headers: { 'Authorization': `Bearer ${token}` } }\ + )).text();\ + await fetch(\ + `http://localhost:${port}/api/w/test-workspace/jobs_u/cancel/${job}/0/${secret}?approver=ruben`,\ + { method: 'POST', body: JSON.stringify('from job'), headers: { 'content-type': 'application/json' } }\ + );\ + return 'a ran';\ + }", + }, + "suspend": { "required_events": 1 }, + }, { + "id": "b", + "value": { + "input_transforms": {}, + "type": "rawscript", + "language": "deno", + "content": "export function main() { return 'b ran' }", + }, + // The gate holds `b` back, so `b` never runs and its error policy describes + // nothing: honouring it here would skip `b` instead of reaching the handler. + "continue_on_error": true, + }], + "failure_module": { + "id": "failure", + "value": { + "input_transforms": {}, + "type": "rawscript", + "language": "deno", + "content": "export function main() { return 'handled' }", + }, + }, + }))?; + + let completed = + RunJob::from(JobPayload::RawFlow { value, path: None, restarted_from: None }) + .arg("port", json!(port)) + .run_until_complete(&db, false, port) + .await; + + server.close().await.unwrap(); + + assert_eq!(json!("handled"), completed.json_result().unwrap()); + + let same_worker: Option = sqlx::query_scalar( + "SELECT same_worker FROM v2_job WHERE parent_job = $1 AND flow_step_id = 'failure'", + ) + .bind(completed.id) + .fetch_one(&db) + .await?; + assert_eq!(Some(true), same_worker); + + Ok(()) + } + /// Test that self-approval is blocked when self_approval_disabled is true. /// /// This test verifies that when a flow has an approval step with self_approval_disabled=true, diff --git a/backend/tests/trigger_listener_queries.rs b/backend/tests/trigger_listener_queries.rs index c3f6357d4c..52088557ac 100644 --- a/backend/tests/trigger_listener_queries.rs +++ b/backend/tests/trigger_listener_queries.rs @@ -173,6 +173,8 @@ fn make_authed() -> windmill_api_auth::ApiAuthed { folders: vec![], scopes: None, username_override: None, + username_override_is_token_label: false, + is_session_token: false, token_prefix: None, read_only: false, } diff --git a/backend/tests/worker.rs b/backend/tests/worker.rs index 3140cd2d7f..67aac0503e 100644 --- a/backend/tests/worker.rs +++ b/backend/tests/worker.rs @@ -587,6 +587,69 @@ async fn test_deno_flow_same_worker(db: Pool) -> anyhow::Result<()> { ); Ok(()) } + +#[cfg(feature = "deno_core")] +#[sqlx::test(fixtures("base"))] +async fn test_same_worker_survives_empty_branch(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server: ApiServer = ApiServer::start(db.clone()).await?; + + // No branch matches and the default is empty, so `a` completes without spawning a job and + // hands the flow back over the UpdateFlow channel. `b` must still be pinned to the worker. + let flow: FlowValue = serde_json::from_value(json!({ + "same_worker": true, + "modules": [ + { + "id": "a", + "value": { + "type": "branchone", + "branches": [{ + "expr": "false", + "modules": [{ + "id": "c", + "value": { + "type": "rawscript", + "language": "deno", + "content": "export function main(){ return 1 }", + } + }] + }], + "default": [] + } + }, + { + "id": "b", + "value": { + "type": "rawscript", + "language": "deno", + "content": "export function main(){ return 42 }", + } + } + ] + })) + .unwrap(); + + let job = run_job_in_new_worker_until_complete( + &db, + false, + JobPayload::RawFlow { value: flow, path: None, restarted_from: None }, + server.addr.port(), + ) + .await; + assert_eq!(job.json_result().unwrap(), json!(42)); + + let same_worker: Option = sqlx::query_scalar( + "SELECT same_worker FROM v2_job WHERE parent_job = $1 AND flow_step_id = 'b'", + ) + .bind(job.id) + .fetch_one(&db) + .await?; + assert_eq!(same_worker, Some(true)); + + Ok(()) +} + #[sqlx::test(fixtures("base"))] async fn test_flow_result_by_id(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; @@ -3992,7 +4055,7 @@ async fn test_failure_module(db: Pool) -> anyhow::Result<()> { /// Push `flow`, run it on a real worker until its first step is running, then simulate /// `monitor::handle_zombie_jobs` reaping that step unrecoverably (its worker crashed/OOM'd) -/// by calling `handle_job_error(..., unrecoverable = true, ...)` exactly as the monitor does. +/// by calling `handle_job_error(..., StepFailureKind::Unrecoverable, ...)` exactly as the monitor does. /// Returns the flow's completed result. #[cfg(feature = "deno_core")] async fn run_flow_until_step_running_then_fail_unrecoverably( @@ -4007,7 +4070,7 @@ async fn run_flow_until_step_running_then_fail_unrecoverably( use windmill_common::client::AuthedClient; use windmill_common::KillpillSender; use windmill_queue::{get_queued_job_v2, MiniCompletedJob, SameWorkerPayload}; - use windmill_worker::{JobCompletedSender, SameWorkerSender}; + use windmill_worker::{JobCompletedSender, SameWorkerSender, StepFailureKind}; let flow_id = RunJob::from(JobPayload::RawFlow { value: flow, path: None, restarted_from: None }) @@ -4072,7 +4135,7 @@ async fn run_flow_until_step_running_then_fail_unrecoverably( windmill_common::error::Error::ExecutionErr( "simulated worker OOM crash".to_string(), ), - true, // unrecoverable + StepFailureKind::Unrecoverable, Some(&sw_tx), "", "test-monitor", diff --git a/backend/windmill-ai/src/providers/anthropic.rs b/backend/windmill-ai/src/providers/anthropic.rs index aafbcfb1df..6ebc555da6 100644 --- a/backend/windmill-ai/src/providers/anthropic.rs +++ b/backend/windmill-ai/src/providers/anthropic.rs @@ -866,6 +866,7 @@ mod tests { user_message: "hello", attachments: None, has_websearch: false, + prompt_cache_key: None, }; AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::Standard) diff --git a/backend/windmill-ai/src/providers/openai.rs b/backend/windmill-ai/src/providers/openai.rs index fd28281864..34185a8852 100644 --- a/backend/windmill-ai/src/providers/openai.rs +++ b/backend/windmill-ai/src/providers/openai.rs @@ -218,6 +218,12 @@ pub struct ResponsesApiRequest<'a> { pub max_output_tokens: Option, #[serde(skip_serializing_if = "Option::is_none")] pub text: Option, + /// From `gpt-5.6` on, the prefix hash alone no longer reliably matches a cached + /// prefix: the key is combined with it to route the request. Omitting it costs the + /// read discount and, since these models bill cache writes, pays to re-write the + /// prefix on every miss. + #[serde(skip_serializing_if = "Option::is_none")] + pub prompt_cache_key: Option<&'a str>, } #[derive(Serialize, Debug)] @@ -434,6 +440,7 @@ impl OpenAIQueryBuilder { .map(|effort| ResponsesApiReasoning { effort: effort.to_string() }), max_output_tokens: args.max_tokens, text, + prompt_cache_key: args.prompt_cache_key, }; serde_json::to_string(&request) @@ -490,6 +497,8 @@ impl OpenAIQueryBuilder { reasoning: None, // Image generation models don't take a reasoning effort max_output_tokens: args.max_tokens, text: None, // No structured output for image generation + // A one-shot image prompt has no reusable prefix to route to a cache + prompt_cache_key: None, }; serde_json::to_string(&request) @@ -609,6 +618,7 @@ mod tests { use crate::query_builder::QueryBuilder; const SYSTEM_PROMPT: &str = "You are a helpful assistant"; + const PROMPT_CACHE_KEY: &str = "test-workspace:f/agent:step_1"; fn client() -> AuthedClient { AuthedClient::new( @@ -641,6 +651,7 @@ mod tests { user_message: "hello", attachments: None, has_websearch: false, + prompt_cache_key: Some(PROMPT_CACHE_KEY), }; OpenAIQueryBuilder::new(AIProvider::OpenAI) @@ -712,4 +723,16 @@ mod tests { assert!(request.get("instructions").is_none()); } + + /// `gpt-5.6` and later only match a cached prefix reliably when the request carries + /// the routing key, so dropping it from the body silently forfeits the cache. + #[tokio::test] + async fn sends_the_prompt_cache_key() { + let messages = vec![message("user", "hi")]; + + let body = build_text_body(&messages, None).await; + let request: serde_json::Value = serde_json::from_str(&body).unwrap(); + + assert_eq!(request["prompt_cache_key"], PROMPT_CACHE_KEY); + } } diff --git a/backend/windmill-ai/src/query_builder.rs b/backend/windmill-ai/src/query_builder.rs index 5a4cecac87..c8f5fbb092 100644 --- a/backend/windmill-ai/src/query_builder.rs +++ b/backend/windmill-ai/src/query_builder.rs @@ -22,6 +22,11 @@ pub struct BuildRequestArgs<'a> { pub user_message: &'a str, pub attachments: Option<&'a [S3Object]>, pub has_websearch: bool, + /// Routing hint for the provider's prompt cache. Requests sharing a prompt prefix + /// must reuse one key to land on the same cache, so it is derived from what pins + /// the prefix (the step), never from the request. `None` retries a key the + /// endpoint rejected. + pub prompt_cache_key: Option<&'a str>, } /// Response from AI provider diff --git a/backend/windmill-api-assets/Cargo.toml b/backend/windmill-api-assets/Cargo.toml index c75715a3ac..62bc3d9c8d 100644 --- a/backend/windmill-api-assets/Cargo.toml +++ b/backend/windmill-api-assets/Cargo.toml @@ -21,3 +21,7 @@ serde.workspace = true serde_json.workspace = true sqlx.workspace = true tracing.workspace = true +uuid.workspace = true + +[dev-dependencies] +tokio = { workspace = true } diff --git a/backend/windmill-api-assets/src/lib.rs b/backend/windmill-api-assets/src/lib.rs index 3a2bf83097..09720b2203 100644 --- a/backend/windmill-api-assets/src/lib.rs +++ b/backend/windmill-api-assets/src/lib.rs @@ -605,9 +605,16 @@ async fn list_favorites( // frontend aggregates into nodes and edges. #[derive(Deserialize)] -struct GraphQuery { +pub struct GraphQuery { pub asset_kinds: Option, pub folder: Option, + /// Render a dbt project as a given deployed version rather than as it is + /// now. A run page passes the version its job ran, so an old run shows the + /// models, SQL and `ref()` lineage of that deploy instead of today's. + /// Absent — the usual case — means the newest live version per path. + /// Hex, like every other script-hash parameter — `ScriptHash` deserializes + /// it, so the run page can pass `job.script_hash` verbatim. + pub dbt_script_hash: Option, } #[derive(Serialize, Debug)] @@ -629,6 +636,70 @@ struct GraphAssetNode { // `AssetGraphAssetNode.derived_from`. #[serde(skip_serializing_if = "Option::is_none")] derived_from: Option, + // Set on a `dbt://` asset produced (or, for a source, consumed) by a dbt + // script: which dbt node it is and what dbt says about it. A dbt project is + // one runnable node with many model assets, so per-model metadata belongs + // here rather than on the script (docs/dbt-runtime.md, decision 15). + // Lockstep with TS `AssetGraphAssetNode.dbt`. + #[serde(skip_serializing_if = "Option::is_none")] + dbt: Option, +} + +#[derive(Serialize, Debug, Clone)] +struct DbtAssetProvenance { + unique_id: String, + // `model` | `snapshot` | `seed` | `source` — a source is read, not written, + // and the canvas distinguishes them. + resource_type: String, + // dbt's own word (`table`, `view`, `incremental`, `snapshot`), kept because + // `view` and `ephemeral` have no Windmill write-strategy analogue. + #[serde(skip_serializing_if = "Option::is_none")] + materialized: Option, + #[serde(skip_serializing_if = "Option::is_none")] + materialize_strategy: Option, + #[serde(skip_serializing_if = "Vec::is_empty")] + tags: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + description: Option, + #[serde(skip_serializing_if = "Vec::is_empty")] + data_tests: Vec, + /// Declared column metadata (name -> description). NOT column lineage — + /// `manifest.json` carries none (docs/dbt-runtime.md, decision 14). + #[serde(skip_serializing_if = "Option::is_none")] + columns: Option, + /// A source's declared freshness policy, for the staleness chip. + #[serde(skip_serializing_if = "Option::is_none")] + freshness: Option, + /// The model's SQL as written. Read-only in Windmill — the file lives in + /// the producing script's bundle, and this is the copy captured when the + /// graph being read was parsed: at deploy, or by a refresh from the + /// editor's buffer. + #[serde(skip_serializing_if = "Option::is_none")] + raw_code: Option, + /// Its path inside the repo, e.g. `models/staging/stg_orders.sql`. + #[serde(skip_serializing_if = "Option::is_none")] + original_file_path: Option, +} + +#[derive(Serialize, Debug, Clone, PartialEq)] +struct DbtDataTest { + // `unique` | `not_null` | `accepted_values` | `relationships` — the four + // generic tests, one-for-one with the `// data_test` kinds — or a package + // test's namespaced name (`dbt_utils.accepted_range`). + kind: String, + #[serde(skip_serializing_if = "Option::is_none")] + column: Option, + #[serde(skip_serializing_if = "Option::is_none")] + args: Option, + // Lowercased. dbt's own severity decides whether a failure fails the run, + // so the canvas shows it rather than assuming every test is blocking. + #[serde(skip_serializing_if = "Option::is_none")] + severity: Option, +} + +#[derive(Serialize, Debug, Clone)] +struct DbtRunnableProvenance { + model_count: usize, } #[derive(Serialize, Debug)] @@ -691,6 +762,11 @@ struct GraphRunnableNode { // signature list. Lockstep with TS `AssetGraphRunnableNode.macros`. #[serde(skip_serializing_if = "Vec::is_empty", default)] macros: Vec, + // Set on a `ScriptLang::Dbt` script: it owns a whole dbt project, so the + // node says how many models it materializes rather than pretending to be a + // single-output script. Lockstep with TS `AssetGraphRunnableNode.dbt`. + #[serde(skip_serializing_if = "Option::is_none", default)] + dbt: Option, } // One macro of a `// macros` library, as surfaced on its graph node. @@ -831,7 +907,7 @@ struct TestEdge { } #[derive(Serialize, Debug)] -struct AssetGraphResponse { +pub struct AssetGraphResponse { assets: Vec, runnables: Vec, edges: Vec, @@ -840,6 +916,34 @@ struct AssetGraphResponse { macro_edges: Vec, #[serde(skip_serializing_if = "Vec::is_empty", default)] test_edges: Vec, + /// `ref()` lineage BETWEEN two dbt models. Without it every model hangs off + /// the one dbt runnable, which reads as a flat source-to-model fan-out + /// instead of the project's actual shape. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + dbt_edges: Vec, + /// The job whose snapshot the dbt half was resolved from, when one was + /// asked for and found. A run page polls the graph while its job runs + /// because a dynamic descriptor's snapshot is written mid-run, and this is + /// what tells it to stop: without it the page cannot distinguish "the + /// snapshot has not been written yet" from "this run has none", and either + /// polls forever or gives up before the ingest. + #[serde(skip_serializing_if = "Option::is_none")] + dbt_snapshot_job: Option, + /// When the dbt half was parsed, for a graph pinned to a job. What the + /// editor labels its provenance with: a buffer refresh and the deployed + /// version's graph are drawn identically, so without saying which one is on + /// screen the ambiguity the explicit refresh removes just moves into the + /// editor. + #[serde(skip_serializing_if = "Option::is_none")] + dbt_graph_ingested_at: Option>, +} + +/// One `ref()`/`source()` edge, in the terms the canvas draws: the two +/// relations, not dbt's node ids. +#[derive(Serialize, Debug, PartialEq, Eq, PartialOrd, Ord)] +struct DbtLineageEdge { + from_asset_path: String, + to_asset_path: String, } async fn asset_graph( @@ -849,15 +953,81 @@ async fn asset_graph( Extension(db): Extension, Query(q): Query, ) -> JsonResult { - let mut tx = user_db.begin(&authed).await?; + // `None`: pinning the graph to one run is job-scoped and this endpoint is + // authorized as `assets:read`. See `asset_graph_for`. + asset_graph_for(&authed, &w_id, user_db, db, q, None).await +} - let kind_filter: Option> = q.asset_kinds.as_ref().map(|s| { - s.split(',') - .filter_map(|k| { - serde_json::from_value::(Value::String(k.trim().into())).ok() - }) - .collect() - }); +/// A run the caller is already authorized to read, and the deployed version it +/// ran. Path and hash come from the job row rather than the query, so a graph +/// cannot be pointed at one project's version while claiming another's run. +pub struct PinnedRun { + pub job_id: uuid::Uuid, + pub script_path: String, + /// `None` for a job that names no deployed version: a `parse` of the + /// EDITOR's buffer, whose graph belongs to that job alone. Such a job pins + /// only because it stored one — every other preview answers with the + /// workspace graph, as before. + pub script_hash: Option, +} + +/// The asset graph, optionally as one run saw it. +/// +/// AUTHORIZES NOTHING BY ITSELF. Every caller owes it two checks, because the +/// answer is workspace asset data reached through a route whose own URL segment +/// decides the scope domain: `assets:read` always, and `require_job_read_access` +/// for the job when passing `Some(pinned)` — which is then taken as already +/// done, since the pinned path and hash come from that job's row. +pub async fn asset_graph_for( + authed: &ApiAuthed, + w_id: &str, + user_db: UserDB, + db: windmill_common::DB, + q: GraphQuery, + pinned: Option, +) -> JsonResult { + let dbt_job_id = pinned.as_ref().map(|p| p.job_id); + // The version is the job's own, not the caller's `dbt_script_hash` — and + // that holds when the job names NONE, so the parameter cannot supply a + // version for a graph that has no business claiming one. + let dbt_script_hash = match pinned.as_ref() { + Some(p) => p.script_hash, + None => q.dbt_script_hash.map(|h| h.0), + }; + // Set only for a pinned run, where it lets `live` resolve without reading + // `script`: a share-link viewer is entitled to the run but usually has no + // grant on the script, and RLS there would empty the graph. `raw_code` has + // its own `script` check, so the body stays hidden either way. + let pinned_path = pinned.as_ref().map(|p| p.script_path.as_str()); + let w_id = w_id.to_string(); + let mut tx = user_db.begin(authed).await?; + // Built once: a scoped token's `scripts:read` paths decide whether a dbt + // node's SQL body may be returned, independently of the `assets:read` scope + // that authorizes this endpoint. + let dbt_source_scope = build_scope_path_predicate(authed, "scripts", "read"); + + // REFUSED, not dropped: a caller that asks for a kind this server does not + // know has asked for something, and answering with the other kinds returns a + // graph missing exactly what they came for — silently. That is how a rename + // of one kind emptied three callers' graphs with nothing logged. + let kind_filter: Option> = q + .asset_kinds + .as_ref() + .map(|s| { + s.split(',') + .map(|k| { + serde_json::from_value::(Value::String(k.trim().into())).map_err( + |_| { + windmill_common::error::Error::BadRequest(format!( + "`{}` is not an asset kind", + k.trim() + )) + }, + ) + }) + .collect::, _>>() + }) + .transpose()?; let kind_filter_ref = kind_filter.as_deref(); let folder_filter = q.folder.as_deref().map(|f| format!("f/{}/%", f)); @@ -1066,6 +1236,267 @@ async fn asset_graph( .fetch_all(&mut *tx) .await?; + // dbt provenance. A dbt script is one runnable node whose models are many + // `dbt://` asset nodes (decision 15), so the per-model metadata has to + // hang off the assets, not off the script. It describes the RELATION, not a + // producer: several dbt scripts (different selections of one project) can + // materialize the same model, and the producer edges already name them. + // + // Scoped to the relations this graph actually renders, so a workspace with + // many dbt projects does not pay for all of them on every request. Not + // scoped by the script's folder: an out-of-folder dbt script still has to + // explain a model an in-scope consumer reads, same as macro definitions. + // Tests come along via `attached_node` — they carry no `asset_path` of + // their own. + let dbt_rows = sqlx::query!( + r#"WITH live AS ( + -- The graph is stored per deployed VERSION, so this endpoint — which + -- describes the project as it is now — takes the newest live one per + -- path. Resolved once here rather than per row: a correlated lookup + -- on every node is what makes these queries fall over. + SELECT * FROM ( + SELECT DISTINCT ON (s.path) s.path, s.hash + FROM script s + WHERE $5::text IS NULL AND s.workspace_id = $1 AND s.language = 'dbt' + AND ($3::bigint IS NULL OR s.hash = $3) + -- A pinned version may be archived by now; that is precisely + -- the case a historical run needs, so the liveness filter + -- applies only when picking the current one. + AND ($3::bigint IS NOT NULL OR (s.deleted = false AND s.archived = false)) + ORDER BY s.path, s.created_at DESC + ) cur + UNION ALL + -- A pinned run names its own version, so `script` is not consulted: + -- under RLS it would answer for the CALLER's grants on the project, + -- emptying the graph for a share-link viewer who is entitled to the + -- run but not the script. A NULL hash here is a job that names no + -- version at all — an editor buffer parse — and matches only the + -- version-less rows that parse stored. + SELECT $5::text, $3::bigint WHERE $5::text IS NOT NULL + ), + -- The run's own snapshot when it left one, the version's graph + -- otherwise. A static descriptor never snapshots, so all of its runs + -- fall through to the same rows. Existence comes from the marker, not + -- from a node row: a dynamic run that disabled every model has a + -- snapshot whose graph is legitimately empty. + chosen AS ( + -- No visibility check on the job here: reaching this with a job at + -- all means the caller passed `require_job_read_access` for it, and + -- re-deciding it under plain RLS can only DISAGREE with that answer + -- — silently, by falling back to the deployed graph rather than + -- erroring. A share-link viewer is entitled to the run and would be + -- shown a different run's model set. See `asset_graph_for`. + SELECT CASE WHEN $4::uuid IS NOT NULL AND EXISTS ( + SELECT 1 FROM dbt_graph_snapshot g + WHERE g.workspace_id = $1 AND g.job_id = $4) + THEN $4::uuid + ELSE '00000000-0000-0000-0000-000000000000'::uuid END AS job_id + ), + scoped AS ( + SELECT n.script_path, n.unique_id FROM dbt_node n + -- `=` still, with the NULL-to-NULL case spelled out and gated on + -- the pin: a version-less row's hash is NULL on both sides, which + -- `=` never matches, but `IS NOT DISTINCT FROM` would cost the + -- equality its index bound on the UNPINNED workspace graph — the + -- hot path. Unpinned, `$5` is NULL and the second arm folds away. + JOIN live l ON l.path = n.script_path + AND (n.script_hash = l.hash + OR ($5::text IS NOT NULL AND l.hash IS NULL + AND n.script_hash IS NULL)) + JOIN chosen ch ON ch.job_id = n.job_id + WHERE n.workspace_id = $1 AND n.asset_path IS NOT NULL + -- Unpinned, the scope is the relations in view: `asset` says + -- which of them this folder touches. Pinned, that table is the + -- WRONG scope — it holds one row set per path, describing the + -- current deploy, so a model this version had and the current one + -- dropped would be filtered out of its own run's graph. The + -- pinned graph's nodes are the scope. Keyed on the pin rather + -- than on the hash: an editor parse pins without naming one, and + -- its models are precisely the ones `asset` does not know yet. + AND ($3::bigint IS NOT NULL OR $5::text IS NOT NULL + OR n.asset_path IN ( + SELECT path FROM asset + WHERE workspace_id = $1 AND kind = 'dbt' + AND ($2::text IS NULL OR usage_path LIKE $2))) + ) + SELECT n.script_path AS "script_path!", n.unique_id AS "unique_id!", + n.resource_type AS "resource_type!", n.name AS "name!", n.asset_path, + n.materialized, n.materialize_strategy, n.tags AS "tags!", n.description, + n.test_kind, n.test_column, n.test_args, n.severity, n.attached_node, + n.columns, n.freshness, + n.raw_code, n.original_file_path, + -- Whether the caller may read the project this row describes. + -- The query deliberately reaches outside the requested folder + -- so an in-scope consumer can explain the relation it reads, + -- and `dbt_node` carries no RLS of its own; the relation's + -- SHAPE is fine to answer that way, everything the project's + -- author WROTE is not. Applied in Rust, over one predicate, so + -- the fields it covers are named in one place. This runs in the + -- authed transaction, so `script`'s RLS answers it. Matched on + -- the HASH as well: `extra_perms` is per row, so a path + -- recreated with narrower ones leaves the archived version + -- readable, and a path-only probe would answer for THAT grant + -- while returning this version's source. + -- + -- A version-less row has no `script` row to ask, and needs + -- none: it exists only because this caller's own parse job + -- created it from a buffer they wrote, and the unpinned `live` + -- branch — fed from `script` — can never join to one. + (n.script_hash IS NULL OR EXISTS ( + SELECT 1 FROM script sc + WHERE sc.workspace_id = n.workspace_id AND sc.path = n.script_path + AND sc.hash = n.script_hash + )) AS "script_visible!" + FROM dbt_node n + JOIN live l ON l.path = n.script_path + AND (n.script_hash = l.hash + OR ($5::text IS NOT NULL AND l.hash IS NULL + AND n.script_hash IS NULL)) + -- Every join onto `dbt_node` needs this, not just the scoping CTE: + -- `job_id` is part of the key, so without it each model comes back + -- once per retained snapshot plus once for the version's graph. + JOIN chosen ch ON ch.job_id = n.job_id + WHERE n.workspace_id = $1 + -- Joined on BOTH columns: a dbt `unique_id` is project-local, so + -- two projects with the same model name would otherwise pull each + -- other's rows. + AND (EXISTS (SELECT 1 FROM scoped s + WHERE s.script_path = n.script_path + AND s.unique_id = n.unique_id) + OR EXISTS (SELECT 1 FROM scoped s + WHERE s.script_path = n.script_path + AND s.unique_id = n.attached_node)) + ORDER BY n.script_path, n.unique_id"#, + &w_id, + folder_filter.as_deref(), + dbt_script_hash, + dbt_job_id, + pinned_path, + ) + .fetch_all(&mut *tx) + .await?; + + // `ref()` lineage between two models, resolved to the relations they + // produce. Joined to `dbt_node` on both key columns because a dbt + // `unique_id` is only unique within its project. + let dbt_edge_rows = sqlx::query!( + r#"WITH live AS ( + SELECT * FROM ( + SELECT DISTINCT ON (s.path) s.path, s.hash + FROM script s + WHERE $5::text IS NULL AND s.workspace_id = $1 AND s.language = 'dbt' + AND ($3::bigint IS NULL OR s.hash = $3) + AND ($3::bigint IS NOT NULL OR (s.deleted = false AND s.archived = false)) + ORDER BY s.path, s.created_at DESC + ) cur + UNION ALL + SELECT $5::text, $3::bigint WHERE $5::text IS NOT NULL + ), + -- The run's own snapshot when it left one, the version's graph + -- otherwise. A static descriptor never snapshots, so all of its runs + -- fall through to the same rows. Existence comes from the marker, not + -- from a node row: a dynamic run that disabled every model has a + -- snapshot whose graph is legitimately empty. + chosen AS ( + -- No visibility check on the job here: reaching this with a job at + -- all means the caller passed `require_job_read_access` for it, and + -- re-deciding it under plain RLS can only DISAGREE with that answer + -- — silently, by falling back to the deployed graph rather than + -- erroring. A share-link viewer is entitled to the run and would be + -- shown a different run's model set. See `asset_graph_for`. + SELECT CASE WHEN $4::uuid IS NOT NULL AND EXISTS ( + SELECT 1 FROM dbt_graph_snapshot g + WHERE g.workspace_id = $1 AND g.job_id = $4) + THEN $4::uuid + ELSE '00000000-0000-0000-0000-000000000000'::uuid END AS job_id + ) + SELECT p.asset_path AS "from_path!", c.asset_path AS "to_path!" + FROM dbt_edge e + JOIN live l ON l.path = e.script_path + AND (e.script_hash = l.hash + OR ($5::text IS NOT NULL AND l.hash IS NULL + AND e.script_hash IS NULL)) + JOIN chosen ch ON ch.job_id = e.job_id + -- Gated the same way as the `live` joins above, and for the same + -- reason twice over: unpinned this folds to a plain equality the + -- versioned key can bound, and pinned it is the only way an editor + -- graph's NULL-to-NULL hashes meet at all. + JOIN dbt_node p ON p.workspace_id = e.workspace_id + AND p.script_path = e.script_path + AND (p.script_hash = e.script_hash + OR ($5::text IS NOT NULL AND e.script_hash IS NULL + AND p.script_hash IS NULL)) + AND p.job_id = ch.job_id + AND p.unique_id = e.parent_unique_id + JOIN dbt_node c ON c.workspace_id = e.workspace_id + AND c.script_path = e.script_path + AND (c.script_hash = e.script_hash + OR ($5::text IS NOT NULL AND e.script_hash IS NULL + AND c.script_hash IS NULL)) + AND c.job_id = ch.job_id + AND c.unique_id = e.child_unique_id + WHERE e.workspace_id = $1 + AND p.asset_path IS NOT NULL AND c.asset_path IS NOT NULL + -- Tests attach to their model as a badge, not as a lineage edge. + AND c.resource_type <> 'test' + -- Scoped by the RELATIONS, like the node provenance above, not by + -- the producing script's folder: two tables consumed in this + -- folder but produced by a dbt project outside it would otherwise + -- both render with their `ref()` edge missing. + -- Same as the node scope: pinned, `asset` describes the CURRENT + -- deploy, so gating on it drops the edges of models this version + -- had and a later one removed. The pinned graph's own edges are + -- the answer. + AND ($3::bigint IS NOT NULL OR $5::text IS NOT NULL OR EXISTS ( + SELECT 1 FROM asset a + WHERE a.workspace_id = $1 AND a.kind = 'dbt' + AND a.path = c.asset_path + AND ($2::text IS NULL OR a.usage_path LIKE $2)))"#, + &w_id, + folder_filter.as_deref(), + dbt_script_hash, + dbt_job_id, + pinned_path, + ) + .fetch_all(&mut *tx) + .await?; + + // The marker `chosen` resolved to, answered once for the caller: its + // EXISTENCE is what lets a run page stop polling, and its timestamp is what + // an editor labels the graph's provenance with — "parsed from the editor at + // 14:32" against "as of last deploy". No `v2_job` recheck, for the reason + // `chosen` gives — disagreeing with the route's decision here would leave a + // share-link viewer with the right graph and a null marker, polling it 40 + // times over. + let dbt_marker = match (dbt_job_id, pinned_path) { + (Some(job), Some(path)) => { + sqlx::query!( + "SELECT g.job_id, g.ingested_at FROM dbt_graph_snapshot g + WHERE g.workspace_id = $1 AND g.script_path = $2 + AND g.script_hash IS NOT DISTINCT FROM $3 + AND g.job_id = CASE WHEN EXISTS ( + SELECT 1 FROM dbt_graph_snapshot s + WHERE s.workspace_id = $1 AND s.job_id = $4) + THEN $4::uuid + ELSE '00000000-0000-0000-0000-000000000000'::uuid END + LIMIT 1", + &w_id, + path, + dbt_script_hash, + job, + ) + .fetch_optional(&mut *tx) + .await? + } + _ => None, + }; + // Only a graph of the RUN's own, never the version's fallback: the run page + // compares this against its job id to decide whether to keep polling. + let dbt_snapshot_job = dbt_marker + .as_ref() + .map(|m| m.job_id) + .filter(|j| !j.is_nil()); + let dbt_graph_ingested_at = dbt_marker.as_ref().map(|m| m.ingested_at); tx.commit().await?; // Parse each pipeline member's body once into its badge annotations, keyed @@ -1110,6 +1541,138 @@ async fn asset_graph( (r.path.clone(), lineage) }) .collect(); + // Model → its provenance, and script → how many models it owns. Tests are + // folded onto the model they are attached to, which is what makes dbt's + // four generic tests render through the existing data-test node. + let mut dbt_by_asset_path: std::collections::HashMap = + Default::default(); + let mut dbt_model_count: std::collections::HashMap = Default::default(); + // Which relations each dbt script actually BUILDS. The sidecar also holds a + // row for a parent kept only to anchor a cross-selection edge, which the + // script reads rather than materializes — counting those would make a script + // selecting one mart claim the staging models upstream of it, and the badge + // says "materializes N models". + let dbt_writes: std::collections::HashSet<(&str, &str)> = rows + .iter() + .filter(|r| { + r.asset_kind == AssetKind::Dbt + && matches!(r.access_type.as_deref(), Some("w") | Some("rw")) + }) + .map(|r| (r.usage_path.as_str(), r.asset_path.as_str())) + .collect(); + // Keyed by (script_path, unique_id) — the sidecar's own primary key — since + // a dbt `unique_id` is only unique within its project. + let dbt_asset_path_by_unique_id: std::collections::HashMap<(&str, &str), &str> = dbt_rows + .iter() + .filter_map(|r| { + Some(( + (r.script_path.as_str(), r.unique_id.as_str()), + r.asset_path.as_deref()?, + )) + }) + .collect(); + for r in &dbt_rows { + let Some(asset_path) = r.asset_path.as_deref() else { + continue; + }; + if r.resource_type != "source" && dbt_writes.contains(&(r.script_path.as_str(), asset_path)) + { + *dbt_model_count.entry(r.script_path.clone()).or_default() += 1; + } + // What the project's author WROTE — the model's SQL, its path in the + // repo, the prose and labels around it — as opposed to the shape of the + // relation it produces. RLS says whether the caller may see the script, + // but not whether a scoped token may: this endpoint is authorized as + // `assets:read`, so a token deliberately narrowed to it would otherwise + // read a project outside its `scripts:read` paths. Both answers gate the + // same set of fields, because a share-link viewer entitled to the RUN is + // not thereby entitled to the documentation of a project they cannot + // open. + let source_allowed = r.script_visible && dbt_source_scope(&r.script_path); + let candidate = DbtAssetProvenance { + raw_code: r.raw_code.clone().filter(|_| source_allowed), + original_file_path: r.original_file_path.clone().filter(|_| source_allowed), + unique_id: r.unique_id.clone(), + resource_type: r.resource_type.clone(), + materialized: r.materialized.clone(), + materialize_strategy: r.materialize_strategy.clone(), + tags: if source_allowed { + r.tags.clone() + } else { + vec![] + }, + description: r.description.clone().filter(|_| source_allowed), + data_tests: vec![], + columns: r.columns.clone().filter(|_| source_allowed), + freshness: r.freshness.clone().filter(|_| source_allowed), + }; + // One relation can carry rows from several projects — typically a model + // in one and a source declaring it in another. The producer describes + // the relation and the source only names it, so which one wins must not + // depend on script-path ordering. The source's freshness policy is + // still worth keeping, so it fills in rather than overwrites. + let existing = dbt_by_asset_path + .entry(asset_path.to_string()) + .or_insert_with(|| candidate.clone()); + if existing.unique_id == candidate.unique_id { + continue; + } + let wins = match ( + existing.resource_type.as_str(), + candidate.resource_type.as_str(), + ) { + ("source", t) if t != "source" => true, + (t, "source") if t != "source" => false, + // Two rows of the same nature: pick by id so the graph does not + // change shape between requests. + _ => candidate.unique_id < existing.unique_id, + }; + let freshness = existing + .freshness + .clone() + .or_else(|| candidate.freshness.clone()); + if wins { + *existing = candidate; + } + existing.freshness = freshness; + } + // Tests fold onto the model they assert, which is what makes dbt's four + // generic tests render through the existing data-test node. A separate pass + // because rows arrive in `unique_id` order, so a test can precede its model. + for r in &dbt_rows { + if r.resource_type != "test" { + continue; + } + let Some(target) = r + .attached_node + .as_deref() + .and_then(|n| dbt_asset_path_by_unique_id.get(&(r.script_path.as_str(), n))) + else { + continue; + }; + let Some(entry) = dbt_by_asset_path.get_mut(*target) else { + continue; + }; + let test = DbtDataTest { + kind: r.test_kind.clone().unwrap_or_else(|| r.name.clone()), + column: r.test_column.clone(), + // The test's kind and column are the badge's shape, already spelled + // out by its `unique_id`. Its arguments are authored data — an + // `accepted_values` list is a column's domain — so they follow the + // same gate as the model's own source. + args: r + .test_args + .clone() + .filter(|_| r.script_visible && dbt_source_scope(&r.script_path)), + // dbt-core 1.x echoes the author's casing, 2.x uppercases; fold so + // the badge reads the same whichever engine deployed the script. + severity: r.severity.as_deref().map(|s| s.to_ascii_lowercase()), + }; + if !entry.data_tests.contains(&test) { + entry.data_tests.push(test); + } + } + let last_success_by_path: std::collections::HashMap> = last_success_rows .into_iter() @@ -1132,6 +1695,18 @@ async fn asset_graph( let mut edges = Vec::with_capacity(rows.len()); let mut asset_set: std::collections::HashSet<(AssetKind, String)> = Default::default(); + // Pinned, the relations come from that graph's own nodes. The `asset` rows + // above are path-keyed — one set per script, always the current deploy — so + // a model this version had and a later one dropped would be missing from its + // own run's graph, and an editor buffer's new models would be missing + // outright, `asset` having never heard of them. + if dbt_script_hash.is_some() || pinned_path.is_some() { + for r in &dbt_rows { + if let Some(p) = r.asset_path.as_deref() { + asset_set.insert((AssetKind::Dbt, p.to_string())); + } + } + } let mut runnable_set: std::collections::HashSet<(AssetUsageKind, String)> = Default::default(); // Every pipeline member in scope goes into the graph, even when the parser @@ -1491,6 +2066,9 @@ async fn asset_graph( }) .flatten(), derived_from: scd2_current_base.get(&(kind, path.clone())).cloned(), + dbt: (kind == AssetKind::Dbt) + .then(|| dbt_by_asset_path.get(&path).cloned()) + .flatten(), kind, path, }) @@ -1562,6 +2140,10 @@ async fn asset_graph( .flatten() .cloned() .unwrap_or_default(), + dbt: (usage_kind == AssetUsageKind::Script) + .then(|| dbt_model_count.get(&path)) + .flatten() + .map(|n| DbtRunnableProvenance { model_count: *n }), path, usage_kind, } @@ -1569,6 +2151,22 @@ async fn asset_graph( .collect(); runnables.sort_by(|a, b| a.path.cmp(&b.path)); + // Only between relations this graph actually renders — an edge to a node + // that is not here has nothing to draw. + let rendered: std::collections::HashSet<&str> = + assets.iter().map(|a| a.path.as_str()).collect(); + let mut dbt_edges: Vec = dbt_edge_rows + .into_iter() + .filter(|r| { + r.from_path != r.to_path + && rendered.contains(r.from_path.as_str()) + && rendered.contains(r.to_path.as_str()) + }) + .map(|r| DbtLineageEdge { from_asset_path: r.from_path, to_asset_path: r.to_path }) + .collect(); + dbt_edges.sort(); + dbt_edges.dedup(); + Ok(Json(AssetGraphResponse { assets, runnables, @@ -1576,6 +2174,9 @@ async fn asset_graph( triggers, macro_edges, test_edges, + dbt_edges, + dbt_snapshot_job, + dbt_graph_ingested_at, })) } diff --git a/backend/windmill-api-assets/tests/dbt_pinned_graph.rs b/backend/windmill-api-assets/tests/dbt_pinned_graph.rs new file mode 100644 index 0000000000..82e32797d0 --- /dev/null +++ b/backend/windmill-api-assets/tests/dbt_pinned_graph.rs @@ -0,0 +1,444 @@ +//! What a caller who cannot see a dbt script gets from a run of it. +//! +//! The share-link case: `require_job_read_access` has already let them through +//! to the job, so `asset_graph_for` is handed the run — but the project itself +//! is not theirs to read. The graph's SHAPE must survive that and its model SQL +//! must not, and the two are decided by different predicates. Resolving the +//! version from the caller's `script` access instead of from the job emptied the +//! whole dbt half, and every later fix in this area re-touched one of the two. + +use sqlx::{Pool, Postgres}; +use windmill_api_assets::{asset_graph_for, GraphQuery, PinnedRun}; +use windmill_api_auth::ApiAuthed; +use windmill_common::db::UserDB; + +const WS: &str = "test-workspace"; +const PATH: &str = "f/private/proj"; +const HASH: i64 = 42; + +/// A member of the workspace with no grant on `f/private` — the reason someone +/// is sent a share link in the first place. +fn outsider() -> ApiAuthed { + ApiAuthed { + email: "outsider@windmill.dev".to_string(), + username: "outsider".to_string(), + is_admin: false, + is_operator: false, + groups: vec![], + folders: vec![], + scopes: None, + username_override: None, + username_override_is_token_label: false, + is_session_token: false, + token_prefix: None, + read_only: false, + } +} + +async fn seed(db: &Pool, job: uuid::Uuid) { + sqlx::query!( + "INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms) + VALUES ($1, 'private', 'private', '{}', '{}')", + WS + ) + .execute(db) + .await + .unwrap(); + sqlx::query!( + "INSERT INTO script (workspace_id, hash, path, summary, description, content, + created_by, language, lock) + VALUES ($1, $2, $3, '', '', 'profile: {}', 'test-user', 'dbt', '')", + WS, + HASH, + PATH, + ) + .execute(db) + .await + .unwrap(); + sqlx::query!( + "INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id, digest) + VALUES ($1, $2, $3, $4, 'd')", + WS, + PATH, + HASH, + job + ) + .execute(db) + .await + .unwrap(); + sqlx::query!( + r#"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id, + resource_type, name, asset_path, raw_code, tags, description, + columns, freshness) + VALUES ($1, $2, $3, $4, 'model.p.orders', 'model', 'orders', + 'u/a/wh/analytics/orders', 'select 1', '{finance}', 'daily order facts', + '{"order_id": {"description": "natural key"}}'::jsonb, + '{"warn_after": {"count": 12, "period": "hour"}}'::jsonb)"#, + WS, + PATH, + HASH, + job + ) + .execute(db) + .await + .unwrap(); + // A test node, for the arguments it carries: `accepted_values` spells out a + // column's domain. + sqlx::query!( + r#"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id, + resource_type, name, tags, test_kind, test_column, test_args, + attached_node) + VALUES ($1, $2, $3, $4, 'test.p.accepted_values_orders_status', 'test', + 'accepted_values_orders_status', '{}', 'accepted_values', 'status', + '{"values": ["gold", "silver"]}'::jsonb, 'model.p.orders')"#, + WS, + PATH, + HASH, + job + ) + .execute(db) + .await + .unwrap(); +} + +/// Everything on a node that the project's author wrote, rather than the shape +/// of the relation it produces. +const AUTHORED: [&str; 5] = [ + "select 1", + "daily order facts", + "finance", + "natural key", + "gold", +]; + +fn query() -> GraphQuery { + GraphQuery { asset_kinds: Some("dbt".to_string()), folder: None, dbt_script_hash: None } +} + +/// Pinned to a run they are entitled to, the outsider gets the model — and +/// nothing its author wrote. Both halves matter: dropping the first is the blank +/// Models panel under working progress rows, dropping the second hands the +/// project's source and documentation to anyone holding a link. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn a_pinned_run_survives_no_access_to_its_script(db: Pool) { + let job = uuid::Uuid::from_u128(7); + seed(&db, job).await; + + let pinned = PinnedRun { job_id: job, script_path: PATH.to_string(), script_hash: Some(HASH) }; + let res = asset_graph_for( + &outsider(), + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + Some(pinned), + ) + .await + .unwrap(); + let body = serde_json::to_value(&res.0).unwrap(); + + let nodes = body["dbt_nodes"] + .as_array() + .or(body["assets"].as_array()) + .unwrap(); + assert!( + !nodes.is_empty(), + "a run the caller may read must render, whatever their access to the project: {body}" + ); + assert_eq!( + body["dbt_snapshot_job"], + serde_json::json!(job), + "and the marker must agree with it, or the page polls the graph 40 times" + ); + for authored in AUTHORED { + assert!( + !body.to_string().contains(authored), + "`{authored}` is the project's, and stays behind access to it: {body}" + ); + } + assert!( + body.to_string().contains("u/a/wh/analytics/orders"), + "while the relation the run wrote is what the page is for: {body}" + ); + + // The same read by someone who may open the project: the gate has to be the + // caller's access, not a field this endpoint stopped serving. + let seen = asset_graph_for( + &ApiAuthed { is_admin: true, ..outsider() }, + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + Some(PinnedRun { job_id: job, script_path: PATH.to_string(), script_hash: Some(HASH) }), + ) + .await + .unwrap(); + let seen = serde_json::to_value(&seen.0).unwrap().to_string(); + for authored in AUTHORED { + assert!( + seen.contains(authored), + "`{authored}` renders for a reader of the project: {seen}" + ); + } +} + +/// Unpinned, the same caller sees nothing of the project: the workspace graph +/// answers for their own access, and this one is not theirs. This is the half +/// that must NOT be relaxed by making the pinned case work. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn without_a_run_the_same_caller_sees_no_dbt_nodes(db: Pool) { + let job = uuid::Uuid::from_u128(7); + seed(&db, job).await; + + let res = asset_graph_for( + &outsider(), + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + None, + ) + .await + .unwrap(); + let body = serde_json::to_value(&res.0).unwrap(); + assert!( + !body.to_string().contains("orders"), + "an unpinned graph is the caller's own view of the workspace: {body}" + ); +} + +/// Archiving retires the script; it must not retire the runs that already +/// happened. The pinned read resolves versions through a CTE that skips archived +/// rows, so nothing about the CURRENT workspace graph depends on the sidecar +/// surviving — which is exactly why deleting it looks safe and is not. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn archiving_the_script_leaves_its_finished_runs_renderable(db: Pool) { + let job = uuid::Uuid::from_u128(7); + seed(&db, job).await; + sqlx::query!( + "UPDATE script SET archived = true WHERE workspace_id = $1 AND hash = $2", + WS, + HASH + ) + .execute(&db) + .await + .unwrap(); + + let admin = ApiAuthed { is_admin: true, ..outsider() }; + let pinned = PinnedRun { job_id: job, script_path: PATH.to_string(), script_hash: Some(HASH) }; + let res = asset_graph_for( + &admin, + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + Some(pinned), + ) + .await + .unwrap(); + assert!( + serde_json::to_value(&res.0) + .unwrap() + .to_string() + .contains("orders"), + "a completed run of an archived version still renders its models" + ); + + let now = asset_graph_for( + &admin, + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + None, + ) + .await + .unwrap(); + assert!( + !serde_json::to_value(&now.0) + .unwrap() + .to_string() + .contains("model.p.orders"), + "while the workspace graph, which describes what is live, drops it" + ); +} + +/// `extra_perms` is a grant on a ROW, so a path recreated with narrower ones +/// leaves the old version readable to whoever the old row named. The source +/// probe therefore has to name the version it is about to return: matched on +/// the path alone, that stale grant answered for the pinned version's SQL and +/// handed a viewer the body of a project they were never given. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn an_old_grant_at_the_same_path_does_not_expose_a_newer_version(db: Pool) { + const V2: i64 = 43; + let v1_job = uuid::Uuid::from_u128(7); + let v2_job = uuid::Uuid::from_u128(8); + seed(&db, v1_job).await; + + // The version the outsider WAS granted, archived — the shape the grant + // outlives the deploy in. + sqlx::query!( + r#"UPDATE script SET archived = true, extra_perms = '{"u/outsider": true}'::jsonb + WHERE workspace_id = $1 AND hash = $2"#, + WS, + HASH + ) + .execute(&db) + .await + .unwrap(); + + // The version they were not. + sqlx::query!( + "INSERT INTO script (workspace_id, hash, path, summary, description, content, + created_by, language, lock) + VALUES ($1, $2, $3, '', '', 'profile: {}', 'test-user', 'dbt', '')", + WS, + V2, + PATH, + ) + .execute(&db) + .await + .unwrap(); + sqlx::query!( + "INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id, digest) + VALUES ($1, $2, $3, $4, 'd2')", + WS, + PATH, + V2, + v2_job + ) + .execute(&db) + .await + .unwrap(); + sqlx::query!( + "INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id, + resource_type, name, asset_path, raw_code, tags) + VALUES ($1, $2, $3, $4, 'model.p.orders', 'model', 'orders', + 'u/a/wh/analytics/orders', 'select 2', '{}')", + WS, + PATH, + V2, + v2_job + ) + .execute(&db) + .await + .unwrap(); + + let pinned = PinnedRun { job_id: v2_job, script_path: PATH.to_string(), script_hash: Some(V2) }; + let res = asset_graph_for( + &outsider(), + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + Some(pinned), + ) + .await + .unwrap(); + let body = serde_json::to_value(&res.0).unwrap().to_string(); + + assert!( + body.contains("orders"), + "the run they were given still renders its models: {body}" + ); + assert!( + !body.contains("select 2"), + "but not the SQL of a version their grant never covered: {body}" + ); +} + +/// The editor's own graph, keyed to the parse job and to no version. +async fn seed_editor_graph(db: &Pool, job: uuid::Uuid) { + sqlx::query!( + "INSERT INTO dbt_graph_snapshot (workspace_id, script_path, script_hash, job_id, digest) + VALUES ($1, $2, NULL, $3, 'd')", + WS, + PATH, + job + ) + .execute(db) + .await + .unwrap(); + sqlx::query!( + r#"INSERT INTO dbt_node (workspace_id, script_path, script_hash, job_id, unique_id, + resource_type, name, asset_path, raw_code, tags) + VALUES ($1, $2, NULL, $3, 'model.p.draft', 'model', 'draft', + 'u/a/wh/analytics/draft', 'select 3', '{}')"#, + WS, + PATH, + job + ) + .execute(db) + .await + .unwrap(); +} + +/// A buffer parse is the third provenance: no deployed version behind it, and +/// its models are ones no `asset` row has heard of. It renders when pinned to +/// the job that produced it — the only way in — and its SQL comes with it, since +/// it exists because this caller's own parse job wrote the buffer. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn an_editor_graph_renders_only_through_its_own_job(db: Pool) { + let job = uuid::Uuid::from_u128(7); + seed(&db, job).await; + let parse = uuid::Uuid::from_u128(9); + seed_editor_graph(&db, parse).await; + + let admin = ApiAuthed { is_admin: true, ..outsider() }; + let pinned = asset_graph_for( + &admin, + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + Some(PinnedRun { job_id: parse, script_path: PATH.to_string(), script_hash: None }), + ) + .await + .unwrap(); + let body = serde_json::to_value(&pinned.0).unwrap(); + assert!( + body.to_string().contains("u/a/wh/analytics/draft"), + "the buffer's own models render: {body}" + ); + assert!( + body.to_string().contains("select 3"), + "and their SQL, which is what the caller just wrote: {body}" + ); + assert_eq!( + body["dbt_snapshot_job"], + serde_json::json!(parse), + "labelled as a graph of its own, so the editor can say where it came from" + ); + assert!( + !body.to_string().contains("select 1"), + "and the deployed version's models are not mixed into it: {body}" + ); + + // Through the PATH — which is what the workspace graph and every run of the + // deployed version ask for — a buffer parse must not appear at all. It + // describes an editor's unsaved state, not what the script owns. + let workspace = asset_graph_for(&admin, WS, UserDB::new(db.clone()), db.clone(), query(), None) + .await + .unwrap(); + let workspace = serde_json::to_value(&workspace.0).unwrap().to_string(); + assert!( + !workspace.contains("u/a/wh/analytics/draft"), + "an editor's buffer is not part of the workspace graph: {workspace}" + ); + + let deployed_run = asset_graph_for( + &admin, + WS, + UserDB::new(db.clone()), + db.clone(), + query(), + Some(PinnedRun { job_id: job, script_path: PATH.to_string(), script_hash: Some(HASH) }), + ) + .await + .unwrap(); + let deployed_run = serde_json::to_value(&deployed_run.0).unwrap().to_string(); + assert!( + !deployed_run.contains("u/a/wh/analytics/draft"), + "nor of a run of the deployed version: {deployed_run}" + ); +} diff --git a/backend/windmill-api-assets/tests/fixtures/base.sql b/backend/windmill-api-assets/tests/fixtures/base.sql new file mode 100644 index 0000000000..412fa1029f --- /dev/null +++ b/backend/windmill-api-assets/tests/fixtures/base.sql @@ -0,0 +1,146 @@ +-- used for backend automated testing +-- https://docs.rs/sqlx/latest/sqlx/attr.test.html + +INSERT INTO workspace + (id, name, owner) + VALUES ('test-workspace', 'test-workspace', 'test-user'); + +INSERT INTO usr(workspace_id, email, username, is_admin, role) VALUES + ('test-workspace', 'test@windmill.dev', 'test-user', true, 'Admin'); + +INSERT INTO workspace_key(workspace_id, kind, key) VALUES + ('test-workspace', 'cloud', 'test-key'); + + +INSERT INTO workspace_settings (workspace_id) VALUES + ('test-workspace'); + +INSERT INTO group_ (workspace_id, name, summary, extra_perms) VALUES + ('test-workspace', 'all', 'All users', '{}'); + +INSERT INTO password(email, password_hash, login_type, super_admin, verified, name, username) + VALUES ('test@windmill.dev', 'not-a-real-hash', 'password', true, true, 'Test User', 'test-user'); + +INSERT INTO password(email, password_hash, login_type, super_admin, verified, name) + VALUES ('test2@windmill.dev', 'not-a-real-hash', 'password', false, true, 'Test User 2'); + +INSERT INTO password(email, password_hash, login_type, super_admin, verified, name) + VALUES ('test3@windmill.dev', 'not-a-real-hash', 'password', false, true, 'Test User 3'); + +INSERT INTO usr(workspace_id, email, username, is_admin, role) VALUES + ('test-workspace', 'test2@windmill.dev', 'test-user-2', false, 'User'); + +INSERT INTO usr(workspace_id, email, username, is_admin, role) VALUES + ('test-workspace', 'test3@windmill.dev', 'test-user-3', false, 'User'); + +insert INTO token(token_hash, token_prefix, token, email, label, super_admin) VALUES (encode(sha256('SECRET_TOKEN'::bytea), 'hex'), 'SECRET_TOK', 'SECRET_TOKEN', 'test@windmill.dev', 'test token', true); +insert INTO token(token_hash, token_prefix, token, email, label, super_admin) VALUES (encode(sha256('SECRET_TOKEN_2'::bytea), 'hex'), 'SECRET_TOK', 'SECRET_TOKEN_2', 'test2@windmill.dev', 'test token 2', false); +insert INTO token(token_hash, token_prefix, token, email, label, super_admin) VALUES (encode(sha256('SECRET_TOKEN_3'::bytea), 'hex'), 'SECRET_TOK', 'SECRET_TOKEN_3', 'test3@windmill.dev', 'test token 3', false); + +GRANT ALL PRIVILEGES ON TABLE workspace_key TO windmill_admin; +GRANT ALL PRIVILEGES ON TABLE workspace_key TO windmill_user; + +CREATE FUNCTION "notify_insert_on_completed_job" () +RETURNS TRIGGER AS $$ +BEGIN + PERFORM pg_notify('completed', NEW.id::text); + RETURN NEW; +END; +$$ LANGUAGE PLPGSQL; + + CREATE TRIGGER "notify_insert_on_completed_job" + AFTER INSERT ON "v2_job_completed" + FOR EACH ROW +EXECUTE FUNCTION "notify_insert_on_completed_job" (); + + +CREATE FUNCTION "notify_queue" () +RETURNS TRIGGER AS $$ +BEGIN + PERFORM pg_notify('queued', NEW.id::text); + RETURN NEW; +END; +$$ LANGUAGE PLPGSQL; + + CREATE TRIGGER "notify_queue_after_insert" + AFTER INSERT ON "v2_job_queue" + FOR EACH ROW +EXECUTE FUNCTION "notify_queue" (); + + CREATE TRIGGER "notify_queue_after_flow_status_update" + AFTER UPDATE ON "v2_job_status" + FOR EACH ROW + WHEN (NEW.flow_status IS DISTINCT FROM OLD.flow_status) +EXECUTE FUNCTION "notify_queue" (); + +-- Apply phase 4: +DROP FUNCTION IF EXISTS v2_job_after_update CASCADE; +DROP FUNCTION IF EXISTS v2_job_completed_before_insert CASCADE; +DROP FUNCTION IF EXISTS v2_job_completed_before_update CASCADE; +DROP FUNCTION IF EXISTS v2_job_queue_after_insert CASCADE; +DROP FUNCTION IF EXISTS v2_job_queue_before_insert CASCADE; +DROP FUNCTION IF EXISTS v2_job_queue_before_update CASCADE; +DROP FUNCTION IF EXISTS v2_job_runtime_before_insert CASCADE; +DROP FUNCTION IF EXISTS v2_job_runtime_before_update CASCADE; +DROP FUNCTION IF EXISTS v2_job_status_before_insert CASCADE; +DROP FUNCTION IF EXISTS v2_job_status_before_update CASCADE; + +DROP VIEW IF EXISTS completed_job, completed_job_view, job, queue, queue_view CASCADE; + +ALTER TABLE v2_job_queue + DROP COLUMN IF EXISTS __parent_job CASCADE, + DROP COLUMN IF EXISTS __created_by CASCADE, + DROP COLUMN IF EXISTS __script_hash CASCADE, + DROP COLUMN IF EXISTS __script_path CASCADE, + DROP COLUMN IF EXISTS __args CASCADE, + DROP COLUMN IF EXISTS __logs CASCADE, + DROP COLUMN IF EXISTS __raw_code CASCADE, + DROP COLUMN IF EXISTS __canceled CASCADE, + DROP COLUMN IF EXISTS __last_ping CASCADE, + DROP COLUMN IF EXISTS __job_kind CASCADE, + DROP COLUMN IF EXISTS __env_id CASCADE, + DROP COLUMN IF EXISTS __schedule_path CASCADE, + DROP COLUMN IF EXISTS __permissioned_as CASCADE, + DROP COLUMN IF EXISTS __flow_status CASCADE, + DROP COLUMN IF EXISTS __raw_flow CASCADE, + DROP COLUMN IF EXISTS __is_flow_step CASCADE, + DROP COLUMN IF EXISTS __language CASCADE, + DROP COLUMN IF EXISTS __same_worker CASCADE, + DROP COLUMN IF EXISTS __raw_lock CASCADE, + DROP COLUMN IF EXISTS __pre_run_error CASCADE, + DROP COLUMN IF EXISTS __email CASCADE, + DROP COLUMN IF EXISTS __visible_to_owner CASCADE, + DROP COLUMN IF EXISTS __mem_peak CASCADE, + DROP COLUMN IF EXISTS __root_job CASCADE, + DROP COLUMN IF EXISTS __leaf_jobs CASCADE, + DROP COLUMN IF EXISTS __concurrent_limit CASCADE, + DROP COLUMN IF EXISTS __concurrency_time_window_s CASCADE, + DROP COLUMN IF EXISTS __timeout CASCADE, + DROP COLUMN IF EXISTS __flow_step_id CASCADE, + DROP COLUMN IF EXISTS __cache_ttl CASCADE; + +LOCK TABLE v2_job_queue IN ACCESS EXCLUSIVE MODE; +ALTER TABLE v2_job_completed + DROP COLUMN IF EXISTS __parent_job CASCADE, + DROP COLUMN IF EXISTS __created_by CASCADE, + DROP COLUMN IF EXISTS __created_at CASCADE, + DROP COLUMN IF EXISTS __success CASCADE, + DROP COLUMN IF EXISTS __script_hash CASCADE, + DROP COLUMN IF EXISTS __script_path CASCADE, + DROP COLUMN IF EXISTS __args CASCADE, + DROP COLUMN IF EXISTS __logs CASCADE, + DROP COLUMN IF EXISTS __raw_code CASCADE, + DROP COLUMN IF EXISTS __canceled CASCADE, + DROP COLUMN IF EXISTS __job_kind CASCADE, + DROP COLUMN IF EXISTS __env_id CASCADE, + DROP COLUMN IF EXISTS __schedule_path CASCADE, + DROP COLUMN IF EXISTS __permissioned_as CASCADE, + DROP COLUMN IF EXISTS __raw_flow CASCADE, + DROP COLUMN IF EXISTS __is_flow_step CASCADE, + DROP COLUMN IF EXISTS __language CASCADE, + DROP COLUMN IF EXISTS __is_skipped CASCADE, + DROP COLUMN IF EXISTS __raw_lock CASCADE, + DROP COLUMN IF EXISTS __email CASCADE, + DROP COLUMN IF EXISTS __visible_to_owner CASCADE, + DROP COLUMN IF EXISTS __tag CASCADE, + DROP COLUMN IF EXISTS __priority CASCADE; diff --git a/backend/windmill-api-auth/src/auth.rs b/backend/windmill-api-auth/src/auth.rs index 5219b41e9b..b3ad811b7f 100644 --- a/backend/windmill-api-auth/src/auth.rs +++ b/backend/windmill-api-auth/src/auth.rs @@ -26,7 +26,8 @@ use windmill_common::DB; use windmill_common::{ auth::{ - get_folders_for_user, get_groups_for_user, hash_token, safe_token_prefix, JWTAuthClaims, + get_folders_for_user, get_groups_for_user, hash_token, is_session_label, safe_token_prefix, + JWTAuthClaims, }, error::{Error, JsonResult}, jwt, @@ -196,7 +197,9 @@ impl AuthCache { tracing::error!("JWT auth error: workspace_id mismatch"); return None; } - let username_override = username_override_from_label(claims.label); + let is_session_token = is_session_label(claims.label.as_deref()); + let (username_override, username_override_is_token_label) = + username_override_from_label(claims.label); let authed = ApiAuthed { email: claims.email, @@ -211,6 +214,8 @@ impl AuthCache { // WM_TOKEN) keeps full user privileges as before. scopes: claims.scopes, username_override, + username_override_is_token_label, + is_session_token, token_prefix: claims.audit_span, read_only: false, }; @@ -265,7 +270,9 @@ impl AuthCache { (Some(owner), Some(email), super_admin, _, label, read_only) if w_id.is_some() => { - let username_override = username_override_from_label(label); + let is_session_token = is_session_label(label.as_deref()); + let (username_override, username_override_is_token_label) = + username_override_from_label(label); if let Some((prefix, name)) = owner.split_once('/') { if prefix == "u" { let lookup = if super_admin { @@ -308,6 +315,8 @@ impl AuthCache { folders, scopes: None, username_override, + username_override_is_token_label, + is_session_token, token_prefix: Some(safe_token_prefix(token)), read_only, }) @@ -358,6 +367,8 @@ impl AuthCache { folders, scopes: None, username_override, + username_override_is_token_label, + is_session_token, token_prefix: Some(safe_token_prefix(token)), read_only, }) @@ -386,7 +397,9 @@ impl AuthCache { } } (_, Some(email), super_admin, scopes, label, read_only) => { - let username_override = username_override_from_label(label); + let is_session_token = is_session_label(label.as_deref()); + let (username_override, username_override_is_token_label) = + username_override_from_label(label); if w_id.is_some() { let row_o = sqlx::query!( "SELECT username, is_admin, operator FROM usr WHERE @@ -429,6 +442,8 @@ impl AuthCache { folders, scopes, username_override, + username_override_is_token_label, + is_session_token, token_prefix: Some(safe_token_prefix(token)), read_only, }) @@ -450,6 +465,8 @@ impl AuthCache { folders: vec![], scopes, username_override, + username_override_is_token_label, + is_session_token, token_prefix: Some(safe_token_prefix(token)), read_only, }), @@ -473,6 +490,8 @@ impl AuthCache { folders: Vec::new(), scopes, username_override, + username_override_is_token_label, + is_session_token, token_prefix: Some(safe_token_prefix(token)), read_only, }) @@ -508,6 +527,8 @@ impl AuthCache { folders: Vec::new(), scopes: None, username_override: None, + username_override_is_token_label: false, + is_session_token: false, token_prefix: Some(safe_token_prefix(token)), read_only: false, }; @@ -715,6 +736,8 @@ fn no_auth_admin_authed() -> ApiAuthed { folders: Vec::new(), scopes: None, username_override: None, + username_override_is_token_label: false, + is_session_token: false, token_prefix: None, read_only: false, } @@ -835,27 +858,47 @@ pub async fn resolve_opt_job_authed( Err((Error::NotAuthorized("Unauthorized".to_string()), parts)) } -fn username_override_from_label(label: Option) -> Option { +/// Returns the override and whether it names the token's *label* rather than the entity that +/// fired the request. Callers must not re-derive the second element from the first: the +/// `ephemeral-script-end-user-` arm forwards a `created_by` verbatim, and `created_by` is +/// unconstrained, so it may itself look like any of these shapes. +/// +/// Only namespaces `create_token` rejects (`is_server_minted_label`) are trusted to name the +/// entity acting, so the label can only have come from a server-side mint. Tokens minted +/// before that guard existed are the remaining hole; closing it needs the token row to record +/// who minted it rather than inferring it from the label. +/// +/// Note that a trigger whose identity is set server-side — the SMTP one builds an `email-*` +/// override directly — does not rely on this at all, so its prefix must not be trusted here. +pub(crate) fn username_override_from_label(label: Option) -> (Option, bool) { match label { + Some(label) if label.starts_with("ephemeral-webhook-") => (Some(label), false), + Some(label) if label.starts_with("ephemeral-script-end-user-") => ( + Some( + label + .trim_start_matches("ephemeral-script-end-user-") + .to_string(), + ), + false, + ), + // User-mintable, so they name nobody in particular — the trigger panels merely + // pre-fill `webhook-`/`http-`, and the editor mints the lsp one. The override keeps + // its value because `require_job_read_access` matches it against the `created_by` of + // jobs launched under it, which these shapes produced while they were trusted. + Some(label) if label == "Ephemeral lsp token" => (Some("lsp".to_string()), true), Some(label) - if label.starts_with("ephemeral-webhook-") - || label.starts_with("webhook-") + if label.starts_with("webhook-") || label.starts_with("http-") || label.starts_with("email-") || label.starts_with("ws-") => { - Some(label) + (Some(label), true) } - Some(label) if label.starts_with("ephemeral-script-end-user-") => Some( - label - .trim_start_matches("ephemeral-script-end-user-") - .to_string(), + Some(label) if label != "ephemeral-script" && label != "session" && !label.is_empty() => ( + Some(format!("{}{label}", crate::GENERIC_TOKEN_LABEL_PREFIX)), + true, ), - Some(label) if label == "Ephemeral lsp token" => Some("lsp".to_string()), - Some(label) if label != "ephemeral-script" && label != "session" && !label.is_empty() => { - Some(format!("label-{label}")) - } - _ => None, + _ => (None, false), } } diff --git a/backend/windmill-api-auth/src/lib.rs b/backend/windmill-api-auth/src/lib.rs index 8120c16605..d92c3a5f2d 100644 --- a/backend/windmill-api-auth/src/lib.rs +++ b/backend/windmill-api-auth/src/lib.rs @@ -23,6 +23,8 @@ use windmill_common::{ }, db::{Authable, Authed, AuthedRef}, error::{self, Error, Result}, + jobs::JobTriggerKind, + triggers::TriggerMetadata, users::username_to_permissioned_as, DB, }; @@ -37,6 +39,11 @@ pub use auth::{ // ------------ ApiAuthed & OptJobAuthed types ------------ +/// Prefix `username_override_from_label` puts on the label of a generic user token. The +/// override keeps this form even though `display_username` skips it: `require_job_read_access` +/// matches it against `created_by` to let a token re-read the jobs it launched. +pub const GENERIC_TOKEN_LABEL_PREFIX: &str = "label-"; + #[derive(Default, Clone, Debug)] pub struct OptJobAuthed { pub job_id: Option, @@ -54,6 +61,15 @@ pub struct ApiAuthed { pub folders: Vec<(String, bool, bool)>, pub scopes: Option>, pub username_override: Option, + /// Whether `username_override` is a generic user-token label rather than a name that + /// identifies the requester. It cannot be recovered from the value: the ephemeral + /// end-user override passes a `created_by` through verbatim, and that may itself be a + /// `label-*` string. Only `username_override_from_label` sets it. + pub username_override_is_token_label: bool, + /// Whether the request authenticated with the session token minted at browser login. + /// Only `trigger_or_fallback` reads it — see `is_session_label` for why it attributes + /// rather than proves, and must not gate authority. + pub is_session_token: bool, pub token_prefix: Option, pub read_only: bool, } @@ -72,8 +88,43 @@ impl ApiAuthed { } } + /// The name a run triggered by this principal is credited to (`v2_job.created_by`). A + /// trigger-token override names the entity that fired the request and wins; a generic + /// token label does not, so the token owner is credited and stays traceable even when + /// `permissioned_as` is an on-behalf-of identity. The audit `end_user` is the override + /// itself, label included, so the two diverge for a labeled token. pub fn display_username(&self) -> &str { - self.username_override.as_ref().unwrap_or(&self.username) + match self.username_override.as_deref() { + Some(o) if !self.username_override_is_token_label => o, + _ => &self.username, + } + } + + /// Set an override that names the entity acting, e.g. a trigger. Assigning + /// `username_override` on its own would keep the provenance flag of whatever this authed + /// was built from, and a stale `true` makes `display_username` ignore the new value. + pub fn set_acting_username_override(&mut self, username_override: Option) { + self.username_override = username_override; + self.username_override_is_token_label = false; + } + + /// The `trigger_kind` a run started through a `/jobs/run*` route is stamped with: a trigger + /// that built its own metadata always wins, and a run driven by any other token — webhooks, + /// the CLI, the SDKs — is `webhook`, matching the `wm_trigger.kind` the preprocessor already + /// reports for these routes. Derived from the token, never from the request, because the + /// column is authority-bearing for other kinds (`app` marks a file as app-produced). + /// + /// A browser session is left unstamped rather than marked [`JobTriggerKind::Ui`]: that label + /// is one a worker built before this release cannot decode, and it would strand the jobs + /// carrying it. `webhook` has always been decodable, so it is safe to write today. + pub fn trigger_or_fallback(&self, trigger: Option) -> Option { + if trigger.is_some() { + return trigger; + } + if self.is_session_token { + return None; + } + Some(TriggerMetadata::new(None, JobTriggerKind::Webhook)) } } @@ -103,6 +154,8 @@ impl From for ApiAuthed { folders: value.folders, scopes: value.scopes, username_override: None, + username_override_is_token_label: false, + is_session_token: false, token_prefix: value.token_prefix, read_only: false, } @@ -414,6 +467,9 @@ fn scope_contains(caller: &ScopeDefinition, requested: &ScopeDefinition) -> bool // write subsumes read; otherwise the action must match exactly. match (caller.action.as_str(), requested.action.as_str()) { (c, r) if c == r || (c == "write" && r == "read") => {} + // Apps only: `write` covers `run` (see `ScopeDefinition::includes`), so an + // app-editor token can mint the narrower run-only credential. + ("write", "run") if caller.domain == "apps" => {} _ => return false, } @@ -852,6 +908,8 @@ pub async fn fetch_api_authed_from_permissioned_as( folders: authed.folders, scopes: authed.scopes, username_override: None, + username_override_is_token_label: false, + is_session_token: false, token_prefix: authed.token_prefix, read_only: false, }; @@ -869,7 +927,8 @@ pub async fn fetch_api_authed_from_permissioned_as( } }; - api_authed.username_override = username_override; + // Callers pass a trigger or app identity here, never a token label. + api_authed.set_acting_username_override(username_override); Ok(api_authed) } @@ -1194,6 +1253,114 @@ mod tests { } } + /// `display_username` is what `push` credits a run to, so a token label standing in for + /// it erases the caller from `created_by` and from the audit trail — irrecoverably when + /// `permissioned_as` is an on-behalf-of identity that also takes the `username` slot. + #[test] + fn generic_token_label_credits_the_token_owner() { + let owner_of = |label: &str| { + let (username_override, username_override_is_token_label) = + auth::username_override_from_label(Some(label.to_string())); + ApiAuthed { + username: "alice".into(), + username_override, + username_override_is_token_label, + ..Default::default() + } + }; + + // Arbitrary user-chosen labels, and the auto-generated MCP OAuth one. + assert_eq!(owner_of("my-personal-token").display_username(), "alice"); + assert_eq!( + owner_of("mcp-oauth-mcp-client-9f3a1c").display_username(), + "alice" + ); + + // A trigger-*shaped* label is just as user-settable as any other, so it is credited + // the same way. Its value is still kept as the override, for `require_job_read_access`. + let webhookish = owner_of("webhook-f/svc/my_script"); + assert_eq!(webhookish.display_username(), "alice"); + assert_eq!( + webhookish.username_override.as_deref(), + Some("webhook-f/svc/my_script") + ); + + // Only labels `create_token` refuses to mint name the entity that fired the request. + assert_eq!( + owner_of("ephemeral-webhook-google-abc12").display_username(), + "ephemeral-webhook-google-abc12" + ); + + // Minted by the editor through the public handler, so it names no principal either. + assert_eq!(owner_of("Ephemeral lsp token").display_username(), "alice"); + + // The SMTP trigger sets its `email-*` identity server-side rather than through a + // label, so a token carrying that prefix is just a user token. + assert_eq!(owner_of("email-f/team/inbox").display_username(), "alice"); + assert_eq!( + owner_of("ephemeral-script-end-user-enduser42").display_username(), + "enduser42" + ); + + // The end-user token forwards a `created_by` verbatim, and `created_by` is not + // constrained to a username — a job launched before the owner was credited still + // carries `label-*`. That is an end user, not this token's label, so it stands. + assert_eq!( + owner_of("ephemeral-script-end-user-label-alice").display_username(), + "label-alice" + ); + } + + /// A browser session is the one shape left unstamped, so the Runs page can say "a token + /// started this" without claiming the converse. Every other label — every shape a member + /// can pass to `create_token` — is `webhook`. + #[test] + fn only_a_browser_session_is_left_unstamped() { + let kind_of = |label: Option<&str>| { + ApiAuthed { + is_session_token: windmill_common::auth::is_session_label(label), + ..Default::default() + } + .trigger_or_fallback(None) + .map(|t| t.trigger_kind.to_string()) + }; + + assert_eq!(kind_of(Some("session")), None); + + for label in [ + Some("my-personal-token"), + Some("webhook-f/svc/my_script"), + Some("Ephemeral lsp token"), + Some("ephemeral-script"), + Some("ephemeral-webhook-google-abc12"), + Some("mcp-oauth-mcp-client-9f3a1c"), + Some(""), + // A label-less token: the job WM_TOKEN, and any token created without one. + None, + ] { + assert_eq!( + kind_of(label).as_deref(), + Some("webhook"), + "label {label:?}" + ); + } + } + + /// A trigger that built its own metadata must survive the fallback, or a scheduled or + /// routed run started under a personal token would be re-attributed to a webhook. + #[test] + fn a_real_trigger_wins_over_the_token_fallback() { + let authed = ApiAuthed::default(); + let schedule = TriggerMetadata::new( + Some("u/alice/nightly".to_string()), + JobTriggerKind::Schedule, + ); + + let kept = authed.trigger_or_fallback(Some(schedule)).unwrap(); + assert_eq!(kept.trigger_kind.to_string(), "schedule"); + assert_eq!(kept.trigger_path.as_deref(), Some("u/alice/nightly")); + } + // Regression tests for the Preview path traversal: a Preview's path skips the // DB `proper_id` CHECK and reaches the worker, where it builds on-disk module // dirs. Traversal must be rejected even for admins, who otherwise bypass the @@ -1484,6 +1651,25 @@ mod tests { opt_scopes(Some(vec!["users:read", "if_jobs:filter_tags:default"])).as_deref() ) .is_ok()); + // Apps `write` covers `run`, so an app-editor token can mint the run-only + // credential for the same app — but only within its own resource subtree, + // and the equivalence stays Apps-only. + let app_editor = authed_with_scopes(Some(vec!["apps:write:u/me/a", "jobs:write"])); + assert!(ensure_scopes_within_caller( + &app_editor, + opt_scopes(Some(vec!["apps:run:u/me/a"])).as_deref() + ) + .is_ok()); + assert!(ensure_scopes_within_caller( + &app_editor, + opt_scopes(Some(vec!["apps:run:u/me/b"])).as_deref() + ) + .is_err()); + assert!(ensure_scopes_within_caller( + &app_editor, + opt_scopes(Some(vec!["jobs:run"])).as_deref() + ) + .is_err()); } #[test] diff --git a/backend/windmill-api-auth/src/scopes.rs b/backend/windmill-api-auth/src/scopes.rs index 75cbc104ac..ed273c7d15 100644 --- a/backend/windmill-api-auth/src/scopes.rs +++ b/backend/windmill-api-auth/src/scopes.rs @@ -120,6 +120,10 @@ impl ScopeDefinition { match (self.action.as_str(), other.action.as_str()) { (a, b) if (a == "write" && b == "read") || (a == b) => {} + // Apps only: `write` can rewrite the app and its policy, so it also covers + // running its components. Not general — `jobs:write` must not grant + // `jobs:run`. The resource check below still confines it to the same app. + ("write", "run") if self.domain == "apps" => {} _ => return false, } @@ -489,6 +493,25 @@ pub fn check_route_access( } } + // Each declared scope must grant what its prompt said and no more: + // `jobs:run` only deployed runnables, `users:read` only the viewer's identity. + if has_raw_app_sdk_sentinel(Some(token_scopes)) { + if let Some(suffix) = route_suffix.as_deref() { + if is_request_supplied_code_route(suffix) { + return Err(Error::PermissionDenied( + "Access denied. A raw app frontend SDK token cannot run request-supplied code." + .to_string(), + )); + } + if required_domain == ScopeDomain::Users && suffix != "users/whoami" { + return Err(Error::PermissionDenied( + "Access denied. A raw app frontend SDK token can only read the viewer's own identity." + .to_string(), + )); + } + } + } + // MCP scopes (mcp:all, mcp:favorites, mcp:hub:*, etc.) use a custom format // that doesn't fit the standard domain:action model. Verify the token has at // least one mcp: scope; MCP handlers do their own fine-grained checking. @@ -701,6 +724,34 @@ pub fn has_app_embed_sentinel(scopes: Option<&[String]>) -> bool { scopes.is_some_and(|s| s.iter().any(|x| x == APP_EMBED_SENTINEL)) } +/// Sentinel in raw-app SDK tokens. Grants nothing; `check_route_access` uses it +/// to narrow the declared scopes to what the viewer's prompt promised. +pub const RAW_APP_SDK_SENTINEL: &str = "raw_app_sdk"; + +pub fn has_raw_app_sdk_sentinel(scopes: Option<&[String]>) -> bool { + scopes.is_some_and(|s| s.iter().any(|x| x == RAW_APP_SDK_SENTINEL)) +} + +/// Endpoints that run code the caller supplies or names by job id (the latter +/// with no ownership check). Their jobs get an unscoped credential as the viewer, +/// so reaching one would make a captured SDK token a full account takeover. +fn is_request_supplied_code_route(suffix: &str) -> bool { + // Prefixes, so the `_async` variants are covered too. + const CODE_ROUTES: [&str; 10] = [ + "jobs/run/preview", + "jobs/run_inline/preview", + "jobs/run_wait_result/preview", + "jobs/run/preview_bundle", + "jobs/run/preview_flow", + "jobs/run_wait_result/preview_flow", + "jobs/run/dependencies", + "jobs/run/flow_dependencies", + "jobs/run/workflow_as_code", + "jobs/restart/f", + ]; + CODE_ROUTES.iter().any(|p| suffix.starts_with(p)) +} + /// Routes an app embed token (sentinel) is denied. Its broad scopes (`apps:run`, /// `jobs:read`, `users:read`, `folders:read`) exist only for a fixed set of routes a /// running app uses, but the whole `/apps`, `/jobs`, `/users`, `/folders` routers are @@ -802,6 +853,15 @@ fn scope_grants_access( && route_path.is_some_and(resource_metadata_route_allowed)); } + // Apps `write` covers `run` (see `ScopeDefinition::includes`). Like every domain + // here this layer is resource-blind; the Run handlers path-check the app. + if scope_domain == ScopeDomain::Apps + && scope_action == ScopeAction::Write + && required_action == ScopeAction::Run + { + return Ok(true); + } + if !scope_action.includes(&required_action) && !(scope_domain == ScopeDomain::Jobs && required_action == ScopeAction::Read @@ -1016,6 +1076,29 @@ mod tests { assert!(check_route_access(&sc, "/api/w/test/data_metrics/list", "GET").is_err()); } + /// `apps_u/execute_component` (and the S3 upload the same components drive) is a + /// Run action, so a scoped token needs `apps:run`. `apps:write` must keep reaching + /// it too: it can rewrite the app and its policy, so withholding execution from it + /// protects nothing while breaking every app-scoped token. + #[test] + fn apps_run_routes_accept_run_and_write_scopes() { + let execute = "/api/w/test/apps_u/execute_component/u/admin/app"; + for scope in ["apps:run", "apps:write"] { + assert!( + check_route_access(&[scope.to_string()], execute, "POST").is_ok(), + "{scope} must reach execute_component" + ); + } + assert!(check_route_access(&["apps:read".to_string()], execute, "POST").is_err()); + // The write-satisfies-run allowance is confined to the apps domain. + assert!(check_route_access( + &["jobs:write".to_string()], + "/api/w/test/jobs/run/p/u/admin/script", + "POST" + ) + .is_err()); + } + #[test] fn test_new_domain_parsing() { // Test that new domains are properly parsed diff --git a/backend/windmill-api-debug/Cargo.toml b/backend/windmill-api-debug/Cargo.toml index 8531407416..c262ac3d32 100644 --- a/backend/windmill-api-debug/Cargo.toml +++ b/backend/windmill-api-debug/Cargo.toml @@ -8,6 +8,10 @@ edition.workspace = true name = "windmill_api_debug" path = "src/lib.rs" +[features] +default = [] +enterprise = [] + [dependencies] windmill-api-auth.workspace = true windmill-common = { workspace = true, default-features = false } diff --git a/backend/windmill-api-debug/src/lib.rs b/backend/windmill-api-debug/src/lib.rs index 3b43a04ad1..7b2f569821 100644 --- a/backend/windmill-api-debug/src/lib.rs +++ b/backend/windmill-api-debug/src/lib.rs @@ -20,15 +20,21 @@ //! - A job entry in v2_job (kind=preview) for traceability //! - A completed job entry in v2_job_completed //! - An audit log entry identical to script preview runs +//! +//! The same signature is what authorizes the debugger's requests back to the API: +//! /api/debug/registry_config serves the instance's dependency-registry settings, which the +//! debugger cannot read for itself, to sessions whose token carries the `registry_config` +//! claim. use axum::{ extract::Path, + http::HeaderMap, routing::{get, post}, Extension, Json, Router, }; use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine}; use chrono::Utc; -use ed25519_dalek::{Signer, SigningKey}; +use ed25519_dalek::{Signature, Signer, SigningKey}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use sqlx::types::Json as SqlxJson; @@ -37,8 +43,18 @@ use tokio::sync::RwLock; use uuid::Uuid; use windmill_audit::{audit_oss::audit_log, ActionKind}; use windmill_common::{ - db::UserDB, error::JsonResult, jobs::JobKind, jwt::JWT_SECRET, scripts::ScriptLang, + db::UserDB, + error::{Error, JsonResult}, + global_settings::{ + BUNFIG_INSTALL_SCOPES_SETTING, EXTRA_PIP_INDEX_URL_SETTING, NPMRC_SETTING, + NPM_CONFIG_REGISTRY_SETTING, PIP_INDEX_URL_SETTING, UV_INDEX_STRATEGY_SETTING, + WORKSPACE_REGISTRIES_SETTING, + }, + jobs::JobKind, + jwt::JWT_SECRET, + scripts::ScriptLang, users::username_to_permissioned_as, + DB, }; use windmill_api_auth::ApiAuthed; @@ -112,7 +128,9 @@ pub async fn reload_debug_signing_key() { } pub fn global_service() -> Router { - Router::new().route("/jwks", get(get_jwks)) + Router::new() + .route("/jwks", get(get_jwks)) + .route("/registry_config", get(get_registry_config)) } pub fn workspaced_service() -> Router { @@ -168,6 +186,220 @@ async fn get_jwks() -> JsonResult { })) } +/// The instance's dependency-registry configuration, as `windmill prepare-deps` consumes it. +/// Field names are the `global_settings` keys, so the debug service forwards this object to +/// the CLI as-is. +#[derive(Serialize, Default)] +pub struct DebugRegistryConfig { + #[serde(skip_serializing_if = "Option::is_none")] + pub npm_config_registry: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub npmrc: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub bunfig_install_scopes: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub pip_index_url: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub pip_extra_index_url: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub uv_index_strategy: Option, + /// Why configured settings were withheld, for the debug service to show the user. + #[serde(skip_serializing_if = "Option::is_none")] + pub message: Option, +} + +/// Placeholder an EE instance puts in an index URL for a token minted per install by +/// `EPHEMERAL_TOKEN_CMD` (`windmill-worker`'s `handle_ephemeral_token`). +const EPHEMERAL_TOKEN_MARKER: &str = "EPHEMERAL_TOKEN"; + +/// Every `global_settings` key [`get_registry_config`] reads, in one query. A setting +/// resolved there but missing here reads as unset, whatever the instance has stored. +const REGISTRY_SETTINGS: [&str; 7] = [ + NPM_CONFIG_REGISTRY_SETTING, + NPMRC_SETTING, + BUNFIG_INSTALL_SCOPES_SETTING, + PIP_INDEX_URL_SETTING, + EXTRA_PIP_INDEX_URL_SETTING, + UV_INDEX_STRATEGY_SETTING, + WORKSPACE_REGISTRIES_SETTING, +]; + +/// Which half of the settings a session installs with, from the language its token was signed +/// for: a session is served only what its own installer runs on, so a token minted for one +/// language cannot be replayed to read the other's credentials. Every language the debugger +/// accepts (`isDebuggableLanguage` in the frontend) has to appear here, or its sessions +/// silently install from the public registries. +fn registry_settings_for_language(language: &str) -> (bool, bool) { + match language { + "bun" | "typescript" | "deno" | "nativets" => (true, false), + "python3" | "python" => (false, true), + _ => (false, false), + } +} + +/// Resolve one setting the way a worker resolves it: a workspace override wins over the +/// instance value, which is `FORCE_` > `global_settings` > `` (the server's +/// `load_option_setting_value`). A blank value means unset from either source, so a +/// workspace can blank one out. +fn resolve_registry_setting( + stored: &std::collections::HashMap, + workspace_overrides: Option<&serde_json::Value>, + key: &str, + env_var: &str, +) -> Option { + let as_str = |v: Option<&serde_json::Value>| v.and_then(|v| v.as_str()).map(|s| s.to_string()); + let instance_value = std::env::var(format!("FORCE_{env_var}")) + .ok() + .or_else(|| as_str(stored.get(key))) + .or_else(|| std::env::var(env_var).ok()); + as_str(workspace_overrides.and_then(|w| w.get(key))) + .or(instance_value) + .filter(|v| !v.trim().is_empty()) +} + +/// Serve the dependency-registry settings to the debug service. +/// +/// `windmill prepare-deps` installs a debug session's imports without a database +/// connection, so the service fetches the settings here and passes them down over the +/// CLI's stdin request. They stop there: a private index URL embeds credentials and the +/// debugged script can read its own process, so nothing served here reaches the session's +/// environment (see `debugger/README.md`). +/// +/// Authorized by the launch token the service verified for that session, and only when the +/// token carries the `registry_config` claim (see [`sign_debug_request`] for what it means). +async fn get_registry_config( + Extension(db): Extension, + headers: HeaderMap, +) -> JsonResult { + let token = headers + .get("authorization") + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.strip_prefix("Bearer ")) + .ok_or_else(|| Error::NotAuthorized("Missing debug token".to_string()))?; + + let claims = verify_debug_token(token).await?; + if !claims.registry_config { + return Err(Error::NotAuthorized( + "This debug session is not allowed to read the registry configuration".to_string(), + )); + } + + let names = REGISTRY_SETTINGS.map(String::from); + let stored = sqlx::query!( + "SELECT name, value FROM global_settings WHERE name = ANY($1)", + &names[..] + ) + .fetch_all(&db) + .await? + .into_iter() + .map(|r| (r.name, r.value)) + .collect::>(); + + let workspace_overrides = stored + .get(WORKSPACE_REGISTRIES_SETTING) + .and_then(|v| v.get(&claims.workspace_id)); + let (npm, python) = registry_settings_for_language(&claims.language); + let resolve = |serve: bool, key: &str, env_var: &str| { + serve + .then(|| resolve_registry_setting(&stored, workspace_overrides, key, env_var)) + .flatten() + }; + + let mut config = DebugRegistryConfig { + npm_config_registry: resolve(npm, NPM_CONFIG_REGISTRY_SETTING, "NPM_CONFIG_REGISTRY"), + npmrc: resolve(npm, NPMRC_SETTING, "NPMRC"), + bunfig_install_scopes: resolve( + npm, + BUNFIG_INSTALL_SCOPES_SETTING, + "BUNFIG_INSTALL_SCOPES", + ), + pip_index_url: resolve(python, PIP_INDEX_URL_SETTING, "PIP_INDEX_URL"), + pip_extra_index_url: resolve(python, EXTRA_PIP_INDEX_URL_SETTING, "PIP_EXTRA_INDEX_URL"), + // Not a private-registry setting: a worker reads it on any edition, so it is + // served below the Enterprise gate too. + uv_index_strategy: resolve(python, UV_INDEX_STRATEGY_SETTING, "UV_INDEX_STRATEGY"), + message: None, + }; + + if cfg!(feature = "enterprise") { + // A worker substitutes this marker with the output of `EPHEMERAL_TOKEN_CMD` + // (`handle_ephemeral_token`), a command the debug service has no way to run. Serving + // the placeholder would install with a literal token, so the value is withheld and + // the service falls back to the index URL in its own environment. + let ephemeral = |url: &Option| { + url.as_ref() + .is_some_and(|u| u.contains(EPHEMERAL_TOKEN_MARKER)) + }; + if ephemeral(&config.pip_index_url) || ephemeral(&config.pip_extra_index_url) { + config.pip_index_url = None; + config.pip_extra_index_url = None; + config.message = Some(format!( + "Python index configuration ignored: an {EPHEMERAL_TOKEN_MARKER} index URL can only be resolved on a worker" + )); + } + } else { + // A private registry is an Enterprise feature, and `read_ee_registry` drops these + // same settings on a CE worker, so a CE debug session installs from the public registries + // and says why, instead of gaining a capability jobs on that instance don't have. + let configured = config.npm_config_registry.is_some() + || config.npmrc.is_some() + || config.bunfig_install_scopes.is_some() + || config.pip_index_url.is_some() + || config.pip_extra_index_url.is_some(); + config.npm_config_registry = None; + config.npmrc = None; + config.bunfig_install_scopes = None; + config.pip_index_url = None; + config.pip_extra_index_url = None; + if configured { + config.message = Some( + "Private registry configuration ignored: this feature requires Windmill Enterprise Edition" + .to_string(), + ); + } + } + + Ok(Json(config)) +} + +/// Verify a token minted by [`sign_debug_request`] and return its claims. +/// +/// The debug service verifies the same token itself against the JWKS public key; this is +/// the server-side half, for the requests the service makes back on a session's behalf. +async fn verify_debug_token(token: &str) -> Result { + let key_guard = DEBUG_SIGNING_KEY.read().await; + let signing_key = key_guard + .as_ref() + .ok_or_else(|| Error::InternalErr("Debug signing key not initialized".to_string()))?; + + let invalid = || Error::NotAuthorized("Invalid debug token".to_string()); + let mut parts = token.split('.'); + let (header_b64, claims_b64, signature_b64) = + match (parts.next(), parts.next(), parts.next(), parts.next()) { + (Some(header), Some(claims), Some(signature), None) => (header, claims, signature), + _ => return Err(invalid()), + }; + + let signature = Signature::from_slice( + &URL_SAFE_NO_PAD + .decode(signature_b64) + .map_err(|_| invalid())?, + ) + .map_err(|_| invalid())?; + signing_key + .verifying_key() + .verify_strict(format!("{header_b64}.{claims_b64}").as_bytes(), &signature) + .map_err(|_| invalid())?; + + let claims: DebugTokenClaims = + serde_json::from_slice(&URL_SAFE_NO_PAD.decode(claims_b64).map_err(|_| invalid())?) + .map_err(|_| invalid())?; + if Utc::now().timestamp() > claims.exp { + return Err(Error::NotAuthorized("Debug token expired".to_string())); + } + Ok(claims) +} + #[derive(Deserialize)] pub struct SignDebugRequest { /// The code to be debugged @@ -193,6 +425,11 @@ pub struct DebugTokenClaims { pub exp: i64, /// Job ID for traceability pub job_id: String, + /// Whether this session may be served the instance's dependency-registry settings + /// (see [`get_registry_config`]). Defaults to `false` so a token that predates the + /// claim is refused rather than silently trusted. + #[serde(default)] + pub registry_config: bool, } #[derive(Serialize)] @@ -248,6 +485,14 @@ async fn sign_debug_request( iat: now_ts, exp, job_id: job_id.to_string(), + // The registry settings embed credentials, and the token reaches the browser, so they + // are only served for a session whose author can already install with them: someone + // who can run a preview job. For npm that discloses nothing new, since a worker leaves + // the same `.npmrc` / `bunfig.toml` in the directory the previewed script runs in; the + // Python index URL only ever appears as uv's argv, so serving it here does widen what + // a member of the workspace can read. Operators cannot run previews at all, so their + // sessions install from the public registries. + registry_config: !authed.is_operator, }; // Create JWT manually with Ed25519 signature @@ -504,3 +749,107 @@ async fn sign_multiplayer( Ok(Json(SignedMultiplayerPayload { token })) } + +#[cfg(test)] +mod tests { + use super::*; + + /// Sign `claims` the way [`sign_debug_request`] does, with a key only this test knows. + async fn signed(claims: &DebugTokenClaims) -> String { + let key = derive_signing_key_from_jwt_secret("test-secret"); + *DEBUG_SIGNING_KEY.write().await = Some(key.clone()); + let header = URL_SAFE_NO_PAD.encode(r#"{"alg":"EdDSA","typ":"JWT"}"#); + let payload = URL_SAFE_NO_PAD.encode(serde_json::to_string(claims).unwrap()); + let message = format!("{header}.{payload}"); + let signature = URL_SAFE_NO_PAD.encode(key.sign(message.as_bytes()).to_bytes()); + format!("{message}.{signature}") + } + + fn claims(exp_in: i64) -> DebugTokenClaims { + DebugTokenClaims { + code_hash: "0".repeat(32), + language: "bun".to_string(), + workspace_id: "test".to_string(), + email: "user@windmill.dev".to_string(), + iat: Utc::now().timestamp(), + exp: Utc::now().timestamp() + exp_in, + job_id: Uuid::nil().to_string(), + registry_config: true, + } + } + + /// The registry settings are credentials, and this signature is the only thing standing + /// between them and any caller of `/api/debug/registry_config`. Each rejection here is a + /// way in if it stops being checked: an edited claim, a session whose author may not read + /// them, or a token replayed long after its session. + #[tokio::test] + async fn only_an_unexpired_token_with_the_claim_verifies() { + let token = signed(&claims(60)).await; + assert!(verify_debug_token(&token).await.is_ok()); + + let no_claim = signed(&DebugTokenClaims { registry_config: false, ..claims(60) }).await; + assert!(!verify_debug_token(&no_claim).await.unwrap().registry_config); + + let expired = signed(&claims(-1)).await; + assert!(verify_debug_token(&expired).await.is_err()); + + // Re-signing is the only way to change a claim: swapping the payload of a valid token + // for one that grants itself the claim must not verify. + let (header, rest) = token.split_once('.').unwrap(); + let (_, signature) = rest.split_once('.').unwrap(); + let forged = URL_SAFE_NO_PAD.encode( + serde_json::to_string(&DebugTokenClaims { exp: i64::MAX, ..claims(60) }).unwrap(), + ); + assert!( + verify_debug_token(&format!("{header}.{forged}.{signature}")) + .await + .is_err() + ); + } + + /// Every language the debugger can start a session for installs dependencies, so each one + /// has to name the settings its installer reads. A language missing here is not a refusal: + /// its sessions quietly install from the public registries instead. + #[test] + fn every_debuggable_language_is_served_its_own_settings() { + // Mirrors `isDebuggableLanguage` in frontend/src/lib/components/debug/debugUtils.ts. + for language in ["bun", "typescript", "deno", "nativets"] { + assert_eq!(registry_settings_for_language(language), (true, false)); + } + assert_eq!(registry_settings_for_language("python3"), (false, true)); + assert_eq!(registry_settings_for_language("go"), (false, false)); + } + + /// A debug session must resolve a registry setting to what a job in the same workspace + /// resolves it to (`read_ee_registry_with_workspace_override`): the workspace override + /// replaces the instance value, and a blank value from either source means unset, which + /// is how a workspace opts out of an instance-wide registry. + #[test] + fn workspace_override_replaces_the_instance_value() { + let stored = [( + NPM_CONFIG_REGISTRY_SETTING.to_string(), + serde_json::json!("https://instance.example/"), + )] + .into_iter() + .collect::>(); + // Never set, so the environment fallback stays out of the comparison. + let env_var = "WM_TEST_DEBUG_REGISTRY_UNSET"; + let resolve = |overrides: Option<&serde_json::Value>| { + resolve_registry_setting(&stored, overrides, NPM_CONFIG_REGISTRY_SETTING, env_var) + }; + + assert_eq!(resolve(None).as_deref(), Some("https://instance.example/")); + let workspace = serde_json::json!({ "npm_config_registry": "https://workspace.example/" }); + assert_eq!( + resolve(Some(&workspace)).as_deref(), + Some("https://workspace.example/") + ); + let blanked = serde_json::json!({ "npm_config_registry": " " }); + assert_eq!(resolve(Some(&blanked)), None); + let unrelated = serde_json::json!({ "npmrc": "//other/:_authToken=x" }); + assert_eq!( + resolve(Some(&unrelated)).as_deref(), + Some("https://instance.example/") + ); + } +} diff --git a/backend/windmill-api-flows/Cargo.toml b/backend/windmill-api-flows/Cargo.toml index 231c00a159..3fc2c7b572 100644 --- a/backend/windmill-api-flows/Cargo.toml +++ b/backend/windmill-api-flows/Cargo.toml @@ -10,8 +10,9 @@ path = "src/lib.rs" [features] default = [] -enterprise = ["windmill-common/enterprise"] +enterprise = ["windmill-common/enterprise", "windmill-native-triggers?/enterprise"] private = ["windmill-common/private", "windmill-dep-map/private"] +native_trigger = ["dep:windmill-native-triggers", "windmill-native-triggers/native_trigger"] [dependencies] windmill-common = { workspace = true, default-features = false } windmill-api-auth.workspace = true @@ -19,6 +20,7 @@ windmill-queue.workspace = true windmill-audit.workspace = true windmill-git-sync.workspace = true windmill-dep-map.workspace = true +windmill-native-triggers = { workspace = true, optional = true } axum.workspace = true hyper.workspace = true diff --git a/backend/windmill-api-flows/src/flows.rs b/backend/windmill-api-flows/src/flows.rs index ca3d65b2ce..71b98b22bd 100644 --- a/backend/windmill-api-flows/src/flows.rs +++ b/backend/windmill-api-flows/src/flows.rs @@ -50,6 +50,7 @@ use windmill_common::{ flows::{EditFlow, Flow, FlowWithStarred, ListFlowQuery, ListableFlow, NewFlow}, jobs::JobPayload, schedule::Schedule, + triggers::MovedNativeTrigger, utils::{http_get_from_hub, not_found_if_none, paginate, Pagination, RunnableKind, StripPath}, }; use windmill_dep_map::scoped_dependency_map::ScopedDependencyMap; @@ -577,17 +578,17 @@ async fn create_flow( // Apply folder default_permissioned_as on create when the caller did not // explicitly preserve a value and the user can preserve. - let explicit_preserve = nf.on_behalf_of_email.is_some() + let explicit_preserve = (nf.on_behalf_of_email.is_some() + || nf.on_behalf_of.is_some()) && nf.preserve_on_behalf_of.unwrap_or(false) && windmill_common::can_preserve_on_behalf_of(&authed); if !explicit_preserve && windmill_common::can_preserve_on_behalf_of(&authed) { - if let Some(default_email) = - windmill_common::folders::resolve_folder_default_on_behalf_of_email( - &db, &w_id, &nf.path, - ) - .await? + if let Some((default_email, default_permissioned_as)) = + windmill_common::folders::resolve_folder_default_on_behalf_of(&db, &w_id, &nf.path) + .await? { nf.on_behalf_of_email = Some(default_email); + nf.on_behalf_of = Some(default_permissioned_as); nf.preserve_on_behalf_of = Some(true); } } @@ -598,19 +599,35 @@ async fn create_flow( check_schedule_conflict(&mut tx, &w_id, &nf.path).await?; let schema_str = nf.schema.and_then(|x| serde_json::to_string(&x.0).ok()); + let resolved_on_behalf_of = + windmill_common::resolve_on_behalf_of( + nf.on_behalf_of_email.as_deref(), + nf.on_behalf_of.as_deref(), + nf.preserve_on_behalf_of.unwrap_or(false), + &authed, + &w_id, + &db, + ) + .await?; + // Written beside the principal only while a worker that still reads it may be live. + let legacy_on_behalf_of_email = + windmill_common::legacy_on_behalf_of_email(resolved_on_behalf_of.as_deref(), &w_id, &db) + .await?; sqlx::query!( r#"INSERT INTO flow ( workspace_id, path, summary, description, dependency_job, lock_error_logs, tag, - dedicated_worker, visible_to_runner_only, on_behalf_of_email, + dedicated_worker, visible_to_runner_only, ws_error_handler_muted, - value, schema, edited_by, edited_at, labels + value, schema, edited_by, edited_at, labels, + on_behalf_of, on_behalf_of_email ) VALUES ( $1, $2, $3, $4, NULL, '', $5, - $6, $7, $8, - $9, - $10, $11::text::json, $12, now(), $13 + $6, $7, + $8, + $9, $10::text::json, $11, now(), $12, + $13, $14 )"#, w_id, nf.path, @@ -619,16 +636,13 @@ async fn create_flow( nf.tag, nf.dedicated_worker, nf.visible_to_runner_only.unwrap_or(false), - windmill_common::resolve_on_behalf_of_email( - nf.on_behalf_of_email.as_deref(), - nf.preserve_on_behalf_of.unwrap_or(false), - &authed, - ), nf.ws_error_handler_muted.unwrap_or(false), sqlx::types::Json(&nf.value) as _, schema_str, &authed.username, nf.labels.as_deref() as Option<&[String]>, + resolved_on_behalf_of, + legacy_on_behalf_of_email, ) .execute(&mut *tx) .await?; @@ -684,10 +698,10 @@ async fn create_flow( ) .await?; if let Some(on_behalf_of) = windmill_common::check_on_behalf_of_preservation( - nf.on_behalf_of_email.as_deref(), + resolved_on_behalf_of.as_deref(), nf.preserve_on_behalf_of.unwrap_or(false), &authed, - &authed.email, + &windmill_common::users::username_to_permissioned_as(&authed.username), ) { audit_log( &mut *tx, @@ -728,6 +742,7 @@ async fn create_flow( &authed.email, windmill_common::users::username_to_permissioned_as(&authed.username), authed.token_prefix.as_deref(), + authed.username_override.as_deref(), None, None, None, @@ -863,8 +878,27 @@ async fn get_latest_version( Ok(Json(version)) } +/// `on_behalf_of_email` is derived rather than selected: the read paths fill it from the +/// principal so clients written against the address keep working. The column itself still +/// exists for the workers that read it — see `legacy_on_behalf_of_email`. +async fn derived_on_behalf_of_email( + db: &DB, + w_id: &str, + flow: &Flow, +) -> error::Result> { + let Some(permissioned_as) = flow.on_behalf_of.as_deref() else { + return Ok(None); + }; + // Uncached, for the reason given on `prefetch_cached_script`: this pair is round-tripped. + Ok(Some( + windmill_common::users::get_email_from_permissioned_as_uncached(permissioned_as, w_id, db) + .await?, + )) +} + async fn get_flow_version( authed: ApiAuthed, + Extension(db): Extension, Extension(user_db): Extension, Path((w_id, version, path)): Path<(String, i64, StripPath)>, ) -> JsonResult { @@ -873,26 +907,28 @@ async fn get_flow_version( let mut tx = user_db.begin(&authed).await?; let flow = sqlx::query_as::<_, Flow>( - "SELECT flow.workspace_id, flow.path, flow.summary, flow.description, flow.archived, flow.extra_perms, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, flow.on_behalf_of_email, flow.labels, flow_version.schema, flow_version.value, flow_version.created_at as edited_at, flow_version.created_by as edited_by + "SELECT flow.workspace_id, flow.path, flow.summary, flow.description, flow.archived, flow.extra_perms, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, flow.on_behalf_of, flow.labels, flow_version.schema, flow_version.value, flow_version.created_at as edited_at, flow_version.created_by as edited_by FROM flow LEFT JOIN flow_version ON flow_version.path = flow.path AND flow_version.workspace_id = flow.workspace_id WHERE flow.path = $1 AND flow.workspace_id = $2 AND flow_version.id = $3", ) .bind(path) - .bind(w_id) + .bind(&w_id) .bind(version) .fetch_optional(&mut *tx) .await?; tx.commit().await?; - let flow = not_found_if_none(flow, "Flow version", version.to_string())?; + let mut flow = not_found_if_none(flow, "Flow version", version.to_string())?; + flow.on_behalf_of_email = derived_on_behalf_of_email(&db, &w_id, &flow).await?; Ok(Json(flow)) } async fn get_flow_version_by_id( authed: ApiAuthed, + Extension(db): Extension, Extension(user_db): Extension, Path((w_id, version)): Path<(String, i64)>, ) -> JsonResult { @@ -929,7 +965,7 @@ async fn get_flow_version_by_id( flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, - flow.on_behalf_of_email, + flow.on_behalf_of, flow.labels, flow_version.schema, flow_version.value, @@ -948,11 +984,12 @@ async fn get_flow_version_by_id( tx.commit().await?; - let flow = not_found_if_none( + let mut flow = not_found_if_none( flow, "Flow", format!("for version {} (flow may have been deleted)", version), )?; + flow.on_behalf_of_email = derived_on_behalf_of_email(&db, &w_id, &flow).await?; Ok(Json(flow)) } @@ -1002,6 +1039,35 @@ async fn update_flow_history( Ok(()) } +/// Re-point the webhooks of the native triggers a rename carried onto the new path. +/// +/// Runs after the deploy transaction commits — repointing a webhook is not undoable — and off the +/// request, because it waits on a third-party service that may be slow or gone, and a deploy that +/// already committed must not look like it failed. The rename itself marked these rows +/// `REREGISTRATION_PENDING`, so nothing is lost silently if this never finishes. +fn reregister_moved_native_triggers( + db: &DB, + authed: &ApiAuthed, + w_id: &str, + moved: Vec, +) { + if moved.is_empty() { + return; + } + #[cfg(feature = "native_trigger")] + { + let (db, authed, w_id) = (db.clone(), authed.clone(), w_id.to_string()); + tokio::spawn(async move { + windmill_native_triggers::rename::reregister_triggers_after_rename( + &db, &authed, &w_id, &moved, + ) + .await; + }); + } + #[cfg(not(feature = "native_trigger"))] + let _ = (db, authed, w_id, moved); +} + async fn update_flow( authed: ApiAuthed, Extension(user_db): Extension, @@ -1021,6 +1087,13 @@ async fn update_flow( // A `<= 0` flow timeout is "unset", not a 0-second limit (see create_flow). nf.timeout = windmill_common::runnable_settings::none_if_non_positive(nf.timeout); check_scopes(&authed, || format!("flows:write:{}", flow_path))?; + // A rename writes the destination as much as the source, so a path-scoped token needs both. + // Checking only the source would let it move a flow onto a path it has no say over — and + // everything that follows the rename, native triggers included, is then acting on a path this + // caller was never authorized for. `create_script` already scopes against its destination. + if nf.path != flow_path { + check_scopes(&authed, || format!("flows:write:{}", nf.path))?; + } if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, @@ -1053,6 +1126,20 @@ async fn update_flow( let old_dep_job = not_found_if_none(old_dep_job, "Flow", flow_path)?; let is_new_path = nf.path != flow_path; let schema_str = schema.and_then(|x| serde_json::to_string(&x).ok()); + let resolved_on_behalf_of = + windmill_common::resolve_on_behalf_of( + nf.on_behalf_of_email.as_deref(), + nf.on_behalf_of.as_deref(), + nf.preserve_on_behalf_of.unwrap_or(false), + &authed, + &w_id, + &db, + ) + .await?; + // Written beside the principal only while a worker that still reads it may be live. + let legacy_on_behalf_of_email = + windmill_common::legacy_on_behalf_of_email(resolved_on_behalf_of.as_deref(), &w_id, &db) + .await?; sqlx::query!( " @@ -1067,26 +1154,22 @@ async fn update_flow( tag = $4, dedicated_worker = $5, visible_to_runner_only = $6, - on_behalf_of_email = $7, - ws_error_handler_muted = $8, - value = $9, - schema = $10::text::json, - edited_by = $11, + ws_error_handler_muted = $7, + value = $8, + schema = $9::text::json, + edited_by = $10, edited_at = now(), - labels = COALESCE($14, labels) + labels = COALESCE($13, labels), + on_behalf_of = $14, + on_behalf_of_email = $15 WHERE - path = $12 AND workspace_id = $13", + path = $11 AND workspace_id = $12", if is_new_path { flow_path } else { &nf.path }, nf.summary, nf.description.as_deref().unwrap_or(""), nf.tag, nf.dedicated_worker, nf.visible_to_runner_only.unwrap_or(false), - windmill_common::resolve_on_behalf_of_email( - nf.on_behalf_of_email.as_deref(), - nf.preserve_on_behalf_of.unwrap_or(false), - &authed, - ), nf.ws_error_handler_muted.unwrap_or(false), sqlx::types::Json(&nf.value) as _, schema_str, @@ -1094,6 +1177,8 @@ async fn update_flow( flow_path, w_id, nf.labels.as_deref() as Option<&[String]>, + resolved_on_behalf_of, + legacy_on_behalf_of_email, ) .execute(&mut *tx) .await @@ -1105,8 +1190,8 @@ async fn update_flow( // if new path, must clone flow to new path and delete old flow for flow_version foreign key constraint sqlx::query!( "INSERT INTO flow - (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels) - SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels + (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels) + SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels FROM flow WHERE path = $2 AND workspace_id = $3", nf.path, @@ -1237,8 +1322,9 @@ async fn update_flow( } } + let mut moved_native_triggers = Vec::new(); if is_new_path { - windmill_common::triggers::update_triggers_script_path( + moved_native_triggers = windmill_common::triggers::update_triggers_script_path( &mut tx, &nf.path, &flow_path, &w_id, true, ) .await @@ -1280,10 +1366,10 @@ async fn update_flow( ) .await?; if let Some(on_behalf_of) = windmill_common::check_on_behalf_of_preservation( - nf.on_behalf_of_email.as_deref(), + resolved_on_behalf_of.as_deref(), nf.preserve_on_behalf_of.unwrap_or(false), &authed, - &authed.email, + &windmill_common::users::username_to_permissioned_as(&authed.username), ) { audit_log( &mut *tx, @@ -1335,6 +1421,7 @@ async fn update_flow( &authed.email, windmill_common::users::username_to_permissioned_as(&authed.username), authed.token_prefix.as_deref(), + authed.username_override.as_deref(), None, None, None, @@ -1410,6 +1497,21 @@ async fn update_flow( new_tx.commit().await?; + // See `tally_rename_vacated_path`. + if flow_path != nf.path { + if let Err(e) = windmill_git_sync::tally_rename_vacated_path( + &db, + &w_id, + DeployedObject::Flow { path: flow_path.to_string(), parent_path: None, version }, + ) + .await + { + tracing::error!(%e, "error tallying the path renamed away from"); + } + } + + reregister_moved_native_triggers(&db, &authed, &w_id, moved_native_triggers); + // Trigger CI tests for items that reference this flow { let db2 = db.clone(); @@ -1512,7 +1614,7 @@ async fn get_flow_by_path( flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, - flow.on_behalf_of_email, + flow.on_behalf_of, flow.labels, folder_labels(flow.workspace_id, flow.path) AS inherited_labels, flow_version.id AS version_id, @@ -1553,7 +1655,7 @@ async fn get_flow_by_path( flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, - flow.on_behalf_of_email, + flow.on_behalf_of, flow.labels, folder_labels(flow.workspace_id, flow.path) AS inherited_labels, flow_version.id AS version_id, @@ -1576,6 +1678,13 @@ async fn get_flow_by_path( tx.commit().await?; + // The primary GET is what the CLI reads before a preserving push, so it must carry the + // derived address: without it the push sends neither half and the identity is cleared. + let mut flow_o = flow_o; + if let Some(fws) = flow_o.as_mut() { + fws.flow.on_behalf_of_email = derived_on_behalf_of_email(&db, &w_id, &fws.flow).await?; + } + // No deployed row + `get_draft`: fall back to the draft table; see scripts.rs. let overlay = overlay_or_draft_only( &db, diff --git a/backend/windmill-api-integration-tests/tests/apps.rs b/backend/windmill-api-integration-tests/tests/apps.rs index 3d70c09ce1..c5b18c61fd 100644 --- a/backend/windmill-api-integration-tests/tests/apps.rs +++ b/backend/windmill-api-integration-tests/tests/apps.rs @@ -335,3 +335,192 @@ async fn test_public_app_by_custom_path(db: Pool) -> anyhow::Result<() Ok(()) } + +/// A raw app's kind lives on its version row, so a value deployed through the +/// low-code endpoint used to convert the app in place and strand its bundle. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_raw_app_kind_is_not_flipped_by_update(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/apps"); + let raw_path = "u/test-user/raw_app"; + let low_code_path = "u/test-user/low_code_app"; + + let raw_app_form = |path: &str| { + reqwest::multipart::Form::new() + .text( + "app", + json!({ + "path": path, + "summary": "Raw app", + "value": { "files": { "index.ts": "export {}" }, "runnables": {} }, + "policy": { "execution_mode": "publisher", "triggerables_v2": {} } + }) + .to_string(), + ) + .part( + "js", + reqwest::multipart::Part::bytes(b"console.log(1)".to_vec()), + ) + }; + + let resp = authed(client().post(format!("{base}/create_raw"))) + .multipart(raw_app_form(raw_path)) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "create_raw: {}", resp.text().await?); + + let resp = authed(client().post(format!("{base}/create"))) + .json(&new_app(low_code_path, "Low code app")) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "create: {}", resp.text().await?); + + let versions_of = |path: &'static str| async move { + let resp = authed_get(port, "get/p", path).await; + let body = resp.json::().await.unwrap(); + body["versions"].as_array().unwrap().len() + }; + let raw_versions = versions_of(raw_path).await; + + // Neither endpoint may deploy a value onto an app of the other kind. + let resp = authed(client().post(format!("{base}/update/{raw_path}"))) + .json(&json!({ "value": { "grid": [] } })) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 400); + assert!( + resp.text().await?.contains("is a raw app"), + "expected the low-code update of a raw app to be refused" + ); + // The refusal has to land before the version insert, not roll one back. + assert_eq!(versions_of(raw_path).await, raw_versions); + + let resp = authed(client().post(format!("{base}/update_raw/{low_code_path}"))) + .multipart(raw_app_form(low_code_path)) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 400); + assert!( + resp.text().await?.contains("is a low-code app"), + "expected the raw update of a low-code app to be refused" + ); + + // Nor may the source endpoints compile a value that isn't a raw app's: both + // refuse up front, without queueing a bundle job no worker would pick up here. + let resp = authed(client().post(format!("{base}/create_raw_source"))) + .json(&json!({ + "path": "u/test-user/from_source", + "summary": "", + "value": { "grid": [] }, + "policy": { "execution_mode": "publisher" } + })) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 400); + assert!( + resp.text().await?.contains("no `files` to bundle"), + "expected a low-code value to be refused before bundling" + ); + + // The source endpoint refuses the same mismatch up front — it must not queue + // a bundle job (which no worker would pick up here) to find that out. + let resp = authed(client().post(format!("{base}/update_raw_source/{low_code_path}"))) + .json(&json!({ "value": { "files": { "/index.tsx": "export {}" } } })) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 400); + assert!( + resp.text().await?.contains("is a low-code app"), + "expected the source deploy of a low-code app to be refused" + ); + + // Metadata-only updates and same-kind deploys still go through. + let resp = authed(client().post(format!("{base}/update/{raw_path}"))) + .json(&json!({ "summary": "Renamed" })) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200, "update summary: {}", resp.text().await?); + + let resp = authed(client().post(format!("{base}/update_raw/{raw_path}"))) + .multipart(raw_app_form(raw_path)) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200, "update_raw: {}", resp.text().await?); + + let resp = authed_get(port, "get/p", raw_path).await; + assert_eq!(resp.json::().await?["raw_app"], true); + + // A caller that means to convert says so, which is how an app converted by + // accident gets restored to what it was. + let resp = authed(client().post(format!("{base}/update/{raw_path}"))) + .json(&json!({ "value": { "grid": [] }, "allow_kind_change": true })) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200, "update: {}", resp.text().await?); + + let resp = authed_get(port, "get/p", raw_path).await; + assert_eq!(resp.json::().await?["raw_app"], false); + + Ok(()) +} + +/// Who may create at a path is the app table's answer, not a rule restated in the +/// handler: a non-admin member of `g/` writes there, and nowhere a +/// non-member does. Both are decided before the sources are compiled, so an empty +/// `files` is enough to say which side of that check the request reached. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_create_raw_source_write_check_follows_app_rls( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + sqlx::query( + "INSERT INTO usr_to_group (workspace_id, usr, group_) VALUES + ('test-workspace', 'test-user-2', 'all')", + ) + .execute(&db) + .await?; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/apps"); + let create_as_member = |path: &'static str| { + client() + .post(format!("{base}/create_raw_source")) + .header("Authorization", "Bearer SECRET_TOKEN_2") + .json(&json!({ + "path": path, + "summary": "", + "value": { "files": {} }, + "policy": { "execution_mode": "publisher" } + })) + .send() + }; + + let resp = create_as_member("g/all/from_source").await.unwrap(); + assert_eq!( + resp.status(), + 400, + "a group member must reach the compile: {}", + resp.text().await? + ); + + let resp = create_as_member("u/test-user/from_source").await.unwrap(); + assert_eq!( + resp.status(), + 403, + "expected another user's path to be refused" + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/audit.rs b/backend/windmill-api-integration-tests/tests/audit.rs index 31994d8165..81c3ae70be 100644 --- a/backend/windmill-api-integration-tests/tests/audit.rs +++ b/backend/windmill-api-integration-tests/tests/audit.rs @@ -33,3 +33,105 @@ async fn test_audit_endpoints(db: Pool) -> anyhow::Result<()> { Ok(()) } + +/// A run fired by a labeled token must be findable both by the token that fired it and by the +/// caller who fired it. `push` builds its own audit author from `(user, permissioned_as)` rather +/// than from the `ApiAuthed`, so the label only reaches the row through the explicit end-user +/// argument; and when the runnable declares `on_behalf_of`, the run-as identity takes `username`, +/// so the caller only stays searchable through the `created_by` parameter. +/// +/// EE-only: the OSS `audit_log` writes nothing, and a lesser plan redacts the `parameters` this +/// matches on. +#[cfg(all(feature = "enterprise", feature = "private"))] +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_job_run_is_searchable_by_token_and_by_caller( + db: Pool, +) -> anyhow::Result<()> { + use serde_json::json; + + initialize_tracing().await; + + // The recorded identity makes a run against this take `u/test-user-2` as its + // permissioned_as, so the audit `username` slot goes to it rather than to the caller. + sqlx::query( + "INSERT INTO script (workspace_id, hash, path, content, language, kind, created_by, + on_behalf_of, schema, summary, description, lock, extra_perms) + VALUES ('test-workspace', 900101, 'u/test-user-2/onbehalf', 'export function main() {}', + 'deno', 'script', 'test-user-2', 'u/test-user-2', '{}', '', '', '', '{\"g/all\": true}')", + ) + .execute(&db) + .await?; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let resp = authed(client().post(format!("http://localhost:{port}/api/users/tokens/create"))) + .json(&json!({ "label": "audit-probe" })) + .send() + .await?; + assert_eq!(resp.status(), 201); + let token = resp.text().await?; + let bearer = |b: reqwest::RequestBuilder| b.header("Authorization", format!("Bearer {token}")); + + let resp = bearer(client().post(format!( + "http://localhost:{port}/api/w/test-workspace/jobs/run/preview" + ))) + .json(&json!({ + "content": "export function main() { return 1; }", + "language": "deno", + "args": {} + })) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /jobs/run/preview", + ); + + let resp = bearer(client().post(format!( + "http://localhost:{port}/api/w/test-workspace/jobs/run/p/u/test-user-2/onbehalf" + ))) + .json(&json!({})) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /jobs/run/p/u/test-user-2/onbehalf", + ); + + let search = |q: &str| { + let url = format!("http://localhost:{port}/api/w/test-workspace/audit/list?username={q}"); + async move { + let resp = authed(client().get(url)).send().await.unwrap(); + assert_eq!(resp.status(), 200); + resp.json::>().await.unwrap() + } + }; + + let by_token = search("label-audit-probe").await; + assert!( + by_token + .iter() + .any(|l| l["operation"] == "jobs.run.preview"), + "the run must be searchable by token label, got {by_token:?}" + ); + assert!( + by_token + .iter() + .any(|l| l["operation"] == "jobs.run.script" && l["username"] == "test-user-2"), + "the on-behalf run must be searchable by token label, got {by_token:?}" + ); + + // `username` is `test-user-2` on that row, so this can only match through `created_by`. + let by_caller = search("test-user").await; + assert!( + by_caller + .iter() + .any(|l| l["operation"] == "jobs.run.script" && l["username"] == "test-user-2"), + "the on-behalf run must stay searchable by the caller who fired it, got {by_caller:?}" + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/fixtures/nested_dev_workspace.sql b/backend/windmill-api-integration-tests/tests/fixtures/nested_dev_workspace.sql new file mode 100644 index 0000000000..76f266bb0f --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/fixtures/nested_dev_workspace.sql @@ -0,0 +1,76 @@ +-- Seven families for the nested dev-workspace (dev of a dev) guards. +-- +-- Family A is rooted at `test-workspace` (base fixture) and is the one a nested dev is attached to: +-- test-workspace -> tw-dev ('dev') +-- plus two standalone attach candidates, one of which already owns a 'dev'-labelled dev workspace. +-- +-- Family B carries a `wm-fork-` workspace re-designated as a dev workspace, which is the shape that +-- returns to being a throwaway fork on detach: +-- prod-b -> wm-fork-redev ('dev') -> redev-dev ('staging') +-- +-- Family C is the ordinary prefix-less nesting. Detaching its middle workspace is fine (it returns +-- to standalone and goes on hosting `c-dev-dev`), but archiving it is not: +-- prod-c -> c-dev ('dev') -> c-dev-dev ('staging') +-- +-- Family E has no nested dev yet, so both "give `wm-fork-edev` a dev" and "detach `wm-fork-edev`" +-- pass their own checks — the pair that must not both commit: +-- prod-e -> wm-fork-edev ('dev'), plus the standalone candidate `e-cand` +-- +-- Family F is three standalone workspaces, so "attach f-mid under prod-f" and "attach f-leaf under +-- f-mid" both pass on their own — adjacent attaches whose labels only collide once both land: +-- prod-f, f-mid, f-leaf +-- +-- Family G already nests, so two attaches at opposite ends of it touch no workspace in common — +-- their labels only collide once both land, three dev workspaces deep: +-- prod-g (root), g-mid -> g-sub ('dev'), and the standalone `g-leaf` +-- +-- Family H is a standalone that already owns a dev: archiving it resolves as "no pairing involved" +-- while still being an operation the pairing lock has to cover: +-- h-cand -> h-sub ('dev') + + +INSERT INTO workspace (id, name, owner, parent_workspace_id, is_dev_workspace, dev_workspace_label) VALUES + ('tw-dev', 'dev of test-workspace', 'test@windmill.dev', 'test-workspace', true, 'dev'), + ('standalone', 'standalone', 'test@windmill.dev', NULL, false, NULL), + ('standalone-dev', 'dev of standalone', 'test@windmill.dev', 'standalone', true, 'dev'), + ('spare', 'spare standalone', 'test@windmill.dev', NULL, false, NULL), + ('prod-b', 'prod b', 'test@windmill.dev', NULL, false, NULL), + ('wm-fork-redev', 'redesignated fork', 'test@windmill.dev', 'prod-b', true, 'dev'), + ('redev-dev', 'dev of the redesignated fork', 'test@windmill.dev', 'wm-fork-redev', true, 'staging'), + ('prod-c', 'prod c', 'test@windmill.dev', NULL, false, NULL), + ('c-dev', 'dev of prod-c', 'test@windmill.dev', 'prod-c', true, 'dev'), + ('c-dev-dev', 'dev of c-dev', 'test@windmill.dev', 'c-dev', true, 'staging'), + ('prod-e', 'prod e', 'test@windmill.dev', NULL, false, NULL), + ('wm-fork-edev', 'redesignated fork with no dev yet', 'test@windmill.dev', 'prod-e', true, 'dev'), + ('e-cand', 'attach candidate', 'test@windmill.dev', NULL, false, NULL), + ('prod-f', 'prod f', 'test@windmill.dev', NULL, false, NULL), + ('f-mid', 'middle attach candidate', 'test@windmill.dev', NULL, false, NULL), + ('f-leaf', 'leaf attach candidate', 'test@windmill.dev', NULL, false, NULL), + ('prod-g', 'prod g', 'test@windmill.dev', NULL, false, NULL), + ('g-mid', 'standalone with a dev of its own', 'test@windmill.dev', NULL, false, NULL), + ('g-sub', 'dev of g-mid', 'test@windmill.dev', 'g-mid', true, 'dev'), + ('g-leaf', 'leaf attach candidate', 'test@windmill.dev', NULL, false, NULL), + ('h-cand', 'standalone owning a dev', 'test@windmill.dev', NULL, false, NULL), + ('h-sub', 'dev of h-cand', 'test@windmill.dev', 'h-cand', true, 'dev'); + +CREATE TEMP VIEW new_workspaces AS SELECT unnest(ARRAY[ + 'tw-dev', 'standalone', 'standalone-dev', 'spare', 'prod-b', 'wm-fork-redev', 'redev-dev', + 'prod-c', 'c-dev', 'c-dev-dev', 'prod-e', 'wm-fork-edev', 'e-cand', + 'prod-f', 'f-mid', 'f-leaf', 'prod-g', 'g-mid', 'g-sub', 'g-leaf', + 'h-cand', 'h-sub' +]) AS id; + +INSERT INTO workspace_settings (workspace_id) + SELECT id FROM new_workspaces; + +INSERT INTO workspace_key (workspace_id, kind, key) + SELECT id, 'cloud', 'test-key' FROM new_workspaces; + +INSERT INTO group_ (workspace_id, name, summary, extra_perms) + SELECT id, 'all', 'All users', '{}' FROM new_workspaces; + +INSERT INTO usr (workspace_id, email, username, is_admin, role) + SELECT id, 'test@windmill.dev', 'test-user', true, 'Admin' + FROM new_workspaces; + +DROP VIEW new_workspaces; diff --git a/backend/windmill-api-integration-tests/tests/fork_clone_on_behalf_of.rs b/backend/windmill-api-integration-tests/tests/fork_clone_on_behalf_of.rs new file mode 100644 index 0000000000..93c9f5adaf --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/fork_clone_on_behalf_of.rs @@ -0,0 +1,118 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; + +use windmill_test_utils::*; + +/// A principal only means something in the workspace whose `usr`/`group_` rows define it, and a +/// fork copies the creator and the groups but not the rest of the membership. Carrying one over +/// blindly would leave a runnable naming somebody who cannot authenticate there; dropping them +/// all would silently hand every configured runnable in the fork to whoever runs it. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_fork_keeps_only_resolvable_on_behalf_of(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base_url = format!("http://localhost:{port}/api"); + + // A superadmin with no `usr` row anywhere: they authenticate from `password` alone, so + // their principal resolves in the fork as much as it did in the parent. + sqlx::query!( + "INSERT INTO password(email, password_hash, login_type, super_admin, verified, name, username) + VALUES ('sa@windmill.dev', 'not-a-real-hash', 'password', true, true, 'Ext', 'ext-sa')" + ) + .execute(&db) + .await?; + + sqlx::query!( + "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of, on_behalf_of_email) + VALUES + ('test-workspace', 'u/test-user/obo_member', 91001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user', 'test@windmill.dev'), + ('test-workspace', 'u/test-user/obo_stranger', 91002, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2', 'test2@windmill.dev'), + ('test-workspace', 'u/test-user/obo_group', 91003, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'g/all', 'group-all@windmill.dev'), + ('test-workspace', 'u/test-user/obo_superadmin', 91004, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/ext-sa', 'sa@windmill.dev'), + ('test-workspace', 'u/test-user/obo_address_only', 91005, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), NULL, 'test2@windmill.dev')" + ) + .execute(&db) + .await?; + + sqlx::query!( + "INSERT INTO flow (workspace_id, path, summary, description, value, edited_by, edited_at, on_behalf_of, on_behalf_of_email) + VALUES ('test-workspace', 'u/test-user/obo_flow', '', '', $1, 'test-user', NOW(), 'u/test-user-2', 'test2@windmill.dev')", + json!({"modules": []}) + ) + .execute(&db) + .await?; + + let resp = reqwest::Client::new() + .post(format!( + "{base_url}/w/test-workspace/workspaces/create_fork" + )) + .header("Authorization", "Bearer SECRET_TOKEN") + .json(&json!({ "id": "wm-fork-obo", "name": "Fork", "color": "#0000ff" })) + .send() + .await?; + assert!( + resp.status().is_success(), + "creating the fork: {}", + resp.text().await? + ); + + let cloned = sqlx::query!( + "SELECT path, on_behalf_of, on_behalf_of_email FROM script WHERE workspace_id = 'wm-fork-obo' ORDER BY path" + ) + .fetch_all(&db) + .await?; + let identity = |path: &str| { + cloned + .iter() + .find(|r| r.path == path) + .unwrap_or_else(|| panic!("{path} was cloned")) + .on_behalf_of + .clone() + }; + + // The creator is the one member the fork always gets, so their principal still resolves. + assert_eq!( + identity("u/test-user/obo_member").as_deref(), + Some("u/test-user") + ); + // Groups are cloned wholesale, so a group principal resolves too. + assert_eq!(identity("u/test-user/obo_group").as_deref(), Some("g/all")); + assert_eq!( + identity("u/test-user/obo_superadmin").as_deref(), + Some("u/ext-sa") + ); + // `test-user-2` is not carried into the fork, so nothing there can run as them — and the + // address has to go with the principal, or a worker that reads only the address still would. + assert_eq!(identity("u/test-user/obo_stranger"), None); + // A row a server predating this release wrote carries the address alone; the clone must not + // mistake it for one it orphaned, because that address is all a later re-derivation has. + assert_eq!( + cloned + .iter() + .find(|r| r.path == "u/test-user/obo_address_only") + .and_then(|r| r.on_behalf_of_email.as_deref()), + Some("test2@windmill.dev"), + ); + + let orphaned = cloned + .iter() + .filter(|r| { + r.on_behalf_of.is_none() + && r.on_behalf_of_email.is_some() + && r.path != "u/test-user/obo_address_only" + }) + .count(); + assert_eq!(orphaned, 0, "a dropped principal leaves no address behind"); + assert_eq!( + sqlx::query_scalar!( + "SELECT on_behalf_of FROM flow WHERE workspace_id = 'wm-fork-obo'" + ) + .fetch_one(&db) + .await?, + None + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/mcp_protocol_versions.rs b/backend/windmill-api-integration-tests/tests/mcp_protocol_versions.rs new file mode 100644 index 0000000000..e684e3ee14 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/mcp_protocol_versions.rs @@ -0,0 +1,221 @@ +//! Protocol-version negotiation for the MCP endpoint. +//! +//! The endpoint is dual-era: legacy revisions keep the `initialize` handshake, +//! while `2026-07-28` carries its version as per-request metadata and is served +//! statelessly. Both are answered on the same URL, so a bump of the rmcp SDK +//! must not silently drop either side. +#![cfg(feature = "mcp")] + +use serde_json::{json, Value}; +use sqlx::{Pool, Postgres}; + +use windmill_test_utils::*; + +/// Every revision the server advertises, oldest first. +const SUPPORTED: [&str; 5] = [ + "2024-11-05", + "2025-03-26", + "2025-06-18", + "2025-11-25", + "2026-07-28", +]; + +const MODERN: &str = "2026-07-28"; + +async fn insert_mcp_token(db: &Pool) -> anyhow::Result<()> { + sqlx::query( + "INSERT INTO token (token_hash, token_prefix, token, email, label, super_admin, scopes) + VALUES (encode(sha256('MCP_TOKEN'::bytea), 'hex'), 'MCP_TOK', 'MCP_TOKEN', 'test@windmill.dev', 'mcp token', true, ARRAY['mcp:all'])", + ) + .execute(db) + .await?; + Ok(()) +} + +/// POST one JSON-RPC message and return the HTTP status plus the decoded body. +/// The endpoint answers either `application/json` or a single-event SSE stream, +/// so strip the `data: ` framing before parsing. +async fn post( + port: u16, + headers: &[(&str, &str)], + body: Value, +) -> anyhow::Result<(reqwest::StatusCode, Value)> { + let mut req = reqwest::Client::new() + .post(format!( + "http://localhost:{port}/api/mcp/w/test-workspace/mcp" + )) + .header("Authorization", "Bearer MCP_TOKEN") + .header("Accept", "application/json, text/event-stream") + .json(&body); + for (k, v) in headers { + req = req.header(*k, *v); + } + let resp = req.send().await?; + let status = resp.status(); + let text = resp.text().await?; + let payload = text + .lines() + .find_map(|l| l.strip_prefix("data: ")) + .unwrap_or(text.trim()); + let parsed = serde_json::from_str(payload) + .map_err(|e| anyhow::anyhow!("status {status}, unparseable body {text:?}: {e}"))?; + Ok((status, parsed)) +} + +fn modern_meta() -> Value { + json!({ + "io.modelcontextprotocol/protocolVersion": MODERN, + "io.modelcontextprotocol/clientInfo": { "name": "test-client", "version": "0.0.1" }, + "io.modelcontextprotocol/clientCapabilities": {}, + }) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_mcp_legacy_initialize_negotiates_requested_version( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + set_jwt_secret().await; + insert_mcp_token(&db).await?; + let server = ApiServer::start_mcp(db.clone()).await?; + let port = server.addr.port(); + + // A legacy client must be answered with the revision it asked for, not with + // whatever the SDK happens to call `LATEST`. + for version in SUPPORTED.iter().filter(|v| **v != MODERN) { + let (status, body) = post( + port, + &[("MCP-Protocol-Version", version)], + json!({ + "jsonrpc": "2.0", "id": 1, "method": "initialize", + "params": { + "protocolVersion": version, + "capabilities": {}, + "clientInfo": { "name": "test-client", "version": "0.0.1" }, + } + }), + ) + .await?; + + assert_eq!(status, 200, "initialize {version} failed: {body}"); + assert_eq!( + body["result"]["protocolVersion"], *version, + "initialize {version} negotiated the wrong revision: {body}" + ); + // `Implementation::from_build_env()` expands its `env!` inside rmcp, so + // the obvious constructor makes the server introduce itself as the SDK. + assert_eq!( + body["result"]["serverInfo"]["name"], "windmill", + "server must identify itself, not the SDK: {body}" + ); + } + + Ok(()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_mcp_modern_requests_are_served_without_initialize( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + set_jwt_secret().await; + insert_mcp_token(&db).await?; + let server = ApiServer::start_mcp(db.clone()).await?; + let port = server.addr.port(); + + // `server/discover` is the modern replacement for the handshake: it must + // exist and advertise exactly the revisions the server implements. + let (status, body) = post( + port, + &[ + ("MCP-Protocol-Version", MODERN), + ("Mcp-Method", "server/discover"), + ], + json!({ + "jsonrpc": "2.0", "id": 1, "method": "server/discover", + "params": { "_meta": modern_meta() } + }), + ) + .await?; + assert_eq!(status, 200, "server/discover failed: {body}"); + assert_eq!( + body["result"]["supportedVersions"], + json!(SUPPORTED), + "server/discover advertised the wrong revisions: {body}" + ); + + // A modern call carries its version in `_meta` and needs no prior session. + let (status, body) = post( + port, + &[ + ("MCP-Protocol-Version", MODERN), + ("Mcp-Method", "tools/list"), + ], + json!({ + "jsonrpc": "2.0", "id": 2, "method": "tools/list", + "params": { "_meta": modern_meta() } + }), + ) + .await?; + assert_eq!(status, 200, "modern tools/list failed: {body}"); + assert!( + body["result"]["tools"] + .as_array() + .is_some_and(|t| !t.is_empty()), + "modern tools/list returned no tools: {body}" + ); + + // SEP-2549 cache hints are required at 2026-07-28 and rmcp omits them unless + // set, which makes strict clients (e.g. the Python SDK) reject the whole + // response rather than degrade. + assert!( + body["result"]["ttlMs"].is_number(), + "modern tools/list is missing ttlMs: {body}" + ); + assert_eq!( + body["result"]["cacheScope"], "private", + "tools/list must not be cached across callers: {body}" + ); + + Ok(()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_mcp_unsupported_version_lists_supported_ones( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + set_jwt_secret().await; + insert_mcp_token(&db).await?; + let server = ApiServer::start_mcp(db.clone()).await?; + let port = server.addr.port(); + + // The client's only way forward is the `supported` list, so an unknown + // version must fail with it rather than with a generic error. + let (status, body) = post( + port, + &[ + ("MCP-Protocol-Version", "1900-01-01"), + ("Mcp-Method", "tools/list"), + ], + json!({ + "jsonrpc": "2.0", "id": 1, "method": "tools/list", + "params": { "_meta": { + "io.modelcontextprotocol/protocolVersion": "1900-01-01", + "io.modelcontextprotocol/clientInfo": { "name": "test-client", "version": "0.0.1" }, + "io.modelcontextprotocol/clientCapabilities": {}, + }} + }), + ) + .await?; + + assert_eq!(status, 400, "expected 400 for unknown version: {body}"); + assert_eq!(body["error"]["code"], -32022, "wrong error code: {body}"); + assert_eq!( + body["error"]["data"]["supported"], + json!(SUPPORTED), + "error did not advertise the supported revisions: {body}" + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/native_triggers.rs b/backend/windmill-api-integration-tests/tests/native_triggers.rs index 1d809ae526..c86ba3154f 100644 --- a/backend/windmill-api-integration-tests/tests/native_triggers.rs +++ b/backend/windmill-api-integration-tests/tests/native_triggers.rs @@ -11,13 +11,21 @@ use serde_json::json; use sqlx::{Pool, Postgres}; +use axum::http::StatusCode; use windmill_api_auth::ApiAuthed; -use windmill_common::variables::{build_crypt, encrypt}; +use windmill_common::{ + error::Error, + variables::{build_crypt, encrypt}, +}; use windmill_native_triggers::{ - decrypt_oauth_data, delete_native_trigger, delete_workspace_integration, - get_workspace_integration, + classify_read_failure, decrypt_oauth_data, delete_native_trigger, + delete_workspace_integration, get_workspace_integration, + github::GitHub, google::{parse_stop_channel_params, should_renew_channel}, - require_native_integration_use, store_native_trigger, store_workspace_integration, + http_error_status, list_native_triggers, map_external_error, + nextcloud::NextCloud, + grant_refused, require_native_integration_use, store_native_trigger, + store_workspace_integration, External, ExternalReadFailure, HttpRequestError, NativeTriggerConfig, OAuthConfig, ServiceName, }; @@ -50,6 +58,8 @@ fn test_authed() -> ApiAuthed { folders: vec![], scopes: None, username_override: None, + username_override_is_token_label: false, + is_session_token: false, token_prefix: None, read_only: false, } @@ -569,6 +579,92 @@ async fn test_cleanup_preserves_triggers(db: Pool) -> anyhow::Result<( Ok(()) } +// ============================================================================ +// 5. Runnable rename +// ============================================================================ + +/// A rename has to carry the trigger row onto the new path and report it as moved: listings only +/// return rows whose runnable still exists, so one left behind on the old path disappears from the +/// UI for good, and one not reported keeps a webhook aimed at the old path. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_rename_moves_native_trigger(db: Pool) -> anyhow::Result<()> { + insert_test_script(&db, "f/test/before").await?; + store_native_trigger( + &db, + "test-workspace", + ServiceName::Nextcloud, + "ext-1", + &NativeTriggerConfig { + script_path: "f/test/before".to_string(), + is_flow: false, + webhook_token: "abcdefghij1234567890".to_string(), + }, + json!({"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent"}), + None, + ) + .await?; + // An unrelated trigger already sitting on the target path must not be reported as moved. + insert_test_script(&db, "f/test/after").await?; + store_native_trigger( + &db, + "test-workspace", + ServiceName::Nextcloud, + "ext-2", + &NativeTriggerConfig { + script_path: "f/test/after".to_string(), + is_flow: false, + webhook_token: "0987654321jihgfedcba".to_string(), + }, + json!({"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent"}), + None, + ) + .await?; + + let mut tx = db.begin().await?; + sqlx::query!( + "UPDATE script SET path = $1 WHERE workspace_id = 'test-workspace' AND path = $2", + "f/test/after", + "f/test/before", + ) + .execute(&mut *tx) + .await?; + let moved = windmill_common::triggers::update_triggers_script_path( + &mut tx, + "f/test/after", + "f/test/before", + "test-workspace", + false, + ) + .await?; + tx.commit().await?; + + assert_eq!( + moved + .iter() + .map(|t| (t.service_name.as_str(), t.external_id.as_str())) + .collect::>(), + vec![("nextcloud", "ext-1")] + ); + + let triggers = list_native_triggers( + &db, + "test-workspace", + ServiceName::Nextcloud, + None, + None, + Some("f/test/after"), + Some(false), + ) + .await?; + assert_eq!( + triggers.len(), + 2, + "the moved trigger should be listed under the new path" + ); + + Ok(()) +} + // --- parse_stop_channel_params --- #[test] @@ -622,3 +718,133 @@ fn test_parse_stop_channel_params_missing_resource_id() { assert!(channel_id.is_none()); assert_eq!(resource_id, ""); } + +// --- provider error reporting --- + +fn nextcloud_error(status: StatusCode, body: &str) -> Error { + NextCloud.external_api_error(HttpRequestError::ApiError { status, body: body.to_string() }) +} + +/// A rejection has to reach the user as the service's own sentence plus what to do about it, +/// never as an internal error carrying the raw envelope. +#[test] +fn test_provider_rejection_is_readable_and_not_internal() { + let err = nextcloud_error( + StatusCode::FORBIDDEN, + r#"{"ocs":{"meta":{"status":"failure","statuscode":403,"message":"Logged in account must be an admin, a sub admin or gotten special right to access this setting"},"data":[]}}"#, + ); + + let message = map_external_error(err).to_string(); + assert!( + message.contains("Logged in account must be an admin"), + "message={message}" + ); + assert!( + !message.contains("\"ocs\""), + "the envelope should not reach the user: {message}" + ); + assert!( + message.contains("Workspace settings > Integrations"), + "the hint should say what to do: {message}" + ); +} + +/// Both the "trigger is gone on the service" path and the delete that tolerates an +/// already-removed webhook branch on this status. +#[test] +fn test_provider_404_is_recognized() { + let err = nextcloud_error(StatusCode::NOT_FOUND, "{}"); + assert_eq!(http_error_status(&err), Some(StatusCode::NOT_FOUND)); + assert!( + matches!(map_external_error(err), Error::NotFound(_)), + "a missing external trigger must map to NotFound" + ); + assert!(matches!( + classify_read_failure(nextcloud_error(StatusCode::NOT_FOUND, "{}")), + ExternalReadFailure::Missing + )); +} + +/// Sending a user to reconnect their integration is only right when the token endpoint refused +/// the grant; a busy or broken endpoint has them fix credentials that are fine. +#[test] +fn test_refresh_failures_blame_only_the_grant_they_refuse() { + let refused = |code: u16| grant_refused(Some(StatusCode::from_u16(code).unwrap()), ""); + for code in [400, 401, 403] { + assert!(refused(code), "{code} refuses the grant"); + } + for code in [404, 408, 429, 500, 503] { + assert!(!refused(code), "{code} says nothing about the grant"); + } + assert!(!grant_refused(None, "")); + + // GitHub answers `bad_refresh_token` with HTTP 200, so the body is the only tell. + let ok = Some(StatusCode::OK); + assert!(grant_refused(ok, r#"{"error":"bad_refresh_token"}"#)); + assert!(grant_refused(ok, r#"{"error":"invalid_grant"}"#)); + assert!(!grant_refused(ok, r#"{"access_token":"t","token_type":"bearer"}"#)); +} + +/// A service that is busy or broken has not refused anything, and callers react differently to +/// the two. GitHub and Google spend a 403 on throttling, where advice about permissions sends +/// the reader after a problem they do not have. +#[test] +fn test_transient_service_failures_are_not_refusals() { + for transient in [408, 429, 503] { + let err = nextcloud_error(StatusCode::from_u16(transient).unwrap(), "{}"); + assert!( + matches!(map_external_error(err), Error::BadGateway(_)), + "{transient} should read as the service failing to serve, not refusing" + ); + } + + // GitHub words its throttle two ways, and neither is a permission problem. + for wording in [ + "API rate limit exceeded for user ID 1.", + "You have exceeded a secondary rate limit.", + "You have triggered an abuse detection mechanism.", + ] { + let throttled = GitHub.external_api_error(HttpRequestError::ApiError { + status: StatusCode::FORBIDDEN, + body: format!(r#"{{"message":"{wording}"}}"#), + }); + let throttled = map_external_error(throttled); + assert!( + matches!(throttled, Error::BadGateway(_)), + "a throttled 403 is the service failing to serve: {throttled:?}" + ); + assert!( + !throttled.to_string().contains("admin rights"), + "a throttled 403 must not advise about permissions: {throttled}" + ); + } + + let refused = GitHub.external_api_error(HttpRequestError::ApiError { + status: StatusCode::FORBIDDEN, + body: r#"{"message":"Must have admin rights to Repository."}"#.to_string(), + }); + assert!( + map_external_error(refused).to_string().contains("admin rights"), + "a real 403 keeps its guidance" + ); +} + +/// A service read degrades to the stored configuration, but only for the service's own +/// failures: `External::get` also runs queries, and reporting one of those as the service's +/// word would hide a Windmill outage behind a 200. +#[test] +fn test_only_service_failures_degrade_the_read() { + assert!(matches!( + classify_read_failure(nextcloud_error(StatusCode::FORBIDDEN, "{}")), + ExternalReadFailure::Unreadable(_) + )); + assert!(matches!( + classify_read_failure(Error::internal_err("connection pool timed out")), + ExternalReadFailure::Internal(_) + )); + let internal = Error::internal_err("connection pool timed out"); + assert!( + matches!(map_external_error(internal), Error::InternalErrLoc { .. }), + "a non-provider error must pass through unmapped" + ); +} diff --git a/backend/windmill-api-integration-tests/tests/nested_dev_workspace.rs b/backend/windmill-api-integration-tests/tests/nested_dev_workspace.rs new file mode 100644 index 0000000000..a9549b91a8 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/nested_dev_workspace.rs @@ -0,0 +1,444 @@ +//! A dev workspace may itself be paired with one (a "dev of a dev"). The guards that keep that +//! shape well-formed are what these tests pin: a chain must stay acyclic, and no two dev workspaces +//! in it may carry the same environment label — they inherit the same git-sync repositories, so an +//! equal label means both deploy to one branch. + +use serde_json::json; +use sqlx::{Pool, Postgres}; + +use windmill_test_utils::*; + +const ADMIN_TOKEN: &str = "SECRET_TOKEN"; + +async fn attach(port: u16, prod: &str, body: serde_json::Value) -> (reqwest::StatusCode, String) { + let resp = reqwest::Client::new() + .post(format!( + "http://localhost:{port}/api/w/{prod}/workspaces/attach_dev_workspace" + )) + .header("Authorization", format!("Bearer {ADMIN_TOKEN}")) + .json(&body) + .send() + .await + .unwrap(); + let status = resp.status(); + (status, resp.text().await.unwrap()) +} + +async fn detach(port: u16, prod: &str, dev: &str) -> (reqwest::StatusCode, String) { + let resp = reqwest::Client::new() + .post(format!( + "http://localhost:{port}/api/w/{prod}/workspaces/detach_dev_workspace" + )) + .header("Authorization", format!("Bearer {ADMIN_TOKEN}")) + .json(&json!({ "dev_workspace_id": dev })) + .send() + .await + .unwrap(); + let status = resp.status(); + (status, resp.text().await.unwrap()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base", "nested_dev_workspace"))] +async fn test_nested_dev_workspace_attach_guards(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // The family root is an ancestor of the dev workspace: reparenting it below would close a + // parent<->child cycle and hang every hierarchy walk. + let (status, body) = attach( + port, + "tw-dev", + json!({ "dev_workspace_id": "test-workspace", "dev_workspace_label": "staging" }), + ) + .await; + assert!( + status.is_client_error(), + "cycle attach returned {status}: {body}" + ); + assert!(body.contains("ancestor"), "unexpected error: {body}"); + + // `tw-dev` is itself the 'dev' workspace of the root, so a dev nested under it cannot be one too. + let (status, body) = attach( + port, + "tw-dev", + json!({ "dev_workspace_id": "spare", "dev_workspace_label": "dev" }), + ) + .await; + assert!( + status.is_client_error(), + "label reuse returned {status}: {body}" + ); + assert!(body.contains("tw-dev"), "unexpected error: {body}"); + + // `standalone` brings its own 'dev'-labelled dev workspace into the chain, which collides with + // `tw-dev` whatever label the candidate itself is given. + let (status, body) = attach( + port, + "tw-dev", + json!({ "dev_workspace_id": "standalone", "dev_workspace_label": "staging" }), + ) + .await; + assert!( + status.is_client_error(), + "subtree label reuse returned {status}: {body}" + ); + assert!(body.contains("standalone-dev"), "unexpected error: {body}"); + + // With a free label and nothing conflicting underneath, the nested pairing goes through. + let (status, body) = attach( + port, + "tw-dev", + json!({ "dev_workspace_id": "spare", "dev_workspace_label": "staging" }), + ) + .await; + assert!( + status.is_success(), + "nested attach returned {status}: {body}" + ); + // Runtime-checked (not `query!`): a macro here would need its own `.sqlx` entry, which + // `cargo sqlx prepare --workspace` does not produce for test targets. + let (parent, is_dev, label): (Option, bool, Option) = sqlx::query_as( + "SELECT parent_workspace_id, is_dev_workspace, dev_workspace_label FROM workspace WHERE id = 'spare'", + ) + .fetch_one(&db) + .await?; + assert_eq!(parent.as_deref(), Some("tw-dev")); + assert!(is_dev); + assert_eq!(label.as_deref(), Some("staging")); + + // A chain runs as deep as there are distinct labels to give it, not two: + // `test-workspace` -> `tw-dev` ('dev') -> `spare` ('staging') -> `e-cand` ('uat'). + let (status, body) = attach( + port, + "spare", + json!({ "dev_workspace_id": "e-cand", "dev_workspace_label": "uat" }), + ) + .await; + assert!( + status.is_success(), + "third-label attach returned {status}: {body}" + ); + + Ok(()) +} + +async fn archive(port: u16, w_id: &str) -> (reqwest::StatusCode, String) { + let resp = reqwest::Client::new() + .post(format!( + "http://localhost:{port}/api/w/{w_id}/workspaces/archive" + )) + .header("Authorization", format!("Bearer {ADMIN_TOKEN}")) + .send() + .await + .unwrap(); + let status = resp.status(); + (status, resp.text().await.unwrap()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base", "nested_dev_workspace"))] +async fn test_teardown_refuses_to_strand_a_nested_dev(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // A `wm-fork-` workspace keeps its parent when it stops being a dev workspace, so it returns to + // being a throwaway fork — which hosts no pairing, leaving `redev-dev` attached with no way to + // reach it. Detach and archive both clear the flag, so both have to refuse. + let (status, body) = detach(port, "prod-b", "wm-fork-redev").await; + assert!( + status.is_client_error(), + "stranding detach returned {status}: {body}" + ); + assert!(body.contains("redev-dev"), "unexpected error: {body}"); + + let (status, body) = archive(port, "wm-fork-redev").await; + assert!( + status.is_client_error(), + "stranding archive returned {status}: {body}" + ); + assert!(body.contains("redev-dev"), "unexpected error: {body}"); + + // Archive soft-deletes whatever the id looks like, so a prefix-less dev workspace strands its + // own dev too — even though detaching that same workspace is fine (it returns to standalone). + let (status, body) = archive(port, "c-dev").await; + assert!( + status.is_client_error(), + "prefix-less stranding archive returned {status}: {body}" + ); + assert!(body.contains("c-dev-dev"), "unexpected error: {body}"); + let (status, body) = detach(port, "prod-c", "c-dev").await; + assert!( + status.is_success(), + "prefix-less detach returned {status}: {body}" + ); + + // Bottom-up is the supported order. + let (status, body) = detach(port, "wm-fork-redev", "redev-dev").await; + assert!(status.is_success(), "leaf detach returned {status}: {body}"); + let (status, body) = detach(port, "prod-b", "wm-fork-redev").await; + assert!( + status.is_success(), + "detach after cleanup returned {status}: {body}" + ); + + Ok(()) +} + +/// Giving a workspace a dev and clearing its own dev flag each decide on state the other mutates, so +/// checked outside a common lock both commit and leave `e-cand` under a throwaway fork. Fired +/// together: whichever lands second must see the first and be rejected. +/// +/// Repeated, because how far each handler gets before the other's mutation lands is timing-dependent +/// — one pass caught an unlocked build only about a fifth of the time, and the runs are cheap. +#[sqlx::test(migrations = "../migrations", fixtures("base", "nested_dev_workspace"))] +async fn test_nested_attach_and_detach_cannot_both_commit( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + for round in 0..12 { + // Back to `prod-e -> wm-fork-edev ('dev')` with `e-cand` standalone, the state in which both + // requests pass their own checks. + sqlx::query( + "UPDATE workspace SET parent_workspace_id = 'prod-e', is_dev_workspace = true, + dev_workspace_label = 'dev' WHERE id = 'wm-fork-edev'", + ) + .execute(&db) + .await?; + sqlx::query( + "UPDATE workspace SET parent_workspace_id = NULL, is_dev_workspace = false, + dev_workspace_label = NULL WHERE id = 'e-cand'", + ) + .execute(&db) + .await?; + + let (attached, detached) = tokio::join!( + attach( + port, + "wm-fork-edev", + json!({ "dev_workspace_id": "e-cand", "dev_workspace_label": "staging" }), + ), + detach(port, "prod-e", "wm-fork-edev"), + ); + assert!( + attached.0.is_success() != detached.0.is_success(), + "round {round}: exactly one must win, got attach={} detach={}\n{}\n{}", + attached.0, + detached.0, + attached.1, + detached.1 + ); + + // Whichever won, `wm-fork-edev` is never left a throwaway fork with a dev workspace beneath it. + let (is_dev, cand_parent): (bool, Option) = sqlx::query_as( + "SELECT (SELECT is_dev_workspace FROM workspace WHERE id = 'wm-fork-edev'), + (SELECT parent_workspace_id FROM workspace WHERE id = 'e-cand')", + ) + .fetch_one(&db) + .await?; + assert!( + is_dev || cand_parent.is_none(), + "round {round}: stranded — wm-fork-edev is_dev={is_dev}, e-cand parent={cand_parent:?}" + ); + } + Ok(()) +} + +/// Two adjacent attaches — `f-mid` under `prod-f` and `f-leaf` under `f-mid` — each see a chain that +/// does not yet contain the other's dev workspace, so both pass their label check. Committing both +/// puts two `dev` workspaces in one chain, deploying to the same branch. Repeated for the same +/// reason as the detach race above. +#[sqlx::test(migrations = "../migrations", fixtures("base", "nested_dev_workspace"))] +async fn test_adjacent_attaches_cannot_both_claim_a_label( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + for round in 0..12 { + sqlx::query( + "UPDATE workspace SET parent_workspace_id = NULL, is_dev_workspace = false, + dev_workspace_label = NULL WHERE id IN ('f-mid', 'f-leaf')", + ) + .execute(&db) + .await?; + + let (upper, lower) = tokio::join!( + attach( + port, + "prod-f", + json!({ "dev_workspace_id": "f-mid", "dev_workspace_label": "dev" }), + ), + attach( + port, + "f-mid", + json!({ "dev_workspace_id": "f-leaf", "dev_workspace_label": "dev" }), + ), + ); + assert!( + upper.0.is_success() != lower.0.is_success(), + "round {round}: exactly one must win, got upper={} lower={}\n{}\n{}", + upper.0, + lower.0, + upper.1, + lower.1 + ); + + // Never both: that is the chain prod-f -> f-mid('dev') -> f-leaf('dev'). + let chained: bool = sqlx::query_scalar( + "SELECT EXISTS( + SELECT 1 FROM workspace mid + JOIN workspace leaf ON leaf.parent_workspace_id = mid.id + WHERE mid.id = 'f-mid' AND leaf.id = 'f-leaf' + AND mid.is_dev_workspace AND leaf.is_dev_workspace + AND mid.parent_workspace_id = 'prod-f' + )", + ) + .fetch_one(&db) + .await?; + assert!(!chained, "round {round}: both attaches committed"); + } + Ok(()) +} + +/// Attaching `g-mid` under `prod-g` and attaching `g-leaf` under `g-sub` touch no workspace in +/// common — `g-sub` already sits under `g-mid`, so the two operations are two hops apart. Each sees +/// a two-workspace chain with a free label; together they make a four-deep one that repeats +/// `staging`. Locking the endpoints alone leaves them free to both commit, which is why the pairing +/// lock covers every workspace its checks read. +#[sqlx::test(migrations = "../migrations", fixtures("base", "nested_dev_workspace"))] +async fn test_attaches_two_hops_apart_cannot_both_claim_a_label( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + for round in 0..12 { + sqlx::query( + "UPDATE workspace SET parent_workspace_id = NULL, is_dev_workspace = false, + dev_workspace_label = NULL WHERE id IN ('g-mid', 'g-leaf')", + ) + .execute(&db) + .await?; + + let (upper, lower) = tokio::join!( + attach( + port, + "prod-g", + json!({ "dev_workspace_id": "g-mid", "dev_workspace_label": "staging" }), + ), + attach( + port, + "g-sub", + json!({ "dev_workspace_id": "g-leaf", "dev_workspace_label": "staging" }), + ), + ); + assert!( + upper.0.is_success() != lower.0.is_success(), + "round {round}: exactly one must win, got upper={} lower={}\n{}\n{}", + upper.0, + lower.0, + upper.1, + lower.1 + ); + + // No chain may carry one label twice. Walk every dev workspace up to its root and count. + let duplicated: Option = sqlx::query_scalar( + "WITH RECURSIVE chain AS ( + SELECT id AS leaf, id, parent_workspace_id, is_dev_workspace, + COALESCE(dev_workspace_label, 'dev') AS label, 0 AS depth + FROM workspace WHERE is_dev_workspace AND NOT deleted + UNION ALL + SELECT c.leaf, w.id, w.parent_workspace_id, w.is_dev_workspace, + COALESCE(w.dev_workspace_label, 'dev'), c.depth + 1 + FROM workspace w JOIN chain c ON w.id = c.parent_workspace_id + WHERE c.depth < 20 + ) + SELECT leaf FROM chain WHERE is_dev_workspace + GROUP BY leaf, label HAVING count(*) > 1 LIMIT 1", + ) + .fetch_optional(&db) + .await?; + assert!( + duplicated.is_none(), + "round {round}: a chain repeats a label, below {duplicated:?}" + ); + } + Ok(()) +} + +/// The pairing lock covers the chains an operation touches, not every pairing on the instance: a +/// transaction holding one family's nodes must not hold up another family's. Pinned because the +/// obvious way to make the races above safe — one key for the whole operation class — would serialize +/// dev-workspace creation database-wide, and creation holds its transaction across a full clone. +#[sqlx::test(migrations = "../migrations", fixtures("base", "nested_dev_workspace"))] +async fn test_pairing_lock_does_not_span_unrelated_families( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Hold family F's nodes the way an in-flight attach on it would, then act on family E. + let mut held = db.begin().await?; + for node in ["prod-f", "f-mid", "f-leaf"] { + sqlx::query("SELECT pg_advisory_xact_lock(hashtext('dev_workspace_pairing:' || $1))") + .bind(node) + .execute(&mut *held) + .await?; + } + // Also the un-suffixed key. Nothing takes it today, so holding it costs the passing case + // nothing — but a lock narrowed back to one key for every family would take it, and without + // this the test would sail through that exact regression. + sqlx::query("SELECT pg_advisory_xact_lock(hashtext('dev_workspace_pairing'))") + .execute(&mut *held) + .await?; + + let (status, body) = tokio::time::timeout( + std::time::Duration::from_secs(20), + detach(port, "prod-e", "wm-fork-edev"), + ) + .await + .map_err(|_| anyhow::anyhow!("an unrelated family's pairing blocked on family F's locks"))?; + assert!( + status.is_success(), + "unrelated detach returned {status}: {body}" + ); + + held.rollback().await?; + Ok(()) +} + +/// Archive resolves the workspace's pairing state before its transaction, so it takes the pairing +/// lock before reading that state again rather than on the strength of it — an attach can be turning +/// the workspace into a dev in the meantime. `h-cand` is standalone, the shape whose resolved state +/// says no pairing is involved: its archive must wait on the lock all the same. +#[sqlx::test(migrations = "../migrations", fixtures("base", "nested_dev_workspace"))] +async fn test_archive_takes_the_pairing_lock_for_a_standalone_workspace( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let mut held = db.begin().await?; + sqlx::query("SELECT pg_advisory_xact_lock(hashtext('dev_workspace_pairing:' || $1))") + .bind("h-cand") + .execute(&mut *held) + .await?; + + let finished = + tokio::time::timeout(std::time::Duration::from_secs(5), archive(port, "h-cand")).await; + assert!( + finished.is_err(), + "archive of a standalone workspace completed while its pairing lock was held: {finished:?}" + ); + + held.rollback().await?; + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/offboarding.rs b/backend/windmill-api-integration-tests/tests/offboarding.rs index be7a94d655..4434e6a2b5 100644 --- a/backend/windmill-api-integration-tests/tests/offboarding.rs +++ b/backend/windmill-api-integration-tests/tests/offboarding.rs @@ -80,6 +80,16 @@ async fn test_offboard_to_user(db: Pool) -> anyhow::Result<()> { .execute(&db) .await?; + // A shared-path script that runs as the departing user. Both halves of its identity have to + // move: a worker predating MIN_VERSION_SUPPORTS_ON_BEHALF_OF_PRINCIPAL reads the address and + // nothing else, so a stale one keeps running it under someone who has just been offboarded. + sqlx::query!( + "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of, on_behalf_of_email) + VALUES ('test-workspace', 'f/shared/obo', 1099, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2', 'test2@windmill.dev')" + ) + .execute(&db) + .await?; + let resp = authed(client().post(ws_url(port, "offboard/test-user-2"))) .json(&json!({ "reassign_to": "u/test-user", @@ -101,6 +111,17 @@ async fn test_offboard_to_user(db: Pool) -> anyhow::Result<()> { assert!(summary["variables_reassigned"].as_i64().unwrap() > 0); assert!(summary["schedules_reassigned"].as_i64().unwrap() > 0); + let obo = sqlx::query!( + "SELECT on_behalf_of, on_behalf_of_email FROM script WHERE path = 'f/shared/obo' AND workspace_id = 'test-workspace'" + ) + .fetch_one(&db) + .await?; + assert_eq!( + (obo.on_behalf_of.as_deref(), obo.on_behalf_of_email.as_deref()), + (Some("u/test-user"), Some("test@windmill.dev")), + "the reassignment moves the whole identity, not just the half this release reads" + ); + // Verify scripts moved let moved = sqlx::query_scalar!( "SELECT COUNT(*) FROM script WHERE path LIKE 'u/test-user/%' AND workspace_id = 'test-workspace' AND NOT archived AND NOT deleted" diff --git a/backend/windmill-api-integration-tests/tests/resources.rs b/backend/windmill-api-integration-tests/tests/resources.rs index 3cc59be473..07afbb3442 100644 --- a/backend/windmill-api-integration-tests/tests/resources.rs +++ b/backend/windmill-api-integration-tests/tests/resources.rs @@ -756,37 +756,26 @@ async fn test_mcp_client_get_job_and_logs(db: Pool) -> anyhow::Result< .auth_header("MCP_TOKEN"); let transport = StreamableHttpClientTransport::from_config(config); - let client_info = ClientInfo { - protocol_version: Default::default(), - capabilities: ClientCapabilities::default(), - client_info: Implementation { - name: "test-client".to_string(), - title: None, - version: "0.0.1".to_string(), - description: None, - website_url: None, - icons: None, - }, - meta: None, - }; + let client_info = ClientInfo::new( + ClientCapabilities::default(), + Implementation::new("test-client", "0.0.1"), + ); let client: RunningService = client_info.serve(transport).await?; // --- Test getJob --- let result = client - .call_tool(CallToolRequestParams { - name: "getJob".into(), - arguments: Some(serde_json::from_value(json!({ "id": job_id.to_string() }))?), - task: None, - meta: None, - }) + .call_tool( + CallToolRequestParams::new("getJob") + .with_arguments(serde_json::from_value(json!({ "id": job_id.to_string() }))?), + ) .await?; let text = result .content .first() - .and_then(|c| c.raw.as_text()) + .and_then(|c| c.as_text()) .expect("getJob should return text content"); let job: serde_json::Value = serde_json::from_str(&text.text)?; assert_eq!(job["id"], job_id.to_string()); @@ -800,18 +789,16 @@ async fn test_mcp_client_get_job_and_logs(db: Pool) -> anyhow::Result< // --- Test getJobLogs --- let result = client - .call_tool(CallToolRequestParams { - name: "getJobLogs".into(), - arguments: Some(serde_json::from_value(json!({ "id": job_id.to_string() }))?), - task: None, - meta: None, - }) + .call_tool( + CallToolRequestParams::new("getJobLogs") + .with_arguments(serde_json::from_value(json!({ "id": job_id.to_string() }))?), + ) .await?; let text = result .content .first() - .and_then(|c| c.raw.as_text()) + .and_then(|c| c.as_text()) .expect("getJobLogs should return text content"); // The logs endpoint returns text/plain, which gets wrapped as a JSON string by call_endpoint let logs: String = serde_json::from_str(&text.text)?; diff --git a/backend/windmill-api-integration-tests/tests/token_hash.rs b/backend/windmill-api-integration-tests/tests/token_hash.rs index 45d324a3d4..285f15a9c6 100644 --- a/backend/windmill-api-integration-tests/tests/token_hash.rs +++ b/backend/windmill-api-integration-tests/tests/token_hash.rs @@ -343,10 +343,17 @@ async fn test_rotate_webhook_token(db: Pool) -> anyhow::Result<()> { .execute(&db) .await?; - // Rotate the token - let rotated = rotate_webhook_token(&db, &original_hash, ServiceName::Google) - .await? - .expect("rotate must return Some for existing token"); + // Rotate onto a different runnable than the original token was minted for: a rename moves the + // trigger, and rotation has to follow it rather than carry the old scopes forward. + let renamed_scopes = vec!["jobs:run:flows:f/test/renamed".to_string()]; + let rotated = rotate_webhook_token( + &db, + &original_hash, + ServiceName::Google, + renamed_scopes.clone(), + ) + .await? + .expect("rotate must return Some for existing token"); // New token should be different assert_ne!(rotated.new_token, original_token); @@ -355,7 +362,7 @@ async fn test_rotate_webhook_token(db: Pool) -> anyhow::Result<()> { // New token's hash should exist in DB with the per-service label and expiration let new_hash = hash_token(&rotated.new_token); let new_row = sqlx::query!( - "SELECT label, expiration FROM token WHERE token_hash = $1", + "SELECT label, expiration, scopes FROM token WHERE token_hash = $1", new_hash ) .fetch_optional(&db) @@ -373,6 +380,13 @@ async fn test_rotate_webhook_token(db: Pool) -> anyhow::Result<()> { new_row.expiration.is_some(), "rotated Google token must carry an expiration" ); + // Carrying the old token's scopes here is what made every post-rename retry mint a token no + // callback could use, while reporting success. + assert_eq!( + new_row.scopes.as_deref(), + Some(renamed_scopes.as_slice()), + "rotation must scope the new token to the runnable it was rotated for" + ); // Old token should still exist (deletion deferred to caller) let old_exists: bool = sqlx::query_scalar!( @@ -402,7 +416,7 @@ async fn test_rotate_webhook_token(db: Pool) -> anyhow::Result<()> { assert!(!old_gone, "old token must be gone after explicit deletion"); // Rotating a non-existent hash should return None - let result = rotate_webhook_token(&db, "nonexistent_hash", ServiceName::Google).await?; + let result = rotate_webhook_token(&db, "nonexistent_hash", ServiceName::Google, vec![]).await?; assert!( result.is_none(), "rotating a non-existent token must return None" diff --git a/backend/windmill-api-integration-tests/tests/token_label_idor.rs b/backend/windmill-api-integration-tests/tests/token_label_idor.rs index 92dada92c2..ddec1a481a 100644 --- a/backend/windmill-api-integration-tests/tests/token_label_idor.rs +++ b/backend/windmill-api-integration-tests/tests/token_label_idor.rs @@ -179,6 +179,9 @@ async fn test_trigger_token_labels_still_creatable(db: Pool) -> anyhow "http-test-user-2-cd34", "email-test-user-2-ef56", "my-ci-token", + // Minted client-side by the editor (every TypeScript editor load) and the debugger. + "Ephemeral lsp token", + "debugger-token", ] { let resp = create_token_with_label(port, "SECRET_TOKEN_2", label).await; assert_eq!( @@ -190,3 +193,31 @@ async fn test_trigger_token_labels_still_creatable(db: Pool) -> anyhow Ok(()) } + +/// The mirror of the above: reserved namespaces must NOT be mintable. `username_override_from_label` +/// trusts these shapes to name the entity acting, so a forged one would stamp an arbitrary +/// name onto `v2_job.created_by` and the audit `end_user` — on an `on_behalf_of` runnable, +/// which also takes the `username`/`email` columns, that leaves no trace of the real caller. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_reserved_token_labels_not_creatable(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + for label in [ + "ephemeral-webhook-forged", + "ephemeral-script-end-user-svcaccount", + "ephemeral-script", + "session", + "mcp-oauth-forged", + ] { + let resp = create_token_with_label(port, "SECRET_TOKEN_2", label).await; + assert_eq!( + resp.status(), + 400, + "creating a token with reserved label {label:?} must be rejected" + ); + } + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/users.rs b/backend/windmill-api-integration-tests/tests/users.rs index 7b3457cde1..fb64d9eb77 100644 --- a/backend/windmill-api-integration-tests/tests/users.rs +++ b/backend/windmill-api-integration-tests/tests/users.rs @@ -512,6 +512,84 @@ async fn test_user_endpoints(db: Pool) -> anyhow::Result<()> { Ok(()) } +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_list_addable_instance_users(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/users/list_addable"); + + // The three fixture accounts are all members of test-workspace already. + sqlx::query( + "INSERT INTO password(email, password_hash, login_type, verified, username, disabled) VALUES + ('addable@windmill.dev', 'x', 'password', true, 'addable-user', false), + ('with_underscore@windmill.dev', 'x', 'password', true, 'underscored', false), + ('gone@windmill.dev', 'x', 'password', true, 'gone-user', true)" + ) + .execute(&db) + .await?; + sqlx::query( + "INSERT INTO usr(workspace_id, email, username, is_admin, role, is_service_account) + VALUES ('test-workspace', 'sa@creator.test-workspace.sa.wm.dev', 'sa', false, 'User', true)" + ) + .execute(&db) + .await?; + + let emails = |query: &str| { + let url = format!("{base}?{query}"); + async move { + let resp = authed(client().get(url)).send().await.unwrap(); + assert_eq!(resp.status(), 200); + resp.json::>() + .await + .unwrap() + .into_iter() + .map(|u| u["email"].as_str().unwrap().to_string()) + .collect::>() + } + }; + + // Members, disabled accounts and service accounts are all out; a service account has no + // `password` row at all, so it can never reach the picker. + assert_eq!( + emails("").await, + vec![ + "addable@windmill.dev", + // seeded by the migrations, not a member of test-workspace + "admin@windmill.dev", + "with_underscore@windmill.dev" + ] + ); + + // The exclusions are part of the query, so the limit counts addable accounts only — a + // workspace whose members sort first must not swallow the whole page. + assert_eq!(emails("per_page=1").await, vec!["addable@windmill.dev"]); + + // Search matches the instance username as well as the email. + assert_eq!( + emails("search=addable-user").await, + vec!["addable@windmill.dev"] + ); + + // Wildcards in the search are matched literally rather than expanded. + assert_eq!( + emails("search=_").await, + vec!["with_underscore@windmill.dev"] + ); + assert!(emails("search=%25").await.is_empty()); + + // Listing instance-wide accounts is superadmin-only, workspace admin is not enough. + let resp = client() + .get(&base) + .header("Authorization", "Bearer SECRET_TOKEN_3") + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 401); + + Ok(()) +} + #[sqlx::test(migrations = "../migrations", fixtures("base"))] async fn test_change_user_email(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; @@ -660,3 +738,175 @@ async fn test_change_user_email(db: Pool) -> anyhow::Result<()> { Ok(()) } + +/// A superadmin acting outside every workspace has no username of their own, so their runnables +/// name them by their address — and an address may contain a `/`, which every reader of a +/// principal splits on. Moving such an account has to leave behind the form that decodes back to +/// them rather than one that reads as a group. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_change_user_email_to_slash_address(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let global_base = format!("http://localhost:{}/api/users", server.addr.port()); + + sqlx::query!( + "INSERT INTO password(email, password_hash, login_type, super_admin, verified, name) + VALUES ('ext@windmill.dev', 'not-a-real-hash', 'password', true, true, 'Ext')" + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of) + VALUES ('test-workspace', 'u/test-user/s', 93001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'ext@windmill.dev')" + ) + .execute(&db) + .await?; + + // The principal follows the address, and a job row carries it in a narrower column than the + // runnable does, so a move that would make it unenqueueable is refused rather than silently + // leaving runnables that look configured and cannot start. + let resp = authed(client().post(format!("{global_base}/change_email/ext@windmill.dev"))) + .json(&json!({ "new_email": "a-very-long-superadmin-address-for-this-test@windmill.dev" })) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!(status, 400, "{body}"); + assert!(body.contains("characters a job can carry"), "{body}"); + + let resp = authed(client().post(format!("{global_base}/change_email/ext@windmill.dev"))) + .json(&json!({ "new_email": "ops/alice@windmill.dev" })) + .send() + .await?; + assert_eq!(resp.status(), 200, "change_email: {}", resp.text().await?); + + assert_eq!( + sqlx::query_scalar!( + "SELECT on_behalf_of FROM script WHERE path = 'u/test-user/s' AND workspace_id = 'test-workspace'" + ) + .fetch_one(&db) + .await? + .as_deref(), + Some("u/ops/alice@windmill.dev"), + "left bare, the new address would come back as group 'alice@windmill.dev'" + ); + + Ok(()) +} + +/// A group's synthetic address (`group-{name}@windmill.dev`) can also be a real user's, and a +/// runnable configured for the *group* carries that address next to `g/{name}`. Moving the +/// colliding user's account must leave it alone: rewriting one half of the pair would leave it +/// naming two different people, which the deploy path rejects. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_change_user_email_leaves_group_identities(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let global_base = format!("http://localhost:{}/api/users", server.addr.port()); + + sqlx::query!("UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'") + .execute(&db) + .await?; + sqlx::query!("UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'") + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO group_(workspace_id, name, summary, extra_perms) VALUES ('test-workspace', 'ops', '', '{}')" + ) + .execute(&db) + .await?; + + // An email change never moves a username, so the principal these rows hold stays put. What + // moves is the address beside it — kept for the workers that still read it — and in every + // pair a group-owned identity keeps the group's synthetic address even though a real account + // now holds it: rewriting one half leaves the pair naming two accounts. + sqlx::query!( + "INSERT INTO app(workspace_id, path, summary, policy, versions) + VALUES ('test-workspace', 'u/test-user/g', '', '{\"on_behalf_of\": \"g/ops\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::jsonb, '{}'), + ('test-workspace', 'u/test-user/u', '', '{\"on_behalf_of\": \"u/test-user-2\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::jsonb, '{}')" + ) + .execute(&db) + .await?; + + sqlx::query!( + "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, on_behalf_of, on_behalf_of_email) + VALUES ('test-workspace', 'u/test-user/sg', 95001, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'g/ops', 'group-ops@windmill.dev'), + ('test-workspace', 'u/test-user/su', 95002, 'def main(): pass', '', '', 'python3', 'test-user', NOW(), 'u/test-user-2', 'group-ops@windmill.dev')" + ) + .execute(&db) + .await?; + + sqlx::query!( + "INSERT INTO draft(workspace_id, path, typ, value, email) + VALUES ('test-workspace', 'u/test-user/dg', 'script', '{\"on_behalf_of\": \"g/ops\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::json, 'test@windmill.dev'), + ('test-workspace', 'u/test-user/du', 'script', '{\"on_behalf_of\": \"u/test-user-2\", \"on_behalf_of_email\": \"group-ops@windmill.dev\"}'::json, 'test@windmill.dev')" + ) + .execute(&db) + .await?; + + let resp = authed(client().post(format!("{global_base}/change_email/group-ops@windmill.dev"))) + .json(&json!({ "new_email": "renamed@windmill.dev" })) + .send() + .await?; + assert_eq!(resp.status(), 200, "change_email: {}", resp.text().await?); + + let apps = sqlx::query!( + "SELECT path, policy->>'on_behalf_of_email' AS email FROM app WHERE workspace_id = 'test-workspace' ORDER BY path" + ) + .fetch_all(&db) + .await?; + assert_eq!( + apps.iter() + .map(|r| (r.path.as_str(), r.email.as_deref())) + .collect::>(), + vec![ + ("u/test-user/g", Some("group-ops@windmill.dev")), + ("u/test-user/u", Some("renamed@windmill.dev")), + ], + "the group-owned app keeps the group's address; the user-owned one moves" + ); + + let scripts = sqlx::query!( + "SELECT path, on_behalf_of_email AS email FROM script WHERE workspace_id = 'test-workspace' AND path LIKE 'u/test-user/s%' ORDER BY path" + ) + .fetch_all(&db) + .await?; + assert_eq!( + scripts + .iter() + .map(|r| (r.path.as_str(), r.email.as_deref())) + .collect::>(), + vec![ + ("u/test-user/sg", Some("group-ops@windmill.dev")), + ("u/test-user/su", Some("renamed@windmill.dev")), + ], + "the group-owned script keeps the group's address; the user-owned one moves" + ); + + let drafts = sqlx::query!( + "SELECT path, value->>'on_behalf_of_email' AS email, value->>'on_behalf_of' AS principal FROM draft WHERE workspace_id = 'test-workspace' ORDER BY path" + ) + .fetch_all(&db) + .await?; + assert_eq!( + drafts + .iter() + .map(|r| (r.path.as_str(), r.principal.as_deref(), r.email.as_deref())) + .collect::>(), + vec![ + ( + "u/test-user/dg", + Some("g/ops"), + Some("group-ops@windmill.dev") + ), + ( + "u/test-user/du", + Some("u/test-user-2"), + Some("renamed@windmill.dev") + ), + ], + "a draft's pair moves as a whole or not at all — either half left behind is a 400 on deploy" + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/workspace_comparison.rs b/backend/windmill-api-integration-tests/tests/workspace_comparison.rs index 392f43d410..11c731d75c 100644 --- a/backend/windmill-api-integration-tests/tests/workspace_comparison.rs +++ b/backend/windmill-api-integration-tests/tests/workspace_comparison.rs @@ -1754,6 +1754,83 @@ async fn test_compare_workspaces_phantom_trigger_shortfuse( Ok(()) } +/// A source-only row is offered to the fork whatever its counters say, so it can +/// carry `behind = 0` — a `behind`-derived tally never sees it, and hiding one must +/// still be reported to the update direction. The merge direction does not carry +/// such a row at all, so hiding one withholds nothing from that side. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_compare_workspaces_hidden_source_only_no_behind( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base_url = format!("http://localhost:{port}/api"); + let non_admin = windmill_api_client::create_client( + &format!("http://localhost:{port}"), + "SECRET_TOKEN_2".to_string(), + ); + + sqlx::query!( + "INSERT INTO workspace (id, name, owner, parent_workspace_id) + VALUES ('wm-fork-test-workspace', 'Fork', 'test-user', 'test-workspace')" + ) + .execute(&db) + .await?; + sqlx::query!("INSERT INTO workspace_settings (workspace_id) VALUES ('wm-fork-test-workspace')") + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO workspace_key(workspace_id, kind, key) + VALUES ('wm-fork-test-workspace', 'cloud', 'test-key')" + ) + .execute(&db) + .await?; + + // Source-only row with no `behind`: the trigger has no backing row, so the + // visibility filter drops it exactly as it would an ACL-hidden one. + sqlx::query!( + "INSERT INTO workspace_diff + (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork) + VALUES ('test-workspace', 'wm-fork-test-workspace', 'f/rt/parent_only', 'http_trigger', 1, 0, true, true, false)" + ) + .execute(&db) + .await?; + + let comparison: serde_json::Value = non_admin + .client() + .get(&format!( + "{base_url}/w/test-workspace/workspaces/compare/wm-fork-test-workspace" + )) + .send() + .await? + .json() + .await?; + assert_eq!( + comparison["all_behind_items_visible"].as_bool(), + Some(false), + "a hidden source-only row must trip the warning even at behind = 0: {comparison}" + ); + assert_eq!( + comparison["hidden_behind"]["total"].as_i64(), + Some(1), + "a hidden source-only row must be counted as withheld from the update direction: {comparison}" + ); + assert_eq!( + comparison["all_ahead_items_visible"].as_bool(), + Some(true), + "the merge direction does not carry a source-only row, so hiding one withholds nothing from it: {comparison}" + ); + assert_eq!( + comparison["hidden_ahead"]["total"].as_i64(), + Some(0), + "a source-only row must not be reported as withheld from the merge direction: {comparison}" + ); + + Ok(()) +} + /// Regression: the "sees everything" guard must require admin of BOTH sides, not /// just the fork. `filter_visible_diffs` keeps a modified/conflict row (one that /// exists in the source AND the fork) only when the caller can see it on both @@ -2129,3 +2206,357 @@ async fn test_full_diff_scan_against_arbitrary_workspace(db: Pool) -> Ok(()) } + +/// Regression for WIN-2289. A fork deletion and a git-sync pull reverting an +/// earlier deploy leave the same `ahead > 0, behind = 0` row with the item gone +/// from the fork; the comparison must carry the recorded origin that tells them +/// apart, and a tally that cannot vouch for its event must not overwrite it. +/// +/// Gated on `private` for the same reason as the tally tests above: the OSS +/// `handle_deployment_metadata` is a no-op, so no rows would ever be written. +#[cfg(feature = "private")] +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_compare_records_fork_removal_origin(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base_url = format!("http://localhost:{port}/api"); + let admin = windmill_api_client::create_client( + &format!("http://localhost:{port}"), + "SECRET_TOKEN".to_string(), + ); + + sqlx::query!("DELETE FROM skip_workspace_diff_tally") + .execute(&db) + .await?; + + // Deployed in the parent before the fork exists, so nothing is tallied yet and + // the fork starts with both scripts. bash needs no lock generation, so the + // create tallies inline instead of deferring to a dependency job (no worker + // runs in this test). + for path in ["u/admin/dropped_by_user", "u/admin/dropped_by_sync"] { + let resp = admin + .client() + .post(&format!("{base_url}/w/test-workspace/scripts/create")) + .json(&json!({ + "path": path, + "summary": "", + "description": "", + "content": "echo 1", + "language": "bash", + "schema": {"type": "object", "properties": {}, "required": []}, + })) + .send() + .await?; + let status = resp.status(); + assert!( + status.is_success(), + "script create failed: {} — {}", + status, + resp.text().await? + ); + } + + let resp = admin + .client() + .post(&format!( + "{base_url}/w/test-workspace/workspaces/create_fork" + )) + .json(&json!({"id": "wm-fork-removal", "name": "Removal Fork"})) + .send() + .await?; + assert!( + resp.status().is_success(), + "fork creation failed: {}", + resp.status() + ); + + // Same act, same resulting row shape; only the header a sync client sets differs. + for (path, origin_header) in [ + ("u/admin/dropped_by_user", None), + ("u/admin/dropped_by_sync", Some("sync")), + ] { + let mut req = admin.client().post(&format!( + "{base_url}/w/wm-fork-removal/scripts/archive/p/{path}" + )); + if let Some(origin) = origin_header { + req = req.header("X-Windmill-Deploy-Origin", origin); + } + let resp = req.send().await?; + let status = resp.status(); + assert!( + status.is_success(), + "archive of {path} failed: {} — {}", + status, + resp.text().await? + ); + } + + // The tally runs in a `tokio::spawn` inside `handle_deployment_metadata`. + let mut recorded = 0; + for _ in 0..40 { + recorded = sqlx::query_scalar!( + "SELECT COUNT(*) AS \"count!\" FROM workspace_diff + WHERE source_workspace_id = 'test-workspace' + AND fork_workspace_id = 'wm-fork-removal' + AND kind = 'script' + AND fork_last_event_origin IS NOT NULL" + ) + .fetch_one(&db) + .await?; + if recorded >= 2 { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } + assert_eq!(recorded, 2, "both archives should have tallied an event"); + + let comparison: serde_json::Value = admin + .client() + .get(&format!( + "{base_url}/w/test-workspace/workspaces/compare/wm-fork-removal" + )) + .send() + .await? + .json() + .await?; + let diffs = comparison["diffs"].as_array().unwrap(); + + for (path, expected_origin) in [ + ("u/admin/dropped_by_user", "authored"), + ("u/admin/dropped_by_sync", "sync"), + ] { + let row = diffs + .iter() + .find(|d| d["path"] == path && d["kind"] == "script") + .unwrap_or_else(|| panic!("{path} should be in the diff; got {diffs:?}")); + assert_eq!(row["exists_in_source"].as_bool(), Some(true), "{row}"); + assert_eq!(row["exists_in_fork"].as_bool(), Some(false), "{row}"); + assert_eq!(row["behind"].as_i64(), Some(0), "{row}"); + assert_eq!( + row["fork_last_event_kind"].as_str(), + Some("delete"), + "{row}" + ); + assert_eq!( + row["fork_last_event_origin"].as_str(), + Some(expected_origin), + "the two removals must stay distinguishable: {row}" + ); + } + + // The call the worker makes on its success path for a lock-generating deploy: + // it reports a deploy that landed before the archive and cannot answer for what + // the path holds now. The archive's record has to survive it, and only the + // counter moves. + windmill_git_sync::handle_deployment_metadata( + "admin@windmill.dev", + "admin", + &db, + "wm-fork-removal", + windmill_git_sync::DeployedObject::Script { + hash: windmill_common::scripts::ScriptHash(0), + path: "u/admin/dropped_by_sync".to_string(), + parent_path: Some("u/admin/dropped_by_sync".to_string()), + }, + None, + true, + // A redeploy that did not move the item passes the same path back; it must + // not be tallied a second time as the path a rename vacated. + Some("u/admin/dropped_by_sync"), + ) + .await?; + let mut after = None; + for _ in 0..40 { + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + let row = sqlx::query!( + "SELECT ahead, fork_last_event_kind, fork_last_event_origin FROM workspace_diff + WHERE source_workspace_id = 'test-workspace' AND fork_workspace_id = 'wm-fork-removal' + AND kind = 'script' AND path = 'u/admin/dropped_by_sync'" + ) + .fetch_one(&db) + .await?; + if row.ahead > 1 { + after = Some(row); + break; + } + } + let after = after.expect("the detached tally should still bump the counter"); + assert_eq!(after.fork_last_event_kind.as_deref(), Some("delete")); + assert_eq!( + after.fork_last_event_origin.as_deref(), + Some("sync"), + "a tally that cannot vouch for its event must not overwrite one that did" + ); + assert_eq!(after.ahead, 2, "and it counts once, not twice"); + + Ok(()) +} + +/// `probe_deploy_event_kind` builds its query for these kinds by interpolating the +/// table name, so a wrong entry compiles fine and only fails once someone deploys +/// that trigger kind in a fork — where the error aborts the tally and the item +/// never reaches `workspace_diff` at all. Sweep the allowlist against a live +/// database instead. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_probe_covers_every_path_keyed_table(db: Pool) -> anyhow::Result<()> { + for kind in windmill_git_sync::PATH_KEYED_TABLES { + let probed = + windmill_git_sync::probe_deploy_event_kind(&db, "test-workspace", kind, "u/a/nothing") + .await + .unwrap_or_else(|e| panic!("probing `{kind}` failed: {e}")); + assert_eq!( + probed, + Some(windmill_common::deploy_origin::DeployEventKind::Delete), + "an absent `{kind}` must probe as a deletion" + ); + } + Ok(()) +} + +/// Flows always generate a lock, so without the request reporting the path a +/// rename left behind, a renamed flow leaves its old path in the parent with +/// nothing to merge. +#[cfg(feature = "private")] +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_rename_records_the_vacated_path(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base_url = format!("http://localhost:{port}/api"); + let admin = windmill_api_client::create_client( + &format!("http://localhost:{port}"), + "SECRET_TOKEN".to_string(), + ); + + sqlx::query!("DELETE FROM skip_workspace_diff_tally") + .execute(&db) + .await?; + let resp = admin + .client() + .post(&format!( + "{base_url}/w/test-workspace/workspaces/create_fork" + )) + .json(&json!({"id": "wm-fork-renamed", "name": "Rename Fork"})) + .send() + .await?; + assert!( + resp.status().is_success(), + "fork creation: {}", + resp.status() + ); + + let flow = |path: &str| { + json!({ + "path": path, + "summary": "", + "description": "", + "value": {"modules": []}, + "schema": {"type": "object", "properties": {}, "required": []}, + }) + }; + for (path, origin) in [ + ("u/admin/moved_by_user", None), + ("u/admin/moved_by_sync", Some("sync")), + ] { + let mut req = admin + .client() + .post(&format!("{base_url}/w/wm-fork-renamed/flows/create")) + .json(&flow(path)); + if let Some(origin) = origin { + req = req.header("X-Windmill-Deploy-Origin", origin); + } + let resp = req.send().await?; + assert!(resp.status().is_success(), "flow create: {}", resp.status()); + + let mut req = admin + .client() + .post(&format!("{base_url}/w/wm-fork-renamed/flows/update/{path}")) + .json(&flow(&format!("{path}_new"))); + if let Some(origin) = origin { + req = req.header("X-Windmill-Deploy-Origin", origin); + } + let resp = req.send().await?; + assert!(resp.status().is_success(), "flow rename: {}", resp.status()); + } + + let mut rows = vec![]; + for _ in 0..40 { + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + rows = sqlx::query!( + "SELECT path, fork_last_event_kind, fork_last_event_origin FROM workspace_diff + WHERE fork_workspace_id = 'wm-fork-renamed' AND kind = 'flow' + AND fork_last_event_origin IS NOT NULL ORDER BY path" + ) + .fetch_all(&db) + .await?; + if rows.len() >= 2 { + break; + } + } + let claimed: Vec<_> = rows + .iter() + .map(|r| { + ( + r.path.as_str(), + r.fork_last_event_kind.as_deref(), + r.fork_last_event_origin.as_deref(), + ) + }) + .collect(); + assert_eq!( + claimed, + vec![ + ("u/admin/moved_by_sync", Some("delete"), Some("sync")), + ("u/admin/moved_by_user", Some("delete"), Some("authored")), + ], + "each vacated path is recorded with the origin of the rename that left it, \ + and the deployed paths — which only the dependency job reports — are not" + ); + + // A script needing no lock deploys inline, so its own request reports both + // paths and can say why the old one is empty rather than only that it is. + let mut parent_hash: Option = None; + for path in ["u/admin/inline", "u/admin/inline_moved"] { + let resp = admin + .client() + .post(&format!("{base_url}/w/wm-fork-renamed/scripts/create")) + .json(&json!({ + "path": path, + "summary": "", + "description": "", + // bash needs no lock, so this deploys inline. + "content": "echo 1", + "language": "bash", + "schema": {"type": "object", "properties": {}, "required": []}, + "parent_hash": parent_hash, + })) + .send() + .await?; + let status = resp.status(); + let hash = resp.text().await?; + assert!(status.is_success(), "script deploy failed: {status} — {hash}"); + parent_hash = Some(hash.trim().trim_matches('"').to_string()); + } + let mut vacated = None; + for _ in 0..40 { + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + vacated = sqlx::query!( + "SELECT fork_last_event_kind, fork_last_event_origin FROM workspace_diff + WHERE fork_workspace_id = 'wm-fork-renamed' AND kind = 'script' + AND path = 'u/admin/inline' AND fork_last_event_kind IS NOT NULL" + ) + .fetch_optional(&db) + .await?; + if vacated.is_some() { + break; + } + } + let vacated = vacated.expect("the inline rename should report the path it left"); + assert_eq!(vacated.fork_last_event_kind.as_deref(), Some("rename_from")); + assert_eq!(vacated.fork_last_event_origin.as_deref(), Some("authored")); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/workspaces.rs b/backend/windmill-api-integration-tests/tests/workspaces.rs index 851dcbb4cd..e9eeb88996 100644 --- a/backend/windmill-api-integration-tests/tests/workspaces.rs +++ b/backend/windmill-api-integration-tests/tests/workspaces.rs @@ -964,3 +964,46 @@ async fn test_get_imports(db: Pool) -> anyhow::Result<()> { Ok(()) } + +/// A dbt project names a warehouse and nothing else, so the workspace setting is +/// the only place the connection is decided. Two things have to hold for that to +/// work at all: the setting round-trips as the MAP the resolver reads (an +/// envelope stored verbatim makes every warehouse name unresolvable), and the +/// route that serves the name to a worker with no database stays job-scoped. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_dbt_warehouses(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace"); + + let resp = authed(client().post(format!("{base}/workspaces/edit_dbt_warehouses"))) + .json(&json!({ + "dbt_warehouses": { "main": { "resource_path": "u/admin/wh", "target": "prod" } } + })) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + + let resp = authed(client().get(format!("{base}/workspaces/get_settings"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + let settings = resp.json::().await?; + assert_eq!( + settings["dbt_warehouses"], + json!({ "main": { "resource_path": "u/admin/wh", "target": "prod" } }) + ); + + // A user token is not a job token: the warehouses a workspace configures are + // a running job's business, not a browsable list. + let resp = authed(client().get(format!("{base}/dbt/warehouse/main"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 400); + + Ok(()) +} diff --git a/backend/windmill-api-jobs/src/execution.rs b/backend/windmill-api-jobs/src/execution.rs index b4707fca9f..ebee1bcbbf 100644 --- a/backend/windmill-api-jobs/src/execution.rs +++ b/backend/windmill-api-jobs/src/execution.rs @@ -691,6 +691,7 @@ pub async fn run_flow<'c>( bool, Option>, )> { + let on_behalf_of = flow_version_info.on_behalf_of(w_id, &db).await?; let FlowVersionInfo { version, tag, @@ -698,8 +699,6 @@ pub async fn run_flow<'c>( has_preprocessor, has_failure_module, chat_input_enabled, - on_behalf_of_email, - edited_by, early_return, labels, .. @@ -719,10 +718,10 @@ pub async fn run_flow<'c>( Some(authed.clone().into()), PushIsolationLevel::Transaction(tx), ) - } else if let Some(on_behalf_of_email) = on_behalf_of_email.as_ref() { + } else if let Some(obo) = on_behalf_of.as_ref() { ( - on_behalf_of_email, - username_to_permissioned_as(&edited_by), + &obo.email, + obo.permissioned_as.clone(), None, PushIsolationLevel::IsolatedRoot(db.clone()), ) @@ -752,6 +751,7 @@ pub async fn run_flow<'c>( email, permissioned_as, authed.token_prefix.as_deref(), + authed.username_override.as_deref(), scheduled_for, None, run_query.parent_job, @@ -769,7 +769,7 @@ pub async fn run_flow<'c>( push_authed.as_ref(), false, None, - trigger, + authed.trigger_or_fallback(trigger), run_query.suspended_mode, ) .await?; @@ -966,6 +966,7 @@ pub async fn push_script_job_by_path_into_queue<'c>( email, permissioned_as, authed.token_prefix.as_deref(), + authed.username_override.as_deref(), scheduled_for, None, run_query.parent_job, @@ -988,7 +989,7 @@ pub async fn push_script_job_by_path_into_queue<'c>( push_authed.as_ref(), false, None, - trigger, + authed.trigger_or_fallback(trigger), run_query.suspended_mode, ) .await?; diff --git a/backend/windmill-api-jobs/src/types.rs b/backend/windmill-api-jobs/src/types.rs index 0d96a66750..6c2d776e30 100644 --- a/backend/windmill-api-jobs/src/types.rs +++ b/backend/windmill-api-jobs/src/types.rs @@ -455,6 +455,7 @@ impl From for Job { resolved_at: None, resolution_note: None, resolved_automatically: None, + trigger_kind: None, }, )), "QueuedJob" => Job::QueuedJob(JobExtended::new( @@ -505,6 +506,7 @@ impl From for Job { runnable_settings_handle: uj.runnable_settings_handle, labels: uj.labels, is_retry: uj.is_retry, + trigger_kind: None, }, )), t => panic!("job type {} not valid", t), diff --git a/backend/windmill-api-scripts/Cargo.toml b/backend/windmill-api-scripts/Cargo.toml index ecf0ff0b61..8b34512872 100644 --- a/backend/windmill-api-scripts/Cargo.toml +++ b/backend/windmill-api-scripts/Cargo.toml @@ -10,12 +10,14 @@ path = "src/lib.rs" [features] default = [] -enterprise = ["windmill-common/enterprise"] +enterprise = ["windmill-common/enterprise", "windmill-native-triggers?/enterprise"] private = ["windmill-common/private", "windmill-dep-map/private"] python = ["dep:windmill-parser-py", "dep:windmill-parser-py-asset"] prometheus = ["dep:prometheus", "windmill-common/prometheus"] +native_trigger = ["dep:windmill-native-triggers", "windmill-native-triggers/native_trigger"] [dependencies] windmill-common = { workspace = true, default-features = false } +windmill-native-triggers = { workspace = true, optional = true } windmill-object-store.workspace = true windmill-api-auth.workspace = true windmill-queue.workspace = true diff --git a/backend/windmill-api-scripts/src/asset_inference.rs b/backend/windmill-api-scripts/src/asset_inference.rs index 5debb0d271..9c503ae1ae 100644 --- a/backend/windmill-api-scripts/src/asset_inference.rs +++ b/backend/windmill-api-scripts/src/asset_inference.rs @@ -42,6 +42,13 @@ fn parse_assets_for_lang( #[cfg(feature = "python")] ScriptLang::Python3 => windmill_parser_py_asset::parse_assets(content), ScriptLang::Ansible => windmill_parser_yaml::parse_assets(content), + // dbt is the one language whose assets are not a function of the script + // content: they come from `manifest.json`, which needs a clone of the + // project's repo and a dbt invocation. The deploy-time dependency job + // does that parse and writes the `asset` rows itself + // (`dbt_executor::dbt_dep`), so returning None here is what keeps this + // path from clobbering them with an empty list. + ScriptLang::Dbt => return None, _ => return None, }; match parsed { diff --git a/backend/windmill-api-scripts/src/scripts.rs b/backend/windmill-api-scripts/src/scripts.rs index c2f023ce5a..2433502537 100644 --- a/backend/windmill-api-scripts/src/scripts.rs +++ b/backend/windmill-api-scripts/src/scripts.rs @@ -71,6 +71,7 @@ use windmill_common::{ ScriptHistory, ScriptHistoryUpdate, ScriptKind, ScriptLang, ScriptModule, ScriptWithStarred, }, + triggers::MovedNativeTrigger, users::username_to_permissioned_as, utils::{not_found_if_none, query_elems_from_hub, require_admin, Pagination, StripPath}, worker::to_raw_value, @@ -499,6 +500,35 @@ async fn get_top_hub_scripts( Ok::<_, Error>((status_code, headers, response)) } +/// Re-point the webhooks of the native triggers a rename carried onto the new path. +/// +/// Runs after the deploy transaction commits — repointing a webhook is not undoable — and off the +/// request, because it waits on a third-party service that may be slow or gone, and a deploy that +/// already committed must not look like it failed. The rename itself marked these rows +/// `REREGISTRATION_PENDING`, so nothing is lost silently if this never finishes. +fn reregister_moved_native_triggers( + db: &DB, + authed: &ApiAuthed, + w_id: &str, + moved: Vec, +) { + if moved.is_empty() { + return; + } + #[cfg(feature = "native_trigger")] + { + let (db, authed, w_id) = (db.clone(), authed.clone(), w_id.to_string()); + tokio::spawn(async move { + windmill_native_triggers::rename::reregister_triggers_after_rename( + &db, &authed, &w_id, &moved, + ) + .await; + }); + } + #[cfg(not(feature = "native_trigger"))] + let _ = (db, authed, w_id, moved); +} + async fn create_snapshot_script( authed: ApiAuthed, Extension(user_db): Extension, @@ -513,6 +543,7 @@ async fn create_snapshot_script( let mut tx = None; let mut uploaded = false; let mut handle_deployment_metadata = None; + let mut moved_native_triggers = Vec::new(); while let Some(field) = multipart.next_field().await.unwrap() { let name = field.name().unwrap().to_string(); let data = field.bytes().await.unwrap(); @@ -520,7 +551,7 @@ async fn create_snapshot_script( let ns: NewScript = Some(serde_json::from_slice(&data).map_err(to_anyhow)?).unwrap(); let is_tar = ns.codebase.as_ref().is_some_and(|x| x.ends_with(".tar")); let use_esm = ns.codebase.as_ref().is_some_and(|x| x.contains(".esm")); - let (new_hash, ntx, hdm) = create_script_internal( + let (new_hash, ntx, hdm, moved) = create_script_internal( ns, w_id.clone(), authed.clone(), @@ -540,6 +571,7 @@ async fn create_snapshot_script( script_hash = Some(nh); tx = Some(ntx); handle_deployment_metadata = hdm; + moved_native_triggers = moved; } if name == "file" { let hash = script_hash.as_ref().ok_or_else(|| { @@ -570,6 +602,7 @@ async fn create_snapshot_script( } tx.unwrap().commit().await?; + reregister_moved_native_triggers(&db, &authed, &w_id, moved_native_triggers); if let Some(hdm) = handle_deployment_metadata { hdm.handle(&db).await?; } @@ -627,7 +660,8 @@ async fn create_script( let script_path = ns.path.clone(); let email = authed.email.clone(); let username = authed.username.clone(); - let (hash, tx, hdm) = create_script_internal( + let authed_for_triggers = authed.clone(); + let (hash, tx, hdm, moved_native_triggers) = create_script_internal( ns, w_id.clone(), authed, @@ -638,51 +672,78 @@ async fn create_script( ) .await?; tx.commit().await?; + reregister_moved_native_triggers(&db, &authed_for_triggers, &w_id, moved_native_triggers); if let Some(hdm) = hdm { - // hdm is Some when no lock generation is needed (script is ready immediately). - // Trigger CI tests for any items that reference this script. + // Only a script that needed no lock generation is deployed and runnable by + // now; one that did gets its CI tests from the dependency job instead, so + // they don't run against a version whose lock does not exist yet — and + // don't run twice. + let ready_to_test = matches!(hdm, PostCommitDeploy::Full { .. }); hdm.handle(&db).await?; let db2 = db.clone(); - tokio::spawn(async move { - if let Err(e) = windmill_dep_map::ci_tests::trigger_ci_tests_for_item( - &db2, - &w_id, - &script_path, - "script", - &email, - &username, - ) - .await - { - tracing::error!(%e, "error triggering CI tests after script deploy"); - } - }); + if ready_to_test { + tokio::spawn(async move { + if let Err(e) = windmill_dep_map::ci_tests::trigger_ci_tests_for_item( + &db2, + &w_id, + &script_path, + "script", + &email, + &username, + ) + .await + { + tracing::error!(%e, "error triggering CI tests after script deploy"); + } + }); + } } Ok((StatusCode::CREATED, format!("{}", hash))) } -struct HandleDeploymentMetadata { - email: String, - created_by: String, - w_id: String, - obj: DeployedObject, - deployment_message: Option, - renamed_from: Option, +/// What a script deploy still has to do once its transaction has committed. +enum PostCommitDeploy { + /// Everything, for a deploy with no dependency job to hand it to. + Full { + email: String, + created_by: String, + w_id: String, + obj: DeployedObject, + deployment_message: Option, + renamed_from: Option, + }, + /// Only the path a rename left behind; the dependency job handles the rest. + /// See `tally_rename_vacated_path`. + VacatedPath { w_id: String, obj: DeployedObject }, } -impl HandleDeploymentMetadata { +impl PostCommitDeploy { async fn handle(self, db: &DB) -> Result<()> { - handle_deployment_metadata( - &self.email, - &self.created_by, - &db, - &self.w_id, - self.obj, - self.deployment_message, - false, - self.renamed_from.as_deref(), - ) - .await + match self { + PostCommitDeploy::Full { + email, + created_by, + w_id, + obj, + deployment_message, + renamed_from, + } => { + handle_deployment_metadata( + &email, + &created_by, + &db, + &w_id, + obj, + deployment_message, + false, + renamed_from.as_deref(), + ) + .await + } + PostCommitDeploy::VacatedPath { w_id, obj } => { + windmill_git_sync::tally_rename_vacated_path(db, &w_id, obj).await + } + } } } @@ -708,6 +769,7 @@ impl HandleDeploymentMetadata { async fn is_noop_deploy_against_parent( ns: &NewScript, parent: &Script, + resolved_on_behalf_of: Option<&str>, db: &DB, ) -> Result { if parent.archived || parent.deleted { @@ -750,7 +812,10 @@ async fn is_noop_deploy_against_parent( auto_kind: _, codebase, has_preprocessor, - on_behalf_of_email, + // both halves are folded into `resolved_on_behalf_of` before the comparison below, + // which is the identity that would actually be stored + on_behalf_of_email: _, + on_behalf_of: _, // caller-intent flag (permission preservation), not script state preserve_on_behalf_of: _, assets, @@ -769,7 +834,20 @@ async fn is_noop_deploy_against_parent( if content != &parent.content { return Ok(false); } - if normalize_optional_text(lock.as_deref()) != normalize_optional_text(parent.lock.as_deref()) { + // A dbt lock is DERIVED, never supplied: the request carries none — the CLI + // sends `lock: undefined` and this route discards any anyway — while a + // deployed parent carries what its dependency job wrote. Comparing them makes + // every unchanged push a new version, a fresh dependency job and the sync + // activity `skip_if_noop` exists to prevent. The parent must actually hold + // one, or a deploy whose dependency job failed could never be retried by + // pushing the same project again. + if matches!(language, ScriptLang::Dbt) { + if normalize_optional_text(parent.lock.as_deref()).is_empty() { + return Ok(false); + } + } else if normalize_optional_text(lock.as_deref()) + != normalize_optional_text(parent.lock.as_deref()) + { return Ok(false); } if summary != &parent.summary { @@ -815,10 +893,25 @@ async fn is_noop_deploy_against_parent( { return Ok(false); } - if on_behalf_of_email != &parent.on_behalf_of_email { + if resolved_on_behalf_of != parent.on_behalf_of.as_deref() { return Ok(false); } - if !schema_opt_eq(schema.as_ref(), parent.schema.as_ref()) { + // Both of a dbt script's derived fields are compared as they WOULD BE STORED, + // not as they arrived: the schema comes from the descriptor and the clients + // cannot derive it (`windmill-parser-wasm` has no dbt arm), so they send the + // previous version's or none at all. Comparing what they sent makes every + // unchanged push differ. + let dbt_schema = matches!(language, ScriptLang::Dbt) + .then(|| windmill_parser_yaml::dbt_arg_schema(content).ok()) + .flatten() + .and_then(|v| serde_json::value::to_raw_value(&v).ok()) + .map(|v| Schema(sqlx::types::Json(v))); + let effective_schema = if matches!(language, ScriptLang::Dbt) { + dbt_schema.as_ref() + } else { + schema.as_ref() + }; + if !schema_opt_eq(effective_schema, parent.schema.as_ref()) { return Ok(false); } if !json_serialize_eq(assets, &parent.assets) { @@ -905,7 +998,8 @@ async fn create_script_internal<'c>( ) -> Result<( ScriptHash, Transaction<'c, Postgres>, - Option, + Option, + Vec, )> { if authed.is_operator { return Err(Error::NotAuthorized( @@ -965,7 +1059,7 @@ async fn create_script_internal<'c>( // Apply folder default_permissioned_as the first time a script is deployed // at this path. Check inside the transaction to avoid TOCTOU with concurrent deploys. - let explicit_preserve = ns.on_behalf_of_email.is_some() + let explicit_preserve = (ns.on_behalf_of_email.is_some() || ns.on_behalf_of.is_some()) && ns.preserve_on_behalf_of.unwrap_or(false) && windmill_common::can_preserve_on_behalf_of(&authed); if !explicit_preserve && windmill_common::can_preserve_on_behalf_of(&authed) { @@ -978,17 +1072,33 @@ async fn create_script_internal<'c>( .await? .unwrap_or(false); if !path_already_exists { - if let Some(default_email) = - windmill_common::folders::resolve_folder_default_on_behalf_of_email( - &db, &w_id, &ns.path, - ) - .await? + if let Some((default_email, default_permissioned_as)) = + windmill_common::folders::resolve_folder_default_on_behalf_of(&db, &w_id, &ns.path) + .await? { ns.on_behalf_of_email = Some(default_email); + ns.on_behalf_of = Some(default_permissioned_as); ns.preserve_on_behalf_of = Some(true); } } } + let authed_principal = windmill_common::users::username_to_permissioned_as(&authed.username); + // Resolved here rather than at the INSERT so the no-op check below compares the identity + // that would actually be stored: the parent row holds only the principal, while a + // preserving push may name that same principal by address alone. + let resolved_on_behalf_of = windmill_common::resolve_on_behalf_of( + ns.on_behalf_of_email.as_deref(), + ns.on_behalf_of.as_deref(), + ns.preserve_on_behalf_of.unwrap_or(false), + &authed, + &w_id, + &db, + ) + .await?; + // Written beside the principal only while a worker that still reads it may be live. + let legacy_on_behalf_of_email = + windmill_common::legacy_on_behalf_of_email(resolved_on_behalf_of.as_deref(), &w_id, &db) + .await?; if sqlx::query_scalar!( "SELECT 1 FROM script WHERE hash = $1 AND workspace_id = $2", hash.0, @@ -1089,17 +1199,26 @@ async fn create_script_internal<'c>( // sync / promotion callbacks — the whole point is that idempotent // CLI pushes must not produce phantom commits on the downstream // git repository. - if skip_if_noop && is_noop_deploy_against_parent(&ns, &ps, &db).await? { + if skip_if_noop + && is_noop_deploy_against_parent(&ns, &ps, resolved_on_behalf_of.as_deref(), &db) + .await? + { tracing::info!( workspace_id = %w_id, path = %ns.path, parent_hash = %p_hash.0, "Skipping no-op script deploy (identical to parent)" ); - return Ok((p_hash.clone(), tx, None)); + return Ok((p_hash.clone(), tx, None, Vec::new())); } if ps.path != ns.path { + // A rename writes the source as much as the destination, and only the destination + // is scope-checked above. `require_owner_of_path` answers whether the *user* owns + // the source, never what their token is scoped to — so without this a path-scoped + // token could move a script it has no say over, taking its native triggers along + // and re-registering them under that token's identity. + check_scopes(&authed, || format!("scripts:write:{}", ps.path))?; require_owner_of_path(&authed, &ps.path)?; } @@ -1143,6 +1262,25 @@ async fn create_script_internal<'c>( .as_ref() .map(|v| v.perms.clone()) .unwrap_or(json!({})); + // A dbt lock names a manifest digest and engine versions that only a + // dependency job can determine, and that job is also what publishes the + // script's manifest graph. Honouring a supplied one would skip it, leaving + // the script with no graph — including on the UI paths that round-trip an + // existing lock, like rename and unarchive. + if matches!(ns.language, ScriptLang::Dbt) { + ns.lock = None; + // A codebase is a bundle of JS/TS sources; a dbt script's project is + // its modules. Accepting one takes the branch below that stands in for + // lock generation, which would suppress the very job that parses the + // project and publishes the graph. + if ns.codebase.is_some() { + return Err(Error::BadRequest( + "a dbt script has no codebase: its project is its modules, the \ + `", + "", + // Postgres' `\w` covers neither join controls nor combining marks; + // Rust's `\w` covers both, so `[\w-]` here would pass these through + // to the constraint and leak its message back to the caller. + "u/admin/a\u{200C}b", + "u/admin/a\u{0301}b", + "u/admin/a\u{203F}b", + ] { + assert!(check_proper_path(bad).is_err(), "{bad} should be rejected"); + } + assert!(check_proper_path(&format!("u/admin/{}", "a".repeat(300))).is_err()); + } + + #[test] + fn proper_type_name_matches_the_db_constraint() { + for ok in ["postgresql", "c_aws_account", "my-type", &"a".repeat(50)] { + assert!( + check_proper_type_name(ok).is_ok(), + "{ok} should be accepted" + ); + } + for bad in [ + "", + &"a".repeat(51), + "", + "a\u{200C}b", + ] { + assert!( + check_proper_type_name(bad).is_err(), + "{bad} should be rejected" + ); + } + } + #[test] fn truncate_handles_multibyte_at_boundary() { // Byte 25 of this string falls inside a 2-byte 'а'; naive `&s[..25]` would panic. diff --git a/backend/windmill-common/src/variables.rs b/backend/windmill-common/src/variables.rs index b43eb07e9c..8310051d75 100644 --- a/backend/windmill-common/src/variables.rs +++ b/backend/windmill-common/src/variables.rs @@ -192,14 +192,25 @@ pub async fn generate_approval_token( Ok(hex::encode(mac.finalize().into_bytes())) } -/// Stateless read-share signature for a job: `HMAC(workspace_key, job_id || "view_token")`. +/// Domain separator of the member-audience view token (see [`generate_view_token`]). +pub const VIEW_TOKEN_DOMAIN: &[u8] = b"view_token"; + +/// Domain separator of the public-audience view token (see [`generate_view_token`]). +/// Distinct from [`VIEW_TOKEN_DOMAIN`] so that already-shared member links keep granting +/// only what they were minted for: going public is always an explicit new mint. +pub const PUBLIC_VIEW_TOKEN_DOMAIN: &[u8] = b"public_view_token"; + +/// Stateless read-share signature for a job: `HMAC(workspace_key, job_id || domain)`. /// Mirrors [`generate_approval_token`] but in a distinct domain so an approval token can /// never be used as a view token (or vice-versa). Used to build a "share read link" that -/// grants an authenticated workspace member read access to a job (and its flow subtree) -/// they otherwise lack ACL on. No expiry/revocation (stateless), like the approval token. +/// grants read access to a job (and its flow subtree) to someone who otherwise lacks ACL +/// on it: an authenticated workspace member under [`VIEW_TOKEN_DOMAIN`], anyone holding +/// the link (logged in or not) under [`PUBLIC_VIEW_TOKEN_DOMAIN`]. No expiry/revocation +/// (stateless), like the approval token. pub async fn generate_view_token( w_id: &str, job_id: uuid::Uuid, + domain: &[u8], db: &DB, ) -> crate::error::Result { use hmac::{Hmac, Mac}; @@ -208,7 +219,7 @@ pub async fn generate_view_token( let mut mac = Hmac::::new_from_slice(key.as_bytes()) .map_err(|e| crate::Error::internal_err(format!("HMAC key error: {e}")))?; mac.update(job_id.as_bytes()); - mac.update(b"view_token"); + mac.update(domain); Ok(hex::encode(mac.finalize().into_bytes())) } @@ -470,7 +481,7 @@ pub async fn get_reserved_variables( ContextualVariable { name: "WM_END_USER_EMAIL".to_string(), value: end_user_email.unwrap_or_else(|| "".to_string()), - description: "Email of the end user that executed the current script. Only available when triggered from an app.".to_string(), + description: "Email of the end user that executed the current script, propagated to flow steps. Only set when the run was triggered from an app (empty otherwise).".to_string(), is_custom: false, }, ContextualVariable { diff --git a/backend/windmill-common/src/worker.rs b/backend/windmill-common/src/worker.rs index b9dd29c00a..acc474c868 100644 --- a/backend/windmill-common/src/worker.rs +++ b/backend/windmill-common/src/worker.rs @@ -217,6 +217,33 @@ pub const CONCURRENCY_KEY_MAX_QUEUED_DEFAULT: u32 = 10_000; /// the setting is cleared or malformed. A workspace spans many keys, so this sits well above /// the per-key cap. pub const WORKSPACE_MAX_QUEUED_JOBS_DEFAULT: u32 = 20_000; +/// Default for [`JOB_OOM_SCORE_ADJ`]; also the value used when the env var is out of range or +/// unparseable. +pub const JOB_OOM_SCORE_ADJ_DEFAULT: i32 = 1000; + +/// procfs accepts -1000..=1000, but a job must never be *less* killable than the worker that +/// supervises it, so negative adjustments are rejected rather than clamped. +fn parse_job_oom_score_adj(raw: Option<&str>) -> i32 { + let Some(raw) = raw else { + return JOB_OOM_SCORE_ADJ_DEFAULT; + }; + match raw.trim().parse::() { + Ok(v) if (0..=1000).contains(&v) => v, + Ok(v) => { + tracing::warn!( + "JOB_OOM_SCORE_ADJ={v} is outside the accepted 0..=1000 range, \ + using {JOB_OOM_SCORE_ADJ_DEFAULT}" + ); + JOB_OOM_SCORE_ADJ_DEFAULT + } + Err(e) => { + tracing::warn!( + "Could not parse JOB_OOM_SCORE_ADJ='{raw}': {e}, using {JOB_OOM_SCORE_ADJ_DEFAULT}" + ); + JOB_OOM_SCORE_ADJ_DEFAULT + } + } +} lazy_static::lazy_static! { pub static ref WORKER_GROUP: String = std::env::var("WORKER_GROUP").unwrap_or_else(|_| { #[cfg(not(feature = "enterprise"))] @@ -240,6 +267,15 @@ lazy_static::lazy_static! { pub static ref LIMIT_WINDOWS_TO_1CU: bool = std::env::var("LIMIT_WINDOWS_TO_1CU").ok().is_some_and(|x| x == "1" || x == "true"); + /// `oom_score_adj` applied to job subprocesses. The kernel adds it to the process's memory + /// use expressed in permille of host RAM, so the job only reliably outranks the worker once + /// the gap between their two adjustments exceeds the worker's own footprint in permille; the + /// default maximizes that margin. Userspace OOM daemons (earlyoom, systemd-oomd, nohang) rank + /// every process on the host by the same score, so at 1000 a tiny job outranks multi-GB + /// processes and gets killed first. Lowering this trades margin over the worker for a fairer + /// ranking against everything else on the host. + pub static ref JOB_OOM_SCORE_ADJ: i32 = parse_job_oom_score_adj(std::env::var("JOB_OOM_SCORE_ADJ").ok().as_deref()); + pub static ref CGROUP_V2_PATH_RE: Regex = Regex::new(r#"(?m)^0::(/.*)$"#).unwrap(); pub static ref CGROUP_V2_CPU_RE: Regex = Regex::new(r#"(?m)^(\d+) \S+$"#).unwrap(); pub static ref CGROUP_V1_INACTIVE_FILE_RE: Regex = Regex::new(r#"(?m)^total_inactive_file (\d+)$"#).unwrap(); @@ -269,6 +305,7 @@ lazy_static::lazy_static! { "ruby".to_string(), "rlang".to_string(), "duckdb".to_string(), + "dbt".to_string(), // for related places search: ADD_NEW_LANG "dependency".to_string(), "flow".to_string(), @@ -2441,6 +2478,26 @@ mod tests { ids.iter().map(|s| s.to_string()).collect() } + #[test] + fn test_parse_job_oom_score_adj() { + assert_eq!(parse_job_oom_score_adj(Some("300")), 300); + assert_eq!(parse_job_oom_score_adj(Some(" 0\n")), 0); + assert_eq!(parse_job_oom_score_adj(None), JOB_OOM_SCORE_ADJ_DEFAULT); + // Out of range and unparseable both fall back rather than weaken the worker's protection. + assert_eq!( + parse_job_oom_score_adj(Some("-500")), + JOB_OOM_SCORE_ADJ_DEFAULT + ); + assert_eq!( + parse_job_oom_score_adj(Some("1001")), + JOB_OOM_SCORE_ADJ_DEFAULT + ); + assert_eq!( + parse_job_oom_score_adj(Some("high")), + JOB_OOM_SCORE_ADJ_DEFAULT + ); + } + #[test] fn test_bash_sandbox_image_annotation() { // `# sandbox ` selects the container runtime and returns the image. diff --git a/backend/windmill-common/src/workspaces.rs b/backend/windmill-common/src/workspaces.rs index 2af5afaf77..d570e7ec32 100644 --- a/backend/windmill-common/src/workspaces.rs +++ b/backend/windmill-common/src/workspaces.rs @@ -70,6 +70,7 @@ bitflags::bitflags! { const DISABLE_WORKSPACE_FORKING = 1 << 1; const RESTRICT_DEPLOY_TO_DEPLOYERS = 1 << 2; const RESTRICT_ANONYMOUS_APP_DEPLOYMENT = 1 << 3; + const RESTRICT_PUBLIC_RUN_SHARING = 1 << 4; } } @@ -81,6 +82,7 @@ pub enum ProtectionRuleKind { DisableWorkspaceForking, RestrictDeployToDeployers, RestrictAnonymousAppDeployment, + RestrictPublicRunSharing, } impl ProtectionRuleKind { @@ -98,6 +100,9 @@ impl ProtectionRuleKind { ProtectionRuleKind::RestrictAnonymousAppDeployment => { ProtectionRules::RESTRICT_ANONYMOUS_APP_DEPLOYMENT } + ProtectionRuleKind::RestrictPublicRunSharing => { + ProtectionRules::RESTRICT_PUBLIC_RUN_SHARING + } } } @@ -113,6 +118,9 @@ impl ProtectionRuleKind { ProtectionRuleKind::RestrictAnonymousAppDeployment => { "Making an app publicly accessible without login (anonymous execution mode) is restricted in this workspace" } + ProtectionRuleKind::RestrictPublicRunSharing => { + "Sharing a run publicly (readable without login) is restricted in this workspace" + } } } } @@ -167,7 +175,7 @@ pub enum ObjectType { DatatableMigration, } -pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28809/sync-script-to-git-repo-windmill"; +pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28871/sync-script-to-git-repo-windmill"; /// Hub script that applies a repository's state back into a workspace /// (the repo → Windmill / "pull" direction). Same script the UI runs from @@ -175,7 +183,7 @@ pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28809/sync-script-to-git-repo /// ignores the slug, so the slug is kept free of characters that would be /// percent-encoded into the run URL (a `:` becomes `%3A`, which some hardened /// reverse proxies reject as double-encoding when the client re-encodes it). -pub const GIT_SYNC_PULL_SCRIPT_PATH: &str = "hub/28808/git-sync-init-repository-windmill"; +pub const GIT_SYNC_PULL_SCRIPT_PATH: &str = "hub/28890/git-sync-init-repository-windmill"; /// Prefix used to identify fork workspaces. A workspace whose id starts with this string is a /// fork of another workspace. @@ -310,7 +318,7 @@ fn is_false(b: &bool) -> bool { !*b } -#[derive(Serialize, Deserialize, Debug)] +#[derive(Serialize, Deserialize, Debug, Clone)] pub struct GitRepositorySettings { #[serde(skip_serializing_if = "Option::is_none")] pub exclude_types_override: Option>, @@ -508,7 +516,7 @@ impl AutoPullSettings { } } -#[derive(Serialize, Deserialize, Debug)] +#[derive(Serialize, Deserialize, Debug, Clone)] pub struct GitSyncSettings { pub include_path: Vec, pub include_type: Vec, @@ -1525,10 +1533,12 @@ pub async fn resolve_fork_branch_target( .fetch_optional(db) .await? } else if branch != expected_base { - // Environment-label branch (`dev`/`staging`) of a dev-workspace child. - // Dev workspaces only exist directly under a root, so no recursion here. - // The tracked-branch guard keeps a label that collides with the tracked - // branch from double-routing (the parent's own pull already covers it). + // Environment-label branch (`dev`, `staging`, ...) of a dev-workspace child. + // Direct children only: a dev nested under another dev is parent-managed + // by that dev, which holds no auto-pull config of its own, so no branch + // pushed to this repo routes to it. The tracked-branch guard keeps a + // label that collides with the tracked branch from double-routing (the + // parent's own pull already covers it). sqlx::query_scalar!( "SELECT id FROM workspace \ WHERE parent_workspace_id = $1 AND NOT deleted AND is_dev_workspace \ @@ -2477,4 +2487,144 @@ mod tests { assert!(msg.contains("staging"), "{msg}"); assert!(msg.contains("tab=windmill_data_tables"), "{msg}"); } + + #[test] + fn test_dbt_warehouse_name_length_bound() { + assert!(validate_dbt_warehouse_name(&"a".repeat(MAX_DBT_WAREHOUSE_NAME_LEN)).is_ok()); + assert!(validate_dbt_warehouse_name(&"a".repeat(MAX_DBT_WAREHOUSE_NAME_LEN + 1)).is_err()); + } +} + +/// Whether a workspace configures this warehouse, without resolving it. +/// +/// For the one caller that needs the NAME and nothing else: a project bringing +/// its own `profiles.yml` names a warehouse to say where its assets belong, and +/// decrypting a connection it will never open to answer that would be waste. +/// +/// NO AUTHORIZATION, like the resolver it delegates to: it reads workspace +/// settings for whatever `w_id` it is given, so callers MUST already be scoped +/// to that workspace. +pub async fn dbt_warehouse_exists(db: &DB, w_id: &str, warehouse: &str) -> Result<()> { + dbt_warehouse_resource(db, w_id, warehouse) + .await + .map(|_| ()) +} + +/// The longest warehouse name a workspace may configure. +/// +/// It is the first segment of every `dbt:////` key, and +/// those land in `asset.path`, a VARCHAR(255). A name near that width would push +/// ordinary relations past the column, and the ingest's own bound would then +/// skip them — a graph quietly missing models. 64 leaves room for two 63-char +/// identifiers, which is Postgres's own limit; warehouses that allow longer ones +/// still rely on that ingest bound as the backstop. +pub const MAX_DBT_WAREHOUSE_NAME_LEN: usize = 64; + +/// A warehouse name is a URL path segment for a worker with no database, so a +/// name that could re-cut the path (or the query) is refused — at every place a +/// name enters, not only where one is written, since a descriptor names one too. +pub fn validate_dbt_warehouse_name(name: &str) -> Result<()> { + if name.is_empty() + || !name + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') + { + return Err(Error::BadRequest(format!( + "`{name}` is not a usable dbt warehouse name: use letters, digits, `_` and `-`" + ))); + } + if name.chars().count() > MAX_DBT_WAREHOUSE_NAME_LEN { + return Err(Error::BadRequest(format!( + "`{name}` is too long for a dbt warehouse name (max {MAX_DBT_WAREHOUSE_NAME_LEN}): it \ + prefixes every asset path this warehouse's models are keyed on" + ))); + } + Ok(()) +} + +/// A dbt warehouse's resolved connection: the value a `profiles.yml` is rendered +/// from, and the target the workspace names for it. +/// +/// Carries CREDENTIALS. dbt is unpermissioned by design (docs/dbt-runtime.md, +/// Decision 24), so this is served to a running job rather than gated on the +/// runner's access to the resource — but it must never reach a route a user can +/// browse. +#[derive(serde::Serialize, serde::Deserialize)] +pub struct DbtWarehouseConnection { + pub value: serde_json::Value, + /// Omitted rather than null when absent, so the wire shape matches the + /// schema generated clients validate against. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub target: Option, + /// What the value IS, which its shape cannot say: a `dbt_profile`'s value is a + /// `profiles.yml` output block and every other type's is a connection to translate, + /// and both are objects carrying a `type`. Defaulted so a worker still resolves + /// against a server predating the field — which serves no `dbt_profile` anyway. + #[serde(default)] + pub resource_type: String, +} + +/// The resource type whose value is a `profiles.yml` output block, taken as it +/// is rather than translated. +pub const DBT_PROFILE_RESOURCE_TYPE: &str = "dbt_profile"; + +/// The warehouse a dbt project runs against, by name — `main` when the +/// descriptor names none. +/// +/// NO AUTHORIZATION: reads workspace settings for whatever `w_id` it is given. +/// Callers MUST already be scoped to that workspace — a running job, or a route +/// that checked its token. What it returns is a POINTER; resolving it is a +/// separate step, and dbt warehouses are unpermissioned there (Decision 24), so +/// the pointer is not the last line of defence. A descriptor cannot name a +/// resource at all, which is what makes the workspace the only place a +/// warehouse is configured — and +/// what lets asset identity key on the NAME (`dbt://main/analytics/orders`), +/// one spelling every project on that warehouse shares. +pub async fn dbt_warehouse_resource( + db: &DB, + w_id: &str, + warehouse: &str, +) -> Result<(String, Option)> { + let cfg = sqlx::query_scalar!( + "SELECT dbt_warehouses FROM workspace_settings WHERE workspace_id = $1", + w_id + ) + .fetch_optional(db) + .await? + .flatten() + .unwrap_or(serde_json::Value::Null); + let entry = cfg.get(warehouse).cloned().ok_or_else(|| { + let configured = cfg + .as_object() + .map(|o| o.keys().cloned().collect::>()) + .unwrap_or_default(); + Error::NotFound(if configured.is_empty() { + format!( + "no dbt warehouse is configured for this workspace — an admin adds one under \ + Settings → dbt, and a project reaches it by name (`{warehouse}` here)" + ) + } else { + format!( + "dbt warehouse `{warehouse}` is not configured for this workspace (configured: \ + {})", + configured.join(", ") + ) + }) + })?; + let path = entry + .get("resource_path") + .and_then(|p| p.as_str()) + .ok_or_else(|| { + Error::internal_err(format!( + "dbt warehouse `{warehouse}` names no resource_path" + )) + })?; + // Stored with or without the prefix, like the storage configs; the caller + // resolves a bare path. + let path = path.strip_prefix("$res:").unwrap_or(path).to_string(); + let target = entry + .get("target") + .and_then(|t| t.as_str()) + .map(|t| t.to_string()); + Ok((path, target)) } diff --git a/backend/windmill-common/tests/dbt_graph_storage.rs b/backend/windmill-common/tests/dbt_graph_storage.rs new file mode 100644 index 0000000000..7b4971312a --- /dev/null +++ b/backend/windmill-common/tests/dbt_graph_storage.rs @@ -0,0 +1,554 @@ +//! The dbt graph's storage invariants, against a real database. +//! +//! Every defect found in this area across review was DB-shaped — which row set +//! a read resolves to, which rows a clear takes, whether a snapshot exists at +//! all — and none of it is reachable from a unit test on a pure function. These +//! pin the answers that took several rounds to settle. + +use sqlx::{Pool, Postgres}; +use windmill_common::dbt_manifest::{ + clear_dbt_editor_graphs, clear_dbt_manifest_version, prune_dbt_run_graphs, + replace_dbt_editor_graph, replace_dbt_manifest, IngestedManifest, IngestedNode, + DBT_EDITOR_GRAPHS_KEPT, DEPLOYED_GRAPH, DEPLOYED_GRAPH_VERSIONS_KEPT, +}; + +const WS: &str = "test-workspace"; +const PATH: &str = "f/test/proj"; +/// The principal an editor parse runs as. Retention is bounded per identity, +/// so the tests have to name one. +const ME: &str = "u/me"; + +async fn deploy_script(db: &Pool, hash: i64) { + sqlx::query!( + "INSERT INTO script (workspace_id, hash, path, summary, description, content, created_by, + language, lock) + VALUES ($1, $2, $3, '', '', 'profile: {}', 'test-user', 'dbt', '')", + WS, + hash, + PATH, + ) + .execute(db) + .await + .unwrap(); +} + +fn manifest(names: &[&str]) -> IngestedManifest { + IngestedManifest { + nodes: names + .iter() + .map(|n| IngestedNode { + unique_id: format!("model.p.{n}"), + resource_type: "model".to_string(), + name: n.to_string(), + asset_path: Some(format!("u/a/wh/s/{n}")), + raw_code: Some(format!("select 1 as {n}")), + ..Default::default() + }) + .collect(), + // Chained, so the edge insert actually runs: `ref()` lineage is in every + // real project, and a fixture with no edges leaves that statement + // unexecuted by every test in this file. + edges: names + .windows(2) + .map(|w| (format!("model.p.{}", w[0]), format!("model.p.{}", w[1]))) + .collect(), + ..Default::default() + } +} + +/// Edges for one version, so a test can assert the batched insert ran at all. +async fn edges_for(db: &Pool, hash: i64) -> i64 { + sqlx::query_scalar!( + "SELECT count(*) FROM dbt_edge WHERE workspace_id = $1 AND script_hash = $2", + WS, + hash + ) + .fetch_one(db) + .await + .unwrap() + .unwrap_or(0) +} + +async fn nodes_for(db: &Pool, hash: i64, job: uuid::Uuid) -> i64 { + sqlx::query_scalar!( + "SELECT count(*) FROM dbt_node WHERE workspace_id = $1 AND script_hash = $2 AND job_id = $3", + WS, + hash, + job + ) + .fetch_one(db) + .await + .unwrap() + .unwrap_or(0) +} + +async fn markers_for_path(db: &Pool) -> i64 { + sqlx::query_scalar!( + "SELECT count(*) FROM dbt_graph_snapshot WHERE workspace_id = $1 AND script_path = $2", + WS, + PATH + ) + .fetch_one(db) + .await + .unwrap() + .unwrap_or(0) +} + +async fn markers(db: &Pool, hash: i64) -> i64 { + sqlx::query_scalar!( + "SELECT count(*) FROM dbt_graph_snapshot WHERE workspace_id = $1 AND script_hash = $2", + WS, + hash + ) + .fetch_one(db) + .await + .unwrap() + .unwrap_or(0) +} + +/// A run whose graph matches the version's stores nothing. Marking a descriptor +/// dynamic is conservative — a `{{ }}` in `vars` says the arguments reach dbt, +/// not that they change which models exist — so the usual dynamic run resolves +/// to exactly the deployed graph, and storing a copy per run is what filled the +/// table with an unchanging picture. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn an_identical_run_stores_no_snapshot(db: Pool) { + deploy_script(&db, 1).await; + let m = manifest(&["a", "b"]); + let job = uuid::Uuid::from_u128(7); + + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &m, "root") + .await + .unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, Some(job), &m, "root") + .await + .unwrap(); + tx.commit().await.unwrap(); + + assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 2); + assert_eq!( + nodes_for(&db, 1, job).await, + 0, + "identical run must not snapshot" + ); + assert_eq!(markers(&db, 1).await, 1, "and leaves no marker of its own"); +} + +/// A run whose model set differs keeps its own, and the version's is untouched: +/// this is what lets an older run page render the project that run built. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn a_differing_run_keeps_its_own_snapshot(db: Pool) { + deploy_script(&db, 1).await; + let job = uuid::Uuid::from_u128(7); + + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &manifest(&["a"]), "root") + .await + .unwrap(); + replace_dbt_manifest( + &mut tx, + WS, + PATH, + 1, + Some(job), + &manifest(&["a", "b"]), + "root", + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + + assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 1); + assert_eq!(nodes_for(&db, 1, job).await, 2); +} + +/// A dynamic run can disable every model. That empty graph is an answer, and it +/// has to be distinguishable from a run that stored nothing — otherwise the run +/// page falls back to the deployed models and shows what the run never built. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn an_empty_run_graph_is_still_a_snapshot(db: Pool) { + deploy_script(&db, 1).await; + let job = uuid::Uuid::from_u128(7); + + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &manifest(&["a"]), "root") + .await + .unwrap(); + replace_dbt_manifest( + &mut tx, + WS, + PATH, + 1, + Some(job), + &IngestedManifest::default(), + "root", + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + + assert_eq!(nodes_for(&db, 1, job).await, 0); + let marker = sqlx::query_scalar!( + "SELECT count(*) FROM dbt_graph_snapshot WHERE workspace_id = $1 AND job_id = $2", + WS, + job + ) + .fetch_one(&db) + .await + .unwrap() + .unwrap_or(0); + assert_eq!(marker, 1, "an empty graph still has a marker"); +} + +/// Archiving or deleting ONE version must leave every other version's graph +/// alone: the routes act on a single hash, and the other versions stay live and +/// keep needing their own models, SQL and lineage. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn clearing_one_version_leaves_the_others(db: Pool) { + deploy_script(&db, 1).await; + deploy_script(&db, 2).await; + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &manifest(&["a"]), "root") + .await + .unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 2, None, &manifest(&["a", "b"]), "root") + .await + .unwrap(); + tx.commit().await.unwrap(); + + let mut tx = db.begin().await.unwrap(); + clear_dbt_manifest_version(&mut tx, WS, PATH, 1) + .await + .unwrap(); + tx.commit().await.unwrap(); + + assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 0); + assert_eq!(markers(&db, 1).await, 0, "the marker goes with the rows"); + assert_eq!( + nodes_for(&db, 2, DEPLOYED_GRAPH).await, + 2, + "the other version survives" + ); + assert_eq!(markers(&db, 2).await, 1); + // The lineage is stored and cleared with its nodes. Asserted here because + // this is where two versions coexist: it pins the batched edge insert + // against a real database as well as the version scoping. + assert_eq!(edges_for(&db, 1).await, 0, "the cleared version's edges go"); + assert_eq!(edges_for(&db, 2).await, 1, "the other version keeps its own"); +} + +/// The routes that hard-delete a path clear no graph rows: they delete the +/// `script` rows and let `ON DELETE CASCADE` take the sidecars, since taking +/// those first would lock them ahead of the script row and deadlock a concurrent +/// publication. So the cascade has to reach every sidecar, snapshots and markers +/// included. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn deleting_the_script_cascades_to_every_sidecar(db: Pool) { + deploy_script(&db, 1).await; + deploy_script(&db, 2).await; + let job = uuid::Uuid::from_u128(7); + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &manifest(&["a", "b"]), "root") + .await + .unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, Some(job), &manifest(&["a"]), "root") + .await + .unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 2, None, &manifest(&["b", "c"]), "root") + .await + .unwrap(); + tx.commit().await.unwrap(); + assert_eq!(markers_for_path(&db).await, 3, "two versions and one run"); + + sqlx::query!( + "DELETE FROM script WHERE workspace_id = $1 AND path = $2", + WS, + PATH + ) + .execute(&db) + .await + .unwrap(); + + assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 0); + assert_eq!(nodes_for(&db, 1, job).await, 0, "a run's snapshot goes too"); + assert_eq!(nodes_for(&db, 2, DEPLOYED_GRAPH).await, 0); + assert_eq!(edges_for(&db, 1).await, 0); + assert_eq!(edges_for(&db, 2).await, 0); + assert_eq!(markers_for_path(&db).await, 0); +} + +/// The sweep ages out run snapshots and never a version's own graph, which is +/// what its runs render from for as long as the version exists. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn the_sweep_takes_old_snapshots_and_spares_the_version(db: Pool) { + deploy_script(&db, 1).await; + let old = uuid::Uuid::from_u128(7); + let recent = uuid::Uuid::from_u128(8); + + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &manifest(&["a"]), "root") + .await + .unwrap(); + replace_dbt_manifest( + &mut tx, + WS, + PATH, + 1, + Some(old), + &manifest(&["a", "b"]), + "root", + ) + .await + .unwrap(); + replace_dbt_manifest( + &mut tx, + WS, + PATH, + 1, + Some(recent), + &manifest(&["a", "c"]), + "root", + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + + // Age one snapshot past the window, rows and marker together. + for t in ["dbt_node", "dbt_edge", "dbt_graph_snapshot"] { + sqlx::query(&format!( + "UPDATE {t} SET ingested_at = now() - interval '400 days' WHERE job_id = $1" + )) + .bind(old) + .execute(&db) + .await + .unwrap(); + } + + prune_dbt_run_graphs(&db, PATH, WS).await.unwrap(); + + assert_eq!(nodes_for(&db, 1, old).await, 0, "the aged snapshot goes"); + assert_eq!(nodes_for(&db, 1, recent).await, 2, "the recent one stays"); + assert_eq!( + nodes_for(&db, 1, DEPLOYED_GRAPH).await, + 1, + "and a version's own graph is never swept" + ); +} + +/// A version's graph cannot expire on a clock — a run page is as old as its job +/// — so growth is bounded by deploy COUNT instead. The newest deploys keep +/// theirs; past that the oldest are reclaimed, which is what stops a CI that +/// deploys on every commit from adding a model set per commit forever. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn only_the_newest_deploys_keep_their_graph(db: Pool) { + let over = DEPLOYED_GRAPH_VERSIONS_KEPT + 3; + for h in 1..=over { + deploy_script(&db, h).await; + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, h, None, &manifest(&["a"]), "root") + .await + .unwrap(); + tx.commit().await.unwrap(); + } + assert_eq!(markers_for_path(&db).await, over, "every deploy stored one"); + + prune_dbt_run_graphs(&db, PATH, WS).await.unwrap(); + + assert_eq!( + markers_for_path(&db).await, + DEPLOYED_GRAPH_VERSIONS_KEPT, + "the bound holds" + ); + // The newest is always among them: losing the live version's graph would + // empty the page of every run of it. + assert_eq!(nodes_for(&db, over, DEPLOYED_GRAPH).await, 1); + assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 0, "the oldest is reclaimed"); +} + +/// The third provenance: a `parse` of the EDITOR's buffer, which names no +/// deployed version. It cannot borrow the deployed hash — the buffer differs +/// from it, and a project being written may have no deployed version at all — so +/// it is keyed to the parse job alone and leaves the version's graph untouched. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn an_editor_parse_is_keyed_to_its_job_and_no_version(db: Pool) { + deploy_script(&db, 1).await; + let job = uuid::Uuid::from_u128(7); + + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &manifest(&["a"]), "root") + .await + .unwrap(); + replace_dbt_editor_graph(&mut tx, WS, PATH, job, ME, &manifest(&["a", "b"]), "root") + .await + .unwrap(); + tx.commit().await.unwrap(); + + assert_eq!( + editor_nodes(&db, job).await, + 2, + "the buffer's own models are stored" + ); + assert_eq!( + nodes_for(&db, 1, DEPLOYED_GRAPH).await, + 1, + "and the deployed version's graph is untouched" + ); + assert_eq!( + nodes_for(&db, 1, job).await, + 0, + "nothing is attributed to the version" + ); +} + +/// A buffer identical to the deploy still stores a graph, unlike a run's +/// snapshot: the editor pins to its own job, so suppressing the write would +/// leave it nothing to pin to and its provenance label would claim a parse that +/// is not on screen. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn an_editor_parse_matching_the_deploy_still_stores(db: Pool) { + deploy_script(&db, 1).await; + let job = uuid::Uuid::from_u128(7); + let m = manifest(&["a"]); + + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &m, "root") + .await + .unwrap(); + replace_dbt_editor_graph(&mut tx, WS, PATH, job, ME, &m, "root") + .await + .unwrap(); + tx.commit().await.unwrap(); + + assert_eq!(editor_nodes(&db, job).await, 1); +} + +/// Bounded per (path, principal) as each refresh lands: the ones before it are +/// dead the moment a newer parse arrives, and a click that stored a full model +/// set forever would be the editor's own leak. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn only_the_newest_editor_parses_are_kept(db: Pool) { + let over = DBT_EDITOR_GRAPHS_KEPT + 2; + for i in 1..=over { + let mut tx = db.begin().await.unwrap(); + replace_dbt_editor_graph( + &mut tx, + WS, + PATH, + uuid::Uuid::from_u128(i as u128), + ME, + &manifest(&["a"]), + "root", + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + assert_eq!( + editor_markers(&db).await, + DBT_EDITOR_GRAPHS_KEPT, + "the bound holds" + ); + assert_eq!( + editor_nodes(&db, uuid::Uuid::from_u128(over as u128)).await, + 1, + "the newest is always among them" + ); + assert_eq!( + editor_nodes(&db, uuid::Uuid::from_u128(1)).await, + 0, + "and its rows go with its marker" + ); +} + +/// Archiving or deleting ONE version must not take an editor's graph with it — +/// that graph describes a buffer, not a version. +/// +/// Retiring the whole PATH must, and cannot rely on the cascade the versioned +/// rows ride: a version-less row leaves `script_hash` NULL, which satisfies the +/// composite foreign key without referencing anything, so deleting the script +/// takes the version's graph and leaves the editor's behind. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn a_version_clear_spares_editor_graphs_and_a_path_clear_does_not(db: Pool) { + deploy_script(&db, 1).await; + let job = uuid::Uuid::from_u128(7); + + let mut tx = db.begin().await.unwrap(); + replace_dbt_manifest(&mut tx, WS, PATH, 1, None, &manifest(&["a"]), "root") + .await + .unwrap(); + replace_dbt_editor_graph(&mut tx, WS, PATH, job, ME, &manifest(&["a"]), "root") + .await + .unwrap(); + clear_dbt_manifest_version(&mut tx, WS, PATH, 1).await.unwrap(); + tx.commit().await.unwrap(); + + assert_eq!(nodes_for(&db, 1, DEPLOYED_GRAPH).await, 0); + assert_eq!(editor_nodes(&db, job).await, 1, "the buffer's graph survives"); + + let mut tx = db.begin().await.unwrap(); + clear_dbt_editor_graphs(&mut tx, WS, PATH).await.unwrap(); + tx.commit().await.unwrap(); + + assert_eq!(editor_nodes(&db, job).await, 0, "retiring the path takes it"); +} + +/// A preview names its own PATH and needs only `jobs:run`, so a bound over the +/// path alone is a way to retire the graphs of whoever is actually editing that +/// script. Each identity reclaims only its own. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn one_principal_cannot_evict_another_s_editor_graphs(db: Pool) { + let mine = uuid::Uuid::from_u128(1); + let mut tx = db.begin().await.unwrap(); + replace_dbt_editor_graph(&mut tx, WS, PATH, mine, ME, &manifest(&["a"]), "root") + .await + .unwrap(); + // Someone else, parsing the same path far more often than the bound allows. + for i in 0..(DBT_EDITOR_GRAPHS_KEPT + 3) { + replace_dbt_editor_graph( + &mut tx, + WS, + PATH, + uuid::Uuid::from_u128(100 + i as u128), + "u/squatter", + &manifest(&["a"]), + "root", + ) + .await + .unwrap(); + } + tx.commit().await.unwrap(); + + assert_eq!( + editor_nodes(&db, mine).await, + 1, + "another principal's parses must not retire this one's graph" + ); +} + +/// Nodes of one editor parse. +async fn editor_nodes(db: &Pool, job: uuid::Uuid) -> i64 { + sqlx::query_scalar!( + "SELECT count(*) FROM dbt_node + WHERE workspace_id = $1 AND job_id = $2 AND script_hash IS NULL", + WS, + job + ) + .fetch_one(db) + .await + .unwrap() + .unwrap_or(0) +} + +async fn editor_markers(db: &Pool) -> i64 { + sqlx::query_scalar!( + "SELECT count(*) FROM dbt_graph_snapshot + WHERE workspace_id = $1 AND script_path = $2 AND script_hash IS NULL", + WS, + PATH + ) + .fetch_one(db) + .await + .unwrap() + .unwrap_or(0) +} diff --git a/backend/windmill-dep-map/src/trigger_dependents.rs b/backend/windmill-dep-map/src/trigger_dependents.rs index 1254e41ed1..51a7fd4b90 100644 --- a/backend/windmill-dep-map/src/trigger_dependents.rs +++ b/backend/windmill-dep-map/src/trigger_dependents.rs @@ -188,6 +188,7 @@ pub async fn trigger_dependents_to_recompute_dependencies( email, permissioned_as.to_string(), Some("trigger.dependents.to.recompute.dependencies"), + None, Some(Utc::now() + Duration::seconds(*DEPENDENCY_JOB_DEBOUNCE_DELAY as i64)), None, None, diff --git a/backend/windmill-git-sync/src/git_sync_oss.rs b/backend/windmill-git-sync/src/git_sync_oss.rs index 5cf1995525..2e04033f35 100644 --- a/backend/windmill-git-sync/src/git_sync_oss.rs +++ b/backend/windmill-git-sync/src/git_sync_oss.rs @@ -29,6 +29,7 @@ pub async fn tally_deployed_object_changes( _obj: &DeployedObject, _db: &DB, _renamed_from: Option<&str>, + _origin: Option, ) -> Result<()> { // Workspace forks are an enterprise feature and not part of the open-source version return Ok(()); diff --git a/backend/windmill-git-sync/src/lib.rs b/backend/windmill-git-sync/src/lib.rs index 1f6e8f5811..2bfc7d2f77 100644 --- a/backend/windmill-git-sync/src/lib.rs +++ b/backend/windmill-git-sync/src/lib.rs @@ -162,6 +162,27 @@ impl DeployedObject { } } + /// The repo-relative path git sync syncs on: what the sync item carries, and + /// therefore what the CLI turns into `--extra-includes` globs and `git add` + /// pathspecs. + /// + /// It only differs from [`Self::get_path`] for data table migrations, whose + /// object path (`/_`, the `workspace_diff` key) is + /// relative to the `migrations/datatable/` prefix the workspace export writes + /// them under. Without the prefix the CLI derives globs that match no file, + /// so nothing gets staged and the push silently commits nothing. + pub fn get_git_sync_path(&self) -> String { + match self { + DeployedObject::DatatableMigration { path } => { + format!("migrations/datatable/{path}") + } + _ => self.get_path(), + } + } + + /// Whether the object skips the repo's include/exclude path filters. True for + /// every kind that lives outside the `f/`/`u/` path namespaces those filters + /// are written against — the object-type filter is what governs them instead. pub fn get_ignore_regex_filter(&self) -> bool { match self { Self::User { .. } @@ -169,7 +190,8 @@ impl DeployedObject { | Self::ResourceType { .. } | Self::Settings { .. } | Self::Key { .. } - | Self::WorkspaceDependencies { .. } => true, + | Self::WorkspaceDependencies { .. } + | Self::DatatableMigration { .. } => true, _ => false, } } @@ -238,6 +260,165 @@ impl DeployedObject { } } +/// Record, from the request that made it, that a rename left `vacated` empty. +/// Only its path and kind are read, so build it naming the path the rename left. +/// +/// A deploy whose metadata is handled by its dependency job needs this: that job +/// runs at an unknown remove from the write, and the row it would write is +/// deleted as soon as the two workspaces agree on the path — so a claim it made +/// could reappear later against an item the parent has since recreated. Call +/// once the rename has committed; the tally reads what the path holds now. +pub async fn tally_rename_vacated_path( + db: &DB, + w_id: &str, + vacated: DeployedObject, +) -> windmill_common::error::Result<()> { + tally_deployed_object_changes( + w_id, + &vacated, + db, + None, + windmill_common::deploy_origin::current(), + ) + .await +} + +/// Item kinds whose `workspace_diff` path is the `path` column of a table named +/// after the kind. Interpolated into SQL, so it must stay a hardcoded allowlist — +/// and, unlike the `sqlx::query!` arms below, a wrong name here is not a compile +/// error, so `workspace_comparison.rs` sweeps this list against a live database. +pub const PATH_KEYED_TABLES: &[&str] = &[ + "resource", + "variable", + "schedule", + "http_trigger", + "websocket_trigger", + "kafka_trigger", + "nats_trigger", + "postgres_trigger", + "mqtt_trigger", + "amqp_trigger", + "sqs_trigger", + "gcp_trigger", + "azure_trigger", + "email_trigger", +]; + +/// What the deploy event that just committed did to `path`: [`DeployEventKind::Write`] +/// if the workspace still holds an item there, [`DeployEventKind::Delete`] if it does +/// not. `None` for a kind this does not know how to probe, so an unmapped kind is +/// recorded as no evidence rather than as a deletion. +/// +/// Existence mirrors the comparison's (`compare_two_*`): an archived script or flow +/// counts as absent. Runs on `&db` (no RLS) — the caller is a post-commit tally, not +/// a user-facing read. +pub async fn probe_deploy_event_kind( + db: &DB, + w_id: &str, + kind: &str, + path: &str, +) -> windmill_common::error::Result> { + use windmill_common::deploy_origin::DeployEventKind; + + let exists: Option = match kind { + "script" => Some( + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM script WHERE workspace_id = $1 AND path = $2 AND archived = false)", + w_id, + path + ) + .fetch_one(db) + .await? + .unwrap_or(false), + ), + "flow" => Some( + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM flow WHERE workspace_id = $1 AND path = $2 AND archived = false)", + w_id, + path + ) + .fetch_one(db) + .await? + .unwrap_or(false), + ), + // Raw apps live in the `app` table too, keyed by path like a regular app. + "app" | "raw_app" => Some( + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM app WHERE workspace_id = $1 AND path = $2)", + w_id, + path + ) + .fetch_one(db) + .await? + .unwrap_or(false), + ), + "folder" => { + let name = path.strip_prefix("f/").unwrap_or(path); + Some( + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM folder WHERE workspace_id = $1 AND name = $2)", + w_id, + name + ) + .fetch_one(db) + .await? + .unwrap_or(false), + ) + } + "resource_type" => Some( + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = $1 AND name = $2)", + w_id, + path + ) + .fetch_one(db) + .await? + .unwrap_or(false), + ), + // Identified by (datatable, timestamp), which survives a rename of the + // migration's `name` segment; the full path does not. + "datatable_migration" => match path + .split_once('/') + .and_then(|(dt, rest)| Some((dt, rest.split_once('_')?.0.parse::().ok()?))) + { + Some((datatable, timestamp)) => Some( + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM datatable_migrations \ + WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3)", + w_id, + datatable, + timestamp + ) + .fetch_one(db) + .await? + .unwrap_or(false), + ), + None => None, + }, + k if PATH_KEYED_TABLES.contains(&k) => { + // SAFETY: `k` comes from the hardcoded PATH_KEYED_TABLES allowlist. + let sql = + format!("SELECT EXISTS(SELECT 1 FROM {k} WHERE workspace_id = $1 AND path = $2)"); + Some( + sqlx::query_scalar(&sql) + .bind(w_id) + .bind(path) + .fetch_one(db) + .await?, + ) + } + _ => None, + }; + + Ok(exists.map(|exists| { + if exists { + DeployEventKind::Write + } else { + DeployEventKind::Delete + } + })) +} + #[cfg(test)] mod tests { use super::*; @@ -297,6 +478,32 @@ mod tests { assert_eq!(obj.get_path(), "workspace-dependencies/python"); } + // --- DeployedObject::get_git_sync_path tests --- + + #[test] + fn test_get_git_sync_path_datatable_migration_is_repo_relative() { + let obj = DeployedObject::DatatableMigration { + path: "mydb/20260101000000_add_users".to_string(), + }; + // The object path keys `workspace_diff`; the git-sync path must carry the + // export's prefix or the CLI stages nothing. + assert_eq!(obj.get_path(), "mydb/20260101000000_add_users"); + assert_eq!( + obj.get_git_sync_path(), + "migrations/datatable/mydb/20260101000000_add_users" + ); + } + + #[test] + fn test_get_git_sync_path_defaults_to_object_path() { + let obj = DeployedObject::Script { + hash: ScriptHash(123), + path: "f/folder/script".to_string(), + parent_path: None, + }; + assert_eq!(obj.get_git_sync_path(), obj.get_path()); + } + // --- DeployedObject::get_ignore_regex_filter tests --- #[test] @@ -337,6 +544,17 @@ mod tests { assert!(obj.get_ignore_regex_filter()); } + #[test] + fn test_ignore_regex_filter_datatable_migration() { + // `migrations/datatable/**` is outside the `f/`/`u/` namespaces the path + // filters are written against; the `datatablemigration` include type is + // what governs these. + let obj = DeployedObject::DatatableMigration { + path: "mydb/20260101000000_add_users".to_string(), + }; + assert!(obj.get_ignore_regex_filter()); + } + #[test] fn test_ignore_regex_filter_script() { let obj = DeployedObject::Script { diff --git a/backend/windmill-mcp/Cargo.toml b/backend/windmill-mcp/Cargo.toml index 36e0d03d18..8ac23e885d 100644 --- a/backend/windmill-mcp/Cargo.toml +++ b/backend/windmill-mcp/Cargo.toml @@ -17,7 +17,7 @@ auth = ["rmcp/auth", "dep:oauth2", "dep:sqlx", "dep:chrono"] oauth2 = { version = "5.0", optional = true } windmill-common = { workspace = true, default-features = false } anyhow.workspace = true -reqwest = { version = "=0.12", features = ["json", "stream", "gzip"] } +reqwest.workspace = true serde.workspace = true serde_json.workspace = true tracing.workspace = true diff --git a/backend/windmill-mcp/src/client/mod.rs b/backend/windmill-mcp/src/client/mod.rs index f61a1c3f85..863d6b0355 100644 --- a/backend/windmill-mcp/src/client/mod.rs +++ b/backend/windmill-mcp/src/client/mod.rs @@ -85,10 +85,7 @@ impl McpClient { // and does not legitimately rely on redirects. .redirect(reqwest::redirect::Policy::none()); // Pin DNS to the address validated above so the connect cannot rebind to - // an internal IP between the check and the request. `apply_dns_pinning` - // lives on windmill-common's reqwest, but this crate resolves a - // different reqwest version (via rmcp), so pin directly with the - // std-typed host/addrs the validation surfaced. Empty addrs (IP literal + // an internal IP between the check and the request. Empty addrs (IP literal // or ALLOW_PRIVATE_MCP_SERVER_URLS) leave resolution untouched. if !validated.addrs.is_empty() { client_builder = client_builder.resolve_to_addrs(&validated.host, &validated.addrs); @@ -102,19 +99,11 @@ impl McpClient { let transport = StreamableHttpClientTransport::with_client(reqwest_client, config); // Set up client info - let client_info = ClientInfo { - protocol_version: Default::default(), - capabilities: ClientCapabilities::default(), - client_info: Implementation { - name: "windmill-ai-agent".to_string(), - title: Some("Windmill AI Agent".to_string()), - version: env!("CARGO_PKG_VERSION").to_string(), - description: None, - website_url: None, - icons: None, - }, - meta: None, - }; + let client_info = ClientInfo::new( + ClientCapabilities::default(), + Implementation::new("windmill-ai-agent", env!("CARGO_PKG_VERSION")) + .with_title("Windmill AI Agent"), + ); // Initialize the connection let client = client_info @@ -143,14 +132,14 @@ impl McpClient { let mcp_args = Self::openai_args_to_mcp_args(arguments).context("Failed to parse tool arguments")?; + let mut params = CallToolRequestParams::new(name.to_string()); + if let Some(args) = mcp_args { + params = params.with_arguments(args); + } + let result = self .client - .call_tool(CallToolRequestParams { - name: name.to_string().into(), - arguments: mcp_args, - task: None, - meta: None, - }) + .call_tool(params) .await .context(format!("Failed to call MCP tool: {}", name))?; diff --git a/backend/windmill-mcp/src/client_registration.rs b/backend/windmill-mcp/src/client_registration.rs index 637b76df80..3aa39e840b 100644 --- a/backend/windmill-mcp/src/client_registration.rs +++ b/backend/windmill-mcp/src/client_registration.rs @@ -170,7 +170,7 @@ pub async fn get_or_refresh_mcp_client( .map_err(|e| error::Error::BadRequest(format!("Failed to create auth manager: {e}")))?; // Discovery hits the well-known endpoint on the MCP server host validated // above; pin to that address so it cannot rebind between check and connect. - // Limitation: rmcp's discover_metadata may additionally follow server-supplied + // Limitation: rmcp's resolve_metadata may additionally follow server-supplied // metadata URLs (resource_metadata / authorization_servers) on other hosts, // which this per-host pin does not cover — a pre-existing gap in rmcp discovery // that a validating resolver would need to close, out of scope for this pin. @@ -181,8 +181,7 @@ pub async fn get_or_refresh_mcp_client( .with_client(discovery_client) .map_err(|e| error::Error::BadRequest(format!("Failed to configure auth manager: {e}")))?; - let metadata = manager - .discover_metadata() + let metadata = crate::oauth::discover_authorization_metadata(&manager) .await .map_err(|e| error::Error::BadRequest(format!("OAuth discovery failed: {e}")))?; diff --git a/backend/windmill-mcp/src/lib.rs b/backend/windmill-mcp/src/lib.rs index d5eb0b9597..33a20be2f0 100644 --- a/backend/windmill-mcp/src/lib.rs +++ b/backend/windmill-mcp/src/lib.rs @@ -40,10 +40,30 @@ pub mod oauth { use std::time::Duration; - pub use rmcp::transport::auth::AuthorizationManager; + use rmcp::transport::auth::AuthorizationMetadataSource; + pub use rmcp::transport::auth::{AuthorizationManager, AuthorizationMetadata}; const DEFAULT_OAUTH_HTTP_TIMEOUT: Duration = Duration::from_secs(30); + /// Discover the MCP server's OAuth metadata, refusing endpoints the server + /// never advertised. + /// + /// When a server publishes no metadata at all, rmcp's `resolve_metadata` + /// falls back to inventing `/authorize`, `/token` and `/register` on the + /// server's own host. Dynamic client registration and the token exchange + /// both carry secrets, so they must only ever reach endpoints the server + /// actually published — a guessed path would send them somewhere the + /// operator never designated as an authorization server. + pub async fn discover_authorization_metadata( + manager: &AuthorizationManager, + ) -> anyhow::Result { + let resolution = manager.resolve_metadata().await?; + if resolution.source == AuthorizationMetadataSource::LegacyEndpointFallback { + anyhow::bail!("MCP server does not publish OAuth authorization metadata"); + } + Ok(resolution.metadata) + } + pub fn no_redirect_http_client() -> Result { no_redirect_http_client_with_timeout(DEFAULT_OAUTH_HTTP_TIMEOUT) } @@ -61,11 +81,8 @@ pub mod oauth { /// guard validated for the request URL so the connect cannot rebind to an /// internal IP after the check (TOCTOU). The OAuth DCR/discovery/token /// requests target author-controlled URLs and carry secrets, so they must - /// go through this rather than the unpinned client. `apply_dns_pinning` - /// lives on windmill-common's reqwest, which this crate resolves at a - /// different version (via rmcp), so pin directly with the std-typed - /// host/addrs. Empty `addrs` (IP literal or ALLOW_PRIVATE_MCP_SERVER_URLS) - /// leaves resolution untouched. + /// go through this rather than the unpinned client. Empty `addrs` (IP literal + /// or ALLOW_PRIVATE_MCP_SERVER_URLS) leaves resolution untouched. pub fn no_redirect_http_client_pinned( target: &windmill_common::ssrf::ValidatedTarget, ) -> Result { @@ -124,5 +141,41 @@ pub mod oauth { handle.join().unwrap(); } + + /// A server publishing no OAuth metadata must be rejected, not have its + /// endpoints guessed: rmcp's own fallback would invent `/authorize`, + /// `/token` and `/register` on that host, and DCR and the token exchange + /// send secrets to whatever comes back. + #[tokio::test] + async fn discovery_refuses_endpoints_the_server_never_published() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + + let handle = thread::spawn(move || { + // Every discovery probe 404s, which is what a plain MCP server + // with no authorization server looks like. + while let Ok((mut stream, _)) = listener.accept() { + let mut buffer = [0u8; 2048]; + let _ = stream.read(&mut buffer); + let _ = std::io::Write::write_all( + &mut stream, + b"HTTP/1.1 404 Not Found\r\ncontent-length: 0\r\nconnection: close\r\n\r\n", + ); + } + }); + + let manager = AuthorizationManager::new(format!("http://{addr}/mcp")) + .await + .expect("manager should construct"); + let err = discover_authorization_metadata(&manager) + .await + .expect_err("must not fall back to guessed endpoints"); + assert!( + err.to_string().contains("does not publish OAuth"), + "unexpected error: {err}" + ); + + drop(handle); + } } } diff --git a/backend/windmill-mcp/src/server/endpoints.rs b/backend/windmill-mcp/src/server/endpoints.rs index 9e3adffc0b..cea2896caa 100644 --- a/backend/windmill-mcp/src/server/endpoints.rs +++ b/backend/windmill-mcp/src/server/endpoints.rs @@ -59,17 +59,13 @@ pub fn endpoint_tool_to_mcp_tool(tool: &EndpointTool) -> Tool { // Create annotations based on HTTP method and endpoint characteristics let annotations = create_endpoint_annotations(tool); - Tool { - name: tool.name.clone(), - description: Some(description.into()), - input_schema: Arc::new(combined_schema.as_object().unwrap().clone()), - title: Some(tool.name.to_string()), - output_schema: None, - icons: None, - annotations: Some(annotations), - meta: None, - execution: None, - } + Tool::new( + tool.name.clone(), + description, + Arc::new(combined_schema.as_object().unwrap().clone()), + ) + .with_title(tool.name.to_string()) + .with_annotations(annotations) } /// Convert an endpoint tool to an MCP tool for multi-workspace mode. @@ -130,26 +126,19 @@ pub fn list_workspaces_tool() -> Tool { "required": [] }); - Tool { - name: Cow::Borrowed("list_workspaces"), - description: Some( - "List the Windmill workspaces this token can access. Use the returned workspace ids as the `workspace_id` argument of the other tools." - .into(), - ), - input_schema: Arc::new(schema.as_object().unwrap().clone()), - title: Some("List accessible workspaces".to_string()), - output_schema: None, - icons: None, - annotations: Some(ToolAnnotations { - title: Some("List accessible workspaces".to_string()), - read_only_hint: Some(true), - destructive_hint: Some(false), - idempotent_hint: Some(true), - open_world_hint: Some(false), - }), - meta: None, - execution: None, - } + Tool::new( + Cow::Borrowed("list_workspaces"), + "List the Windmill workspaces this token can access. Use the returned workspace ids as the `workspace_id` argument of the other tools.", + Arc::new(schema.as_object().unwrap().clone()), + ) + .with_title("List accessible workspaces") + .with_annotations( + ToolAnnotations::with_title("List accessible workspaces") + .read_only(true) + .destructive(false) + .idempotent(true) + .open_world(false), + ) } /// Create appropriate annotations for endpoint tools based on HTTP method @@ -166,13 +155,11 @@ fn create_endpoint_annotations(tool: &EndpointTool) -> ToolAnnotations { _ => (false, true, false, true), // Default: assume can modify and be destructive }; - ToolAnnotations { - title: Some(format!("{} {}", method, tool.path)), - read_only_hint: Some(read_only), - destructive_hint: Some(destructive), - idempotent_hint: Some(idempotent), - open_world_hint: Some(open_world), - } + ToolAnnotations::with_title(format!("{} {}", method, tool.path)) + .read_only(read_only) + .destructive(destructive) + .idempotent(idempotent) + .open_world(open_world) } /// Merge schema into combined properties and required fields diff --git a/backend/windmill-mcp/src/server/mod.rs b/backend/windmill-mcp/src/server/mod.rs index da7032418b..19c7e43df6 100644 --- a/backend/windmill-mcp/src/server/mod.rs +++ b/backend/windmill-mcp/src/server/mod.rs @@ -23,10 +23,10 @@ pub use tools::create_tool_from_item; // Re-export rmcp types for convenience pub use rmcp::handler::server::ServerHandler; pub use rmcp::model::{ - Annotated, CallToolRequestParams, CallToolResult, Content, Implementation, - InitializeRequestParams, InitializeResult, ListPromptsResult, ListResourceTemplatesResult, - ListResourcesResult, ListToolsResult, PaginatedRequestParams, ProtocolVersion, RawContent, - RawTextContent, ServerCapabilities, ServerInfo, Tool, ToolAnnotations, + CallToolRequestParams, CallToolResult, ContentBlock, Implementation, InitializeRequestParams, + InitializeResult, ListPromptsResult, ListResourceTemplatesResult, ListResourcesResult, + ListToolsResult, PaginatedRequestParams, ProtocolVersion, ServerCapabilities, ServerInfo, Tool, + ToolAnnotations, }; pub use rmcp::service::{RequestContext, RoleServer}; pub use rmcp::transport::streamable_http_server::{ diff --git a/backend/windmill-mcp/src/server/runner.rs b/backend/windmill-mcp/src/server/runner.rs index f9d8c5d876..e8248920ed 100644 --- a/backend/windmill-mcp/src/server/runner.rs +++ b/backend/windmill-mcp/src/server/runner.rs @@ -17,16 +17,42 @@ use crate::server::endpoints::{ use crate::server::tools::create_tool_from_item; use rmcp::handler::server::ServerHandler; use rmcp::model::{ - CallToolRequestParams, CallToolResult, Content, Implementation, InitializeRequestParams, - InitializeResult, ListPromptsResult, ListResourceTemplatesResult, ListResourcesResult, - ListToolsResult, PaginatedRequestParams, ProtocolVersion, ServerCapabilities, ServerInfo, + CacheScope, CallToolRequestParams, CallToolResponse, CallToolResult, ContentBlock, + Implementation, InitializeResult, ListPromptsResult, ListResourceTemplatesResult, + ListResourcesResult, ListToolsResult, PaginatedRequestParams, ProtocolVersion, + ServerCapabilities, ServerInfo, }; use rmcp::service::{RequestContext, RoleServer}; use rmcp::ErrorData; use serde_json::Value; +use std::borrow::Cow; use std::collections::{HashMap, HashSet}; use std::sync::Arc; +/// Protocol revisions this server is willing to speak. `2026-07-28` is served +/// statelessly with per-request metadata; the older revisions keep the +/// `initialize` handshake, so both eras are answered on the same endpoint. +const SUPPORTED_PROTOCOL_VERSIONS: &[ProtocolVersion] = &[ + ProtocolVersion::V_2024_11_05, + ProtocolVersion::V_2025_03_26, + ProtocolVersion::V_2025_06_18, + ProtocolVersion::V_2025_11_25, + ProtocolVersion::V_2026_07_28, +]; + +/// SEP-2549 cache hints, required on every list result at `2026-07-28` — rmcp +/// leaves them unset, and a strict client rejects the response without them. +/// +/// Zero because nothing here is cacheable: the listing is rebuilt from the +/// workspace's scripts and flows, which change at any time, and this server +/// advertises no `listChanged` capability, so a client that cached a stale list +/// would have no way to learn it had gone stale. +const LIST_TTL_MS: u64 = 0; +/// Every listing is filtered by the caller's token scopes and workspace +/// membership, so no two callers necessarily see the same tools — a shared +/// cache entry would leak one token's view to another. +const LIST_CACHE_SCOPE: CacheScope = CacheScope::Private; + // Re-export from http crate for extracting request parts use http::request::Parts as HttpParts; @@ -127,7 +153,9 @@ impl Runner { } /// How an endpoint tool interacts with the token's `mcp:scripts:` / `mcp:flows:` -/// path scopes. +/// path scopes. Mirrored in +/// `frontend/src/lib/components/mcp/endpointScopePolicy.ts` so the scope picker +/// only advertises tools this policy will actually list — keep the two in sync. enum EndpointPathPolicy { /// Executes the script/flow named by the `path` argument. Gated by the /// script/flow scope alone (an endpoint scope is not enough to run things): @@ -305,24 +333,26 @@ fn find_matching_path(candidates: Vec, request_name: &str) - impl ServerHandler for Runner { fn get_info(&self) -> ServerInfo { - ServerInfo { - protocol_version: ProtocolVersion::default(), - capabilities: ServerCapabilities::builder().enable_tools().build(), - server_info: Implementation::from_build_env(), - instructions: Some( + // Not `Implementation::from_build_env()`: its `env!` expands inside rmcp, so it + // would name the SDK crate rather than this server. + let server_info = Implementation::new("windmill", env!("CARGO_PKG_VERSION")) + .with_title("Windmill") + .with_website_url("https://windmill.dev"); + + InitializeResult::new(ServerCapabilities::builder().enable_tools().build()) + .with_server_info(server_info) + .with_instructions( "This server provides a list of scripts and flows the user can run on Windmill. \ - Each flow and script is a tool callable with their respective arguments." - .to_string(), - ), - } + Each flow and script is a tool callable with their respective arguments.", + ) } - async fn initialize( - &self, - _request: InitializeRequestParams, - _context: RequestContext, - ) -> Result { - Ok(self.get_info()) + /// Pinned rather than left to rmcp's default (every version the SDK knows), so a + /// future SDK revision cannot start advertising a version this server has not been + /// exercised against. Bounds `initialize` negotiation, `server/discover`, and + /// per-request version validation alike. + fn supported_protocol_versions(&self) -> Cow<'static, [ProtocolVersion]> { + Cow::Borrowed(SUPPORTED_PROTOCOL_VERSIONS) } async fn list_tools( @@ -357,7 +387,7 @@ impl ServerHandler for Runner { &self, request: CallToolRequestParams, context: RequestContext, - ) -> Result { + ) -> Result { let (auth, mode) = Self::extract_context(&context)?; // Parse MCP scopes for authorization @@ -368,7 +398,9 @@ impl ServerHandler for Runner { let args = request.arguments.map(Value::Object).unwrap_or(Value::Null); - match mode { + // Every tool here runs to completion in one round trip: none of them ask the + // client for input, so the MRTR variants of `CallToolResponse` are never built. + let result = match mode { McpMode::Single(workspace_id) => { self.call_tool_single( &auth, @@ -384,7 +416,8 @@ impl ServerHandler for Runner { self.call_tool_multi(&auth, &token, &scope_config, read_only, request.name, args) .await } - } + }?; + Ok(result.into()) } async fn list_resources( @@ -392,7 +425,9 @@ impl ServerHandler for Runner { _request: Option, _context: RequestContext, ) -> Result { - Ok(ListResourcesResult { resources: vec![], next_cursor: None, meta: None }) + Ok(ListResourcesResult::with_all_items(vec![]) + .with_ttl_ms(LIST_TTL_MS) + .with_cache_scope(LIST_CACHE_SCOPE)) } async fn list_prompts( @@ -400,7 +435,9 @@ impl ServerHandler for Runner { _request: Option, _context: RequestContext, ) -> Result { - Ok(ListPromptsResult::default()) + Ok(ListPromptsResult::default() + .with_ttl_ms(LIST_TTL_MS) + .with_cache_scope(LIST_CACHE_SCOPE)) } async fn list_resource_templates( @@ -408,7 +445,9 @@ impl ServerHandler for Runner { _request: Option, _context: RequestContext, ) -> Result { - Ok(ListResourceTemplatesResult::default()) + Ok(ListResourceTemplatesResult::default() + .with_ttl_ms(LIST_TTL_MS) + .with_cache_scope(LIST_CACHE_SCOPE)) } } @@ -545,7 +584,9 @@ impl Runner { tools.push(endpoint_tool_to_mcp_tool(&endpoint_tool)); } - Ok(ListToolsResult { tools, next_cursor: None, meta: None }) + Ok(ListToolsResult::with_all_items(tools) + .with_ttl_ms(LIST_TTL_MS) + .with_cache_scope(LIST_CACHE_SCOPE)) } /// Handle a tool call for a single, bound workspace. @@ -573,7 +614,7 @@ impl Runner { .await .map_err(|e| ErrorData::internal_error(e.message, None))?; - return Ok(CallToolResult::success(vec![Content::text( + return Ok(CallToolResult::success(vec![ContentBlock::text( truncate_tool_result( serde_json::to_string_pretty(&result).unwrap_or_else(|_| "{}".to_string()), ), @@ -697,7 +738,7 @@ impl Runner { }; match result { - Ok(value) => Ok(CallToolResult::success(vec![Content::text( + Ok(value) => Ok(CallToolResult::success(vec![ContentBlock::text( truncate_tool_result( serde_json::to_string_pretty(&value).unwrap_or_else(|_| "{}".to_string()), ), @@ -731,7 +772,9 @@ impl Runner { tools.push(endpoint_tool_to_mcp_tool_multi(&endpoint_tool)); } - ListToolsResult { tools, next_cursor: None, meta: None } + ListToolsResult::with_all_items(tools) + .with_ttl_ms(LIST_TTL_MS) + .with_cache_scope(LIST_CACHE_SCOPE) } /// Handle a tool call for a multi-workspace session. `base_auth` is the @@ -752,7 +795,7 @@ impl Runner { .list_accessible_workspaces(base_auth) .await .map_err(|e| ErrorData::internal_error(e.message, None))?; - return Ok(CallToolResult::success(vec![Content::text( + return Ok(CallToolResult::success(vec![ContentBlock::text( serde_json::to_string_pretty(&workspaces).unwrap_or_else(|_| "[]".to_string()), )])); } @@ -825,7 +868,7 @@ impl Runner { .await .map_err(|e| ErrorData::internal_error(e.message, None))?; - Ok(CallToolResult::success(vec![Content::text( + Ok(CallToolResult::success(vec![ContentBlock::text( truncate_tool_result( serde_json::to_string_pretty(&result).unwrap_or_else(|_| "{}".to_string()), ), diff --git a/backend/windmill-mcp/src/server/tools.rs b/backend/windmill-mcp/src/server/tools.rs index 50ed03426b..efdf86ed22 100644 --- a/backend/windmill-mcp/src/server/tools.rs +++ b/backend/windmill-mcp/src/server/tools.rs @@ -190,21 +190,17 @@ pub fn create_tool_from_item( } }; - Tool { - name: Cow::Owned(path), - description: Some(Cow::Owned(description)), - input_schema: Arc::new(input_schema_map), - title: Some(title.clone()), - output_schema: None, - icons: None, - annotations: Some(ToolAnnotations { - title: Some(title), - read_only_hint: Some(false), // Can modify environment - destructive_hint: Some(true), // Can potentially be destructive - idempotent_hint: Some(false), // Are not guaranteed to be idempotent - open_world_hint: Some(true), // Can interact with external services - }), - meta: None, - execution: None, - } + Tool::new( + Cow::Owned(path), + Cow::Owned(description), + Arc::new(input_schema_map), + ) + .with_title(title.clone()) + .with_annotations( + ToolAnnotations::with_title(title) + .read_only(false) // Can modify environment + .destructive(true) // Can potentially be destructive + .idempotent(false) // Are not guaranteed to be idempotent + .open_world(true), // Can interact with external services + ) } diff --git a/backend/windmill-native-triggers/src/github/external.rs b/backend/windmill-native-triggers/src/github/external.rs index d31b2ae207..2c23e02bef 100644 --- a/backend/windmill-native-triggers/src/github/external.rs +++ b/backend/windmill-native-triggers/src/github/external.rs @@ -232,7 +232,10 @@ impl External for GitHub { Err(e) => { errors.push(crate::sync::SyncError { resource_path: trigger.script_path.clone(), - error_message: format!("Failed to verify GitHub webhook: {}", e), + error_message: format!( + "Failed to verify GitHub webhook: {}", + crate::external_error_message(&e) + ), error_type: "api_error".to_string(), }); } @@ -288,6 +291,31 @@ impl External for GitHub { fn additional_routes(&self) -> axum::Router { routes::github_routes(self.clone()) } + + fn error_hint(&self, status: StatusCode) -> Option<&'static str> { + match status { + StatusCode::UNAUTHORIZED => { + Some("reconnect the GitHub integration from Workspace settings > Integrations.") + } + StatusCode::FORBIDDEN => Some( + "check that the OAuth app is authorized for the organization and that the \ + connected GitHub account has the access this needs — managing webhooks requires \ + admin rights on the repository.", + ), + _ => None, + } + } + + /// GitHub answers 403 to both throttling and a permission failure, and words the throttle + /// two ways: the primary and secondary limits say "rate limit", the older secondary + /// response says "abuse detection mechanism". + fn is_transient_response(&self, status: StatusCode, body: &str) -> bool { + if status != StatusCode::FORBIDDEN { + return false; + } + let body = body.to_ascii_lowercase(); + body.contains("rate limit") || body.contains("abuse detection") + } } impl GitHub { diff --git a/backend/windmill-native-triggers/src/github/routes.rs b/backend/windmill-native-triggers/src/github/routes.rs index b2196f2249..68de93b6a9 100644 --- a/backend/windmill-native-triggers/src/github/routes.rs +++ b/backend/windmill-native-triggers/src/github/routes.rs @@ -5,7 +5,10 @@ use http::Method; use windmill_api_auth::ApiAuthed; use windmill_common::{error::JsonResult, DB}; -use crate::{get_workspace_integration, require_native_integration_use, External, ServiceName}; +use crate::{ + get_workspace_integration, map_external_error, require_native_integration_use, External, + ServiceName, +}; use super::{GitHub, GithubApiRepoResponse, GithubRepoEntry}; @@ -33,7 +36,8 @@ async fn list_repos( let repos: Vec = handler .http_client_request::<_, ()>(&url, Method::GET, &workspace_id, &db, None, None) - .await?; + .await + .map_err(map_external_error)?; let count = repos.len(); all_entries.extend(repos.into_iter().map(|r| GithubRepoEntry { diff --git a/backend/windmill-native-triggers/src/google/external.rs b/backend/windmill-native-triggers/src/google/external.rs index 616f8d3f63..1c5a2dd9ec 100644 --- a/backend/windmill-native-triggers/src/google/external.rs +++ b/backend/windmill-native-triggers/src/google/external.rs @@ -11,7 +11,8 @@ use windmill_common::{ use windmill_queue::PushArgsOwned; use crate::{ - generate_webhook_service_url, rotate_webhook_token, + generate_webhook_service_url, + lock::TriggerLock, sync::{SyncAction, SyncError, TriggerSyncInfo}, update_native_trigger_error, update_native_trigger_service_config, External, NativeTrigger, NativeTriggerData, ServiceName, @@ -198,6 +199,26 @@ impl External for Google { fn additional_routes(&self) -> axum::Router { routes::google_routes(self.clone()) } + + fn error_hint(&self, status: http::StatusCode) -> Option<&'static str> { + match status { + http::StatusCode::UNAUTHORIZED => { + Some("reconnect the Google integration from Workspace settings > Integrations.") + } + http::StatusCode::FORBIDDEN => Some( + "the connected Google account is missing access to this resource or the scope the \ + integration was granted does not cover it.", + ), + _ => None, + } + } + + /// Google answers 403 to a quota being spent as well as to a permission failure. Every + /// throttling reason it defines sits in the `usageLimits` domain, so matching the domain + /// covers the group without an allowlist to extend each time one is added. + fn is_transient_response(&self, status: http::StatusCode, body: &str) -> bool { + status == http::StatusCode::FORBIDDEN && body.contains("usageLimits") + } } // Helper methods for creating trigger type-specific watches @@ -342,10 +363,11 @@ impl Google { .transpose()? .ok_or_else(|| Error::InternalErr("Missing service config".to_string()))?; - let rotated = match rotate_webhook_token( + let rotated = match crate::rotate_webhook_token( db, &trigger.webhook_token_hash, ServiceName::Google, + crate::webhook_token_scopes(&trigger.script_path, trigger.is_flow), ) .await? { @@ -477,7 +499,7 @@ enum RenewOutcome { Skipped, } -/// Renew one Google watch channel under a row lock. +/// Renew one Google watch channel under `TriggerLock`. /// `sync_all_triggers` runs on every replica with no leader election — without /// the lock, parallel renewals orphan the losers' new tokens and Google channels. async fn try_renew_channel_locked( @@ -486,22 +508,31 @@ async fn try_renew_channel_locked( workspace_id: &str, trigger: &NativeTrigger, ) -> Result { - let mut tx = db.begin().await?; + // Renewal stops the live channel and creates a replacement, exactly like a re-registration + // does — so the two must not overlap, or each stops the channel the other just made and one + // is left orphaned, delivering a second copy of every event. Skipping is fine: the sweep runs + // again. Excluding them through this lock rather than by holding the row itself is what keeps + // a rename's `UPDATE native_trigger` from having to wait on a third party's API. + let Some(lock) = + TriggerLock::try_acquire(db, workspace_id, ServiceName::Google, &trigger.external_id) + .await? + else { + return Ok(RenewOutcome::Skipped); + }; let row = sqlx::query!( r#" - SELECT service_config, webhook_token_hash + SELECT service_config, webhook_token_hash, script_path, is_flow FROM native_trigger WHERE workspace_id = $1 AND service_name = $2 AND external_id = $3 - FOR UPDATE SKIP LOCKED "#, workspace_id, ServiceName::Google as ServiceName, trigger.external_id, ) - .fetch_optional(&mut *tx) + .fetch_optional(db) .await?; let Some(row) = row else { @@ -522,10 +553,14 @@ async fn try_renew_channel_locked( return Ok(RenewOutcome::Skipped); } - // Use freshly-read fields — webhook_token_hash may have rotated since list time. + // Use freshly-read fields — the listing this came from predates the lock, so a rename may + // have moved the runnable since. Renewing against the listed path would build the channel's + // callback URL from a path that no longer resolves. let fresh_trigger = NativeTrigger { service_config: Some(service_config), webhook_token_hash: row.webhook_token_hash, + script_path: row.script_path, + is_flow: row.is_flow, ..trigger.clone() }; @@ -533,6 +568,8 @@ async fn try_renew_channel_locked( .renew_channel(workspace_id, &fresh_trigger, db) .await?; + let mut tx = db.begin().await?; + // Past this point a new Google channel exists. Any failure leaks it. if let Err(e) = update_native_trigger_service_config( &mut *tx, @@ -577,6 +614,8 @@ async fn try_renew_channel_locked( ), } + lock.release().await?; + Ok(RenewOutcome::Renewed) } @@ -637,7 +676,10 @@ async fn renew_expiring_channels( workspace_id, ServiceName::Google, &trigger.external_id, - Some(&format!("Channel renewal failed: {}", e)), + Some(&format!( + "Channel renewal failed: {}", + crate::external_error_message(&e) + )), ) .await; diff --git a/backend/windmill-native-triggers/src/google/routes.rs b/backend/windmill-native-triggers/src/google/routes.rs index c5f7611f10..306a7e54e2 100644 --- a/backend/windmill-native-triggers/src/google/routes.rs +++ b/backend/windmill-native-triggers/src/google/routes.rs @@ -10,7 +10,10 @@ use serde::{Deserialize, Serialize}; use windmill_api_auth::ApiAuthed; use windmill_common::{error::JsonResult, DB}; -use crate::{get_workspace_integration, require_native_integration_use, External, ServiceName}; +use crate::{ + get_workspace_integration, map_external_error, require_native_integration_use, External, + ServiceName, +}; use super::Google; @@ -100,7 +103,8 @@ async fn list_calendars( let response: GoogleCalendarListResponse = handler .http_client_request::<_, ()>(&url, Method::GET, &workspace_id, &db, None, None) - .await?; + .await + .map_err(map_external_error)?; let calendars = response .items @@ -153,7 +157,8 @@ async fn list_drive_files( let response: DriveApiResponse = handler .http_client_request::<_, ()>(&url, Method::GET, &workspace_id, &db, None, None) - .await?; + .await + .map_err(map_external_error)?; let files = response .files @@ -206,7 +211,8 @@ async fn list_shared_drives( let response: SharedDrivesApiResponse = handler .http_client_request::<_, ()>(&url, Method::GET, &workspace_id, &db, None, None) - .await?; + .await + .map_err(map_external_error)?; let drives = response .drives diff --git a/backend/windmill-native-triggers/src/handler.rs b/backend/windmill-native-triggers/src/handler.rs index fda28af407..cc6c61be81 100644 --- a/backend/windmill-native-triggers/src/handler.rs +++ b/backend/windmill-native-triggers/src/handler.rs @@ -1,7 +1,10 @@ use crate::{ - decrypt_oauth_data, delete_native_trigger, delete_token_by_hash, get_native_trigger, - list_native_triggers, rotate_webhook_token, store_native_trigger, update_native_trigger_error, - webhook_token_label, External, NativeTrigger, NativeTriggerConfig, NativeTriggerData, + classify_read_failure, decrypt_oauth_data, delete_native_trigger, delete_token_by_hash, + get_native_trigger, list_native_triggers, lock::TriggerLock, map_external_error, + map_external_error_with, rotate_webhook_token, store_native_trigger, + sync::EXTERNAL_TRIGGER_MISSING_ERROR, update_native_trigger_error, + update_native_trigger_if_runnable_unchanged, webhook_token_label, webhook_token_scopes, + External, ExternalReadFailure, NativeTrigger, NativeTriggerConfig, NativeTriggerData, ServiceName, }; use axum::{ @@ -52,6 +55,51 @@ async fn require_is_writer_on_runnable( } } +/// A trigger may only point at a live runnable. +/// +/// The webhook URL is built from this path, so a trigger pointed at a path the runnable has left +/// keeps delivering there: for a flow the row is gone and the trigger vanishes from listings, +/// for a script the abandoned version is still resolvable and fires stale code indefinitely. A +/// client that loaded before a rename submits the old path in good faith, so this needs no +/// concurrency to happen. The writer checks above do not cover it: they return early for admins +/// and path owners without ever looking at the runnable. +async fn require_runnable_exists(db: &DB, w_id: &str, path: &str, is_flow: bool) -> Result<()> { + let exists = if is_flow { + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM flow WHERE path = $1 AND workspace_id = $2)", + path, + w_id + ) + .fetch_one(db) + .await? + } else { + // Renaming a script archives the version at the old path instead of removing it, so a + // plain existence check would still accept a path the script has moved off. Every deploy + // archives its parent, leaving exactly one non-archived version at a live path and none + // at an abandoned one. Soft-delete sets `archived` too, and `deleted` is checked because + // it, not `archived`, is what stops a version from being resolved for execution. + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2 \ + AND archived = false AND deleted = false)", + path, + w_id + ) + .fetch_one(db) + .await? + }; + + if exists.unwrap_or(false) { + Ok(()) + } else { + Err(Error::BadRequest(format!( + "There is no {kind} at {path} to trigger. If the {kind} was renamed since this page \ + was loaded, reload and try again; otherwise point this trigger at an existing {kind} \ + or delete it.", + kind = if is_flow { "flow" } else { "script" } + ))) + } +} + #[derive(Debug, Deserialize)] pub struct ListQuery { pub page: Option, @@ -65,6 +113,10 @@ pub struct FullTriggerResponse { #[serde(flatten)] pub windmill_data: NativeTrigger, pub external_data: Option, + /// Why `external_data` is missing, when the service could not be read. The stored + /// configuration is still returned so the trigger stays viewable and editable. + #[serde(skip_serializing_if = "Option::is_none")] + pub external_error: Option, } #[derive(Debug, Serialize)] @@ -72,7 +124,7 @@ pub struct CreateTriggerResponse { pub external_id: String, } -async fn new_webhook_token( +pub(crate) async fn new_webhook_token( tx: &mut PgConnection, db: &DB, authed: &ApiAuthed, @@ -81,9 +133,7 @@ async fn new_webhook_token( workspace_id: &str, service_name: ServiceName, ) -> Result { - let kind = if is_flow { "flows" } else { "scripts" }; - - let scopes = vec![format!("jobs:run:{kind}:{script_path}")]; + let scopes = webhook_token_scopes(script_path, is_flow); let label = webhook_token_label(service_name); let expiration = service_name .webhook_token_expiration() @@ -121,6 +171,7 @@ async fn create_native_trigger( db.clone(), ) .await?; + require_runnable_exists(&db, &workspace_id, &data.script_path, data.is_flow).await?; let mut tx = user_db.begin(&authed).await?; @@ -148,7 +199,8 @@ async fn create_native_trigger( &db, &mut tx, ) - .await?; + .await + .map_err(map_external_error)?; let (external_id, _) = handler.external_id_and_metadata_from_response(&resp); @@ -169,7 +221,8 @@ async fn create_native_trigger( &db, &mut tx, ) - .await? + .await + .map_err(map_external_error)? }; let config = NativeTriggerConfig { @@ -225,11 +278,14 @@ async fn update_native_trigger_handler( db.clone(), ) .await?; + require_runnable_exists(&db, &workspace_id, &data.script_path, data.is_flow).await?; let integration_service = service_name.integration_service(); let oauth_data: T::OAuthData = decrypt_oauth_data(&db, &workspace_id, integration_service).await?; + let lock = TriggerLock::acquire(&db, &workspace_id, service_name, &external_id).await?; + let mut tx = user_db.clone().begin(&authed).await?; let existing = get_native_trigger(&mut *tx, &workspace_id, service_name, &external_id) @@ -260,7 +316,14 @@ async fn update_native_trigger_handler( token } else { // Same runnable — rotate the token (mints a fresh label + expiration) - match rotate_webhook_token(&db, &existing.webhook_token_hash, service_name).await? { + match rotate_webhook_token( + &db, + &existing.webhook_token_hash, + service_name, + webhook_token_scopes(&data.script_path, data.is_flow), + ) + .await? + { Some(rotated) => { old_token_hash_to_delete = Some(rotated.old_token_hash); rotated.new_token @@ -294,7 +357,8 @@ async fn update_native_trigger_handler( &db, &mut tx, ) - .await?; + .await + .map_err(map_external_error)?; let config = NativeTriggerConfig { script_path: data.script_path.clone(), @@ -302,7 +366,11 @@ async fn update_native_trigger_handler( webhook_token, }; - store_native_trigger( + // `existing` was read before the network call, and a rename writes `script_path` from the + // deploy transaction, which this lock does not cover. Writing a stale path back would undo the + // rename and drop the trigger out of every listing, so refuse the edit instead. The rename's + // own re-registration is queued behind this lock and puts the service back in step. + let applied = update_native_trigger_if_runnable_unchanged( &mut *tx, &workspace_id, service_name, @@ -310,9 +378,18 @@ async fn update_native_trigger_handler( &config, service_config, data.summary.as_deref(), + &existing.script_path, + existing.is_flow, ) .await?; + if !applied { + return Err(Error::BadRequest(format!( + "The runnable of {external_id} was renamed while this trigger was being saved, so the \ + edit was not applied. Reload and save again." + ))); + } + audit_log( &mut *tx, &authed, @@ -326,6 +403,8 @@ async fn update_native_trigger_handler( tx.commit().await?; + lock.release().await?; + // Everything succeeded — clean up old token (best-effort) if let Some(old_hash) = old_token_hash_to_delete { if let Err(e) = delete_token_by_hash(&db, &old_hash).await { @@ -373,10 +452,14 @@ async fn get_native_trigger_handler( .get(&workspace_id, &oauth_data, &external_id, &db, &mut tx) .await; + let mut external_error = None; + let external_data = match native_trigger { Ok(Some(native_cfg)) => { - // Clear error if it was set - if windmill_trigger.error.is_some() { + // Only the "no longer exists" error is disproven by the trigger being there; other + // paths record failures (e.g. a webhook still aimed at a pre-rename path) that a + // successful fetch says nothing about. + if windmill_trigger.error.as_deref() == Some(EXTERNAL_TRIGGER_MISSING_ERROR) { update_native_trigger_error( &mut *tx, &workspace_id, @@ -389,33 +472,54 @@ async fn get_native_trigger_handler( Some(native_cfg) } Ok(None) => None, - Err(Error::NotFound(_)) => { - let error_msg = "Trigger no longer exists on external service".to_string(); - tracing::warn!( - "Native trigger no longer exists on external service {}, setting error", - service_name - ); + Err(e) => match classify_read_failure(e) { + ExternalReadFailure::Missing => { + let error_msg = EXTERNAL_TRIGGER_MISSING_ERROR.to_string(); + tracing::warn!( + "Native trigger no longer exists on external service {}, setting error", + service_name + ); - update_native_trigger_error( - &mut *tx, - &workspace_id, - service_name, - &external_id, - Some(&error_msg), - ) - .await?; + update_native_trigger_error( + &mut *tx, + &workspace_id, + service_name, + &external_id, + Some(&error_msg), + ) + .await?; - tx.commit().await?; + tx.commit().await?; - return Err(Error::NotFound(format!( - "Trigger '{}' no longer exists on external service {}", - external_id, service_name - ))); - } - Err(e) => return Err(e), + return Err(Error::NotFound(format!( + "Trigger '{}' no longer exists on external service {}", + external_id, service_name + ))); + } + // The service being unreadable says nothing about the trigger Windmill stores, and + // failing here would leave the editor with no configuration to show at all. Report + // what the service said alongside the stored configuration instead. + ExternalReadFailure::Unreadable(message) => { + tracing::warn!( + "Could not read trigger '{}' from {}, returning the stored configuration: {}", + external_id, + service_name, + message + ); + external_error = Some(message); + None + } + ExternalReadFailure::Internal(e) => return Err(e), + }, }; - let full_resp = Json(FullTriggerResponse { windmill_data: windmill_trigger, external_data }); + tx.commit().await?; + + let full_resp = Json(FullTriggerResponse { + windmill_data: windmill_trigger, + external_data, + external_error, + }); Ok(full_resp) } @@ -428,6 +532,8 @@ async fn delete_native_trigger_handler( Extension(user_db): Extension, Path((workspace_id, external_id)): Path<(String, String)>, ) -> Result { + let lock = TriggerLock::acquire(&db, &workspace_id, service_name, &external_id).await?; + let mut tx = user_db.begin(&authed).await?; let existing = get_native_trigger(&mut *tx, &workspace_id, service_name, &external_id) @@ -452,7 +558,17 @@ async fn delete_native_trigger_handler( handler .delete(&workspace_id, &oauth_data, &external_id, &db, &mut tx) - .await?; + .await + .map_err(|e| { + map_external_error_with(e, |m| { + let end = if m.ends_with(['.', '!', '?']) { + "" + } else { + "." + }; + format!("{m}{end} The trigger was kept in Windmill, so it can still fire.") + }) + })?; let deleted = delete_native_trigger(&mut *tx, &workspace_id, service_name, &external_id).await?; @@ -482,6 +598,7 @@ async fn delete_native_trigger_handler( .await?; tx.commit().await?; + lock.release().await?; Ok(format!("Native trigger deleted")) } diff --git a/backend/windmill-native-triggers/src/lib.rs b/backend/windmill-native-triggers/src/lib.rs index a81217d8d5..4400091ed6 100644 --- a/backend/windmill-native-triggers/src/lib.rs +++ b/backend/windmill-native-triggers/src/lib.rs @@ -54,13 +54,17 @@ use windmill_common::{ use windmill_queue::PushArgsOwned; #[cfg(feature = "native_trigger")] -use windmill_oauth::{OClient, RefreshToken, Url, OAUTH_HTTP_CLIENT}; +use windmill_oauth::{ErrorField, ExecuteError, OClient, RefreshToken, Url, OAUTH_HTTP_CLIENT}; use windmill_api_auth::ApiAuthed; pub mod handler; +pub(crate) mod lock; pub mod sync; pub mod workspace_integrations; +#[cfg(feature = "native_trigger")] +pub mod rename; + // Service modules - add new services here: #[cfg(feature = "native_trigger")] pub mod github; @@ -361,6 +365,23 @@ pub trait External: Send + Sync + 'static { axum::Router::new() } + /// Pull the human-readable message out of an error body, when the service wraps it in an + /// envelope. Returning `None` (default) shows the body as-is. + fn describe_error_body(&self, _body: &str) -> Option { + None + } + + /// What the user has to do about a rejection, appended to the service's own message. + fn error_hint(&self, _status: StatusCode) -> Option<&'static str> { + None + } + + /// Whether the service is telling us it cannot serve the request right now, when the + /// status alone would read as a refusal. GitHub and Google both answer 403 to throttling. + fn is_transient_response(&self, _status: StatusCode, _body: &str) -> bool { + false + } + async fn http_client_request( &self, url: &str, @@ -384,13 +405,14 @@ pub trait External: Send + Sync + 'static { match result { Ok(response) => Ok(response), - Err(err) - if err.status() == Some(StatusCode::UNAUTHORIZED) - || err.status() == Some(StatusCode::FORBIDDEN) => - { + // Only an expired or revoked token is worth a refresh. A 403 means the account + // behind the token is authenticated and still not allowed, which minting a new + // token for that same account cannot change; retrying would only burn a refresh + // rotation per request and bury the service's own explanation. + Err(err) if err.status() == Some(StatusCode::UNAUTHORIZED) => { tracing::info!( - "HTTP auth error ({}), attempting token refresh", - err.status().unwrap() + "HTTP 401 from {}, attempting token refresh", + Self::DISPLAY_NAME ); let refreshed_oauth_config = refresh_oauth_tokens( @@ -398,7 +420,16 @@ pub trait External: Send + Sync + 'static { Self::REFRESH_ENDPOINT, Self::AUTH_ENDPOINT, ) - .await?; + .await + .map_err(|f| { + self.external_error( + f.rejected.then_some(StatusCode::UNAUTHORIZED), + format!( + "the stored credentials could not be refreshed: {}", + f.message + ), + ) + })?; task::spawn({ let db_clone = db.clone(); @@ -418,7 +449,7 @@ pub trait External: Send + Sync + 'static { } }); - let response = make_http_request( + make_http_request( url, method, headers, @@ -426,12 +457,122 @@ pub trait External: Send + Sync + 'static { &refreshed_oauth_config.access_token, ) .await - .map_err(to_anyhow)?; - Ok(response) + .map_err(|e| self.external_api_error(e)) } - Err(e) => Err(to_anyhow(e).into()), + Err(e) => Err(self.external_api_error(e)), } } + + /// Wrap a failed provider call so the status stays inspectable by internal callers (a 404 + /// means the trigger is gone, not that the call broke) and the message stays readable by + /// the time it reaches a user. + fn external_api_error(&self, e: HttpRequestError) -> Error { + let (detail, transient) = match &e { + HttpRequestError::ApiError { status, body } => ( + self.describe_error_body(body) + .unwrap_or_else(|| body.to_string()), + self.is_transient_response(*status, body), + ), + // A reqwest error names the request it was making, never why it failed: the reason + // (refused, DNS, TLS) lives one link down the source chain. + other => (error_source_chain(other), false), + }; + self.external_error_inner(e.status(), detail, transient) + } + + fn external_error(&self, status: Option, detail: String) -> Error { + self.external_error_inner(status, detail, false) + } + + fn external_error_inner( + &self, + status: Option, + detail: String, + transient: bool, + ) -> Error { + to_anyhow(ExternalApiError { + service: Self::DISPLAY_NAME, + status, + detail: truncate_detail(&detail), + // Guidance about permissions on a throttled request sends the reader after a + // problem they do not have. + hint: if transient { + None + } else { + status.and_then(|s| self.error_hint(s)) + }, + transient, + }) + .into() + } +} + +/// A native trigger provider refused or could not serve a request. +#[derive(Debug)] +pub struct ExternalApiError { + pub service: &'static str, + pub status: Option, + pub detail: String, + pub hint: Option<&'static str>, + /// The service could not serve the request now, whatever the status suggests: providers + /// overload 403 for throttling, and a refusal and an outage want opposite reactions. + pub transient: bool, +} + +impl ExternalApiError { + fn is_transient(&self) -> bool { + self.transient || self.status.is_some_and(is_transient_status) + } +} + +impl std::fmt::Display for ExternalApiError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self.status { + Some(status) if self.is_transient() => write!( + f, + "{} failed to serve the request ({}): {}", + self.service, status, self.detail + )?, + Some(status) => write!( + f, + "{} rejected the request ({}): {}", + self.service, status, self.detail + )?, + // No status covers both never reaching the service and not being able to read what + // it answered, so the wording may not commit to either. + None => write!(f, "Request to {} failed: {}", self.service, self.detail)?, + } + if let Some(hint) = self.hint { + write!(f, " — {}", hint)?; + } + Ok(()) + } +} + +impl std::error::Error for ExternalApiError {} + +fn error_source_chain(e: &dyn std::error::Error) -> String { + let mut msg = e.to_string(); + let mut source = e.source(); + while let Some(cause) = source { + let cause = cause.to_string(); + // reqwest repeats "error sending request for url (…)" at two levels of the chain. + if !msg.contains(&cause) { + msg.push_str(&format!(": {cause}")); + } + source = source.and_then(|c| c.source()); + } + msg +} + +/// Provider bodies are unbounded and end up in toasts and `native_trigger.error`. +fn truncate_detail(detail: &str) -> String { + const MAX: usize = 400; + let detail = detail.trim(); + match detail.char_indices().nth(MAX) { + Some((idx, _)) => format!("{}…", &detail[..idx]), + None => detail.to_string(), + } } #[derive(Debug, Serialize, Deserialize)] @@ -533,17 +674,83 @@ impl HttpRequestError { } } -/// Extract the HTTP status code from an error returned by `http_client_request`. -/// Returns `None` if the error didn't originate from an HTTP call. -pub fn http_error_status(e: &windmill_common::error::Error) -> Option { +/// `Some` only for a failure the service itself produced. Everything else — a query, a +/// decryption, a serialization — is Windmill's own and must not be reported as the service's. +pub fn as_external_error(e: &Error) -> Option<&ExternalApiError> { match e { - windmill_common::error::Error::Anyhow { error, .. } => error - .downcast_ref::() - .and_then(|e| e.status()), + Error::Anyhow { error, .. } => error.downcast_ref::(), _ => None, } } +/// Extract the HTTP status code from an error returned by `http_client_request`. +/// Returns `None` if the error didn't originate from an HTTP call. +pub fn http_error_status(e: &Error) -> Option { + as_external_error(e).and_then(|e| e.status) +} + +/// The message to show a user for a failed provider call, without the internal decoration +/// `Error`'s own `Display` adds. Non-provider errors are rendered as-is. +pub fn external_error_message(e: &Error) -> String { + as_external_error(e).map_or_else(|| e.to_string(), |ext| ext.to_string()) +} + +/// What a failed read of a trigger from its service means for the response. +#[derive(Debug)] +pub enum ExternalReadFailure { + /// The service answered that the trigger is gone. + Missing, + /// The service could not be read, which says nothing about the stored trigger. + Unreadable(String), + /// Windmill's own failure, wearing no service's name. + Internal(Error), +} + +pub fn classify_read_failure(e: Error) -> ExternalReadFailure { + match as_external_error(&e) { + Some(ext) if ext.status == Some(StatusCode::NOT_FOUND) => ExternalReadFailure::Missing, + Some(ext) => ExternalReadFailure::Unreadable(ext.to_string()), + None => ExternalReadFailure::Internal(e), + } +} + +/// Turn a provider failure into something a client can read and act on. +/// +/// Without this every rejection reaches the browser as a 500 carrying a raw upstream body. +/// The provider's status is deliberately not mirrored: a 401/403 answered by Windmill reads as +/// a Windmill permission problem, when the account that lacks rights is the one connected to +/// the *provider*. Errors that did not come from a provider call pass through untouched. +pub fn map_external_error(e: Error) -> Error { + map_external_error_with(e, |message| message) +} + +/// `map_external_error` with a chance to add what the failure means for Windmill's own state. +pub fn map_external_error_with(e: Error, decorate: impl FnOnce(String) -> String) -> Error { + let Some((status, transient, message)) = + as_external_error(&e).map(|ext| (ext.status, ext.is_transient(), ext.to_string())) + else { + return e; + }; + let message = decorate(message); + match status { + // A refusal is the caller's to fix; being unable to serve the request now is not, and + // the two want different reactions from whoever reads it. + _ if transient => Error::BadGateway(message), + Some(StatusCode::NOT_FOUND) => Error::NotFound(message), + Some(_) => Error::BadRequest(message), + // No status at all: the provider was unreachable or answered something undecodable. + None => Error::BadGateway(message), + } +} + +fn is_transient_status(status: StatusCode) -> bool { + status.is_server_error() + || matches!( + status, + StatusCode::REQUEST_TIMEOUT | StatusCode::TOO_MANY_REQUESTS + ) +} + /// Read OAuth client_id and client_secret from instance-level global settings. /// Used when a workspace integration has `instance_shared: true`. async fn get_instance_oauth_credentials( @@ -648,24 +855,98 @@ struct RefreshTokenResponse { refresh_token: Option, } +/// Why an OAuth token refresh did not produce a new token. +#[derive(Debug)] +pub struct RefreshFailure { + /// The token endpoint refused the grant, so the stored credentials are what to fix. + /// + /// Nothing else about the answer is carried further. The status a caller reads off an + /// `ExternalApiError` says what the call for the *trigger* answered, and 404 there means + /// the trigger is gone — it records that on the row and lets a delete drop it. A wrong + /// base URL 404s the token endpoint while the webhook is perfectly alive. + pub rejected: bool, + pub message: String, +} + +#[cfg(feature = "native_trigger")] +impl RefreshFailure { + fn rejected(message: String) -> Self { + RefreshFailure { rejected: true, message } + } + + fn outage(message: String) -> Self { + RefreshFailure { rejected: false, message } + } +} + +/// Whether a token endpoint's answer means reconnecting the integration is the fix, rather +/// than the endpoint failing to serve a grant that may well be valid. +/// +/// Used when the answer carried no OAuth error code to read. The body is checked because no +/// status reliably reports a refusal: GitHub answers `bad_refresh_token` with HTTP 200. The +/// status is the last resort — on a token endpoint these three most often mean the stored +/// grant is spent, and there is nothing better to go on. +pub fn grant_refused(status: Option, body: &str) -> bool { + body.contains("invalid_grant") + || body.contains("bad_refresh_token") + || matches!( + status, + Some(StatusCode::BAD_REQUEST | StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) + ) +} + +/// Whether an OAuth error code (RFC 6749 §5.2) means the credentials Windmill stored are what +/// needs replacing. The rest — a malformed request, an unsupported grant type, a scope the +/// server will not grant — are faults in how the integration asks, which reconnecting as the +/// same user will reproduce exactly. +#[cfg(feature = "native_trigger")] +fn code_means_reconnect(code: &ErrorField) -> bool { + matches!(code, ErrorField::InvalidGrant | ErrorField::InvalidClient) +} + +#[cfg(feature = "native_trigger")] +fn refresh_failure_from(e: ExecuteError) -> RefreshFailure { + let rejected = match &e { + // A code says exactly what was refused, so nothing else needs guessing at. + ExecuteError::ErrorResponse { error, .. } => code_means_reconnect(&error.error), + // A body that would not deserialize is where a refusal hides when the status says + // nothing, and it is the only place the reason is written down. + ExecuteError::BadResponse { status, body, .. } => { + grant_refused(Some(*status), &String::from_utf8_lossy(body)) + } + _ => grant_refused(e.status(), ""), + }; + let message = match &e { + ExecuteError::BadResponse { body, .. } => { + format!("{e}: {}", String::from_utf8_lossy(body)) + } + _ => error_source_chain(&e), + }; + if rejected { + RefreshFailure::rejected(message) + } else { + RefreshFailure::outage(message) + } +} + /// Refresh OAuth tokens using windmill-oauth. #[cfg(feature = "native_trigger")] pub async fn refresh_oauth_tokens( oauth_config: &OAuthConfig, refresh_endpoint: &str, auth_endpoint: &str, -) -> Result { +) -> std::result::Result { let refresh_token_str = oauth_config .refresh_token .as_ref() - .ok_or_else(|| Error::InternalErr("No refresh token available".to_string()))?; + .ok_or_else(|| RefreshFailure::rejected("no refresh token is stored".to_string()))?; // Build OAuth client for token refresh // Auth URL is not used for refresh, but required by the client constructor let auth_url = Url::parse(&resolve_endpoint(&oauth_config.base_url, auth_endpoint)) - .map_err(|e| Error::InternalErr(format!("Invalid auth URL: {}", e)))?; + .map_err(|e| RefreshFailure::outage(format!("invalid auth URL: {e}")))?; let token_url = Url::parse(&resolve_endpoint(&oauth_config.base_url, refresh_endpoint)) - .map_err(|e| Error::InternalErr(format!("Invalid token URL: {}", e)))?; + .map_err(|e| RefreshFailure::outage(format!("invalid token URL: {e}")))?; let mut client = OClient::new(oauth_config.client_id.clone(), auth_url, token_url); client.set_client_secret(oauth_config.client_secret.clone()); @@ -675,7 +956,7 @@ pub async fn refresh_oauth_tokens( .with_client(&*OAUTH_HTTP_CLIENT) .execute() .await - .map_err(|e| Error::InternalErr(format!("Failed to refresh token: {:?}", e)))?; + .map_err(refresh_failure_from)?; Ok(OAuthConfig { base_url: oauth_config.base_url.clone(), @@ -694,10 +975,11 @@ pub async fn refresh_oauth_tokens( _oauth_config: &OAuthConfig, _refresh_endpoint: &str, _auth_endpoint: &str, -) -> Result { - Err(Error::InternalErr( - "Native triggers feature is not enabled".to_string(), - )) +) -> std::result::Result { + Err(RefreshFailure { + rejected: false, + message: "the native_trigger feature is not enabled".to_string(), + }) } async fn update_oauth_token_resource( @@ -770,23 +1052,37 @@ async fn update_oauth_token_resource( } } +/// The scopes a webhook token must carry to run one runnable, and nothing else. +/// +/// Always derived from the runnable the trigger points at *now*. A token that outlives a rename +/// carries the old path, and a webhook presenting it is refused however correct the URL is. +pub fn webhook_token_scopes(script_path: &str, is_flow: bool) -> Vec { + let kind = if is_flow { "flows" } else { "scripts" }; + vec![format!("jobs:run:{kind}:{script_path}")] +} + /// Create a new webhook token, minting a fresh `ephemeral-webhook-{service}-{rd5}` /// label and the per-service expiration (see `ServiceName::webhook_token_expiration`). /// The old token is **not** deleted — callers must call `delete_token_by_hash` on /// `old_token_hash` after the trigger row has been successfully updated. /// /// Returns `Ok(None)` if the old token no longer exists (e.g. manually deleted by user). +/// +/// `scopes` is applied rather than carried over: the old token's scopes name whatever runnable it +/// was minted for, which a rename may since have moved. Copying them is how a re-save "succeeds" +/// while every callback it authorises is refused. pub async fn rotate_webhook_token( db: &DB, old_token_hash: &str, service_name: ServiceName, + scopes: Vec, ) -> Result> { use windmill_common::auth::{hash_token, TOKEN_PREFIX_LEN}; use windmill_common::min_version::MIN_VERSION_SUPPORTS_TOKEN_HASH; use windmill_common::utils::rd_string; let old = match sqlx::query!( - "SELECT email, scopes, workspace_id, super_admin, owner FROM token WHERE token_hash = $1", + "SELECT email, workspace_id, super_admin, owner FROM token WHERE token_hash = $1", old_token_hash ) .fetch_optional(db) @@ -825,7 +1121,7 @@ pub async fn rotate_webhook_token( old.email, new_label, old.super_admin, - old.scopes.as_deref(), + Some(scopes.as_slice()), old.workspace_id, old.owner, new_expiration, @@ -920,6 +1216,114 @@ pub async fn store_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres> Ok(()) } +/// Record the outcome of re-registering a webhook: the token that now authenticates it and the +/// config the service resolved. +/// +/// Deliberately not `store_native_trigger`: the runnable a trigger points at belongs to whoever +/// renamed or edited it, not to the re-registration, which only ever holds the path as it stood +/// before its network call. Writing that path back would undo a rename that landed in between. +/// +/// Conditional on `updated_at` — the row version — for the same reason. `TriggerLock` does not +/// cover the rename's own `UPDATE`, because deploys must never block on a third party, so another +/// write can land *during* this registration's network call. Recording then would attach a token +/// for state that no longer holds and, worse, clear the `REREGISTRATION_PENDING` a newer rename +/// set to protect itself. Comparing the path alone would not catch it: a save at the same path, or +/// a rename away and back, leaves the path equal while the token has moved on. +/// +/// Returns `false` when the row changed, leaving it untouched for whoever wrote it to finish. +pub(crate) async fn record_reregistration<'c, E: sqlx::Executor<'c, Database = Postgres>>( + db: E, + workspace_id: &str, + service_name: ServiceName, + external_id: &str, + webhook_token: &str, + service_config: serde_json::Value, + expected_updated_at: DateTime, +) -> Result { + use windmill_common::auth::hash_token; + + let applied = sqlx::query_scalar!( + r#" + UPDATE native_trigger + SET webhook_token_hash = $1, service_config = $2, error = NULL, updated_at = NOW() + WHERE + workspace_id = $3 + AND service_name = $4 + AND external_id = $5 + AND updated_at = $6 + RETURNING 1 AS "applied!" + "#, + hash_token(webhook_token), + sqlx::types::Json(service_config) as _, + workspace_id, + service_name as ServiceName, + external_id, + expected_updated_at, + ) + .fetch_optional(db) + .await? + .is_some(); + + Ok(applied) +} + +/// Apply a trigger edit, unless the runnable moved under it in the meantime. +/// +/// A rename writes `native_trigger.script_path` from inside the deploy transaction, which is not +/// under `TriggerLock` — so an edit holding the lock across its network call can still have the +/// ground shift beneath it. Writing its snapshot's path back would undo the rename and hide the +/// trigger from every listing, which is the bug this whole change exists to fix, so refuse instead. +/// +/// Callers MUST have verified write access to `config.script_path`. +/// +/// Returns `false` when the runnable moved; the edit is then stale and the caller should say so. +pub(crate) async fn update_native_trigger_if_runnable_unchanged< + 'c, + E: sqlx::Executor<'c, Database = Postgres>, +>( + db: E, + workspace_id: &str, + service_name: ServiceName, + external_id: &str, + config: &NativeTriggerConfig, + service_config: serde_json::Value, + summary: Option<&str>, + expected_script_path: &str, + expected_is_flow: bool, +) -> Result { + use windmill_common::auth::hash_token; + + let applied = sqlx::query_scalar!( + r#" + UPDATE native_trigger + SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, + summary = $5, error = NULL, updated_at = NOW() + WHERE + workspace_id = $6 + AND service_name = $7 + AND external_id = $8 + AND script_path = $9 + AND is_flow = $10 + RETURNING 1 AS "applied!" + "#, + config.script_path, + config.is_flow, + hash_token(&config.webhook_token), + sqlx::types::Json(service_config) as _, + summary, + workspace_id, + service_name as ServiceName, + external_id, + expected_script_path, + expected_is_flow, + ) + .fetch_optional(db) + .await? + .is_some(); + + Ok(applied) +} + pub async fn update_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres>>( db: E, workspace_id: &str, diff --git a/backend/windmill-native-triggers/src/lock.rs b/backend/windmill-native-triggers/src/lock.rs new file mode 100644 index 0000000000..577025ada3 --- /dev/null +++ b/backend/windmill-native-triggers/src/lock.rs @@ -0,0 +1,136 @@ +//! Serializing the operations that mutate a trigger's external registration. + +use std::sync::Arc; + +use sqlx::{Connection, PgConnection}; +use tokio::sync::{OwnedSemaphorePermit, Semaphore}; +use windmill_common::{ + error::{Error, Result}, + DB, +}; + +use crate::ServiceName; + +/// How long to wait for whoever holds the lock before giving up. +/// +/// A waiter parks on its own connection, and that connection is outside the pool and so outside +/// its limits — enough of them queued on one trigger would eat into what the server has for +/// everything else. Waiting is bounded instead: the holder is doing a network call with its own +/// timeout, so anything longer than this is not worth a connection, and the caller gets a plain +/// "busy, try again" rather than an open-ended stall. +const LOCK_WAIT: &str = "45s"; + +/// Ceiling on how many of these connections can exist at once. +/// +/// Each is opened outside the pool, so nothing else caps them: a burst of renames or repeated +/// requests against one trigger would otherwise eat into what the server has for every other +/// workload. Waiting for a permit costs nothing — the operation was going to queue on the lock +/// anyway — and it is generous enough that ordinary use never reaches it. +const MAX_CONCURRENT_LOCKS: usize = 32; + +lazy_static::lazy_static! { + static ref LOCK_SLOTS: Arc = Arc::new(Semaphore::new(MAX_CONCURRENT_LOCKS)); +} + +/// Held for the whole of a read → register → record cycle on one trigger. +/// +/// Registering a webhook is a read-modify-write spanning a network round-trip, against state held +/// both here and on the external service. Two of them interleaving desynchronises the two: whoever +/// writes the row last wins in the database, whoever calls the service last wins there, and they +/// need not be the same operation — which is how a trigger ends up pointing somewhere the service +/// is not calling, or a deleted trigger keeps firing. +/// +/// A Postgres advisory lock makes them take turns. It is deliberately not a row lock: a rename's +/// own `UPDATE native_trigger` must not block behind a re-registration's network call, and only +/// code that takes the same key here waits. +/// +/// The lock lives on its own connection, opened outside the pool. Holders keep it for as long as +/// the external call takes and go on to need pool connections of their own to finish and release +/// it, so taking it from the pool would let a handful of concurrent renames hold every connection +/// while each waits for one more. Being off-pool also makes the lock impossible to strand: it is +/// session-scoped, and a dropped or crashed connection releases it, whereas a pooled connection +/// would carry it back into the pool. +pub(crate) struct TriggerLock { + conn: Option, + _slot: OwnedSemaphorePermit, +} + +impl TriggerLock { + pub(crate) async fn acquire( + db: &DB, + w_id: &str, + service_name: ServiceName, + external_id: &str, + ) -> Result { + let (mut conn, slot) = Self::open(db).await?; + sqlx::query_scalar!( + "SELECT pg_advisory_lock(hashtextextended($1, 0))", + Self::key(w_id, service_name, external_id) + ) + .fetch_one(&mut conn) + .await + .map_err(|e| { + Error::BadRequest(format!( + "Another operation on {external_id} is still running after {LOCK_WAIT}; try again \ + shortly ({e})" + )) + })?; + Ok(Self { conn: Some(conn), _slot: slot }) + } + + /// Take the lock only if it is free, for callers that would rather come back later than wait + /// out someone else's network call — the background renewal sweep, which runs again shortly. + pub(crate) async fn try_acquire( + db: &DB, + w_id: &str, + service_name: ServiceName, + external_id: &str, + ) -> Result> { + let (mut conn, slot) = Self::open(db).await?; + let acquired = sqlx::query_scalar!( + r#"SELECT pg_try_advisory_lock(hashtextextended($1, 0)) AS "acquired!""#, + Self::key(w_id, service_name, external_id) + ) + .fetch_one(&mut conn) + .await?; + Ok(acquired.then_some(Self { conn: Some(conn), _slot: slot })) + } + + /// The lock's own connection, lent to the work it protects. + /// + /// Callers need a connection to hand the external service adapter, and it is held for the + /// whole network call. Taking that from the pool is what starves it: a handful of concurrent + /// renames would each pin one for as long as the remote takes to answer. This one is already + /// dedicated and idle for exactly that window. + pub(crate) fn conn(&mut self) -> &mut PgConnection { + self.conn + .as_mut() + .expect("lock connection is only taken on release") + } + + pub(crate) async fn release(mut self) -> Result<()> { + if let Some(conn) = self.conn.take() { + // Closing the connection is what ends the session and its locks; unlocking first only + // makes that explicit at the call site. + conn.close().await?; + } + Ok(()) + } + + async fn open(db: &DB) -> Result<(PgConnection, OwnedSemaphorePermit)> { + let slot = LOCK_SLOTS + .clone() + .acquire_owned() + .await + .map_err(|e| Error::internal_err(format!("trigger lock slots closed: {e}")))?; + let mut conn = PgConnection::connect_with(&db.connect_options()).await?; + sqlx::query(&format!("SET lock_timeout = '{LOCK_WAIT}'")) + .execute(&mut conn) + .await?; + Ok((conn, slot)) + } + + fn key(w_id: &str, service_name: ServiceName, external_id: &str) -> String { + format!("native_trigger:{w_id}:{service_name}:{external_id}") + } +} diff --git a/backend/windmill-native-triggers/src/nextcloud/external.rs b/backend/windmill-native-triggers/src/nextcloud/external.rs index b296ddb15b..763ad6fbb6 100644 --- a/backend/windmill-native-triggers/src/nextcloud/external.rs +++ b/backend/windmill-native-triggers/src/nextcloud/external.rs @@ -9,13 +9,14 @@ use windmill_common::{ }; use crate::{ - generate_webhook_service_url, + generate_webhook_service_url, http_error_status, nextcloud::{ routes, NextCloud, NextCloudOAuthData, NextCloudTriggerData, NextcloudServiceConfig, OcsResponse, }, External, NativeTriggerData, ServiceName, }; +use http::StatusCode; lazy_static::lazy_static! { pub static ref TOKEN_NEEDED: Box = to_raw_value(&serde_json::json!({ @@ -172,19 +173,28 @@ impl External for NextCloud { ) .await?; - // Fetch back the updated state and convert to JSON config - let trigger_data = self - .get(w_id, oauth_data, external_id, db, tx) - .await? - .ok_or_else(|| { - Error::InternalErr(format!( - "Failed to fetch back trigger {} after update", - external_id - )) - })?; - serde_json::to_value(&trigger_data).map_err(|e| { - Error::internal_err(format!("Failed to convert trigger data to JSON: {}", e)) - }) + // The webhook is already updated at this point, so the read-back is an enrichment (it + // picks up whatever Nextcloud resolved server-side), not a second chance to fail. Failing + // here would report a webhook that is in fact installed as un-installed, and callers would + // then unwind a token the service is already using. + let read_back = match self.get(w_id, oauth_data, external_id, db, tx).await { + Ok(Some(trigger_data)) => serde_json::to_value(&trigger_data).ok(), + Ok(None) => None, + Err(e) => { + tracing::warn!( + "Nextcloud webhook {external_id} was updated but could not be read back, \ + storing the requested config: {e:#}" + ); + None + } + }; + + match read_back { + Some(config) => Ok(config), + None => serde_json::to_value(&data.service_config).map_err(|e| { + Error::internal_err(format!("Failed to convert trigger data to JSON: {}", e)) + }), + } } async fn get( @@ -226,11 +236,13 @@ impl External for NextCloud { let mut headers = HashMap::new(); headers.insert("OCS-APIRequest".to_string(), "true".to_string()); + // A webhook already removed in Nextcloud is the outcome this call wants, so only 404 is + // swallowed; anything else would leave a live webhook behind while Windmill forgets it. let _: serde_json::Value = self .http_client_request::<_, ()>(&url, Method::DELETE, w_id, db, Some(headers), None) .await - .or_else(|e| match &e { - Error::InternalErr(msg) if msg.contains("404") => Ok(serde_json::Value::Null), + .or_else(|e| match http_error_status(&e) { + Some(StatusCode::NOT_FOUND) => Ok(serde_json::Value::Null), _ => Err(e), })?; @@ -256,7 +268,10 @@ impl External for NextCloud { ); errors.push(crate::sync::SyncError { resource_path: format!("workspace:{}", workspace_id), - error_message: format!("Failed to fetch external triggers: {}", e), + error_message: format!( + "Failed to fetch external triggers: {}", + crate::external_error_message(&e) + ), error_type: "external_service_error".to_string(), }); return; @@ -294,6 +309,28 @@ impl External for NextCloud { fn additional_routes(&self) -> axum::Router { routes::nextcloud_routes(self.clone()) } + + fn describe_error_body(&self, body: &str) -> Option { + let ocs: OcsResponse = serde_json::from_str(body).ok()?; + Some(ocs.ocs.meta.message).filter(|m| !m.is_empty()) + } + + /// Appended to every failed call of this service, so a hint may not assert what the caller + /// was doing — only name the requirement that most often explains the status. + fn error_hint(&self, status: StatusCode) -> Option<&'static str> { + match status { + StatusCode::UNAUTHORIZED => { + Some("reconnect the Nextcloud integration from Workspace settings > Integrations.") + } + StatusCode::FORBIDDEN => Some( + "Nextcloud grants webhook management to administrators only, and a Windmill \ + admin is a different thing. If the connected Nextcloud account is not an admin \ + there and holds no delegated admin rights for the Webhooks setting, reconnect \ + the integration from Workspace settings > Integrations with one that does.", + ), + _ => None, + } + } } impl NextCloud { diff --git a/backend/windmill-native-triggers/src/nextcloud/routes.rs b/backend/windmill-native-triggers/src/nextcloud/routes.rs index 2f54124d64..a1c679602c 100644 --- a/backend/windmill-native-triggers/src/nextcloud/routes.rs +++ b/backend/windmill-native-triggers/src/nextcloud/routes.rs @@ -10,7 +10,7 @@ use windmill_common::{ use windmill_api_auth::ApiAuthed; use crate::{ - get_workspace_integration, + get_workspace_integration, map_external_error, nextcloud::{NextCloudEventType, OcsResponse}, require_native_integration_use, External, ServiceName, }; @@ -47,7 +47,8 @@ async fn list_available_events( Some(headers), None, ) - .await?; + .await + .map_err(map_external_error)?; let events = serde_json::from_str(&ocs_response.ocs.data) .map_err(|e| Error::InternalErr(format!("Failed to parse NextCloud events data: {}", e)))?; diff --git a/backend/windmill-native-triggers/src/rename.rs b/backend/windmill-native-triggers/src/rename.rs new file mode 100644 index 0000000000..a7168f5c7e --- /dev/null +++ b/backend/windmill-native-triggers/src/rename.rs @@ -0,0 +1,244 @@ +//! Keeping native triggers usable when their runnable is renamed. + +use std::collections::HashMap; + +use windmill_api_auth::ApiAuthed; +use windmill_audit::{audit_oss::audit_log, ActionKind}; +use windmill_common::{ + error::{Error, Result}, + triggers::MovedNativeTrigger, + DB, +}; + +use crate::{ + decrypt_oauth_data, delete_token_by_hash, get_native_trigger, github::GitHub, google::Google, + handler::new_webhook_token, lock::TriggerLock, nextcloud::NextCloud, record_reregistration, + update_native_trigger_error, External, NativeTriggerData, ServiceName, +}; + +/// Re-register the webhooks of the native triggers a rename moved onto a new runnable path. +/// +/// The URL held by the external service embeds the runnable path and a token scoped to it, so +/// after a rename the registration points at a path that no longer resolves. +/// +/// `moved` comes from `windmill_common::triggers::update_triggers_script_path`. Callers MUST have +/// already committed that rename and verified the caller's write access to the path the rows now +/// carry — this mints fresh `jobs:run:*` tokens for it. Run it *after* the commit: repointing a +/// webhook is not undoable, so doing it while the deploy transaction can still roll back would +/// strand the trigger on a path and token that never existed. +/// +/// The replacement token belongs to `authed`, so the runnable now executes as whoever deployed the +/// rename rather than as the trigger's previous owner — the same identity swap a manual trigger +/// edit performs, since a path-scoped token cannot outlive the path it names. +/// +/// A service that rejects the update gets the failure recorded on its trigger row rather than +/// propagated: the runnable is renamed either way, and the user can retry by re-saving the trigger. +pub async fn reregister_triggers_after_rename( + db: &DB, + authed: &ApiAuthed, + w_id: &str, + moved: &[MovedNativeTrigger], +) { + let mut by_service: HashMap> = HashMap::new(); + for trigger in moved { + match ServiceName::try_from(trigger.service_name.clone()) { + Ok(service) => by_service.entry(service).or_default().push(trigger), + Err(e) => tracing::error!( + "Unknown native trigger service '{}' on trigger '{}' in workspace '{w_id}': {e:#}", + trigger.service_name, + trigger.external_id + ), + } + } + + for (service, of_service) in by_service { + match service { + ServiceName::Nextcloud => { + reregister_service(db, authed, w_id, &of_service, NextCloud).await + } + ServiceName::Google => reregister_service(db, authed, w_id, &of_service, Google).await, + ServiceName::Github => reregister_service(db, authed, w_id, &of_service, GitHub).await, + } + } +} + +async fn reregister_service( + db: &DB, + authed: &ApiAuthed, + w_id: &str, + moved: &[&MovedNativeTrigger], + handler: T, +) { + let oauth_data: T::OAuthData = + match decrypt_oauth_data(db, w_id, T::SERVICE_NAME.integration_service()).await { + Ok(oauth_data) => oauth_data, + Err(e) => { + for trigger in moved { + record_failure::(db, w_id, &trigger.external_id, &e).await; + } + return; + } + }; + + for trigger in moved { + if let Err(e) = reregister_one(db, authed, w_id, &handler, &oauth_data, trigger).await { + record_failure::(db, w_id, &trigger.external_id, &e).await; + } + } +} + +/// Point one trigger's webhook at the runnable the rename moved it to. +/// +/// `TriggerLock` excludes the other operations that touch a registration — edits, deletes, channel +/// renewal, another re-registration — for the whole span including the network call. Renames are +/// not among them: they move the row from inside the runnable's own transaction without taking +/// this lock, so a further rename can still land mid-call. That is why the write-back is +/// additionally conditional on `updated_at` rather than trusting the row read here. +async fn reregister_one( + db: &DB, + authed: &ApiAuthed, + w_id: &str, + handler: &T, + oauth_data: &T::OAuthData, + moved: &MovedNativeTrigger, +) -> Result<()> { + let external_id = moved.external_id.as_str(); + let mut lock = TriggerLock::acquire(db, w_id, T::SERVICE_NAME, external_id).await?; + + let trigger = get_native_trigger(db, w_id, T::SERVICE_NAME, external_id) + .await? + .ok_or_else(|| Error::NotFound(format!("Native trigger not found: {external_id}")))?; + + // Something took the lock first and pointed the trigger elsewhere. That operation authorized + // its own destination and installed its own registration; re-registering here would overwrite + // both with a token minted for a runnable this rename never authorized. + if trigger.script_path != moved.script_path || trigger.is_flow != moved.is_flow { + tracing::info!( + "Skipping re-registration of the {} trigger '{external_id}': it was moved to '{}' \ + after the rename that queued this", + T::SERVICE_NAME, + trigger.script_path + ); + return lock.release().await; + } + + let service_config: T::ServiceConfig = serde_json::from_value( + trigger + .service_config + .clone() + .unwrap_or(serde_json::Value::Null), + ) + .map_err(|e| Error::internal_err(format!("stored trigger config cannot be read back: {e}")))?; + + let data = NativeTriggerData { + script_path: trigger.script_path.clone(), + is_flow: trigger.is_flow, + service_config, + summary: trigger.summary.clone(), + }; + + // The token is scoped to the runnable path and only its hash is kept, so pointing the webhook + // at the new path means minting a replacement rather than reusing the old one. Commit it + // before handing it out: a service may call back the moment it accepts the new URL. + let mut tx = db.begin().await?; + let webhook_token = new_webhook_token( + &mut tx, + db, + authed, + &trigger.script_path, + trigger.is_flow, + w_id, + T::SERVICE_NAME, + ) + .await?; + tx.commit().await?; + + let updated = handler + .update( + w_id, + oauth_data, + external_id, + &webhook_token, + &data, + db, + lock.conn(), + ) + .await; + + // A failed `update` does not mean the service never installed the token — Nextcloud mutates and + // then reads back, and any service can fail on the response after committing the mutation. So + // the token stays valid; deleting one the service did install would turn a webhook that still + // works into one that 401s. + let service_config = updated.inspect_err(|_| { + tracing::warn!( + "The webhook token minted for the {} trigger '{external_id}' was kept even though no \ + row references it, since the service may have installed it", + T::SERVICE_NAME + ) + })?; + + let mut tx = db.begin().await?; + + let applied = record_reregistration( + &mut *tx, + w_id, + T::SERVICE_NAME, + external_id, + &webhook_token, + service_config, + trigger.updated_at, + ) + .await?; + + if !applied { + // Written again while this was on the network — another rename, a save, or a disconnect + // that removed the row. Leave whatever is there now, including a newer rename's pending + // marker, for its own writer to finish, and keep this token: the service may be holding + // it. If the row is gone outright, the disconnect path revokes every token it deletes. + tx.rollback().await?; + tracing::info!( + "Discarding the re-registration of the {} trigger '{external_id}': the row changed \ + while the service was being updated", + T::SERVICE_NAME + ); + return lock.release().await; + } + + delete_token_by_hash(&mut *tx, &trigger.webhook_token_hash).await?; + + audit_log( + &mut *tx, + authed, + &format!("native_triggers.{}.update", T::SERVICE_NAME), + ActionKind::Update, + w_id, + Some(external_id), + Some([("reason", "runnable renamed")].into()), + ) + .await?; + + tx.commit().await?; + lock.release().await?; + + Ok(()) +} + +async fn record_failure(db: &DB, w_id: &str, external_id: &str, err: &Error) { + tracing::error!( + "Failed to re-register the {} trigger '{external_id}' in workspace '{w_id}' after rename: {err:#}", + T::SERVICE_NAME, + ); + let message = format!( + "Could not re-point the webhook registered on {} at the renamed runnable, so it is no \ + longer known to be delivering: {err}. Save the trigger again to retry.", + T::DISPLAY_NAME + ); + if let Err(e) = + update_native_trigger_error(db, w_id, T::SERVICE_NAME, external_id, Some(&message)).await + { + tracing::error!( + "Failed to record the re-registration failure of the {} trigger '{external_id}': {e:#}", + T::SERVICE_NAME, + ); + } +} diff --git a/backend/windmill-native-triggers/src/sync.rs b/backend/windmill-native-triggers/src/sync.rs index a544ea1625..bc259f048e 100644 --- a/backend/windmill-native-triggers/src/sync.rs +++ b/backend/windmill-native-triggers/src/sync.rs @@ -12,6 +12,11 @@ use crate::{ update_native_trigger_service_config, External, NativeTrigger, }; +/// The one `native_trigger.error` an existence check owns. Finding the trigger on the service +/// may only clear this value: every other failure recorded there — a webhook left aimed at a +/// pre-rename path, a channel that failed to renew — is one existence cannot disprove. +pub const EXTERNAL_TRIGGER_MISSING_ERROR: &str = "Trigger no longer exists on external service"; + #[derive(Debug, Serialize)] pub struct TriggerSyncInfo { pub external_id: String, @@ -290,7 +295,7 @@ pub async fn reconcile_with_external_state( for trigger in windmill_triggers { if !external_trigger_map.contains_key(&trigger.external_id) { // Trigger no longer exists on external service - set error - let error_msg = "Trigger no longer exists on external service".to_string(); + let error_msg = EXTERNAL_TRIGGER_MISSING_ERROR.to_string(); if trigger.error.as_deref() != Some(&error_msg) { tracing::info!( @@ -336,8 +341,10 @@ pub async fn reconcile_with_external_state( // Trigger exists on external service let external_service_config = external_trigger_map.get(&trigger.external_id).unwrap(); - // Clear error if it was set - if trigger.error.is_some() { + // Only clear the error this reconciler itself sets. Existence says nothing about the + // failures other paths record — a webhook left aimed at a pre-rename path is present + // here yet still broken, and wiping its error would erase the only signal the user has. + if trigger.error.as_deref() == Some(EXTERNAL_TRIGGER_MISSING_ERROR) { tracing::info!( "Trigger (external_id: '{}', script_path: '{}') exists on external service, clearing error", trigger.external_id, diff --git a/backend/windmill-native-triggers/src/workspace_integrations.rs b/backend/windmill-native-triggers/src/workspace_integrations.rs index 18f431812a..eccfa56f19 100644 --- a/backend/windmill-native-triggers/src/workspace_integrations.rs +++ b/backend/windmill-native-triggers/src/workspace_integrations.rs @@ -344,24 +344,26 @@ async fn delete_triggers_for_service(db: &DB, workspace_id: &str, service_name: } // For Google: skip remote cleanup (watch channels expire naturally) - // Bulk delete all triggers - if let Err(e) = sqlx::query!( - "DELETE FROM native_trigger WHERE workspace_id = $1 AND service_name = $2", + // Revoke the tokens the deleted rows actually named, not the ones listed above: a + // re-registration running concurrently mints a replacement, and leaving that alive would let a + // disconnected integration keep starting jobs. + let deleted_token_hashes = sqlx::query_scalar!( + "DELETE FROM native_trigger WHERE workspace_id = $1 AND service_name = $2 RETURNING webhook_token_hash", workspace_id, service_name as ServiceName ) - .execute(db) + .fetch_all(db) .await - { + .unwrap_or_else(|e| { tracing::error!("Failed to delete native triggers for service {service_name:?} in workspace {workspace_id}: {e}"); - } + Vec::new() + }); - // Delete all associated webhook tokens - for trigger in &triggers { - if let Err(e) = delete_token_by_hash(db, &trigger.webhook_token_hash).await { + for webhook_token_hash in &deleted_token_hashes { + if let Err(e) = delete_token_by_hash(db, webhook_token_hash).await { tracing::error!( "Failed to delete webhook token with hash {}: {e}", - trigger.webhook_token_hash + webhook_token_hash ); } } diff --git a/backend/windmill-oauth/src/lib.rs b/backend/windmill-oauth/src/lib.rs index e314f13d75..32648a2115 100644 --- a/backend/windmill-oauth/src/lib.rs +++ b/backend/windmill-oauth/src/lib.rs @@ -32,7 +32,8 @@ pub type DB = sqlx::Pool; // Re-export oauth2 types that consumers need (also used internally) pub use oauth2::{ - helpers, AccessToken, AuthType, Client as OClient, RefreshToken, Scope, State, Url, + helpers, AccessToken, AuthType, Client as OClient, ErrorField, ExecuteError, RefreshToken, + Scope, State, Url, }; // Re-export reqwest Client (version 0.12 compatible with async-oauth2) diff --git a/backend/windmill-object-store/src/lib.rs b/backend/windmill-object-store/src/lib.rs index 9121f6183d..2482cf14dd 100644 --- a/backend/windmill-object-store/src/lib.rs +++ b/backend/windmill-object-store/src/lib.rs @@ -355,10 +355,41 @@ pub async fn attempt_fetch_bytes( return Ok(bytes); } +/// Whether an S3 resource carries static credentials. When it does not, the +/// ambient AWS chain (env, profile, ECS/EC2 instance role) is used instead. +/// Shared so callers that sign requests by hand resolve credentials on exactly the +/// same condition as `build_s3_client`. +#[cfg(feature = "parquet")] +pub fn s3_resource_has_static_credentials(s3_resource: &S3Resource) -> bool { + s3_resource.access_key.as_ref().is_some_and(|x| x != "") + || s3_resource.secret_key.as_ref().is_some_and(|x| x != "") +} + +/// Ambient AWS credentials from the shared, cached provider backing +/// `build_s3_client`. Callers that sign their own requests must go through this +/// rather than resolving the default chain themselves: the cache is what keeps a +/// burst of requests from hitting the instance metadata service once each. +/// +/// These are the **instance's own** credentials, not any caller's, and they are +/// returned in the clear. A caller therefore MUST: +/// - authorize the request target itself — reaching this function implies no +/// permission check, and the credentials typically outrank the requesting user; +/// - use them only to sign a request it has already authorized, never surface them +/// in a response, log, or error message, and never hand them to a caller-supplied +/// endpoint. +/// +/// Prefer `build_s3_client`, which confines them to the object-store client; reach +/// for this only where a request must be signed by hand. +#[cfg(feature = "parquet")] +pub async fn ambient_aws_credentials( + region: &str, +) -> anyhow::Result { + ambient_aws_credentials_provider(region).await.get().await +} + #[cfg(feature = "parquet")] pub async fn build_s3_client(s3_resource_ref: &S3Resource) -> error::Result> { - let static_creds = s3_resource_ref.access_key.as_ref().is_some_and(|x| x != "") - || s3_resource_ref.secret_key.as_ref().is_some_and(|x| x != ""); + let static_creds = s3_resource_has_static_credentials(s3_resource_ref); let credentials_provider = if !static_creds { Some(ambient_aws_credentials_provider(&s3_resource_ref.region).await) diff --git a/backend/windmill-queue/src/asset_dispatch.rs b/backend/windmill-queue/src/asset_dispatch.rs index 29f44b153d..af9ca957b9 100644 --- a/backend/windmill-queue/src/asset_dispatch.rs +++ b/backend/windmill-queue/src/asset_dispatch.rs @@ -66,7 +66,7 @@ use windmill_common::error::{self, Result}; use windmill_common::get_latest_deployed_hash_for_path; use windmill_common::jobs::{JobKind, JobPayload, JobTriggerKind}; use windmill_common::partition::PARTITION_ARG; -use windmill_common::scripts::ScriptHash; +use windmill_common::scripts::{ScriptHash, ScriptLang}; use windmill_common::triggers::TriggerMetadata; use windmill_common::users::{get_email_from_permissioned_as, username_to_permissioned_as}; use windmill_common::worker::to_raw_value; @@ -245,6 +245,18 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result if job.parent_job.is_some() && !is_native_retry_attempt(db, job).await? { return Ok(DispatchResult::default()); } + // A dbt run records the relations it builds, so it looks like a producer + // here — but dbt does not trigger downstream runs. Its own DAG is dbt's to + // order; the only thing a cascade would add is waking Windmill scripts that + // read a mart, and nothing outside dbt can declare a `dbt://` write, so + // that edge exists in one direction only. Cascading from a project whose + // per-run selection can build any subset of itself needs a per-run write set + // to be correct, which is a design worth doing deliberately rather than + // inferring. Until then dbt materializes and reports; it does not dispatch. + if job.script_lang == Some(ScriptLang::Dbt) { + return Ok(DispatchResult::default()); + } + let runnable_path = match job.runnable_path.as_deref() { Some(p) if !p.is_empty() => p, _ => return Ok(DispatchResult::default()), @@ -263,6 +275,7 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result return Ok(DispatchResult::default()); }; + let args = fetch_args(db, &job.workspace_id, job.id).await?; if read_skip_arg(args.as_ref()) { return Ok(DispatchResult::default()); @@ -753,6 +766,9 @@ async fn push_subscriber( .await? .prefetch_cached(db) .await?; + let on_behalf_of = script + .on_behalf_of(&producer.workspace_id, db) + .await?; let hash = ScriptHash(script.hash); let tag = script.tag; let concurrency_settings = script.runnable_settings.concurrency_settings; @@ -811,15 +827,17 @@ async fn push_subscriber( // producer's. Subscriptions are workspace-wide, so attributing the run // to the producer would let anyone who can deploy a `// on` script // execute code with the permissions of whoever happens to write the - // asset (e.g. an admin's scheduled job). `on_behalf_of_email` (an - // explicit service-account opt-in at deploy) takes precedence for the - // email; otherwise the deployer's email is resolved from their - // username. - let permissioned_as = username_to_permissioned_as(&script.created_by); - let email = match script.on_behalf_of_email { - Some(obo) => obo, + // asset (e.g. an admin's scheduled job). An on-behalf-of identity (an + // explicit service-account opt-in at deploy) takes precedence; otherwise + // the deployer's email is resolved from their username. + let (email, permissioned_as) = match on_behalf_of { + Some(obo) => (obo.email, obo.permissioned_as), None => { - get_email_from_permissioned_as(&permissioned_as, &producer.workspace_id, db).await? + let permissioned_as = username_to_permissioned_as(&script.created_by); + let email = + get_email_from_permissioned_as(&permissioned_as, &producer.workspace_id, db) + .await?; + (email, permissioned_as) } }; @@ -862,6 +880,7 @@ async fn push_subscriber( permissioned_as, Some(producer_path), None, + None, Some(producer_path.to_string()), None, None, diff --git a/backend/windmill-queue/src/ducklake_maintenance_oss.rs b/backend/windmill-queue/src/ducklake_maintenance_oss.rs index ab6e035847..8f983c344a 100644 --- a/backend/windmill-queue/src/ducklake_maintenance_oss.rs +++ b/backend/windmill-queue/src/ducklake_maintenance_oss.rs @@ -72,7 +72,7 @@ pub async fn sync_ducklake_maintenance_schedules<'c>( /// Build the job payload for one occurrence of a managed maintenance schedule /// (`push_scheduled_job` calls this for reserved-prefix schedule paths). -/// Returns `(payload, tag, timeout, on_behalf_of_email, created_by)`. +/// Returns `(payload, tag, timeout, on_behalf_of)`. /// /// Always `Ok(None)` in the public build: the caller falls through to normal /// script resolution, so a pre-existing user schedule under a real @@ -88,8 +88,7 @@ pub async fn build_maintenance_schedule_payload<'c>( JobPayload, Option, Option, - Option, - String, + Option, )>, > { Ok(None) diff --git a/backend/windmill-queue/src/jobs.rs b/backend/windmill-queue/src/jobs.rs index c4f83f4c4b..fd80a741dc 100644 --- a/backend/windmill-queue/src/jobs.rs +++ b/backend/windmill-queue/src/jobs.rs @@ -41,7 +41,7 @@ use windmill_common::audit::AuditAuthor; use windmill_common::auth::JobPerms; #[cfg(feature = "benchmark")] use windmill_common::bench::BenchmarkIter; -use windmill_common::jobs::{JobTriggerKind, EMAIL_ERROR_HANDLER_USER_EMAIL}; +use windmill_common::jobs::{JobTriggerKind, TriggerKindLabel, EMAIL_ERROR_HANDLER_USER_EMAIL}; use windmill_common::min_version::{ MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2, }; @@ -486,6 +486,7 @@ pub async fn push_init_job<'c>( None, None, None, + None, false, true, None, @@ -545,6 +546,7 @@ pub async fn push_periodic_bash_job<'c>( None, None, None, + None, false, true, None, @@ -1694,6 +1696,7 @@ async fn restart_job_if_perpetual_inner( &queued_job.permissioned_as_email, queued_job.permissioned_as.clone(), Some(&format!("add.completed.job{}", queued_job.id)), + None, scheduled_for, queued_job.schedule_path(), None, @@ -1996,6 +1999,7 @@ pub async fn maybe_enqueue_native_script_retry( &job.permissioned_as_email, job.permissioned_as.clone(), Some(&format!("retry.{}", job.id)), + None, Some(scheduled_for), None, Some(root), @@ -2839,6 +2843,7 @@ pub async fn push_error_handler<'a, 'c, T: Serialize + Send + Sync>( Some(&format!("error.handler.{job_id}")), None, None, + None, Some(job_id), None, Some(job_id), @@ -2927,6 +2932,7 @@ pub async fn push_success_handler<'a, 'c, T: Serialize + Send + Sync>( Some(&format!("success.handler.{job_id}")), None, None, + None, Some(job_id), // parent_job Some(job_id), // root_job Some(job_id), // flow_innermost_root_job @@ -2998,7 +3004,7 @@ pub struct MiniPulledJob { pub preprocessed: Option, pub script_entrypoint_override: Option, pub trigger: Option, - pub trigger_kind: Option, + pub trigger_kind: Option, pub visible_to_owner: bool, pub permissioned_as_end_user_email: Option, pub runnable_settings_handle: Option, @@ -3074,7 +3080,7 @@ pub struct MiniCompletedJob { pub script_lang: Option, pub permissioned_as_email: String, pub flow_step_id: Option, - pub trigger_kind: Option, + pub trigger_kind: Option, pub trigger: Option, pub priority: Option, pub concurrent_limit: Option, @@ -3101,7 +3107,7 @@ impl From for MiniCompletedJob { script_lang: job.script_lang, permissioned_as_email: job.permissioned_as_email, flow_step_id: job.flow_step_id, - trigger_kind: job.trigger_kind, + trigger_kind: job.trigger_kind.map(Into::into), trigger: job.trigger, priority: job.priority, concurrent_limit: job.concurrent_limit, @@ -3190,12 +3196,12 @@ impl MiniCompletedJob { } fn schedule_path( - trigger_kind: &Option, + trigger_kind: &Option, trigger: &Option, ) -> Option { if trigger_kind .as_ref() - .is_some_and(|t| matches!(t, JobTriggerKind::Schedule)) + .is_some_and(|t| t.is(JobTriggerKind::Schedule)) { trigger.clone() } else { @@ -3273,11 +3279,10 @@ impl MiniPulledJob { preprocessed: job.preprocessed.clone(), script_entrypoint_override: job.script_entrypoint_override.clone(), trigger: job.schedule_path.clone(), - trigger_kind: if job.schedule_path.is_some() { - Some(JobTriggerKind::Schedule) - } else { - None - }, + trigger_kind: job + .schedule_path + .is_some() + .then(|| JobTriggerKind::Schedule.into()), visible_to_owner: job.visible_to_owner.clone(), permissioned_as_end_user_email: None, } @@ -3375,7 +3380,15 @@ impl PulledJob { Some(is_operator), Some(groups), Some(folders), - ) => Some(JobPerms { email, username, is_admin, is_operator, groups, folders }), + ) => Some(JobPerms { + email, + username, + is_admin, + is_operator, + groups, + folders, + end_user_email: self.job.permissioned_as_end_user_email.clone(), + }), _ => None, }; @@ -3397,13 +3410,10 @@ impl PulledJob { pub async fn create_token(db: &DB, job: &MiniPulledJob, perms: Option) -> String { // skipping test runs if job.workspace_id != "" { - let label = if job.permissioned_as != format!("u/{}", job.created_by) - && job.permissioned_as != job.created_by - { - format!("ephemeral-script-end-user-{}", job.created_by) - } else { - "ephemeral-script".to_string() - }; + let label = windmill_common::auth::ephemeral_script_token_label( + &job.permissioned_as, + &job.created_by, + ); windmill_common::auth::create_token_for_owner( db, &job.workspace_id, @@ -3482,7 +3492,7 @@ pub async fn get_mini_pulled_job<'c>( preprocessed, script_entrypoint_override, trigger, - trigger_kind as \"trigger_kind: JobTriggerKind\", + trigger_kind as \"trigger_kind: TriggerKindLabel\", visible_to_owner, NULL as permissioned_as_end_user_email FROM v2_job_queue INNER JOIN v2_job ON v2_job.id = v2_job_queue.id LEFT JOIN v2_job_status ON v2_job_status.id = v2_job_queue.id WHERE v2_job_queue.id = $1", @@ -3509,7 +3519,7 @@ pub struct QueuedJobV2 { pub script_lang: Option, pub permissioned_as_email: String, pub flow_step_id: Option, - pub trigger_kind: Option, + pub trigger_kind: Option, pub trigger: Option, pub priority: Option, pub concurrent_limit: Option, @@ -3551,7 +3561,7 @@ pub async fn get_queued_job_v2<'c>( script_lang as "script_lang: ScriptLang", permissioned_as_email, flow_step_id, - trigger_kind as "trigger_kind: JobTriggerKind", + trigger_kind as "trigger_kind: TriggerKindLabel", trigger, q.priority, concurrent_limit, @@ -5051,7 +5061,7 @@ pub fn get_mini_completed_job<'a, 'e, A: sqlx::Acquire<'e, Database = Postgres> MiniCompletedJob, "SELECT j.id, j.workspace_id, j.runnable_id AS \"runnable_id: ScriptHash\", q.scheduled_for, q.started_at, j.parent_job, j.flow_innermost_root_job, j.runnable_path, j.kind as \"kind!: JobKind\", j.permissioned_as, - j.created_by, j.script_lang AS \"script_lang: ScriptLang\", j.permissioned_as_email, j.flow_step_id, j.trigger_kind AS \"trigger_kind: JobTriggerKind\", j.trigger, j.priority, j.concurrent_limit, j.tag, j.cache_ttl, q.cache_ignore_s3_path, q.runnable_settings_handle + j.created_by, j.script_lang AS \"script_lang: ScriptLang\", j.permissioned_as_email, j.flow_step_id, j.trigger_kind AS \"trigger_kind: TriggerKindLabel\", j.trigger, j.priority, j.concurrent_limit, j.tag, j.cache_ttl, q.cache_ignore_s3_path, q.runnable_settings_handle FROM v2_job j LEFT JOIN v2_job_queue q ON j.id = q.id WHERE j.id = $1 AND j.workspace_id = $2", id, @@ -5276,6 +5286,11 @@ pub async fn push<'c, 'd>( email: &str, permissioned_as: String, token_prefix: Option<&str>, + // Identifies the caller in the audit trail — distinct from the `end_user_email` below, + // which the job itself reads. The API sources it from `ApiAuthed::username_override`; + // a token label reaches audit through here alone, since `user` is the token owner. + // Callers that know no caller beyond `user` pass None. + audit_end_user: Option<&str>, scheduled_for_o: Option>, schedule_path: Option, parent_job: Option, @@ -5306,6 +5321,7 @@ pub async fn push<'c, 'd>( email, permissioned_as, token_prefix, + audit_end_user, scheduled_for_o, schedule_path, parent_job, @@ -5340,6 +5356,7 @@ async fn push_inner<'c, 'd>( mut email: &str, mut permissioned_as: String, token_prefix: Option<&str>, + audit_end_user: Option<&str>, #[allow(unused_mut)] mut scheduled_for_o: Option>, schedule_path: Option, //should be removed in favor of the trigger param below parent_job: Option, @@ -6841,20 +6858,27 @@ async fn push_inner<'c, 'd>( JobKind::UnassignedSinglestepFlow => "jobs.run.unassigned_singlestepflow", }; - let audit_author = if format!("u/{user}") != permissioned_as && user != permissioned_as { - AuditAuthor { - email: email.to_string(), - username: windmill_common::auth::permissioned_as_to_username(&permissioned_as), - username_override: Some(user.to_string()), - token_prefix: token_prefix.map(|s| s.to_string()), - } - } else { - AuditAuthor { - email: email.to_string(), - username: user.to_string(), - username_override: None, - token_prefix: token_prefix.map(|s| s.to_string()), - } + let runs_on_behalf = format!("u/{user}") != permissioned_as && user != permissioned_as; + // `user` is the end user of `permissioned_as` when the job runs as someone else, and + // `audit_end_user` takes the one `end_user` slot ahead of it. Record it here so it stays + // searchable: `username` holds the run-as identity in that case, not the caller. + if runs_on_behalf && audit_end_user.is_some_and(|e| e != user) { + hm.insert("created_by", user); + } + let audit_author = AuditAuthor { + email: email.to_string(), + username: if runs_on_behalf { + windmill_common::auth::permissioned_as_to_username(&permissioned_as) + } else { + user.to_string() + }, + // `username` is the identity the job runs as; `end_user` identifies the caller + // behind it, as it does for an app run. An explicitly passed one therefore wins + // over `user`, which only stands in for a caller when the job runs as someone else. + username_override: audit_end_user + .map(|s| s.to_string()) + .or_else(|| runs_on_behalf.then(|| user.to_string())), + token_prefix: token_prefix.map(|s| s.to_string()), }; if let Some(ref stringified_args) = stringified_args { diff --git a/backend/windmill-queue/src/schedule.rs b/backend/windmill-queue/src/schedule.rs index 55944b08ca..10f4a2b408 100644 --- a/backend/windmill-queue/src/schedule.rs +++ b/backend/windmill-queue/src/schedule.rs @@ -22,6 +22,7 @@ use windmill_common::get_latest_flow_version_id_for_path; use windmill_common::jobs::check_tag_available_for_workspace_internal; use windmill_common::jobs::JobPayload; use windmill_common::jobs::JobTriggerKind; +use windmill_common::jobs::OnBehalfOf; use windmill_common::runnable_settings::ConcurrencySettings; use windmill_common::runnable_settings::DebouncingSettings; use windmill_common::schedule::schedule_to_user; @@ -34,19 +35,18 @@ use windmill_common::DB; use windmill_common::{ error::{self, Result}, schedule::Schedule, - users::username_to_permissioned_as, utils::{now_from_db, ScheduleType, StripPath}, }; /// Helper to fetch metadata for a schedule's script or flow async fn get_schedule_metadata<'c>( tx: &mut sqlx::Transaction<'c, sqlx::Postgres>, + db: &DB, schedule: &Schedule, ) -> Result<( Option, // tag Option, // timeout - Option, // on_behalf_of_email - String, // created_by + Option, // identity the runnable is deployed to run as Option, // hash (for scripts) Option, // flow_version (for flows) Option, // retry @@ -66,20 +66,18 @@ async fn get_schedule_metadata<'c>( ) .await?; - let FlowVersionInfo { tag, on_behalf_of_email, edited_by, .. } = - get_flow_version_info_from_version( - &mut **tx, - version, - &schedule.workspace_id, - &schedule.script_path, - ) - .await?; + let flow_info = get_flow_version_info_from_version( + &mut **tx, + version, + &schedule.workspace_id, + &schedule.script_path, + ) + .await?; Ok(( - tag, + flow_info.tag.clone(), None, - on_behalf_of_email, - edited_by, + flow_info.on_behalf_of(&schedule.workspace_id, db).await?, None, Some(version), parsed_retry, @@ -99,27 +97,19 @@ async fn get_schedule_metadata<'c>( _dedicated_worker, _priority, timeout, - on_behalf_of_email, - created_by, + on_behalf_of, _runnable_settings_handle, _labels, ) = windmill_common::get_latest_hash_for_path( &mut **tx, + db, &schedule.workspace_id, &schedule.script_path, false, ) .await?; - Ok(( - tag, - timeout, - on_behalf_of_email, - created_by, - Some(hash), - None, - parsed_retry, - )) + Ok((tag, timeout, on_behalf_of, Some(hash), None, parsed_retry)) } } @@ -246,7 +236,7 @@ pub async fn push_scheduled_job<'c>( // If schedule handler is defined, wrap the scheduled job in a synthetic flow // with the handler as the first step (with stop_after_if to skip if handler returns false) - let (payload, tag, timeout, on_behalf_of_email, created_by) = if let Some(maintenance_payload) = + let (payload, tag, timeout, on_behalf_of) = if let Some(maintenance_payload) = maintenance_payload { maintenance_payload @@ -266,8 +256,8 @@ pub async fn push_scheduled_job<'c>( ); // Get metadata from the scheduled script/flow for tag, timeout, etc. - let (tag, timeout, on_behalf_of_email, created_by, hash, flow_version, retry) = - get_schedule_metadata(&mut tx, schedule).await?; + let (tag, timeout, on_behalf_of, hash, flow_version, retry) = + get_schedule_metadata(&mut tx, db, schedule).await?; ( JobPayload::SingleStepFlow { @@ -300,8 +290,7 @@ pub async fn push_scheduled_job<'c>( tag }, timeout, - on_behalf_of_email, - created_by, + on_behalf_of, ) } else if schedule.is_flow { let version = get_latest_flow_version_id_for_path( @@ -314,15 +303,7 @@ pub async fn push_scheduled_job<'c>( .warn_after_seconds_with_sql(1, "get_latest_flow_version_id_for_path".to_string()) .await?; - let FlowVersionInfo { - version, - tag, - dedicated_worker, - on_behalf_of_email, - edited_by, - labels, - .. - } = get_flow_version_info_from_version( + let flow_info = get_flow_version_info_from_version( &mut *tx, version, &schedule.workspace_id, @@ -330,6 +311,8 @@ pub async fn push_scheduled_job<'c>( ) .warn_after_seconds_with_sql(1, "get_flow_version_info_from_version".to_string()) .await?; + let on_behalf_of = flow_info.on_behalf_of(&schedule.workspace_id, db).await?; + let FlowVersionInfo { version, tag, dedicated_worker, labels, .. } = flow_info; ( JobPayload::Flow { @@ -341,8 +324,7 @@ pub async fn push_scheduled_job<'c>( }, tag, None, - on_behalf_of_email, - edited_by, + on_behalf_of, ) } else { let ( @@ -359,12 +341,12 @@ pub async fn push_scheduled_job<'c>( dedicated_worker, priority, timeout, - on_behalf_of_email, - created_by, + on_behalf_of, runnable_settings_handle, labels, ) = windmill_common::get_latest_hash_for_path( &mut *tx, + db, &schedule.workspace_id, &schedule.script_path, false, @@ -434,8 +416,7 @@ pub async fn push_scheduled_job<'c>( tag }, timeout, - on_behalf_of_email, - created_by, + on_behalf_of.clone(), ) } else { ( @@ -463,8 +444,7 @@ pub async fn push_scheduled_job<'c>( tag }, timeout, - on_behalf_of_email, - created_by, + on_behalf_of, ) } }; @@ -485,8 +465,8 @@ pub async fn push_scheduled_job<'c>( ); }; - let (email, permissioned_as, push_authed, revert_to_windmill_user) = if let Some(email) = - on_behalf_of_email.as_ref() + let (email, permissioned_as, push_authed, revert_to_windmill_user) = if let Some(obo) = + on_behalf_of.as_ref() { let is_windmill_user = sqlx::query_scalar!("SELECT CURRENT_USER = 'windmill_user' as \"is_windmill_user!\"") @@ -500,8 +480,8 @@ pub async fn push_scheduled_job<'c>( .await?; } ( - email.clone(), - username_to_permissioned_as(&created_by), + obo.email.clone(), + obo.permissioned_as.clone(), None, is_windmill_user, ) @@ -561,6 +541,7 @@ pub async fn push_scheduled_job<'c>( &email, permissioned_as, Some(&schedule.path), + None, Some(next), Some(schedule.path.clone()), None, diff --git a/backend/windmill-queue/tests/fixtures/schedule_push.sql b/backend/windmill-queue/tests/fixtures/schedule_push.sql index 9312d0bbd9..5dcc9e1528 100644 --- a/backend/windmill-queue/tests/fixtures/schedule_push.sql +++ b/backend/windmill-queue/tests/fixtures/schedule_push.sql @@ -22,13 +22,13 @@ VALUES ( 'Test script', '', 'f/system/test_script', 100001, 'deno', '', 'script' ); --- A script with on_behalf_of_email -INSERT INTO script (workspace_id, created_by, content, schema, summary, description, path, hash, language, lock, kind, on_behalf_of_email) +-- A script with an on-behalf-of identity +INSERT INTO script (workspace_id, created_by, content, schema, summary, description, path, hash, language, lock, kind, on_behalf_of) VALUES ( 'test-workspace', 'test-user', 'export async function main() { return "obo"; }', '{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"required":[],"type":"object"}', - 'OBO script', '', 'f/system/obo_script', 100002, 'deno', '', 'script', 'obo@windmill.dev' + 'OBO script', '', 'f/system/obo_script', 100002, 'deno', '', 'script', 'u/obo-user' ); -- A script with a tag @@ -66,13 +66,13 @@ VALUES ( 'test-user' ); --- A flow with on_behalf_of_email -INSERT INTO flow (workspace_id, summary, description, path, versions, schema, value, edited_by, on_behalf_of_email) +-- A flow with an on-behalf-of identity +INSERT INTO flow (workspace_id, summary, description, path, versions, schema, value, edited_by, on_behalf_of) VALUES ( 'test-workspace', 'OBO flow', '', 'f/system/obo_flow', '{200002}', '{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"required":[],"type":"object"}', '{"modules": [{"id": "a", "value": {"path": "f/system/test_script", "type": "script", "input_transforms": {}}}]}', - 'test-user', 'obo@windmill.dev' + 'test-user', 'u/obo-user' ); INSERT INTO flow_version (id, workspace_id, path, schema, value, created_by) diff --git a/backend/windmill-queue/tests/native_retry_test.rs b/backend/windmill-queue/tests/native_retry_test.rs index 6b6c2d65ce..8da40b7cad 100644 --- a/backend/windmill-queue/tests/native_retry_test.rs +++ b/backend/windmill-queue/tests/native_retry_test.rs @@ -36,7 +36,7 @@ mod native_retry { script_lang: Some(ScriptLang::Deno), permissioned_as_email: "test@windmill.dev".to_string(), flow_step_id: None, - trigger_kind: Some(JobTriggerKind::Schedule), + trigger_kind: Some(JobTriggerKind::Schedule.into()), trigger: Some(SCHED.to_string()), priority: None, concurrent_limit: None, diff --git a/backend/windmill-queue/tests/schedule_push.rs b/backend/windmill-queue/tests/schedule_push.rs index 3ac4cd2bd9..2069ce06b7 100644 --- a/backend/windmill-queue/tests/schedule_push.rs +++ b/backend/windmill-queue/tests/schedule_push.rs @@ -83,7 +83,7 @@ mod schedule_push { script_lang: None, permissioned_as_email: schedule.email.clone(), flow_step_id: None, - trigger_kind: Some(JobTriggerKind::Schedule), + trigger_kind: Some(JobTriggerKind::Schedule.into()), trigger: Some(schedule.path.clone()), priority: None, concurrent_limit: None, diff --git a/backend/windmill-store/Cargo.toml b/backend/windmill-store/Cargo.toml index 1409e30ebf..31cd0fbe0f 100644 --- a/backend/windmill-store/Cargo.toml +++ b/backend/windmill-store/Cargo.toml @@ -52,6 +52,7 @@ sql-builder.workspace = true async-recursion.workspace = true futures.workspace = true chrono.workspace = true +dashmap.workspace = true reqwest.workspace = true anyhow.workspace = true base64.workspace = true diff --git a/backend/windmill-store/src/resources.rs b/backend/windmill-store/src/resources.rs index 4ccf364cf8..b2191cf729 100644 --- a/backend/windmill-store/src/resources.rs +++ b/backend/windmill-store/src/resources.rs @@ -6,8 +6,11 @@ * LICENSE-AGPL for a copy of the license. */ +use dashmap::DashMap; use std::collections::HashMap; use std::net::IpAddr; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::LazyLock; use windmill_api_auth::{ build_scope_path_predicate, check_scopes, maybe_refresh_folders, require_owner_of_path, @@ -18,7 +21,10 @@ use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult}; use crate::secret_backend_ext::rename_vault_secret; use crate::var_resource_cache::{auth_identity, cache_resource, get_cached_resource}; -use windmill_common::utils::{escape_ilike_pattern, BulkDeleteRequest}; +use windmill_common::utils::{ + check_proper_path, check_proper_type_name, escape_ilike_pattern, sanitize_db_error, + BulkDeleteRequest, +}; use windmill_common::webhook::{WebhookMessage, WebhookShared}; use axum::{ @@ -71,6 +77,15 @@ pub fn workspaced_service() -> Router { ) .route("/update/{*path}", post(update_resource)) .route("/update_value/{*path}", post(update_resource_value)) + .route( + "/history/p/{*path}", + get(get_resource_history).delete(clear_resource_history), + ) + .route("/history/v/{version}", get(get_resource_version)) + .route( + "/history/restore/v/{version}", + post(restore_resource_version), + ) .route("/delete/{*path}", delete(delete_resource)) .route("/delete_bulk", delete(delete_resources_bulk)) .route("/create", post(create_resource)) @@ -1009,6 +1024,79 @@ async fn check_path_conflict<'c>( return Ok(()); } +/// Resource types the platform writes on its own behalf rather than users editing them: +/// `state` backs `setState` and `cache` backs cached job results, so both are rewritten on +/// every job run. Workspace export skips them for the same reason the `record_resource_version` +/// trigger does; that trigger repeats this list in SQL, so the two have to move together. +pub const INTERNAL_RESOURCE_TYPES: [&str; 2] = ["state", "cache"]; + +/// Versions retained per resource path. The monitor sweep enforces it eventually; the history +/// endpoint caps its own read at the same number so a burst of writes between two sweeps cannot +/// make the drawer fetch an unbounded list. +pub const MAX_RESOURCE_VERSIONS: i64 = 100; + +/// Above any plausible hand-editing cadence, and the rate at which the retained history stops +/// being useful: at this many writes a minute, MAX_RESOURCE_VERSIONS covers only five minutes. +/// Counting is per process, so N servers raise the effective threshold to N times this; erring +/// low is the safe direction for something that only ever logs. +const RESOURCE_WRITE_ADVISORY_PER_MIN: u32 = 20; + +struct ResourceWriteRate { + count: u32, + minute_bucket: i64, +} + +/// Writes seen per (workspace, path) per minute. Purely advisory, and deliberately so: nothing +/// is throttled, the count is per process and resets on restart, so it undercounts across +/// servers. That is affordable for a log line and is what keeps this off the write path proper. +static RESOURCE_WRITE_RATES: LazyLock> = + LazyLock::new(DashMap::new); +static RESOURCE_WRITES_SEEN: AtomicU64 = AtomicU64::new(0); + +/// Notice a caller rewriting one resource in a loop and point them at a store meant for it. +/// Counts writes rather than versions: an unchanged value records nothing, but it still costs a +/// row rewrite, an audit entry and a webhook, and is still a caller who wants a different store. +fn note_resource_write(w_id: &str, path: &str, resource_type: &str) { + // `state` and `cache` are rewritten once per job by design. Counting them would fire the + // advisory hardest on exactly the traffic it is not about, which is how a warning becomes + // noise people filter out. + if INTERNAL_RESOURCE_TYPES.contains(&resource_type) { + return; + } + let minute_bucket = chrono::Utc::now().timestamp() / 60; + // The entry guard holds a lock on its DashMap shard, and `retain` below takes every shard. + // Keeping the guard alive across that call deadlocks the request handler, so this block is + // load-bearing: it must end before the eviction, not be flattened into the function body. + let reached_cap = { + let mut rate = RESOURCE_WRITE_RATES + .entry((w_id.to_string(), path.to_string())) + .or_insert(ResourceWriteRate { count: 0, minute_bucket }); + if rate.minute_bucket != minute_bucket { + rate.minute_bucket = minute_bucket; + rate.count = 0; + } + rate.count += 1; + rate.count == RESOURCE_WRITE_ADVISORY_PER_MIN + }; + // Bounded without a background task: periodically drop what neither the current nor the + // previous minute can still need. + if RESOURCE_WRITES_SEEN.fetch_add(1, Ordering::Relaxed) % 256 == 0 { + RESOURCE_WRITE_RATES.retain(|_, rate| rate.minute_bucket >= minute_bucket - 1); + } + // Once per minute per path: `==` rather than `>=` so a sustained loop logs at the crossing + // and then stays quiet until the bucket rolls over. + if reached_cap { + tracing::warn!( + workspace_id = %w_id, + path = %path, + "resource written {} times in a minute; each write rewrites the whole value, and a \ + changed one also records a version. High-frequency writes belong in a state resource, \ + object storage, or a datatable.", + RESOURCE_WRITE_ADVISORY_PER_MIN + ); + } +} + #[derive(Deserialize)] struct CreateResourceQuery { update_if_exists: Option, @@ -1023,6 +1111,8 @@ async fn create_resource( Json(resource): Json, ) -> Result<(StatusCode, String)> { check_scopes(&authed, || format!("resources:write:{}", resource.path))?; + check_proper_path(&resource.path)?; + check_proper_type_name(&resource.resource_type)?; if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, AuditAuthorable::username(&authed), @@ -1100,7 +1190,8 @@ async fn create_resource( resource.labels.as_deref() as Option<&[String]> ) .execute(&mut *tx) - .await?; + .await + .map_err(sanitize_db_error)?; } else { // Create-only (the default): DO NOTHING + a row-count guard, so a path that appears between // check_path_conflict above and this insert is rejected rather than overwritten. A plain @@ -1119,7 +1210,8 @@ async fn create_resource( resource.labels.as_deref() as Option<&[String]> ) .execute(&mut *tx) - .await?; + .await + .map_err(sanitize_db_error)?; if inserted.rows_affected() == 0 { return Err(Error::BadRequest(format!( "Resource {} already exists", @@ -1691,6 +1783,10 @@ async fn update_resource( // source path. if let Some(npath) = ns.path.as_deref() { check_scopes(&authed, || format!("resources:write:{}", npath))?; + check_proper_path(npath)?; + } + if let Some(nrt) = ns.resource_type.as_deref() { + check_proper_type_name(nrt)?; } if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, @@ -1810,7 +1906,10 @@ async fn update_resource( } let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; - let npath_o: Option = sqlx::query_scalar(&sql).fetch_optional(&mut *tx).await?; + let npath_o: Option = sqlx::query_scalar(&sql) + .fetch_optional(&mut *tx) + .await + .map_err(sanitize_db_error)?; let npath = not_found_if_none(npath_o, "Resource", path)?; @@ -1961,44 +2060,79 @@ async fn update_resource_value( ) -> Result { let path = path.to_path(); check_scopes(&authed, || format!("resources:write:{}", path))?; - if let RuleCheckResult::Blocked(msg) = check_deploy_rules( + set_resource_value( + &authed, + &db, + &user_db, + &webhook, &w_id, - AuditAuthorable::username(&authed), + path, + nv.value, + "resources.update", + ) + .await?; + Ok(format!("value of resource {} updated", path)) +} + +/// Write a resource's value and run everything that has to follow it: a version row, the audit +/// entry, deployment metadata, the webhook, and dependent CI tests. Shared with version restore +/// so a restored value is indistinguishable downstream from any other edit. +async fn set_resource_value( + authed: &ApiAuthed, + db: &DB, + user_db: &UserDB, + webhook: &WebhookShared, + w_id: &str, + path: &str, + value: Option, + audit_action: &str, +) -> Result<()> { + if let RuleCheckResult::Blocked(msg) = check_deploy_rules( + w_id, + AuditAuthorable::username(authed), &authed.groups, authed.is_admin, - &db, + db, ) .await? { return Err(Error::PermissionDenied(msg)); } - let mut tx = user_db.begin(&authed).await?; + let mut tx = user_db.clone().begin(authed).await?; - sqlx::query!( - "UPDATE resource SET value = $1, edited_at = now() WHERE path = $2 AND workspace_id = $3", - nv.value, + // `RETURNING resource_type` rather than a second lookup: the advisory below has to know the + // type to leave `state` and `cache` alone, and this statement already runs. + let updated = sqlx::query_scalar!( + "UPDATE resource SET value = $1, edited_at = now() WHERE path = $2 AND workspace_id = $3 + RETURNING resource_type", + value, path, w_id ) - .execute(&mut *tx) + .fetch_optional(&mut *tx) .await?; + let Some(resource_type) = updated else { + return Err(Error::NotFound(format!("Resource {} not found", path))); + }; audit_log( &mut *tx, - &authed, - "resources.update", + authed, + audit_action, ActionKind::Update, - &w_id, + w_id, Some(path), None, ) .await?; tx.commit().await?; + note_resource_write(w_id, path, &resource_type); + handle_deployment_metadata( &authed.email, &authed.username, - &db, - &w_id, + db, + w_id, DeployedObject::Resource { path: path.to_string(), parent_path: Some(path.to_string()) }, None, true, @@ -2007,9 +2141,9 @@ async fn update_resource_value( .await?; webhook.send_message( - w_id.clone(), + w_id.to_string(), WebhookMessage::UpdateResource { - workspace: w_id.clone(), + workspace: w_id.to_string(), old_path: path.to_owned(), new_path: path.to_owned(), }, @@ -2018,12 +2152,13 @@ async fn update_resource_value( // Trigger CI tests for items that reference this resource { let db2 = db.clone(); + let w_id2 = w_id.to_string(); let path2 = path.to_string(); let email2 = authed.email.clone(); let username2 = authed.username.clone(); tokio::spawn(async move { if let Err(e) = windmill_dep_map::ci_tests::trigger_ci_tests_for_item( - &db2, &w_id, &path2, "resource", &email2, &username2, + &db2, &w_id2, &path2, "resource", &email2, &username2, ) .await { @@ -2032,7 +2167,299 @@ async fn update_resource_value( }); } - Ok(format!("value of resource {} updated", path)) + Ok(()) +} + +#[derive(Serialize)] +struct ResourceVersion { + id: i64, + created_at: chrono::DateTime, + created_by: Option, +} + +#[derive(Serialize)] +struct ResourceVersionWithValue { + id: i64, + created_at: chrono::DateTime, + created_by: Option, + value: Option, + /// `$var:`/`$res:` paths in this version that no longer resolve. Restoring is still allowed — + /// the reference may be about to be recreated, and refusing would make a version unrestorable + /// for a reason outside the resource — but the caller should say so before confirming. + missing_references: Vec, +} + +/// Deliberately does not carry the resource's live value. Every write mints a version, so the +/// newest one already holds it, and `resource_version` rows are immutable — reading the live +/// value from `resource` instead would mean comparing a mutable row against this list, which at +/// READ COMMITTED can disagree with it. The drawer reads only versions, and by id, so there is +/// nothing for a concurrent write to make incoherent. +#[derive(Serialize)] +pub struct ResourceHistory { + versions: Vec, + /// Whether this resource's history can ever fill, so the drawer can say an empty one is + /// permanent rather than promising versions from the next edit. Decided here rather than by + /// shipping the type out, so INTERNAL_RESOURCE_TYPES is not restated in TypeScript too. + /// Compared against nothing, so unlike the value it carries no coherence risk. + versioned: bool, +} + +async fn get_resource_history( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, path)): Path<(String, StripPath)>, +) -> JsonResult { + let path = path.to_path(); + check_scopes(&authed, || format!("resources:read:{}", path))?; + let mut tx = user_db.begin(&authed).await?; + + let versions = sqlx::query_as!( + ResourceVersion, + "SELECT id, created_at, created_by FROM resource_version + WHERE workspace_id = $1 AND path = $2 ORDER BY id DESC LIMIT $3", + w_id, + path, + MAX_RESOURCE_VERSIONS + ) + .fetch_all(&mut *tx) + .await?; + let resource_type = sqlx::query_scalar!( + "SELECT resource_type FROM resource WHERE workspace_id = $1 AND path = $2", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + + Ok(Json(ResourceHistory { + versions, + versioned: resource_type + .map(|t| !INTERNAL_RESOURCE_TYPES.contains(&t.as_str())) + .unwrap_or(true), + })) +} + +/// Collect the references in `value` that no longer point at anything. Covers the three +/// path-addressed forms — `$var:` and `$jsonvar:` resolve against `variable`, `$res:` against +/// `resource`. `$encrypted:` carries its payload inline so there is nothing to look up, and +/// neither resolution nor this check follows references transitively. +/// +/// Runs on the caller's RLS-scoped transaction, so a referenced item the caller cannot read is +/// reported as missing. That errs towards warning rather than staying silent, and the reference +/// is unusable to them either way. +async fn missing_references( + tx: &mut sqlx::PgConnection, + w_id: &str, + value: Option<&serde_json::Value>, +) -> Result> { + const VAR_PREFIXES: [&str; 2] = ["$var:", "$jsonvar:"]; + const RES_PREFIX: &str = "$res:"; + + let mut refs: Vec = vec![]; + fn collect(v: &serde_json::Value, out: &mut Vec) { + match v { + // Stated once: adding a form to the consts above must not need a second edit here, + // or the new form would be collected but never checked (or the reverse). + serde_json::Value::String(s) + if VAR_PREFIXES.iter().any(|p| s.starts_with(p)) || s.starts_with(RES_PREFIX) => + { + out.push(s.clone()) + } + serde_json::Value::Array(a) => a.iter().for_each(|x| collect(x, out)), + serde_json::Value::Object(o) => o.values().for_each(|x| collect(x, out)), + _ => {} + } + } + if let Some(v) = value { + collect(v, &mut refs); + } + refs.sort(); + refs.dedup(); + + let mut var_paths: Vec = vec![]; + let mut res_paths: Vec = vec![]; + for r in &refs { + match VAR_PREFIXES.iter().find_map(|p| r.strip_prefix(p)) { + Some(p) => var_paths.push(p.to_string()), + None => { + if let Some(p) = r.strip_prefix(RES_PREFIX) { + res_paths.push(p.to_string()) + } + } + } + } + + // Sets, not Vecs: a value carries as many references as its author put in it, and a linear + // scan per reference would make the cost of one version fetch quadratic in caller-controlled + // input. + let existing_vars: std::collections::HashSet = sqlx::query_scalar!( + "SELECT path FROM variable WHERE workspace_id = $1 AND path = ANY($2)", + w_id, + &var_paths[..] + ) + .fetch_all(&mut *tx) + .await? + .into_iter() + .collect(); + let existing_res: std::collections::HashSet = sqlx::query_scalar!( + "SELECT path FROM resource WHERE workspace_id = $1 AND path = ANY($2)", + w_id, + &res_paths[..] + ) + .fetch_all(&mut *tx) + .await? + .into_iter() + .collect(); + + Ok(refs + .into_iter() + .filter( + |r| match VAR_PREFIXES.iter().find_map(|p| r.strip_prefix(p)) { + Some(p) => !existing_vars.contains(p), + None => match r.strip_prefix(RES_PREFIX) { + Some(p) => !existing_res.contains(p), + None => false, + }, + }, + ) + .collect()) +} + +async fn get_resource_version( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, version)): Path<(String, i64)>, +) -> JsonResult { + let mut tx = user_db.begin(&authed).await?; + + let row = sqlx::query!( + "SELECT id, path, created_at, created_by, value FROM resource_version + WHERE workspace_id = $1 AND id = $2", + w_id, + version + ) + .fetch_optional(&mut *tx) + .await?; + let row = not_found_if_none(row, "ResourceVersion", version.to_string())?; + check_scopes(&authed, || format!("resources:read:{}", row.path))?; + + let missing = missing_references(&mut tx, &w_id, row.value.as_ref()).await?; + tx.commit().await?; + + Ok(Json(ResourceVersionWithValue { + id: row.id, + created_at: row.created_at, + created_by: row.created_by, + value: row.value, + missing_references: missing, + })) +} + +/// Drop every version of a resource except the one matching its current value. +/// +/// The remediation for a credential that was stored inline in a resource value (which `wmill` +/// pushes and `setResource` writes can do, unlike the UI form) and has since been rotated: +/// without this, rotating leaves the old secret readable in the history by anyone who can read +/// the resource. All-but-current rather than per-version because such a secret sits in every +/// version from its introduction to the rotation, so deleting them one at a time would silently +/// leave copies behind. +async fn clear_resource_history( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, path)): Path<(String, StripPath)>, +) -> Result { + let path = path.to_path(); + check_scopes(&authed, || format!("resources:write:{}", path))?; + require_owner_of_path(&authed, path)?; + + // Confirm the caller can see the resource under their own RLS before deleting anything with + // the unrestricted pool — resource_version grants users SELECT only, so the delete cannot run + // through user_db. + let mut tx = user_db.begin(&authed).await?; + let exists = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM resource WHERE workspace_id = $1 AND path = $2)", + w_id, + path + ) + .fetch_one(&mut *tx) + .await?; + tx.commit().await?; + if !exists.unwrap_or(false) { + return Err(Error::NotFound(format!("Resource {} not found", path))); + } + + let mut tx = db.begin().await?; + let deleted = sqlx::query!( + "DELETE FROM resource_version rv + WHERE rv.workspace_id = $1 AND rv.path = $2 + AND rv.id != ( + SELECT max(id) FROM resource_version l + WHERE l.workspace_id = rv.workspace_id AND l.path = rv.path + )", + w_id, + path + ) + .execute(&mut *tx) + .await?; + audit_log( + &mut *tx, + &authed, + "resources.clear_history", + ActionKind::Delete, + &w_id, + Some(path), + None, + ) + .await?; + tx.commit().await?; + + Ok(format!( + "cleared {} past versions of resource {}", + deleted.rows_affected(), + path + )) +} + +async fn restore_resource_version( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Extension(webhook): Extension, + Path((w_id, version)): Path<(String, i64)>, +) -> Result { + let mut tx = user_db.clone().begin(&authed).await?; + let row = sqlx::query!( + "SELECT path, value FROM resource_version WHERE workspace_id = $1 AND id = $2", + w_id, + version + ) + .fetch_optional(&mut *tx) + .await?; + let row = not_found_if_none(row, "ResourceVersion", version.to_string())?; + tx.commit().await?; + + check_scopes(&authed, || format!("resources:write:{}", row.path))?; + + // Writes the old value forward as a new version rather than rewinding, so the history stays + // append-only, the restore is itself attributable, and the restore can be undone in turn. + set_resource_value( + &authed, + &db, + &user_db, + &webhook, + &w_id, + &row.path, + row.value, + "resources.restore_version", + ) + .await?; + + Ok(format!( + "resource {} restored to version {}", + row.path, version + )) } #[derive(Serialize)] @@ -2161,6 +2588,8 @@ async fn create_resource_type( return Err(Error::PermissionDenied(msg)); } + check_proper_type_name(&resource_type.name)?; + let mut tx = user_db.begin(&authed).await?; check_rt_path_conflict(&mut tx, &w_id, &resource_type.name).await?; @@ -2449,7 +2878,10 @@ fn is_private_or_reserved_ip(ip: &IpAddr) -> bool { v4.is_loopback() || v4.is_private() || v4.is_link_local() - || v4.is_unspecified() + // RFC 1122 "this network": the whole /8, not just the + // unspecified address `is_unspecified()` matches — stacks that + // map 0.x.y.z onto the local host make `0.0.0.1` a bypass. + || v4.octets()[0] == 0 // 0.0.0.0/8 || v4.is_broadcast() // 100.64.0.0/10 (Carrier-grade NAT / CGNAT) || (v4.octets()[0] == 100 && (v4.octets()[1] & 0xC0) == 64) @@ -3400,10 +3832,13 @@ mod tests { assert!(is_private_or_reserved_ip( &"100.64.0.1".parse::().unwrap() )); - // Unspecified + // "This network" 0.0.0.0/8, not just the unspecified address assert!(is_private_or_reserved_ip( &"0.0.0.0".parse::().unwrap() )); + assert!(is_private_or_reserved_ip( + &"0.0.0.1".parse::().unwrap() + )); // IPv6 loopback assert!(is_private_or_reserved_ip(&"::1".parse::().unwrap())); // IPv6 unique local address (fc00::/7) diff --git a/backend/windmill-store/src/variables.rs b/backend/windmill-store/src/variables.rs index 55047ec0c2..8f288dc85e 100644 --- a/backend/windmill-store/src/variables.rs +++ b/backend/windmill-store/src/variables.rs @@ -10,14 +10,16 @@ use windmill_api_auth::{ build_scope_path_predicate, check_scopes, maybe_refresh_folders, require_owner_of_path, ApiAuthed, }; -use windmill_common::db::DB; +use windmill_common::db::{Authable, DB}; use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult}; use crate::secret_backend_ext::{ delete_secret_from_backend, get_secret_value, is_external_stored_value, is_vault_stored_value, rename_vault_secret, store_secret_value, }; -use windmill_common::utils::{escape_ilike_pattern, BulkDeleteRequest}; +use windmill_common::utils::{ + check_proper_path, escape_ilike_pattern, sanitize_db_error, BulkDeleteRequest, +}; use windmill_common::webhook::{WebhookMessage, WebhookShared}; use axum::{ @@ -367,10 +369,10 @@ async fn get_variable( { return Ok(Json(overlay)); } - explain_variable_perm_error(&path, &w_id, &db).await?; + explain_variable_perm_error(&path, &w_id, &db, Some(&authed)).await?; unreachable!() } else { - explain_variable_perm_error(&path, &w_id, &db).await?; + explain_variable_perm_error(&path, &w_id, &db, Some(&authed)).await?; unreachable!() }; @@ -461,10 +463,27 @@ async fn get_value( .map(Json); } +/// The grants alone can't explain a denial: a job started on behalf of another +/// user is authorized as that user, so the error has to name who was actually +/// asking or it reads as a grant bug. +fn describe_authed(authed: Option<&(impl Authable + Sync)>) -> String { + match authed { + Some(authed) => format!( + "username: {}, email: {}, groups: {:?}, folders: {:?}", + authed.username(), + authed.email(), + authed.groups(), + authed.folders() + ), + None => "unauthenticated".to_string(), + } +} + async fn explain_variable_perm_error( path: &str, w_id: &str, db: &sqlx::Pool, + authed: Option<&(impl Authable + Sync)>, ) -> windmill_common::error::Result<()> { let extra_perms = sqlx::query_scalar!( "SELECT extra_perms from variable WHERE path = $1 AND workspace_id = $2", @@ -489,13 +508,14 @@ async fn explain_variable_perm_error( .fetch_optional(db) .await?; return Err(Error::NotAuthorized(format!( - "Variable exists but you don't have access to it:\nvariable perms: {}\nfolder perms: {}", - serde_json::to_string_pretty(&extra_perms).unwrap_or_default(), serde_json::to_string_pretty(&folder_extra_perms).unwrap_or_default() + "Variable exists but you don't have access to it:\nvariable perms: {}\nfolder perms: {}\nauthed as: {}", + serde_json::to_string_pretty(&extra_perms).unwrap_or_default(), serde_json::to_string_pretty(&folder_extra_perms).unwrap_or_default(), describe_authed(authed) ))); } else { return Err(Error::NotAuthorized(format!( - "Variable exists but you don't have access to it:\nvariable perms: {}", - serde_json::to_string_pretty(&extra_perms).unwrap_or_default() + "Variable exists but you don't have access to it:\nvariable perms: {}\nauthed as: {}", + serde_json::to_string_pretty(&extra_perms).unwrap_or_default(), + describe_authed(authed) ))); } } @@ -575,6 +595,7 @@ async fn create_variable( Json(variable): Json, ) -> Result<(StatusCode, String)> { check_scopes(&authed, || format!("variables:write:{}", variable.path))?; + check_proper_path(&variable.path)?; if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, AuditAuthorable::username(&authed), @@ -641,7 +662,8 @@ async fn create_variable( &authed.username ) .execute(&mut *tx) - .await?; + .await + .map_err(sanitize_db_error)?; if variable.ws_specific.unwrap_or(false) { sqlx::query!( @@ -1077,6 +1099,7 @@ async fn update_variable( // source path. if let Some(npath) = ns.path.as_deref() { check_scopes(&authed, || format!("variables:write:{}", npath))?; + check_proper_path(npath)?; } let authed = maybe_refresh_folders(&path, &w_id, authed, &db).await; @@ -1271,7 +1294,10 @@ async fn update_variable( sqlb.set_str("edited_by", &authed.username); sqlb.returning("path"); let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; - let npath_o: Option = sqlx::query_scalar(&sql).fetch_optional(&mut *tx).await?; + let npath_o: Option = sqlx::query_scalar(&sql) + .fetch_optional(&mut *tx) + .await + .map_err(sanitize_db_error)?; not_found_if_none(npath_o, "Variable", path)? } else { // `has_sql_updates` is guaranteed true whenever `ns.path` is provided @@ -1498,7 +1524,13 @@ pub async fn get_value_internal<'a>( let variable = if let Some(variable) = variable_o { variable } else { - explain_variable_perm_error(path, w_id, &db_with_opt_authed.db()).await?; + explain_variable_perm_error( + path, + w_id, + &db_with_opt_authed.db(), + db_with_opt_authed.authed(), + ) + .await?; unreachable!() }; diff --git a/backend/windmill-test-utils/src/lib.rs b/backend/windmill-test-utils/src/lib.rs index c9c019c6e8..060e8af983 100644 --- a/backend/windmill-test-utils/src/lib.rs +++ b/backend/windmill-test-utils/src/lib.rs @@ -223,6 +223,7 @@ impl RunJob { /* email */ &email, /* permissioned_as */ "u/test-user".to_string(), /* token_prefix */ None, + /* audit_end_user */ None, scheduled_for_o, /* schedule_path */ None, /* parent_job */ None, @@ -269,6 +270,7 @@ impl RunJob { email, format!("u/{}", username), /* token_prefix */ None, + /* audit_end_user */ None, scheduled_for_o, /* schedule_path */ None, /* parent_job */ None, diff --git a/backend/windmill-trigger/src/global_handler.rs b/backend/windmill-trigger/src/global_handler.rs index 00727eac68..6e1c92d588 100644 --- a/backend/windmill-trigger/src/global_handler.rs +++ b/backend/windmill-trigger/src/global_handler.rs @@ -37,10 +37,13 @@ async fn get_suspended_trigger( path: &str, ) -> Result { // Only trigger kinds backed by a `_trigger` table support reassignment. - // `app` (and webhook/schedule) have no such table, so reject them with a clear + // `app`/`ui` (and webhook/schedule) have no such table, so reject them with a clear // error instead of failing on a missing-relation database error below. match trigger_kind { - JobTriggerKind::Webhook | JobTriggerKind::Schedule | JobTriggerKind::App => { + JobTriggerKind::Webhook + | JobTriggerKind::Schedule + | JobTriggerKind::App + | JobTriggerKind::Ui => { return Err(Error::BadRequest(format!( "{} triggers do not support job reassignment", trigger_kind diff --git a/backend/windmill-trigger/src/handler.rs b/backend/windmill-trigger/src/handler.rs index 0c46ff25fd..ad3a6b097a 100644 --- a/backend/windmill-trigger/src/handler.rs +++ b/backend/windmill-trigger/src/handler.rs @@ -361,7 +361,17 @@ pub trait TriggerCrud: Send + Sync + 'static { .bind(script_path) .fetch_one(&mut *tx) .await - .unwrap_or(0); + // Falling back to 0 keeps one unreadable table from failing the whole count + // endpoint, but the cause must still reach the logs: a silent 0 is + // indistinguishable from "no triggers" in the UI. + .unwrap_or_else(|err| { + tracing::error!( + "failed to count {} triggers of {} {script_path} in {workspace_id}: {err:#}", + Self::TABLE_NAME, + if is_flow { "flow" } else { "script" } + ); + 0 + }); count } diff --git a/backend/windmill-trigger/src/trigger_helpers.rs b/backend/windmill-trigger/src/trigger_helpers.rs index e0bfaf3d34..6a37bd0da1 100644 --- a/backend/windmill-trigger/src/trigger_helpers.rs +++ b/backend/windmill-trigger/src/trigger_helpers.rs @@ -957,6 +957,7 @@ async fn trigger_script_with_retry_and_error_handler<'c>( email, permissioned_as, authed.token_prefix.as_deref(), + authed.username_override.as_deref(), None, None, None, diff --git a/backend/windmill-types/src/assets.rs b/backend/windmill-types/src/assets.rs index fb624382c4..463a365b06 100644 --- a/backend/windmill-types/src/assets.rs +++ b/backend/windmill-types/src/assets.rs @@ -14,6 +14,18 @@ pub enum AssetKind { Ducklake, DataTable, Volume, + /// A warehouse relation a dbt project builds or reads, + /// `dbt:////`, where `` is the name the + /// workspace configures it under. + /// + /// The SCHEME names the producer — dbt is the only thing that creates one — + /// while the PATH stays the physical relation, because that is what two + /// projects agree on: a mart one builds is a `source` the next reads, and + /// their dbt `unique_id`s differ (`model.a.orders` vs + /// `source.b.analytics.orders`) where the relation does not + /// (docs/dbt-runtime.md, decision 11). A dbt run does not trigger that + /// reader — the shared node is lineage, not a cascade edge. + Dbt, } impl AssetKind { @@ -28,6 +40,7 @@ impl AssetKind { AssetKind::Ducklake => Some("ducklake://"), AssetKind::DataTable => Some("datatable://"), AssetKind::Volume => Some("volume://"), + AssetKind::Dbt => Some("dbt://"), AssetKind::Variable => None, } } diff --git a/backend/windmill-types/src/flows.rs b/backend/windmill-types/src/flows.rs index 017484caeb..02feeb421a 100644 --- a/backend/windmill-types/src/flows.rs +++ b/backend/windmill-types/src/flows.rs @@ -39,9 +39,14 @@ pub struct Flow { pub timeout: Option, #[serde(skip_serializing_if = "is_none_or_false")] pub visible_to_runner_only: Option, + /// Derived from `on_behalf_of` on the read paths, not selected. Kept in + /// the response so clients written against the old shape keep working. #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] pub on_behalf_of_email: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub on_behalf_of: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, /// Labels inherited from the parent folder, computed at read time. Not stored on the flow row. #[sqlx(default)] @@ -127,6 +132,10 @@ pub struct NewFlow { pub deployment_message: Option, pub visible_to_runner_only: Option, pub on_behalf_of_email: Option, + /// Authorization identity to run as, paired with `on_behalf_of_email`. Both move + /// together under the same `preserve_on_behalf_of` gate and must name the same user + /// or group; `None` has it derived from that email rather than left unset. + pub on_behalf_of: Option, pub preserve_on_behalf_of: Option, pub ws_error_handler_muted: Option, #[serde(default)] @@ -164,6 +173,7 @@ pub struct EditFlow { pub deployment_message: Option, pub visible_to_runner_only: Option, pub on_behalf_of_email: Option, + pub on_behalf_of: Option, pub preserve_on_behalf_of: Option, pub ws_error_handler_muted: Option, #[serde(default)] @@ -188,6 +198,7 @@ impl EditFlow { deployment_message: self.deployment_message, visible_to_runner_only: self.visible_to_runner_only, on_behalf_of_email: self.on_behalf_of_email, + on_behalf_of: self.on_behalf_of, preserve_on_behalf_of: self.preserve_on_behalf_of, ws_error_handler_muted: self.ws_error_handler_muted, labels: self.labels, diff --git a/backend/windmill-types/src/jobs.rs b/backend/windmill-types/src/jobs.rs index 8385e57bd7..42f1285644 100644 --- a/backend/windmill-types/src/jobs.rs +++ b/backend/windmill-types/src/jobs.rs @@ -55,11 +55,17 @@ pub enum JobTriggerKind { // direct `/jobs/run` cannot set it, so it distinguishes files an app actually // produced from files a viewer forged by running a declared runnable directly. App, + // A run started from the browser (the `/jobs/run*` request carried the session token + // minted at login). Nothing writes this yet — it exists so every binary from this release + // can decode it, which is what a later release needs before it can start stamping it. + // When that happens: attribution, not authority, unlike `App` — a member can have a + // session token minted for a script (`/users/refresh_token`), so nothing may gate on it. + Ui, } -impl std::fmt::Display for JobTriggerKind { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let kind = match self { +impl JobTriggerKind { + pub const fn as_str(&self) -> &'static str { + match self { JobTriggerKind::Webhook => "webhook", JobTriggerKind::Http => "http", JobTriggerKind::Websocket => "websocket", @@ -80,8 +86,52 @@ impl std::fmt::Display for JobTriggerKind { JobTriggerKind::Asset => "asset", JobTriggerKind::Freshness => "freshness", JobTriggerKind::App => "app", - }; - write!(f, "{}", kind) + JobTriggerKind::Ui => "ui", + } + } +} + +impl std::fmt::Display for JobTriggerKind { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}", self.as_str()) + } +} + +/// `v2_job.trigger_kind` as a worker reads it back, kept as the raw label rather than as +/// [`JobTriggerKind`]. A worker has to keep running jobs stamped with a kind added after it was +/// built, and decoding into the enum turns an unfamiliar label into a hard error — sqlx on the +/// pull query, serde on the agent-worker payload — which strands the job instead of running it. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(transparent)] +pub struct TriggerKindLabel(pub String); + +impl TriggerKindLabel { + pub fn is(&self, kind: JobTriggerKind) -> bool { + self.0 == kind.as_str() + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl From for TriggerKindLabel { + fn from(kind: JobTriggerKind) -> Self { + Self(kind.as_str().to_string()) + } +} + +impl sqlx::Type for TriggerKindLabel { + fn type_info() -> sqlx::postgres::PgTypeInfo { + sqlx::postgres::PgTypeInfo::with_name("JOB_TRIGGER_KIND") + } +} + +impl<'r> sqlx::Decode<'r, sqlx::Postgres> for TriggerKindLabel { + fn decode(value: sqlx::postgres::PgValueRef<'r>) -> Result { + Ok(Self( + <&str as sqlx::Decode>::decode(value)?.to_owned(), + )) } } @@ -252,6 +302,13 @@ pub struct QueuedJob { #[serde(default, skip_serializing_if = "Option::is_none")] #[sqlx(default)] pub is_retry: Option, + // How the run was started. NULL on every job pushed before the API began stamping it, and + // on the paths that still don't, so the run page treats it as "unknown" rather than + // "manual". `#[sqlx(default)]` lets the queries that don't select it omit the column, and + // the label type keeps a kind added after this binary was built readable instead of fatal. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub trigger_kind: Option, } impl QueuedJob { @@ -328,6 +385,7 @@ impl Default for QueuedJob { runnable_settings_handle: None, labels: None, is_retry: None, + trigger_kind: None, } } } @@ -415,6 +473,13 @@ pub struct CompletedJob { #[serde(default, skip_serializing_if = "Option::is_none")] #[sqlx(default)] pub resolved_automatically: Option, + // How the run was started. NULL on every job pushed before the API began stamping it, and + // on the paths that still don't, so the run page treats it as "unknown" rather than + // "manual". `#[sqlx(default)]` lets the queries that don't select it omit the column, and + // the label type keeps a kind added after this binary was built readable instead of fatal. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub trigger_kind: Option, } impl CompletedJob { @@ -660,7 +725,7 @@ pub fn generate_dynamic_input_key(workspace_id: &str, path: &str) -> String { #[cfg(test)] mod tests { - use super::is_valid_entrypoint_name; + use super::{is_valid_entrypoint_name, JobTriggerKind, TriggerKindLabel}; #[test] fn valid_entrypoint_names_are_accepted() { @@ -705,4 +770,22 @@ mod tests { // Over-long names are rejected. assert!(!is_valid_entrypoint_name(&"a".repeat(256))); } + + /// The point of [`TriggerKindLabel`]: a worker built before a trigger kind existed must + /// still be able to read — and therefore run — a job stamped with it. Decoding into + /// [`JobTriggerKind`] instead makes an unfamiliar label a hard error and strands the job. + #[test] + fn an_unknown_trigger_kind_label_survives_deserialization() { + #[derive(serde::Deserialize)] + struct Job { + trigger_kind: Option, + } + + let job: Job = serde_json::from_str(r#"{"trigger_kind":"a_kind_from_the_future"}"#).unwrap(); + assert_eq!(job.trigger_kind.as_ref().map(|k| k.as_str()), Some("a_kind_from_the_future")); + assert!(!job.trigger_kind.unwrap().is(JobTriggerKind::Schedule)); + + let job: Job = serde_json::from_str(r#"{"trigger_kind":"schedule"}"#).unwrap(); + assert!(job.trigger_kind.unwrap().is(JobTriggerKind::Schedule)); + } } diff --git a/backend/windmill-types/src/scripts.rs b/backend/windmill-types/src/scripts.rs index 5e5c7f313e..414650e846 100644 --- a/backend/windmill-types/src/scripts.rs +++ b/backend/windmill-types/src/scripts.rs @@ -66,6 +66,7 @@ pub enum ScriptLang { Java, Ruby, Rlang, + Dbt, // for related places search: ADD_NEW_LANG } @@ -96,6 +97,7 @@ impl ScriptLang { ScriptLang::Java => "java", ScriptLang::Ruby => "ruby", ScriptLang::Rlang => "rlang", + ScriptLang::Dbt => "dbt", // for related places search: ADD_NEW_LANG } } @@ -134,7 +136,7 @@ impl ScriptLang { use ScriptLang::*; match self { Nativets | Bun | Bunnative | Deno | Go | Php | CSharp | Java => "//", - Python3 | Bash | Powershell | Graphql | Ansible | Nu | Ruby | Rlang => "#", + Python3 | Bash | Powershell | Graphql | Ansible | Nu | Ruby | Rlang | Dbt => "#", Postgresql | Mysql | Bigquery | Snowflake | Mssql | OracleDB | DuckDb => "--", Rust => "//!", // for related places search: ADD_NEW_LANG @@ -170,6 +172,7 @@ impl FromStr for ScriptLang { "java" => ScriptLang::Java, "ruby" => ScriptLang::Ruby, "rlang" => ScriptLang::Rlang, + "dbt" => ScriptLang::Dbt, // for related places search: ADD_NEW_LANG language => return Err(anyhow::anyhow!("{} is currently not supported", language)), }; @@ -326,7 +329,8 @@ pub const SCRIPT_COLUMNS: &str = concat!( "lock, lock_error_logs, language, kind, tag, envs, ", "dedicated_worker, ws_error_handler_muted, priority, cache_ttl, cache_ignore_s3_path, ", "timeout, delete_after_use, delete_after_secs, restart_unless_cancelled, ", - "visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, ", + "visible_to_runner_only, auto_kind, codebase, has_preprocessor, ", + "on_behalf_of, ", "assets, modules, labels, concurrency_key, concurrent_limit, ", "concurrency_time_window_s, debounce_key, debounce_delay_s, runnable_settings_handle", ); @@ -382,9 +386,14 @@ pub struct Script { pub codebase: Option, #[serde(skip_serializing_if = "Option::is_none")] pub has_preprocessor: Option, + /// Derived from `on_behalf_of` on the read paths, not selected. Kept in the response so + /// clients written against the old shape keep working. #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] pub on_behalf_of_email: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub on_behalf_of: Option, + #[serde(skip_serializing_if = "Option::is_none")] #[sqlx(json(nullable))] pub assets: Option>, #[serde(skip_serializing_if = "Option::is_none")] @@ -554,6 +563,10 @@ pub struct NewScript { pub codebase: Option, pub has_preprocessor: Option, pub on_behalf_of_email: Option, + /// Authorization identity to run as, paired with `on_behalf_of_email`. Both move + /// together under the same `preserve_on_behalf_of` gate and must name the same user + /// or group; `None` has it derived from that email rather than left unset. + pub on_behalf_of: Option, pub preserve_on_behalf_of: Option, #[serde(skip_serializing_if = "Option::is_none")] pub assets: Option>, @@ -606,6 +619,7 @@ impl Hash for NewScript { self.codebase.hash(state); self.has_preprocessor.hash(state); self.on_behalf_of_email.hash(state); + self.on_behalf_of.hash(state); self.preserve_on_behalf_of.hash(state); self.assets.hash(state); self.labels.hash(state); diff --git a/backend/windmill-worker/Cargo.toml b/backend/windmill-worker/Cargo.toml index f4659f7d82..90b15fd1b5 100644 --- a/backend/windmill-worker/Cargo.toml +++ b/backend/windmill-worker/Cargo.toml @@ -82,6 +82,7 @@ tokio.workspace = true tokio-stream.workspace = true serde.workspace = true serde_json.workspace = true +serde_yml.workspace = true futures.workspace = true async-recursion.workspace = true async-trait.workspace = true diff --git a/backend/windmill-worker/nsjail/run.dbt.config.proto b/backend/windmill-worker/nsjail/run.dbt.config.proto new file mode 100644 index 0000000000..192a12b3a5 --- /dev/null +++ b/backend/windmill-worker/nsjail/run.dbt.config.proto @@ -0,0 +1,138 @@ +name: "dbt run project" + +mode: ONCE +hostname: "dbt" +log_level: ERROR +time_limit: {TIMEOUT} + +{RLIMIT_AS} +rlimit_cpu: 1000 +rlimit_fsize: 1000 +rlimit_nofile: 10000 + +cwd: "{PROJECT_DIR}" + +# The warehouse is remote for every adapter but DuckDB, so the network stays. +clone_newnet: false +clone_newuser: {CLONE_NEWUSER} + +skip_setsid: true +keep_caps: false +keep_env: true +mount_proc: true + +mount { + src: "/bin" + dst: "/bin" + is_bind: true +} + +mount { + src: "/lib" + dst: "/lib" + is_bind: true +} + +mount { + src: "/lib64" + dst: "/lib64" + is_bind: true + mandatory: false +} + +mount { + src: "/usr" + dst: "/usr" + is_bind: true +} + +mount { + src: "/dev/null" + dst: "/dev/null" + is_bind: true + rw: true +} + +mount { + src: "/dev/random" + dst: "/dev/random" + is_bind: true +} + +mount { + src: "/dev/urandom" + dst: "/dev/urandom" + is_bind: true +} + +mount { + dst: "/dev/shm" + fstype: "tmpfs" + rw: true + is_bind: false +} + +{TMP_MOUNT_BLOCK} + +# The job's own directory: the project, the rendered profile, the artifacts +# dbt writes and the logs the progress reporter tails all live under it. +mount { + src: "{JOB_DIR}" + dst: "{JOB_DIR}" + is_bind: true + rw: true +} + +# The engine and the interpreter it was built against, read-only: a project +# must not be able to rewrite the dbt it is run by, or the one the next job +# on this worker will be. +mount { + src: "{ENGINE_DIR}" + dst: "{ENGINE_DIR}" + is_bind: true +} + +mount { + src: "{PY_INSTALL_DIR}" + dst: "{PY_INSTALL_DIR}" + is_bind: true + mandatory: false +} + +mount { + src: "/etc" + dst: "/etc" + is_bind: true +} + +# Container runtimes bind exactly these 3 files as separate submounts over +# /etc; nsjail's ro remount of /etc is non-recursive so they stay writable. +# Load-bearing -- do not remove as redundant with the /etc bind above. +mount { + src: "/etc/resolv.conf" + dst: "/etc/resolv.conf" + is_bind: true + mandatory: false +} + +mount { + src: "/etc/hosts" + dst: "/etc/hosts" + is_bind: true + mandatory: false +} + +mount { + src: "/etc/ssl/certs" + dst: "/etc/ssl/certs" + is_bind: true + mandatory: false +} + +{SHARED_MOUNT} + +# The project's own environment. Set here rather than on the process that +# execs nsjail: the loader honours `LD_PRELOAD` before the jail exists. +{ENVARS} + +iface_no_lo: true diff --git a/backend/windmill-worker/nsjail/run.docker.config.proto b/backend/windmill-worker/nsjail/run.docker.config.proto index 5cc1e4ca03..a8e327927e 100644 --- a/backend/windmill-worker/nsjail/run.docker.config.proto +++ b/backend/windmill-worker/nsjail/run.docker.config.proto @@ -46,6 +46,17 @@ gidmap { # and avoid it. Generated from the extracted rootfs. {ROOTFS_MOUNTS} +# `./result.json` (relative to the image's WorkingDir) bound to a host file outside the +# image rootfs, so the JSON result the container writes flows back to the job without the +# worker ever reading a path the image controls. +# +# Placed immediately after the rootfs binds and BEFORE every other mount: nsjail resolves +# a mount destination against its temporary root before pivot_root, so whatever is mounted +# earlier can steer where this lands. Keeping only the rootfs ahead of it means the image +# is the single influence on that resolution — and `result_mount_dst` verifies the whole +# path against the extracted rootfs on the host. Empty when it can't be verified. +{RESULT_MOUNT} + # Pseudo-filesystems the image expects. /tmp honors the same instance settings as # every other nsjail job (nsjail_tmp_backing tmpfs/disk, nsjail_tmpfs_size_mb); # /dev gets the standard nodes; /proc comes from mount_proc (the jail's own pid ns). diff --git a/backend/windmill-worker/src/agent_workers.rs b/backend/windmill-worker/src/agent_workers.rs index 5320c8ef6c..8dccf691fa 100644 --- a/backend/windmill-worker/src/agent_workers.rs +++ b/backend/windmill-worker/src/agent_workers.rs @@ -80,8 +80,6 @@ pub async fn get_datatable_resource_from_agent_http( /// Record a materialization outcome from an agent worker (no direct DB) via the /// API, so `materialized_partition` state lands the same as on a Sql worker. -// Only called from the duckdb executor, which is itself `#[cfg(feature = "duckdb")]`. -#[cfg(feature = "duckdb")] pub async fn record_materialization_from_agent_http( client: &HttpClient, w_id: &str, diff --git a/backend/windmill-worker/src/ai/tools.rs b/backend/windmill-worker/src/ai/tools.rs index 8adc3bb180..9c15229ba6 100644 --- a/backend/windmill-worker/src/ai/tools.rs +++ b/backend/windmill-worker/src/ai/tools.rs @@ -489,6 +489,7 @@ async fn execute_windmill_tool( permissioned_as, Some(&format!("job-span-{}", ctx.job.id)), None, + None, ctx.job.schedule_path(), Some(ctx.job.id), ctx.job.root_job.or(Some(ctx.job.id)), diff --git a/backend/windmill-worker/src/ai_executor.rs b/backend/windmill-worker/src/ai_executor.rs index ec470fb966..b11ecebc6c 100644 --- a/backend/windmill-worker/src/ai_executor.rs +++ b/backend/windmill-worker/src/ai_executor.rs @@ -11,6 +11,7 @@ use crate::worker_flow::{get_previous_job_result, get_transform_context}; use async_recursion::async_recursion; use regex::Regex; use serde_json::value::RawValue; +use sha2::Digest; use std::{collections::HashMap, sync::Arc}; use uuid::Uuid; #[cfg(feature = "bedrock")] @@ -589,6 +590,7 @@ pub async fn handle_ai_agent_job( Ok(Some(hub_script.schema)) } else { let hash = get_latest_hash_for_path( + db, db, &job.workspace_id, path.as_str(), @@ -798,6 +800,24 @@ pub async fn handle_ai_agent_job( } } +/// OpenAI rejects a `prompt_cache_key` over 64 characters +/// (`Invalid 'prompt_cache_key': string too long`), and a runnable path alone can pass +/// that. Fold an over-long key into a digest of itself: same step still yields the same +/// key across runs, which is the whole property that routes them to one cache. +fn bounded_prompt_cache_key(raw: &str) -> String { + const MAX_LEN: usize = 64; + if raw.len() <= MAX_LEN { + return raw.to_string(); + } + let suffix = hex::encode(&sha2::Sha256::digest(raw.as_bytes())[..16]); + // Keep a readable head so a key stays traceable to its workspace in provider logs. + let mut head = MAX_LEN - suffix.len() - 1; + while head > 0 && !raw.is_char_boundary(head) { + head -= 1; + } + format!("{}:{}", &raw[..head], suffix) +} + #[async_recursion] pub async fn run_agent( // connection @@ -843,9 +863,10 @@ pub async fn run_agent( { query_builder = create_chat_completions_query_builder(&credentials); } - // Both outlive the iteration that discovers them: a request shape or a route the + // These outlive the iteration that discovers them: a request shape or a route the // endpoint rejected once stays rejected for the whole step. let mut include_usage = true; + let mut include_prompt_cache_key = true; // Initialize messages let mut messages = @@ -863,6 +884,17 @@ pub async fn run_agent( let effective_flow_step_id: Option<&str> = flow_step_id_override.or(job.flow_step_id.as_deref()); + // Keyed on the step, not the run: every run of this step opens with the same system + // prompt and tool definitions, and each agent-loop iteration extends the previous + // one's prefix. Above ~15 requests/minute one key starts missing again, which is a + // reason to split it further, never to make it per-run. + let prompt_cache_key = bounded_prompt_cache_key(&format!( + "{}:{}:{}", + job.workspace_id, + job.runnable_path(), + effective_flow_step_id.unwrap_or_default() + )); + // Fetch flow context for input transforms context, chat and memory let mut flow_context = get_flow_context(db, job).await; @@ -1145,7 +1177,7 @@ pub async fn run_agent( } } else { // For all other providers, use the HTTP client approach - let build_args = BuildRequestArgs { + let mut build_args = BuildRequestArgs { messages: &messages, tools: tool_defs.as_deref(), model: args.provider.get_model(), @@ -1158,6 +1190,7 @@ pub async fn run_agent( user_message: args.user_message.as_deref().unwrap_or(""), attachments: args.user_attachments.as_deref(), has_websearch, + prompt_cache_key: include_prompt_cache_key.then_some(prompt_cache_key.as_str()), }; // A worker cannot run the client credentials exchange, so an OAuth resource @@ -1207,9 +1240,10 @@ pub async fn run_agent( }; // An endpoint can reject the request shape rather than the model: - // `stream_options`, which not every OpenAI-compatible provider accepts, and - // the route itself, when an Azure resource is outside the Responses API's - // model/region matrix. Each is retried once with that part dropped. + // `stream_options` and `prompt_cache_key`, which not every OpenAI-compatible + // gateway accepts, and the route itself, when an Azure resource is outside + // the Responses API's model/region matrix. Each is retried once with that + // part dropped. // Set where the route is found to be absent, and read once the fallback has // answered: a rejection it did not resolve says nothing about the deployment. let mut rerouted_by_a_route_rejection = false; @@ -1257,6 +1291,13 @@ pub async fn run_agent( || text.contains("include_usage") || text.contains("Additional properties are not allowed")); + // An OpenAI-compatible gateway that validates the body strictly + // names the offending field, whether it calls it an unrecognized + // argument or an unexpected additional property. + let rejects_prompt_cache_key = build_args.prompt_cache_key.is_some() + && status.as_u16() == 400 + && text.contains("prompt_cache_key"); + // Only the first call of the step may re-route: an endpoint that // does not serve this API rejects that one already, whereas a // rejection once the conversation is under way is about the @@ -1271,6 +1312,15 @@ pub async fn run_agent( "Retrying request without stream_options due to provider incompatibility" ); include_usage = false; + } else if rejects_prompt_cache_key { + // Checked before the route fallback: the endpoint serves this + // route, it just refuses one optional field, and re-routing + // the whole step over that would give up far more. + tracing::info!( + "Retrying request without prompt_cache_key due to provider incompatibility" + ); + include_prompt_cache_key = false; + build_args.prompt_cache_key = None; } else if route_unserved { tracing::info!( "Endpoint rejected the request ({}), falling back to chat/completions", @@ -1663,6 +1713,41 @@ mod tests { } } + /// Over 64 characters OpenAI rejects the key outright, which costs a wasted round + /// trip per run and silently leaves that step with no prompt caching at all. + #[test] + fn prompt_cache_key_stays_within_the_provider_bound() { + let long = format!("my-workspace:f/{}/agent:step_12", "nested_folder".repeat(8)); + assert!(long.len() > 64); + + let bounded = bounded_prompt_cache_key(&long); + + assert!( + bounded.len() <= 64, + "got {} chars: {bounded}", + bounded.len() + ); + // Stable for the same step, or every run would land on a different cache. + assert_eq!(bounded, bounded_prompt_cache_key(&long)); + assert_ne!( + bounded, + bounded_prompt_cache_key(&long.replace("step_12", "step_13")) + ); + } + + #[test] + fn prompt_cache_key_passes_short_keys_through_unchanged() { + let short = "admins:f/agent/step:a"; + assert_eq!(bounded_prompt_cache_key(short), short); + } + + /// Truncation on a byte index would panic mid-character. + #[test] + fn prompt_cache_key_truncates_on_a_char_boundary() { + let long = format!("workspace:f/{}/agent:step", "é".repeat(80)); + assert!(bounded_prompt_cache_key(&long).len() <= 64); + } + #[test] fn overlay_tool_inputs_binds_matching_flowmodule_tool_only() { fn js(expr: &str) -> InputTransform { diff --git a/backend/windmill-worker/src/ansible_executor.rs b/backend/windmill-worker/src/ansible_executor.rs index d3e71d8fdb..5e151824f0 100644 --- a/backend/windmill-worker/src/ansible_executor.rs +++ b/backend/windmill-worker/src/ansible_executor.rs @@ -31,8 +31,9 @@ use crate::{ bash_executor::BIN_BASH, common::{ build_command_with_isolation, check_executor_binary_exists, get_reserved_variables, - read_and_check_result, render_nsjail_rlimit_as, resolve_nsjail_timeout, - resolve_nsjail_tmp_mount_block, start_child_process, transform_json, OccupancyMetrics, + interpolate_template, read_and_check_result, render_nsjail_rlimit_as, + resolve_nsjail_timeout, resolve_nsjail_tmp_mount_block, start_child_process, + transform_json, validate_relative_path, OccupancyMetrics, }, handle_child::handle_child, is_sandboxing_enabled, @@ -240,97 +241,6 @@ async fn prepare_socket_root(root: &str, stale_after: std::time::Duration) { } } -lazy_static::lazy_static! { - static ref TEMPLATE_RE: regex::Regex = regex::Regex::new(r"\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}").unwrap(); -} - -/// Substitute `{{ arg_name }}` placeholders with values from `args`. -/// Strings are used raw; numbers/bools are stringified. Other types are rejected. -fn interpolate_template( - template: &str, - args: Option<&HashMap>>, - field_name: &str, -) -> error::Result { - let mut last_err: Option = None; - let result = TEMPLATE_RE.replace_all(template, |caps: ®ex::Captures| { - let name = &caps[1]; - let raw = args.and_then(|a| a.get(name)); - let Some(raw) = raw else { - last_err = Some(error::Error::BadRequest(format!( - "`{}` references `{{{{ {} }}}}` but no such argument was provided", - field_name, name - ))); - return String::new(); - }; - let json: serde_json::Value = match serde_json::from_str(raw.get()) { - Ok(v) => v, - Err(e) => { - last_err = Some(error::Error::BadRequest(format!( - "`{}` could not parse argument `{}` as JSON: {e}", - field_name, name - ))); - return String::new(); - } - }; - match json { - serde_json::Value::String(s) => s, - serde_json::Value::Number(n) => n.to_string(), - serde_json::Value::Bool(b) => b.to_string(), - serde_json::Value::Null => { - last_err = Some(error::Error::BadRequest(format!( - "`{}` references `{{{{ {} }}}}` but the argument is null", - field_name, name - ))); - String::new() - } - _ => { - last_err = Some(error::Error::BadRequest(format!( - "`{}` references `{{{{ {} }}}}` but the argument is not a primitive (string/number/bool)", - field_name, name - ))); - String::new() - } - } - }); - if let Some(e) = last_err { - return Err(e); - } - Ok(result.into_owned()) -} - -/// Reject absolute paths and `..` segments to prevent escaping the cloned repo directory. -fn validate_relative_path(path: &str, field_name: &str) -> error::Result<()> { - let trimmed = path.trim(); - if trimmed.is_empty() { - return Err(error::Error::BadRequest(format!( - "`{}` resolved to an empty path", - field_name - ))); - } - let p = std::path::Path::new(trimmed); - for component in p.components() { - match component { - // RootDir catches leading `/` or `\`; Prefix catches Windows drive - // letters and UNC paths. `Path::is_absolute()` alone misses - // RootDir-only paths on Windows (e.g. `/etc/passwd`). - std::path::Component::RootDir | std::path::Component::Prefix(_) => { - return Err(error::Error::BadRequest(format!( - "`{}` must be a relative path inside the cloned repo, got: {}", - field_name, trimmed - ))); - } - std::path::Component::ParentDir => { - return Err(error::Error::BadRequest(format!( - "`{}` must not contain `..` segments, got: {}", - field_name, trimmed - ))); - } - _ => {} - } - } - Ok(()) -} - async fn clone_repo( repo: &GitRepo, job_dir: &str, diff --git a/backend/windmill-worker/src/bun_executor.rs b/backend/windmill-worker/src/bun_executor.rs index 353cc59836..d64499878b 100644 --- a/backend/windmill-worker/src/bun_executor.rs +++ b/backend/windmill-worker/src/bun_executor.rs @@ -571,12 +571,8 @@ async fn gen_bunfig( NPMRC.read().await.clone() }; - if let Some(ref npmrc_content) = npmrc { - if !npmrc_content.trim().is_empty() { - tracing::debug!("Writing .npmrc for bun from npmrc setting"); - write_file(job_dir, ".npmrc", npmrc_content)?; - return Ok(()); - } + if npmrc.as_ref().is_some_and(|c| !c.trim().is_empty()) { + return write_bun_registry_config(job_dir, npmrc, None, None); } let (registry, bunfig_install_scopes) = if let Some(conn) = db { @@ -606,6 +602,35 @@ async fn gen_bunfig( BUNFIG_INSTALL_SCOPES.read().await.clone(), ) }; + write_bun_registry_config(job_dir, None, registry, bunfig_install_scopes) +} + +/// The files [`write_bun_registry_config`] may create in the directory bun installs from. +/// Both can hold a registry auth token, so `prepare-deps` deletes them by these names once +/// the install is over. +pub(crate) const BUN_NPMRC_FILE: &str = ".npmrc"; +pub(crate) const BUN_CONFIG_FILE: &str = "bunfig.toml"; + +/// Write the registry configuration `bun install` picks up from its working directory: the +/// `npmrc` setting verbatim as `.npmrc` when set, otherwise a `bunfig.toml` holding the +/// registry URL, its auth token and the install scopes. +/// +/// Shared with the debugger's `prepare-deps`, which resolves the same settings without a +/// database (see `prepare_deps.rs`), so the two install paths configure bun identically. +pub(crate) fn write_bun_registry_config( + job_dir: &str, + npmrc: Option, + registry: Option, + bunfig_install_scopes: Option, +) -> Result<()> { + if let Some(ref npmrc_content) = npmrc { + if !npmrc_content.trim().is_empty() { + tracing::debug!("Writing .npmrc for bun from npmrc setting"); + write_file(job_dir, BUN_NPMRC_FILE, npmrc_content)?; + return Ok(()); + } + } + if registry.is_some() || bunfig_install_scopes.is_some() { let (url, token_opt) = if let Some(ref s) = registry { let url = s.trim(); @@ -635,7 +660,7 @@ registry = {} .unwrap_or("".to_string()) ); tracing::debug!("Writing following bunfig.toml: {bunfig_toml}"); - let _ = write_file(&job_dir, "bunfig.toml", &bunfig_toml)?; + let _ = write_file(&job_dir, BUN_CONFIG_FILE, &bunfig_toml)?; } Ok(()) } @@ -2803,86 +2828,97 @@ pub async fn handle_wac_v2_output( let push_result: error::Result<()> = async { for (step, (_, child_uuid)) in steps.iter().zip(job_ids.iter()) { // Resolve job payload based on dispatch_type - let (job_payload, child_args, is_external) = match step.dispatch_type.as_str() { - "script" if step.script.starts_with("./") => { - // Module-relative path: resolve from parent script's modules - let module_key = step.script.strip_prefix("./").unwrap(); - let module = resolve_parent_module(modules, module_key)?; - let payload = JobPayload::Code(RawCode { - content: module.content, - path: job.runnable_path.clone(), - hash: None, - language: module.language, - lock: module.lock, - cache_ttl: job.cache_ttl, - cache_ignore_s3_path: job.cache_ignore_s3_path, - dedicated_worker: None, - concurrency_settings: ConcurrencySettingsWithCustom::default(), - debouncing_settings: DebouncingSettings::default(), - modules: None, - tag: None, - }); - let step_args: HashMap> = step - .args - .iter() - .map(|(k, v)| { - let raw = serde_json::value::to_raw_value(v).unwrap(); - (k.clone(), raw) - }) - .collect(); - (payload, step_args, true) - } - "script" => { - // Resolve script path to job payload (handles hash, lang, etc.) - let (payload, _, _, _, _, _) = script_path_to_payload( - &step.script, - None, // no authed db for background workers - db.clone(), - &job.workspace_id, - Some(true), // skip preprocessor - ) - .await?; - let step_args: HashMap> = step - .args - .iter() - .map(|(k, v)| { - let raw = serde_json::value::to_raw_value(v).unwrap(); - (k.clone(), raw) - }) - .collect(); - (payload, step_args, true) - } - "flow" => { - let flow_info = get_latest_flow_version_info_for_path( - None, - db, - &job.workspace_id, - &step.script, - true, - ) - .await?; - let payload = JobPayload::Flow { - path: step.script.clone(), - dedicated_worker: flow_info.dedicated_worker, - apply_preprocessor: false, - version: flow_info.version, - labels: flow_info.labels.clone(), - }; - let step_args: HashMap> = step - .args - .iter() - .map(|(k, v)| { - let raw = serde_json::value::to_raw_value(v).unwrap(); - (k.clone(), raw) - }) - .collect(); - (payload, step_args, true) - } - _ => { - // "inline" — re-run parent with _executing_key - (job_payload_template.clone(), parent_args.clone(), false) - } - }; + let (job_payload, child_args, is_external, on_behalf_of) = + match step.dispatch_type.as_str() { + "script" if step.script.starts_with("./") => { + // Module-relative path: resolve from parent script's modules + let module_key = step.script.strip_prefix("./").unwrap(); + let module = resolve_parent_module(modules, module_key)?; + let payload = JobPayload::Code(RawCode { + content: module.content, + path: job.runnable_path.clone(), + hash: None, + language: module.language, + lock: module.lock, + cache_ttl: job.cache_ttl, + cache_ignore_s3_path: job.cache_ignore_s3_path, + dedicated_worker: None, + concurrency_settings: ConcurrencySettingsWithCustom::default(), + debouncing_settings: DebouncingSettings::default(), + modules: None, + tag: None, + }); + let step_args: HashMap> = step + .args + .iter() + .map(|(k, v)| { + let raw = serde_json::value::to_raw_value(v).unwrap(); + (k.clone(), raw) + }) + .collect(); + // Inline module code, not a separate runnable: it has no + // identity of its own and runs as the parent. + (payload, step_args, true, None) + } + "script" => { + // Resolve script path to job payload (handles hash, lang, etc.) + let (payload, _, _, _, _, on_behalf_of) = script_path_to_payload( + &step.script, + None, // no authed db for background workers + db.clone(), + &job.workspace_id, + Some(true), // skip preprocessor + ) + .await?; + let step_args: HashMap> = step + .args + .iter() + .map(|(k, v)| { + let raw = serde_json::value::to_raw_value(v).unwrap(); + (k.clone(), raw) + }) + .collect(); + (payload, step_args, true, on_behalf_of) + } + "flow" => { + let flow_info = get_latest_flow_version_info_for_path( + None, + db, + &job.workspace_id, + &step.script, + true, + ) + .await?; + let payload = JobPayload::Flow { + path: step.script.clone(), + dedicated_worker: flow_info.dedicated_worker, + apply_preprocessor: false, + version: flow_info.version, + labels: flow_info.labels.clone(), + }; + let on_behalf_of = flow_info + .on_behalf_of(&job.workspace_id, db) + .await?; + let step_args: HashMap> = step + .args + .iter() + .map(|(k, v)| { + let raw = serde_json::value::to_raw_value(v).unwrap(); + (k.clone(), raw) + }) + .collect(); + (payload, step_args, true, on_behalf_of) + } + _ => { + // "inline" — re-run parent with _executing_key + ( + job_payload_template.clone(), + parent_args.clone(), + false, + None, + ) + } + }; let push_args = PushArgs { args: &child_args, extra: None }; @@ -2930,6 +2966,21 @@ pub async fn handle_wac_v2_output( } } + // A target runnable that opts into on-behalf-of runs under its own + // identity, never the caller's, so a step that reaches it through a + // workflow cannot widen or narrow its permissions. `created_by` still + // credits the caller, matching how the run API pushes these jobs. + let (child_email, child_permissioned_as) = match on_behalf_of.as_ref() { + Some(on_behalf_of) => ( + on_behalf_of.email.as_str(), + on_behalf_of.permissioned_as.clone(), + ), + None => ( + job.permissioned_as_email.as_str(), + job.permissioned_as.clone(), + ), + }; + let (_, mut tx) = push( db, PushIsolationLevel::IsolatedRoot(db.clone()), @@ -2937,8 +2988,9 @@ pub async fn handle_wac_v2_output( job_payload, push_args, &job.created_by, - &job.permissioned_as_email, - job.permissioned_as.clone(), + child_email, + child_permissioned_as, + None, None, None, None, diff --git a/backend/windmill-worker/src/common.rs b/backend/windmill-worker/src/common.rs index a406f3610c..cd493d969e 100644 --- a/backend/windmill-worker/src/common.rs +++ b/backend/windmill-worker/src/common.rs @@ -528,7 +528,7 @@ pub async fn get_reserved_variables( }; let tested_runnable = match (&job.trigger_kind, &job.trigger) { - (Some(windmill_common::jobs::JobTriggerKind::CiTest), Some(t)) => Some(t.clone()), + (Some(k), Some(t)) if k.is(windmill_common::jobs::JobTriggerKind::CiTest) => Some(t.clone()), _ => None, }; @@ -2339,3 +2339,95 @@ mod tests { assert!(result.is_err()); } } + +lazy_static::lazy_static! { + static ref TEMPLATE_RE: regex::Regex = + regex::Regex::new(r"\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}").unwrap(); +} + +/// Substitute `{{ arg_name }}` placeholders with values from `args`. +/// Strings are used raw; numbers/bools are stringified. Other types are rejected. +pub(crate) fn interpolate_template( + template: &str, + args: Option<&HashMap>>, + field_name: &str, +) -> error::Result { + let mut last_err: Option = None; + let result = TEMPLATE_RE.replace_all(template, |caps: ®ex::Captures| { + let name = &caps[1]; + let raw = args.and_then(|a| a.get(name)); + let Some(raw) = raw else { + last_err = Some(error::Error::BadRequest(format!( + "`{}` references `{{{{ {} }}}}` but no such argument was provided", + field_name, name + ))); + return String::new(); + }; + let json: serde_json::Value = match serde_json::from_str(raw.get()) { + Ok(v) => v, + Err(e) => { + last_err = Some(error::Error::BadRequest(format!( + "`{}` could not parse argument `{}` as JSON: {e}", + field_name, name + ))); + return String::new(); + } + }; + match json { + serde_json::Value::String(s) => s, + serde_json::Value::Number(n) => n.to_string(), + serde_json::Value::Bool(b) => b.to_string(), + serde_json::Value::Null => { + last_err = Some(error::Error::BadRequest(format!( + "`{}` references `{{{{ {} }}}}` but the argument is null", + field_name, name + ))); + String::new() + } + _ => { + last_err = Some(error::Error::BadRequest(format!( + "`{}` references `{{{{ {} }}}}` but the argument is not a primitive (string/number/bool)", + field_name, name + ))); + String::new() + } + } + }); + if let Some(e) = last_err { + return Err(e); + } + Ok(result.into_owned()) +} + +/// Reject absolute paths and `..` segments to prevent escaping the cloned repo directory. +pub(crate) fn validate_relative_path(path: &str, field_name: &str) -> error::Result<()> { + let trimmed = path.trim(); + if trimmed.is_empty() { + return Err(error::Error::BadRequest(format!( + "`{}` resolved to an empty path", + field_name + ))); + } + let p = std::path::Path::new(trimmed); + for component in p.components() { + match component { + // RootDir catches leading `/` or `\`; Prefix catches Windows drive + // letters and UNC paths. `Path::is_absolute()` alone misses + // RootDir-only paths on Windows (e.g. `/etc/passwd`). + std::path::Component::RootDir | std::path::Component::Prefix(_) => { + return Err(error::Error::BadRequest(format!( + "`{}` must be a relative path inside the cloned repo, got: {}", + field_name, trimmed + ))); + } + std::path::Component::ParentDir => { + return Err(error::Error::BadRequest(format!( + "`{}` must not contain `..` segments, got: {}", + field_name, trimmed + ))); + } + _ => {} + } + } + Ok(()) +} diff --git a/backend/windmill-worker/src/csharp_executor.rs b/backend/windmill-worker/src/csharp_executor.rs index 944236cf2b..132936a807 100644 --- a/backend/windmill-worker/src/csharp_executor.rs +++ b/backend/windmill-worker/src/csharp_executor.rs @@ -27,8 +27,8 @@ use windmill_queue::CanceledBy; use crate::{ common::{ build_command_with_isolation, check_executor_binary_exists, create_args_and_out_file, - get_reserved_variables, read_result, resolve_nsjail_timeout, resolve_nsjail_tmp_mount_block, - start_child_process, DEV_CONF_NSJAIL, + get_reserved_variables, read_result, resolve_nsjail_timeout, + resolve_nsjail_tmp_mount_block, start_child_process, DEV_CONF_NSJAIL, }, get_proxy_envs_for_lang, handle_child::handle_child, diff --git a/backend/windmill-worker/src/dbt_engine.rs b/backend/windmill-worker/src/dbt_engine.rs new file mode 100644 index 0000000000..2da3481b12 --- /dev/null +++ b/backend/windmill-worker/src/dbt_engine.rs @@ -0,0 +1,824 @@ +//! Provisioning the three dbt engines on a worker. +//! +//! No engine ships in a Windmill image. `dbt-core-1x` and `dbt-core-2x` are +//! Apache 2.0 and an operator may pre-stage either (`DBT_BUNDLED_DIR`); the +//! Fusion engine is **never bundled**. Its license grants only a +//! non-transferable, non-sublicensable redistribution right and forbids +//! interposing on Provider-to-End-User communication, which is exactly what +//! shipping it inside a job runner would do. The mitigation is that the user's +//! own instance fetches it from dbt Labs on first use, so Windmill never +//! redistributes it (docs/dbt-runtime.md, decision 1). +//! +//! Everything lands in a worker-global cache keyed by engine and version, so +//! the fetch happens once per worker rather than once per job. + +use std::path::{Path, PathBuf}; +use std::process::Stdio; + +use tokio::process::Command; +use uuid::Uuid; +use windmill_common::error::{self, Error}; +use windmill_common::worker::{write_file, Connection, ROOT_CACHE_NOMOUNT_DIR}; +use windmill_parser_yaml::DbtEngine; +use windmill_queue::append_logs; + +use crate::dbt_executor::digest; +use crate::dbt_profiles::DbtAdapter; +use crate::handle_child::{get_mem_peak, run_future_with_polling_update_job_poller, JobCtx}; + +lazy_static::lazy_static! { + pub static ref DBT_CACHE_DIR: String = format!("{}dbt", *ROOT_CACHE_NOMOUNT_DIR); + /// Adapters an operator vouches for beyond `PUBLISHED_ADAPTERS`, comma-separated — so a + /// brand-new adapter needs an admin's decision, not a Windmill release. + static ref DBT_EXTRA_ADAPTERS: Vec = std::env::var("DBT_EXTRA_ADAPTERS") + .unwrap_or_default() + .split(',') + .map(|a| a.trim().to_ascii_lowercase()) + .filter(|a| !a.is_empty()) + .collect(); + /// Where an operator may pre-stage an Apache-2.0 engine in a derived image. + /// A persistent image path, unlike the runtime caches, which are a fresh + /// volume at start — which is the whole reason it is a separate directory. + static ref DBT_BUNDLED_DIR: String = + std::env::var("DBT_BUNDLED_DIR").unwrap_or_else(|_| "/usr/local/dbt".to_string()); + static ref UV_PATH: String = + std::env::var("UV_PATH").unwrap_or_else(|_| "/usr/local/bin/uv".to_string()); + /// Bounds on the dbt-core the 1.x engine resolves. A RANGE, not a pin: the + /// adapter decides which core it can take, and several cap below the newest + /// (dbt-oracle and dbt-databricks below 1.12), so pinning core independently + /// makes those projects unprovisionable. The floor is the CLI this runtime + /// invokes -- 1.7 rejects `--target` on `parse`, so resolving down to it + /// produces a working venv that then fails on flags, which is worse than + /// not resolving at all. + /// + /// Every static in this group is env-overridable per instance, so an + /// operator can move an engine without waiting on a release. + static ref DBT_CORE_1X_FLOOR: String = + std::env::var("DBT_CORE_1X_FLOOR").unwrap_or_else(|_| "1.8".to_string()); + static ref DBT_CORE_1X_CEILING: String = + std::env::var("DBT_CORE_1X_CEILING").unwrap_or_else(|_| "2.0.0".to_string()); + /// Reported as the engine version only when the installed `dist-info` cannot + /// be read. It pins no install and is not in the cache key -- the resolved + /// range is -- so it must not be mistaken for the version that gets used. + static ref DBT_CORE_1X_VERSION: String = + std::env::var("DBT_CORE_1X_VERSION").unwrap_or_else(|_| "1.12.0".to_string()); + /// Pinned, not ranged: the Rust engine ships its adapters in the binary, so + /// there is no adapter resolution to accommodate. + static ref DBT_CORE_2X_VERSION: String = + std::env::var("DBT_CORE_2X_VERSION").unwrap_or_else(|_| "2.0.0-alpha.5".to_string()); + static ref DBT_PYTHON_VERSION: String = + std::env::var("DBT_PYTHON_VERSION").unwrap_or_else(|_| "3.12".to_string()); + /// Where the Fusion engine is fetched from. Never a Windmill-hosted mirror: + /// the point of runtime fetch is that the binary comes from dbt Labs. + static ref DBT_FUSION_INSTALL_URL: String = std::env::var("DBT_FUSION_INSTALL_URL") + .unwrap_or_else(|_| "https://public.cdn.getdbt.com/fs/install/install.sh".to_string()); +} + +pub struct ProvisionedEngine { + /// Absolute path of the dbt binary to invoke. + pub bin: PathBuf, + /// The engine's own directory — the ONLY part of the cache a sandboxed job + /// may see. Its siblings hold other workspaces' package trees, which are + /// scoped by cache key, not by permissions. + pub root: PathBuf, + pub version: String, + pub engine: DbtEngine, + /// The adapter version this venv resolved, for dbt-core 1.x where the + /// adapter is a separate package that versions independently of core. + /// `None` for the Rust engines, which ship their adapters in the binary. + pub adapter_version: Option, +} + +/// Ensure the engine is present on this worker and return how to invoke it. +pub async fn provision_engine( + engine: DbtEngine, + adapter: DbtAdapter, + // What makes the lockfile a lockfile: without it a script silently changes dbt + // version when the instance upgrades or lands on a different worker. `None` for + // a deploy, which is what writes the pin. + pinned_version: Option<&str>, + pinned_adapter_version: Option<&str>, + job_id: &Uuid, + w_id: &str, + conn: &Connection, + ctx: &mut JobCtx<'_>, +) -> error::Result { + let pinned_version = checked_version(pinned_version, "engine_version")?; + let pinned_adapter_version = checked_version(pinned_adapter_version, "adapter_version")?; + tokio::fs::create_dir_all(&*DBT_CACHE_DIR).await.ok(); + match engine { + DbtEngine::DbtCore1x => { + provision_core_1x( + adapter, + pinned_version, + pinned_adapter_version, + job_id, + w_id, + conn, + ctx, + ) + .await + } + DbtEngine::DbtCore2x => provision_core_2x(pinned_version, job_id, w_id, conn, ctx).await, + DbtEngine::Fusion => provision_fusion(pinned_version, job_id, w_id, conn, ctx).await, + } +} + +/// A version out of the lockfile, which is CALLER DATA: a preview run submits +/// its own `lock` alongside its content, so nothing about this string has been +/// through a deploy. +/// +/// It is interpolated into the engine cache path and into a pip requirement, +/// and provisioning runs on the host rather than inside the dbt jail — a +/// `../..` in it would download an archive and extract it anywhere the worker +/// can write. Accepted only as a plain version token, which every version any +/// of the three engines publishes already is. +fn checked_version<'a>(v: Option<&'a str>, field: &str) -> error::Result> { + if let Some(v) = v { + let plain = v.len() <= 64 + && v.starts_with(|c: char| c.is_ascii_alphanumeric()) + && v.chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_' | '+')); + if !plain { + return Err(Error::BadRequest(format!( + "the lockfile's `{field}` is not a version: expected a token of letters, digits, \ + `.`, `-`, `_` or `+` starting with a letter or digit, got `{v}`" + ))); + } + } + Ok(v) +} + +/// dbt adapters whose package this worker installs unasked. A PUBLISHED-PACKAGE list, not a +/// capability one: an adapter absent from it still renders a profile and still runs under an +/// engine that already carries it. See `ensure_adapter_installable`. +const PUBLISHED_ADAPTERS: &[&str] = &[ + "athena", "clickhouse", "databricks", "decodable", "doris", "dremio", "duckdb", "exasol", + "extrica", "fabric", "fabricspark", "firebolt", "glue", "greenplum", "hive", "ibmdb2", + "impala", "materialize", "mysql", "oracle", "postgres", "redshift", "risingwave", "rockset", + "singlestore", "snowflake", "spark", "sqlite", "sqlserver", "starrocks", "synapse", "teradata", + "tidb", "trino", "vertica", "yellowbrick", +]; + +/// Refuse to install a package nobody vouched for. `dbt-` comes from an adapter name a +/// SCRIPT AUTHOR chooses, `dbt-` is not a reserved PyPI prefix, and this install runs outside +/// the nsjail ordinary dependency installation uses — so an unbounded name would run a PEP +/// 517 backend as the worker. The admin decides what is trusted, via `DBT_EXTRA_ADAPTERS`. +fn ensure_adapter_installable(adapter: &DbtAdapter) -> error::Result<()> { + let name = adapter.name(); + if adapter.known().is_some() + || PUBLISHED_ADAPTERS.contains(&name) + || DBT_EXTRA_ADAPTERS.iter().any(|a| a == name) + { + return Ok(()); + } + Err(Error::BadRequest(format!( + "`{name}` is not an adapter this instance installs: the dbt-core 1.x engine would have \ + to fetch `dbt-{name}` from PyPI, and `dbt-` is not a reserved name there. An admin adds \ + it to DBT_EXTRA_ADAPTERS, or use an engine that ships its adapters (`engine: fusion`)" + ))) +} + +/// A uv venv per (dbt version, adapter): the adapter is a separate pip package +/// and installing every adapter into one venv would make their transitive +/// dependency sets fight. +async fn provision_core_1x( + adapter: DbtAdapter, + pinned_version: Option<&str>, + pinned_adapter_version: Option<&str>, + job_id: &Uuid, + w_id: &str, + conn: &Connection, + ctx: &mut JobCtx<'_>, +) -> error::Result { + ensure_adapter_installable(&adapter)?; + if adapter.pip_package().is_empty() { + return Err(Error::BadRequest(format!( + "the {} adapter has no dbt-core 1.x package: it exists only inside the Fusion \ + engine. Set `engine: fusion` to use it", + adapter.name() + ))); + } + // A RANGE, not an `==`. A locked version still pins exactly, because that + // one was resolved for this adapter in the first place. + let version_spec = match pinned_version { + Some(v) => format!("dbt-core=={v}"), + None => format!( + "dbt-core>={},<{}", + DBT_CORE_1X_FLOOR.as_str(), + DBT_CORE_1X_CEILING.as_str() + ), + }; + let version = pinned_version + .map(str::to_string) + .unwrap_or_else(|| DBT_CORE_1X_VERSION.clone()); + // The adapter is in the cache key: pinning core alone would let a rebuilt + // cache resolve a newer adapter than the deploy did, which changes runtime + // behavior under a lockfile that claims to prevent exactly that. + let adapter_spec = match pinned_adapter_version { + Some(v) => format!("{}=={v}", adapter.pip_package()), + None => adapter.pip_package().to_string(), + }; + let dir = PathBuf::from(&*DBT_CACHE_DIR).join(format!( + "core1x-{}-{}", + digest(&version_spec), + digest(&adapter_spec) + )); + let bin = dir.join("bin").join("dbt"); + if bin.exists() { + let adapter_version = installed_adapter_version(&dir, adapter).await; + return Ok(ProvisionedEngine { + root: dir.clone(), + bin, + // What the resolver actually chose, so the lock pins the version + // this adapter can take rather than the one we asked for. + version: installed_package_version(&dir, "dbt_core") + .await + .unwrap_or(version), + engine: DbtEngine::DbtCore1x, + adapter_version, + }); + } + + append_logs( + job_id, + w_id, + format!( + "\nProvisioning {version_spec} with {}...\n", + adapter.pip_package() + ), + conn, + ) + .await; + // Build beside the target and rename, so two jobs racing on one worker cannot + // observe a half-installed venv through `bin.exists()`. `--relocatable` makes + // the rename safe: without it uv bakes the staging path into every shebang and + // the moved venv's `dbt` fails with ENOENT. + let staging_guard = Scratch::new(staging_path(&dir, job_id)); + let staging = staging_guard.path().to_path_buf(); + tokio::fs::remove_dir_all(&staging).await.ok(); + run_tool( + Command::new(UV_PATH.as_str()) + // The interpreter goes in Windmill's own cache rather than the + // worker user's home: a sandboxed job can only see the paths that + // are mounted into it, and this is one of them. + .env("UV_PYTHON_INSTALL_DIR", crate::PY_INSTALL_DIR.as_str()) + .args([ + "venv", + "--relocatable", + "--python", + DBT_PYTHON_VERSION.as_str(), + ]) + .arg(&staging), + "uv venv", + job_id, + w_id, + conn, + ctx, + ) + .await?; + run_tool( + Command::new(UV_PATH.as_str()) + .env("UV_PYTHON_INSTALL_DIR", crate::PY_INSTALL_DIR.as_str()) + .env("VIRTUAL_ENV", &staging) + .args(["pip", "install", &version_spec, &adapter_spec]), + "uv pip install", + job_id, + w_id, + conn, + ctx, + ) + .await + .map_err(|e| { + // The common cause is an adapter with no release for this dbt-core + // range: dbt-mysql, for one, has not shipped past `~=1.7`. uv reports + // that as a resolver dump, which does not say what to do about it. + Error::ExecutionErr(format!( + "{e}\n\ninstalling the {} adapter: it must have a release compatible with \ + {version_spec}, which is the dbt-core CLI this engine invokes. If the adapter has \ + not kept up, use `engine: dbt-core-2x` or `engine: fusion`", + adapter.name() + )) + })?; + match tokio::fs::rename(&staging, &dir).await { + Ok(()) => staging_guard.keep(), + // Lost the race: the winner's venv is equivalent, so use it. The guard + // removes ours. + Err(_) if bin.exists() => {} + Err(e) => return Err(Error::internal_err(format!("installing dbt-core: {e}"))), + } + let adapter_version = installed_adapter_version(&dir, adapter).await; + let version = installed_package_version(&dir, "dbt_core") + .await + .unwrap_or(version); + Ok( + ProvisionedEngine { + root: dir, + bin, + version, + engine: DbtEngine::DbtCore1x, + adapter_version, + }, + ) +} + +/// The adapter version a venv actually resolved, read from its dist-info so a +/// deploy can lock it and later runs can ask for the same one. +async fn installed_adapter_version(dir: &Path, adapter: DbtAdapter) -> Option { + installed_package_version(dir, &adapter.pip_package().replace('-', "_")).await +} + +/// The version of an installed distribution, read from its `.dist-info`. +async fn installed_package_version(dir: &Path, dist: &str) -> Option { + let prefix = format!("{dist}-"); + let mut entries = tokio::fs::read_dir(dir.join("lib")).await.ok()?; + while let Ok(Some(py)) = entries.next_entry().await { + let mut pkgs = tokio::fs::read_dir(py.path().join("site-packages")) + .await + .ok()?; + while let Ok(Some(e)) = pkgs.next_entry().await { + let name = e.file_name().to_string_lossy().to_string(); + if let Some(rest) = name.strip_prefix(&prefix) { + if let Some(v) = rest.strip_suffix(".dist-info") { + return Some(v.to_string()); + } + } + } + } + None +} + +async fn provision_core_2x( + pinned_version: Option<&str>, + job_id: &Uuid, + w_id: &str, + conn: &Connection, + ctx: &mut JobCtx<'_>, +) -> error::Result { + let version = pinned_version + .map(str::to_string) + .unwrap_or_else(|| DBT_CORE_2X_VERSION.clone()); + // No Windmill image ships an engine, so this is the operator's own + // pre-stage: `DBT_BUNDLED_DIR` populated in a derived image, for an + // air-gapped instance or a fleet that should not fetch per worker. Checked + // before the cache because it is read-only and shared, where the cache is a + // per-worker volume. + let bundled = PathBuf::from(&*DBT_BUNDLED_DIR) + .join(format!("core2x-{version}")) + .join("dbt-sa-cli"); + if bundled.exists() { + return Ok(ProvisionedEngine { + root: bundled.parent().map(Path::to_path_buf).unwrap_or_default(), + bin: bundled, + version, + engine: DbtEngine::DbtCore2x, + adapter_version: None, + }); + } + let dir = PathBuf::from(&*DBT_CACHE_DIR).join(format!("core2x-{version}")); + let bin = dir.join("dbt-sa-cli"); + if bin.exists() { + return Ok(ProvisionedEngine { + root: dir.clone(), + bin, + version, + engine: DbtEngine::DbtCore2x, + adapter_version: None, + }); + } + let target = format!("{}-unknown-linux-gnu", std::env::consts::ARCH); + let url = format!( + "https://github.com/dbt-labs/dbt-core/releases/download/v{version}/dbt-core-{version}-{target}.tar.gz" + ); + append_logs( + job_id, + w_id, + format!("\nFetching dbt-core {version}...\n"), + conn, + ) + .await; + fetch_and_extract(&url, &dir, "dbt-sa-cli", job_id, w_id, conn, ctx).await?; + Ok(ProvisionedEngine { + root: dir, + bin, + version, + engine: DbtEngine::DbtCore2x, + adapter_version: None, + }) +} + +/// Fusion is fetched from dbt Labs at runtime and cached — see the module docs +/// for why it must never be baked into an image. +async fn provision_fusion( + pinned_version: Option<&str>, + job_id: &Uuid, + w_id: &str, + conn: &Connection, + ctx: &mut JobCtx<'_>, +) -> error::Result { + // Version-keyed like the other two: one shared `fusion` directory means a + // run landing on a clean worker fetches whatever is current rather than + // what the deploy locked. + let dir = PathBuf::from(&*DBT_CACHE_DIR).join(match pinned_version { + Some(v) => format!("fusion-{v}"), + None => "fusion".to_string(), + }); + // The installer places the binary directly in the directory `--to` names. + let bin = dir.join("dbt"); + if bin.exists() { + return Ok(ProvisionedEngine { + root: dir.clone(), + bin, + version: fusion_version(&dir).await, + engine: DbtEngine::Fusion, + adapter_version: None, + }); + } + append_logs( + job_id, + w_id, + "\nFetching the dbt Fusion engine from dbt Labs (not bundled with Windmill; \ + subject to the dbt Fusion engine license agreement)...\n" + .to_string(), + conn, + ) + .await; + let script = fetch_under_job( + "the Fusion installer", + async { + let net = + |e: reqwest::Error| Error::internal_err(format!("fetching the installer: {e}")); + Ok(windmill_common::utils::HTTP_CLIENT + .get(&*DBT_FUSION_INSTALL_URL) + .send() + .await + .and_then(|r| r.error_for_status()) + .map_err(net)? + .text() + .await + .map_err(net)?) + }, + job_id, + w_id, + conn, + ctx, + ) + .await?; + // Install into a per-job sibling and rename, like the other two engines: + // pointing the installer straight at the shared cache lets a second job + // observe `bin/dbt` and execute it while the first is still writing. + let staging_guard = Scratch::new(staging_path(&dir, job_id)); + let staging = staging_guard.path().to_path_buf(); + tokio::fs::remove_dir_all(&staging).await.ok(); + let script_guard = + Scratch::new(std::env::temp_dir().join(format!("wm-fusion-install-{job_id}.sh"))); + let tmp = script_guard.path().to_path_buf(); + write_file( + tmp.parent().unwrap().to_str().unwrap(), + tmp.file_name().unwrap().to_str().unwrap(), + &script, + )?; + // `--to` and `--version` are the installer's own flags (`install.sh + // --help`); it takes no positional arguments and ignores unknown ones, so + // an approximation here fails by silently installing the latest release + // into the user's $HOME instead of the cache. + let mut install = Command::new("sh"); + install.arg(&tmp).arg("--to").arg(&staging); + if let Some(v) = pinned_version { + install.args(["--version", v]); + } else { + install.arg("--update"); + } + run_tool(&mut install, "fusion install", job_id, w_id, conn, ctx).await?; + drop(script_guard); + if !staging.join("dbt").exists() { + return Err(Error::internal_err( + "the Fusion installer did not produce a dbt binary".to_string(), + )); + } + if tokio::fs::rename(&staging, &dir).await.is_ok() { + staging_guard.keep(); + } else { + if !bin.exists() { + return Err(Error::internal_err( + "could not install the Fusion engine".to_string(), + )); + } + } + Ok(ProvisionedEngine { + root: dir.clone(), + bin, + version: fusion_version(&dir).await, + engine: DbtEngine::Fusion, + adapter_version: None, + }) +} + +/// From the binary itself: the installer writes no version file, and a pinned +/// directory name would only echo back what was asked for rather than what was +/// installed. +async fn fusion_version(dir: &Path) -> String { + Command::new(dir.join("dbt")) + .arg("--version") + .output() + .await + .ok() + .filter(|o| o.status.success()) + .and_then(|o| { + String::from_utf8_lossy(&o.stdout) + .split_whitespace() + .nth(1) + .map(|v| v.to_string()) + }) + .unwrap_or_else(|| "unknown".to_string()) +} + +/// A sibling of `dir` to build in before renaming into place. Appended to the +/// whole file name rather than via `with_extension`, which would eat everything +/// after the version's last dot. +/// A per-job path removed on drop unless the install claimed it. +/// +/// Provisioning is a sequence of fallible awaits — a download, an installer, an +/// extraction — and each one can also be cancelled. Cleaning up after the `?` +/// only covers the paths someone remembered, so a run of failed or cancelled +/// first-use installs accumulates venvs and tarballs until the worker's disk is +/// gone. Dropping is the one exit every path takes. +struct Scratch(Option); + +impl Scratch { + fn new(path: PathBuf) -> Self { + Scratch(Some(path)) + } + + fn path(&self) -> &Path { + self.0.as_deref().unwrap_or(Path::new("")) + } + + /// The install succeeded and moved it: stop owning it. + fn keep(mut self) { + self.0 = None; + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let Some(p) = self.0.take() else { + return; + }; + let remove = move || { + let _ = std::fs::remove_dir_all(&p); + let _ = std::fs::remove_file(&p); + }; + // Off the runtime thread: this removes thousands of files and `Drop` runs + // inside the job future, so doing it synchronously stalls every other job + // on that thread. Detached because a `Drop` cannot await. + match tokio::runtime::Handle::try_current() { + Ok(handle) => { + handle.spawn_blocking(remove); + } + // No runtime to hand it to (a test, or shutdown): do it here rather + // than leave the directory behind. + Err(_) => remove(), + } + } +} + +fn staging_path(dir: &Path, job_id: &Uuid) -> PathBuf { + let name = dir.file_name().unwrap_or_default().to_string_lossy(); + dir.with_file_name(format!("{name}.staging-{job_id}")) +} + +/// Fetch under the job's cancellation and timeout. The shared `HTTP_CLIENT` +/// sets no request timeout, so a stalled release download would otherwise hold +/// the worker for as long as the connection stays open. +async fn fetch_under_job( + what: &str, + fetch: F, + job_id: &Uuid, + w_id: &str, + conn: &Connection, + ctx: &mut JobCtx<'_>, +) -> error::Result +where + F: std::future::Future>, +{ + run_future_with_polling_update_job_poller( + *job_id, + ctx.timeout(), + conn, + ctx.mem_peak, + ctx.canceled_by, + async move { + fetch + .await + .map_err(|e| Error::internal_err(format!("fetching {what}: {e}"))) + }, + ctx.worker_name, + w_id, + &mut Some(ctx.occupancy_metrics), + Box::pin(futures::stream::once(async { 0 })), + ) + .await +} + +async fn fetch_and_extract( + url: &str, + dir: &Path, + expected_bin: &str, + job_id: &Uuid, + w_id: &str, + conn: &Connection, + ctx: &mut JobCtx<'_>, +) -> error::Result<()> { + let tarball_guard = Scratch::new(std::env::temp_dir().join(format!("wm-dbt-{job_id}.tar.gz"))); + let tarball = tarball_guard.path().to_path_buf(); + // STREAMED to disk, never held whole: an engine archive is ~290 MB, this + // runs in the shared worker process rather than the job's subprocess, and + // several cold jobs provision at once — buffering would multiply that until + // the worker dies, taking every job on it along. + fetch_under_job( + url, + async { + use futures::StreamExt; + use tokio::io::AsyncWriteExt; + let net = |e: reqwest::Error| Error::internal_err(format!("fetching {url}: {e}")); + let mut resp = windmill_common::utils::HTTP_CLIENT + .get(url) + .send() + .await + .and_then(|r| r.error_for_status()) + .map_err(net)? + .bytes_stream(); + let mut file = tokio::fs::File::create(&tarball) + .await + .map_err(|e| Error::internal_err(format!("writing {url}: {e}")))?; + while let Some(chunk) = resp.next().await { + let chunk = chunk.map_err(net)?; + file.write_all(&chunk) + .await + .map_err(|e| Error::internal_err(format!("writing {url}: {e}")))?; + } + file.flush() + .await + .map_err(|e| Error::internal_err(format!("writing {url}: {e}")))?; + Ok(()) + }, + job_id, + w_id, + conn, + ctx, + ) + .await?; + let staging_guard = Scratch::new(staging_path(dir, job_id)); + let staging = staging_guard.path().to_path_buf(); + tokio::fs::create_dir_all(&staging) + .await + .map_err(|e| Error::internal_err(format!("creating {staging:?}: {e}")))?; + run_tool( + Command::new("tar") + .arg("xzf") + .arg(&tarball) + .arg("-C") + .arg(&staging) + // The release tarballs wrap the binary in a versioned directory. + .args(["--strip-components", "1"]), + "tar", + job_id, + w_id, + conn, + ctx, + ) + .await?; + drop(tarball_guard); + if !staging.join(expected_bin).exists() { + return Err(Error::internal_err(format!( + "{url} did not contain the expected `{expected_bin}` binary" + ))); + } + if tokio::fs::rename(&staging, dir).await.is_ok() { + staging_guard.keep(); + } else { + if !dir.join(expected_bin).exists() { + return Err(Error::internal_err(format!("could not install {url}"))); + } + } + Ok(()) +} + +/// Run a provisioning command to completion. Provisioning happens inside the +/// job that needs the engine, so it runs under that job's cancellation and +/// timeout: a cold `uv pip install` or Fusion download is the longest thing a +/// dbt job does, and a cancel that could not reach it would hold the worker +/// slot for the rest of the install. Its output is not streamed to the job log +/// — these are worker-level setup steps with no node to attribute progress to, +/// and failures surface with the tool's own stderr. +async fn run_tool( + cmd: &mut Command, + name: &str, + job_id: &Uuid, + w_id: &str, + conn: &Connection, + ctx: &mut JobCtx<'_>, +) -> error::Result<()> { + let child = cmd + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + // The wait future owns the child, so cancellation dropping that future + // is what terminates the install — dropping it alone would not. + .kill_on_drop(true) + .spawn() + .map_err(|e| Error::internal_err(format!("{name} could not be started: {e}")))?; + let pid = child.id(); + let out = run_future_with_polling_update_job_poller( + *job_id, + ctx.timeout(), + conn, + ctx.mem_peak, + ctx.canceled_by, + async move { + child + .wait_with_output() + .await + .map_err(|e| Error::internal_err(format!("{name} failed: {e}"))) + }, + ctx.worker_name, + w_id, + &mut Some(ctx.occupancy_metrics), + Box::pin(futures::stream::unfold((), move |_| async move { + Some((get_mem_peak(pid, false).await, ())) + })), + ) + .await?; + if !out.status.success() { + return Err(Error::internal_err(format!( + "{name} failed: {}", + String::from_utf8_lossy(&out.stderr) + ))); + } + Ok(()) +} + +#[cfg(test)] +mod core1x_tests { + use super::*; + use crate::dbt_profiles::KnownAdapter; + + // Several adapters cap dbt-core below what this runtime would ask for + // (dbt-mysql ~=1.7, dbt-oracle and dbt-databricks below 1.12) and + // dbt-salesforce has no 1.x package at all, so pinning core independently made + // those projects fail at provisioning. The install names a ceiling instead. + #[test] + fn every_adapter_either_names_a_package_or_is_fusion_only() { + for a in KnownAdapter::ALL { + if matches!(a, KnownAdapter::Salesforce) { + continue; + } + assert!( + !a.pip_package().is_empty(), + "{} must name a pip package for dbt-core 1.x", + a.name() + ); + } + // Fusion has it built in, and there is no package to install. + assert!(KnownAdapter::Salesforce.pip_package().is_empty()); + assert_eq!(KnownAdapter::Salesforce.name(), "salesforce"); + } + + // `dbt-` is not a reserved prefix on PyPI and this install is not sandboxed, + // so the name a script author picks decides which package runs its build + // backend as the worker. + #[test] + fn an_unvouched_adapter_is_not_installed() { + let known = DbtAdapter::from_dbt_type("postgres").unwrap(); + assert!(ensure_adapter_installable(&known).is_ok()); + let published = DbtAdapter::from_dbt_type("trino").unwrap(); + assert!(ensure_adapter_installable(&published).is_ok()); + let squatted = DbtAdapter::from_dbt_type("totally-legit-adapter").unwrap(); + assert!(ensure_adapter_installable(&squatted).is_err()); + } + + /// A preview submits its own lockfile, so this string reaches a path join + /// and a pip requirement straight from the caller, on the host and outside + /// the jail. + #[test] + fn a_lockfile_version_cannot_leave_the_cache_directory() { + for v in ["1.12.0", "2.0.0-alpha.5", "2.0.0-preview.202"] { + assert_eq!(checked_version(Some(v), "engine_version").unwrap(), Some(v)); + } + for v in [ + "../../../../etc", + "1.0/../..", + "..", + ".ssh", + "-rf", + "1.0 --index-url=http://x", + "1.0\n", + ] { + assert!( + checked_version(Some(v), "engine_version").is_err(), + "`{v}` must be refused before it reaches a path" + ); + } + assert!(checked_version(None, "engine_version").unwrap().is_none()); + } +} diff --git a/backend/windmill-worker/src/dbt_executor.rs b/backend/windmill-worker/src/dbt_executor.rs new file mode 100644 index 0000000000..33ddd6c32e --- /dev/null +++ b/backend/windmill-worker/src/dbt_executor.rs @@ -0,0 +1,5783 @@ +//! Running a dbt project as a Windmill job. +//! +//! One `dbt build` per job, not one job per model. That is the shape +//! astronomer-cosmos arrived at with `ExecutionMode.WATCHER` after per-model +//! Airflow tasks proved roughly 6x slower on a real project; dbt's own +//! threading provides the parallelism and Windmill provides the observability +//! (docs/dbt-runtime.md). Per-model status comes from dbt's JSON event stream +//! while the run is in flight, and the structured job result comes from +//! `run_results.json` at the end. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use std::process::Stdio; + +use serde::{Deserialize, Serialize}; +use serde_json::value::RawValue; +use sha2::{Digest, Sha256}; +use tokio::process::Command; +use uuid::Uuid; +use windmill_common::client::AuthedClient; +use windmill_common::error::{self, Error}; +use windmill_common::materialization::{ + record_materialization, MaterializationStatus, RecordMaterializationRequest, +}; +use windmill_common::worker::{to_raw_value, write_file, Connection}; +use windmill_parser_yaml::{ + parse_dbt_descriptor, DbtDescriptor, DbtTestBehavior, DBT_COMMANDS, DBT_COMMAND_ARG, + DBT_COMMAND_LABEL, DBT_DEFAULT_WAREHOUSE, +}; +use windmill_queue::{append_logs, CanceledBy, MiniPulledJob}; + +use crate::common::{ + render_nsjail_rlimit_as, resolve_nsjail_timeout, resolve_nsjail_tmp_mount_block, +}; +use crate::common::{start_child_process, OccupancyMetrics}; +use crate::dbt_engine::{provision_engine, ProvisionedEngine, DBT_CACHE_DIR}; +use crate::dbt_profiles::{ + ensure_adapter_licensed, render_dbt_profile, render_profile, DbtAdapter, KnownAdapter, +}; +use crate::handle_child::{ + get_mem_peak, handle_child, run_future_with_polling_update_job_poller, JobCtx, JobDeadline, +}; +use crate::worker::write_module_files; +use crate::{ + is_sandboxing_enabled, GIT_PATH, NSJAIL_DBT_RLIMIT_AS_MB, NSJAIL_PATH, PATH_ENV, PROXY_ENVS, + TZ_ENV, +}; + +/// The profile name Windmill renders into `profiles.yml`. dbt takes the profile +/// to use from `dbt_project.yml`, so the rendered file must answer to whatever +/// name the project declares — resolved from the project file, with this as the +/// fallback for the (invalid) case where it declares none. +const FALLBACK_PROFILE_NAME: &str = "windmill"; + +/// Written to the script's lockfile at deploy. +#[derive(Serialize, Deserialize, Debug, Clone, Default)] +pub struct DbtDependencyLocks { + /// The `/` the profile resolved to at deploy. The + /// resource is re-read on every run, so a schema or catalog changed on it + /// afterwards moves every relation the project builds — and the stored + /// graph, which still names the old ones, has to be re-ingested. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub profile_relation_root: Option, + /// dbt-core 1.x only: its adapter is a separate package versioning + /// independently of core, so pinning core alone still lets a rebuilt cache + /// resolve different runtime behavior. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub adapter_version: Option, + /// Digest of the `package-lock.yml` produced at deploy. Package trees are + /// worker-local, so a cache miss on another worker must prove it resolved + /// the same dependencies before it may run this script version. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub package_lock_digest: Option, + pub manifest_digest: String, + pub engine: String, + pub engine_version: String, +} + +/// Per-node outcome, from `run_results.json`. +#[derive(Serialize, Debug, Clone)] +pub struct DbtNodeResult { + pub unique_id: String, + /// dbt's own status word, verbatim (`success`, `error`, `partial success`, + /// `no-op`, …). Kept because it is what the log and dbt's docs say, but it + /// is dbt's vocabulary to change — read `outcome` to make a decision. + pub status: String, + /// The same result in Windmill's terms, which is the stable half of this + /// contract: `passed` | `failed` | `warned` | `skipped` | `no_op` | + /// `unknown`. A dbt release that renames a status, or adds one, moves + /// `status` and leaves this alone. + pub outcome: &'static str, + pub execution_time: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub rows_affected: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub relation_name: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub message: Option, + /// Test nodes: how many rows violated the assertion. + #[serde(skip_serializing_if = "Option::is_none")] + pub failures: Option, +} + +/// The job result. Partial failure is dbt's normal case, so the result has to +/// be legible without reading the log: which models succeeded, which failed, +/// which tests failed and at what severity. +#[derive(Serialize, Debug)] +pub struct DbtRunResult { + pub engine: String, + pub engine_version: String, + pub command: String, + pub totals: DbtTotals, + pub nodes: Vec, + /// The arguments this invocation ran with, as SUBMITTED — a `$var:` stays a + /// reference, so no resolved value (and no secret) is published. + /// + /// Present because a `dbt retry` restores the failed run's arguments inside + /// the worker and they are never written back to the retry job: its own + /// args are just `{"dbt_command": "retry"}`. Anything that needs to act on + /// what the run actually used — the row preview, which is a `dbt show` of + /// the same project — cannot get them from the job. + #[serde(skip_serializing_if = "std::collections::HashMap::is_empty")] + pub invocation_args: std::collections::HashMap>, +} + +#[derive(Serialize, Debug, Default)] +pub struct DbtTotals { + pub total: usize, + pub success: usize, + pub error: usize, + pub warn: usize, + pub skipped: usize, +} + +#[derive(Deserialize, Debug)] +struct RunResults { + #[serde(default)] + results: Vec, +} + +#[derive(Deserialize, Debug)] +struct RunResultNode { + unique_id: String, + status: String, + #[serde(default)] + execution_time: Option, + #[serde(default)] + adapter_response: serde_json::Value, + #[serde(default)] + relation_name: Option, + #[serde(default)] + message: Option, + #[serde(default)] + failures: Option, +} + +pub(crate) async fn handle_dbt_job( + requirements_o: Option<&String>, + job_dir: &str, + worker_name: &str, + job: &MiniPulledJob, + mem_peak: &mut i32, + canceled_by: &mut Option, + conn: &Connection, + client: &AuthedClient, + inner_content: &str, + envs: HashMap, + occupancy_metrics: &mut OccupancyMetrics, + modules: Option<&HashMap>, +) -> error::Result> { + let descriptor = parse_dbt_descriptor(inner_content)?; + let locks: Option = requirements_o + .map(|s| { + serde_json::from_str(s) + .map_err(|e| Error::internal_err(format!("reading the dbt lockfile: {e}"))) + }) + .transpose()?; + + // Through `build_args_map`, like every other executor: dbt cannot resolve a + // `$var:` / `$res:` / `$encrypted:` reference, so passing one raw sends the + // literal string to `--vars` — a placeholder holding a schema or an `enabled` + // flag would then build a different slice of the project than was asked for. + let args = flatten_command( + crate::common::build_args_map(job, client, conn) + .await? + .unwrap_or_else(|| job.args.as_ref().map(|a| a.0.clone()).unwrap_or_default()), + )?; + // As submitted, command block and all: this is what the state saves and the + // result publishes, and both describe an invocation of this script, not one + // executor's view of it. + let raw_args = job.args.as_ref().map(|a| a.0.clone()).unwrap_or_default(); + let inv = Invocation { args: args.clone(), raw_args, envs: envs.clone(), strict: true }; + // One wall clock for the whole job. A dbt job is a sequence of + // subprocesses — provision, deps, parse, ls, build, then the + // `after_all` tests — and each would otherwise resolve the job's full + // timeout for itself. + let deadline = JobDeadline::start(conn, &job.workspace_id, job.id, job.timeout).await; + // Built once and reborrowed into every phase. The five fields travel + // together through the whole executor, so passing them apart means each new + // phase grows another five parameters and another copy of this literal. + let mut ctx = JobCtx { mem_peak, canceled_by, occupancy_metrics, worker_name, deadline }; + // From EVERY dbt run, not from the progress reporter: that reporter runs only + // for engines emitting node events, so hanging the prune off it leaves a + // Fusion-only or dbt-core-2x instance accumulating rows nothing deletes. + if let Connection::Sql(pool) = conn { + let (pool, prune_w_id) = (pool.clone(), job.workspace_id.clone()); + let prune_path = job.runnable_path.clone().unwrap_or_default(); + tokio::spawn(async move { + windmill_common::dbt_manifest::prune_run_progress(&pool, &prune_w_id).await; + if let Err(e) = + windmill_common::dbt_manifest::prune_dbt_run_graphs(&pool, &prune_path, &prune_w_id) + .await + { + tracing::warn!("pruning dbt run graph snapshots: {e:#}"); + } + }); + } + let prepared = prepare_project( + &descriptor, + inner_content, + locks.as_ref(), + job_dir, + &job.id, + &job.workspace_id, + job.runnable_path.as_deref().unwrap_or_default(), + job.runnable_id.map(|h| h.0), + conn, + client, + &mut ctx, + &envs, + modules, + ) + .await; + // A preparation failure leaves the saved run alone: nothing up to here — the + // bundle, `dbt deps`, the engine, the profile — touches a relation, so the + // previous run's failures still describe the warehouse. Only an interrupted + // BUILD invalidates that, which the save at the end of the job decides. + let mut prepared = prepared?; + + // A `vars` override drives `enabled`, alias, schema, database and + // materialization, so the run's models are not the deployed graph's: it + // snapshots under its own job id. Re-asked below for a retry, which submits + // only `dbt_command` until the failed run's arguments are restored. + prepared.graph_refresh.add_caller_args(&descriptor, &args)?; + + let command = match arg_str(&args, "dbt_command")? { + // An allowlist, not a passthrough: the value becomes the dbt subcommand, + // and running a script needs weaker permission than editing it — so an + // unchecked arg would let a runner invoke `clean` or `seed` on the + // descriptor's warehouse. + Some(c) if DBT_COMMANDS.contains(&c.as_str()) => c, + Some(c) => { + return Err(Error::BadRequest(format!( + "`dbt_command` must be one of {}, got `{c}`", + DBT_COMMANDS.join(", ") + ))) + } + None => windmill_parser_yaml::default_dbt_command(&descriptor).to_string(), + }; + // A parse is the whole job: it resolves the project into a manifest, stores + // the graph and stops. Handled before everything below because none of it + // applies — nothing is built, so there is no test phase, no materialization, + // no retry state and no ownership to publish. + if command == "parse" { + return run_parse_only( + &prepared, + &descriptor, + // Tolerant of the `{{ }}` placeholders only a run can fill, exactly + // as the deploy's own parse is: refreshing an editor buffer must not + // require filling the run form in first, and the graph it produces is + // the one the deploy would store. + &Invocation { strict: false, ..inv }, + &mut ctx, + job, + conn, + ) + .await; + } + + // `dbt retry` resumes from the previous run's `run_results.json`, which is what + // makes one-job-per-invocation defensible. Each attempt gets a fresh job dir, so + // that state is restored along with the ARGUMENTS it ran with: dbt reuses the + // failed invocation's selection and vars, so every phase must agree with them. + let mut restored_results_digest: Option = None; + let inv = if command == "retry" { + // Read BEFORE the restore replaces the arguments. A retry must name the run + // it resumes: only the latest failure of this script is kept, so an unnamed + // one would mean "whatever failed last" and quietly resume a different run + // than the caller was looking at. + let expected = arg_str(&inv.args, "dbt_retry_job")?.filter(|s| !s.trim().is_empty()); + let Some(expected) = expected else { + return Err(Error::BadRequest( + "a `retry` needs `dbt_retry_job`, the id of the run to resume. Open that run and \ + use `Resume this run`, or pass its id: only the latest failure of this script is \ + kept, so a retry names which one it means" + .to_string(), + )); + }; + // Parsed, not compared as text: the saved run is a `uuid`, and an id + // that differs only in case or in braces names the same run. + let expected = Uuid::parse_str(expected.trim()).map_err(|_| { + Error::BadRequest(format!( + "`dbt_retry_job` must be the id of the run to resume, got `{expected}`" + )) + })?; + let restored = restore_run_state( + &prepared, + &job.workspace_id, + &job.permissioned_as, + &inv, + expected, + conn, + ) + .await?; + restored_results_digest = Some(restored.results_digest.clone()); + // Restored args are the ones SUBMITTED, so the references they carry are + // resolved again now — against this caller's access, not the original's. + let inv = Invocation { + args: flatten_command( + crate::common::transform_json(client, &job.workspace_id, &restored.args, job, conn) + .await? + .unwrap_or_else(|| restored.args.clone()), + )?, + raw_args: restored.args, + ..inv + }; + // Compared only now that they are resolved: a `$var:` whose value moved + // selects a different node set, so the saved failures no longer describe + // what a retry would build. Refused rather than resumed, since which + // graph it would use depends on which worker it lands on. + if let Some(saved) = restored.args_digest.as_deref() { + if saved != inv.resolved_args_digest() { + return Err(Error::BadRequest( + "the values this run's arguments resolve to have changed since the run \ + being retried, so its failures no longer describe what a retry would \ + build; run the script normally instead" + .to_string(), + )); + } + } + // The restored arguments decide this retry's graph, and RESOLVED ones: a + // saved `select` spelled `$res:` is a string until resolved, so reading the + // raw form refuses the retry as "must be a list of strings" for a reference + // that resolves to the very list the failed run built with. + prepared + .graph_refresh + .add_caller_args(&descriptor, &inv.args)?; + if restored.needs_parse { + // After the resolution above, so the manifest describes the project + // the build is about to retry. + run_dbt_parse( + &prepared, + &descriptor, + &inv, + &mut ctx, + &job.id, + &job.workspace_id, + conn, + ) + .await?; + } + inv + } else { + inv + }; + + // Ingested BEFORE the build, from a `dbt parse` with this run's vars, so the + // models shown are the ones about to be built. Rows are keyed by path, version + // AND job so no two runs collide; the path-keyed `asset` usage belongs to one + // version, which `claim_graph_publication` arbitrates (docs/dbt-runtime.md). + if prepared.graph_refresh.needed() && !windmill_parser_yaml::dbt::is_read_only_command(&command) + { + // A parse and an ingest write no relation either, so a failure in either + // leaves the saved run as accurate as the preparation exits above do. + if command != "retry" { + run_dbt_parse( + &prepared, + &descriptor, + &inv, + &mut ctx, + &job.id, + &job.workspace_id, + conn, + ) + .await?; + } + // For a retry the restored manifest already describes the invocation + // being resumed, so only the ingest runs — with that invocation's + // arguments, which the selection resolver needs to interpolate. + ingest_from_run(&prepared, &descriptor, &inv, &mut ctx, job, conn).await?; + } + + // A read-only command prints rows to stdout, so it is captured rather than + // streamed, and nothing below applies: nothing was built, so there is no graph + // to publish, no materialization, no test phase and nothing to retry. + if windmill_parser_yaml::dbt::is_read_only_command(&command) { + return run_show( + &prepared, + &descriptor, + &inv, + &mut ctx, + &job.id, + &job.workspace_id, + conn, + ) + .await; + } + + let mut run = run_dbt( + &prepared, + &command, + &descriptor, + &inv, + job, + conn, + &mut ctx, + true, + ) + .await; + + // `after_all` is two invocations, models then tests, and each REWRITES + // `run_results.json` — so the model results are read before the test phase + // overwrites them, or the job reports tests alone and nothing settles the + // models' materializations. + let mut results = read_run_results(&prepared.project_dir).await; + + // In-job node retry: rebuilding only the failed and skipped nodes, while the + // previous attempt's `run_results.json` is still in the job directory. Never on + // an agent worker, which cannot read `v2_job_queue` — the wait below would be + // uninterruptible, so a cancelled job would hold its slot and then start dbt. + let node_retry = descriptor + .retry_failed_nodes + .filter(|_| matches!(conn, Connection::Sql(_))); + let mut retries_left = node_retry.map(|p| p.attempts()).unwrap_or(0); + if let Some(policy) = node_retry.filter(|_| run.is_err()) { + retry_failed_nodes( + policy, + &prepared, + &descriptor, + &inv, + job, + conn, + &mut ctx, + &mut run, + &mut results, + &mut retries_left, + ) + .await; + } + // A `retry` whose saved results were tests alone IS the test phase: dbt reran + // exactly those tests, so running the suite after it would execute every test + // a second time and report each one twice. `test_behavior: after_all` is how + // `run_results.json` comes to hold tests alone. + let retry_was_the_test_phase = command == "retry" + && !results.is_empty() + && results + .iter() + .all(|n| n.unique_id.starts_with("test.") || n.unique_id.starts_with("unit_test.")); + // `retry` counts as the model phase too: a run that failed midway and was + // retried to success would otherwise return green having never tested. + if run.is_ok() + && matches!(descriptor.test_behavior, DbtTestBehavior::AfterAll) + && matches!(command.as_str(), "build" | "retry") + && !retry_was_the_test_phase + { + run = run_dbt( + &prepared, + "test", + &descriptor, + &inv, + job, + conn, + &mut ctx, + // The tests must be scoped exactly like the models were: testing + // the whole project would assert against models this script never + // builds, the same failure the ingest-side scoping fixes. + true, + ) + .await; + // Merged, not appended: the model phase and the test phase can name the + // same node, and a duplicate would double its totals and collide as a key + // in the result table. + merge_results(&mut results, read_run_results(&prepared.project_dir).await); + // The same policy applies to a failing test. `dbt retry` redoes test + // nodes, and the descriptor promises to retry the ones that failed — + // retrying the model phase alone would exempt the failure mode + // `test_behavior: after_all` exists to produce. + if let Some(policy) = node_retry.filter(|_| run.is_err()) { + retry_failed_nodes( + policy, + &prepared, + &descriptor, + &inv, + job, + conn, + &mut ctx, + &mut run, + &mut results, + &mut retries_left, + ) + .await; + } + } + + // Best-effort: losing the state costs a retry, not the run that just + // finished. Logged rather than dropped — without it the only symptom is + // `dbt retry` reporting nothing to resume, which reads as a bug in retry. + if let Err(e) = save_run_state( + &prepared, + &job.workspace_id, + &job.permissioned_as, + job.visible_to_owner, + &job.id, + &inv, + restored_results_digest.as_deref(), + conn, + ) + .await + { + tracing::warn!("dbt: could not save retry state for job {}: {e:#}", job.id); + } + let reconciled = reconcile_materializations(&prepared, &results, job, conn, client).await; + terminalize_running_relations(job, &reconciled, conn).await; + + let result = build_result(&prepared, &command, results, &inv); + match run { + Ok(()) => Ok(to_raw_value(&result)), + Err(e) => { + // dbt's exit code already honors each test's own `severity`: a + // failing `warn` test leaves the run successful. Overriding that + // would make the same project behave differently on Windmill than + // it does locally, which is the promise this feature is built on. + append_logs( + &job.id, + &job.workspace_id, + format!("\n{}\n", render_failures(&result)), + conn, + ) + .await; + Err(Error::ExecutionErr(format!( + "{e}\n\n{}", + serde_json::to_string_pretty(&result).unwrap_or_default() + ))) + } + } +} + +/// Deploy-time lock: materialise the script's project and parse it, so its +/// models land in the asset graph before it has ever run. +/// +/// This is the one place where dbt does not fit the shape every other language +/// uses. `parse_assets_for_lang` is a pure function of the script content, and +/// dbt's assets are not derivable from the descriptor — they need the project +/// on disk and a dbt invocation. So the dependency job, which already runs on a +/// worker with the engine available, does the parse and writes the `asset` rows +/// itself; `parse_assets_for_lang` returns `None` for dbt and leaves them +/// alone. That also makes redeploy the graph-refresh mechanism, with no +/// separate concept (docs/dbt-runtime.md, decision 12). +#[allow(clippy::too_many_arguments)] +pub(crate) async fn dbt_dep( + content: &str, + // The project this dependency job is deploying: a dependency job has no + // generic module-writing step, so the executor materialises them. + modules: Option<&HashMap>, + job_id: &Uuid, + mem_peak: &mut i32, + canceled_by: &mut Option, + job_dir: &str, + db: &sqlx::Pool, + worker_name: &str, + w_id: &str, + script_path: &str, + occupancy_metrics: &mut OccupancyMetrics, + token: &str, + base_internal_url: &str, +) -> error::Result { + let descriptor = parse_dbt_descriptor(content)?; + // A DEPLOY writes a whole node set of its own, `raw_code` included, so it + // has to reclaim as well: hung off runs alone, a project redeployed on every + // push by CI and run nightly kept one full graph per push until the next + // run, and one deployed but never run kept them for good. + { + let (pool, prune_w_id) = (db.clone(), w_id.to_string()); + let prune_path = script_path.to_string(); + tokio::spawn(async move { + if let Err(e) = + windmill_common::dbt_manifest::prune_dbt_run_graphs(&pool, &prune_path, &prune_w_id) + .await + { + tracing::warn!("pruning dbt graphs at deploy: {e:#}"); + } + }); + } + // The script's own `envs`, exactly as a run gets them. A project can drive a + // model's schema, alias or `enabled` from `env_var()`, so parsing with an + // empty environment would record one relation at deploy and build another + // at run time — with no per-run refresh to correct it. + let envs = script_envs(db, job_id, w_id).await; + let conn = Connection::Sql(db.clone()); + let client = AuthedClient::new( + base_internal_url.to_string(), + w_id.to_string(), + token.to_string(), + None, + ); + // A dependency job carries no per-job timeout of its own, but its phases + // still share one wall clock rather than each getting the instance-wide + // one. + let deadline = JobDeadline::start(&conn, w_id, *job_id, None).await; + let mut ctx = JobCtx { mem_peak, canceled_by, occupancy_metrics, worker_name, deadline }; + let prepared = prepare_project( + &descriptor, + content, + None, + job_dir, + job_id, + w_id, + script_path, + deploying_script_hash(db, job_id).await, + &conn, + &client, + &mut ctx, + &envs, + modules, + ) + .await?; + + // A deploy has no job arguments, so it tolerates the `{{ }}` placeholders + // only a run can fill (see `Invocation::strict`). Its environment is the + // script's, matching what the run will parse with. + let inv = Invocation { envs: envs.clone(), strict: false, ..Default::default() }; + run_dbt_parse(&prepared, &descriptor, &inv, &mut ctx, job_id, w_id, &conn).await?; + + let selected = + resolve_selection(&prepared, &descriptor, &inv, &mut ctx, job_id, w_id, &conn).await?; + let manifest = read_manifest(&prepared.project_dir).await?; + let manifest_digest = digest( + &tokio::fs::read_to_string( + prepared + .project_dir + .join(ARTIFACTS_DIR) + .join("manifest.json"), + ) + .await + .unwrap_or_default(), + ); + + // Two deploys of one path can run concurrently — nothing serializes + // dependency jobs. The GRAPH is keyed by version so both may write theirs; + // the path-keyed asset usages are claimed by the newest. + let publisher = match deploying_script_hash(db, job_id).await { + Some(hash) => GraphPublisher::Version(hash), + None => GraphPublisher::Unversioned, + }; + let superseded = if let Some(warehouse) = prepared.warehouse.as_deref() { + let ingested = windmill_common::dbt_manifest::ingest_manifest( + &manifest, + warehouse, + prepared.default_database.as_deref(), + selected.as_ref(), + ); + let published = persist_ingest( + db, + w_id, + script_path, + &ingested, + &prepared.relation_root(), + publisher, + None, + true, + ) + .await?; + if published { + append_logs( + job_id, + w_id, + format!( + "\nIngested {} dbt nodes and {} edges into the asset graph\n", + ingested.nodes.len(), + ingested.edges.len() + ), + &conn, + ) + .await; + } + !published + } else { + // No warehouse identity, so nothing can be ingested — but this version's + // rows must still go, or a descriptor moved to its own profiles.yml keeps + // claiming relations it no longer describes. + let mut tx = db.begin().await?; + // Clearing is a publication too: an older job that no longer describes the + // script must not wipe a newer deploy's graph. + let published = claim_graph_publication(&mut tx, w_id, script_path, publisher).await?; + if published { + // This VERSION's rows, never the path's: what is given up is the + // path-keyed usage cleared below, while an older version's graph is + // what its own finished runs still render. + if let GraphPublisher::Version(hash) = publisher { + windmill_common::dbt_manifest::clear_dbt_manifest_version( + &mut tx, + w_id, + script_path, + hash, + ) + .await?; + } + windmill_common::assets::replace_static_asset_usage(&mut tx, w_id, script_path, &[]) + .await?; + tx.commit().await?; + append_logs( + job_id, + w_id, + "\nNo asset-graph ingest: the descriptor names no `profile.warehouse` beside \ + its own `profile.profiles_yml`, so there is no warehouse identity to key \ + `dbt://` assets on. Any previously ingested nodes for this script have been \ + cleared.\n" + .to_string(), + &conn, + ) + .await; + } + !published + }; + if superseded { + append_logs( + job_id, + w_id, + "\nA newer version of this script was deployed while this job ran, so the asset \ + graph was left describing that one.\n" + .to_string(), + &conn, + ) + .await; + } + + serde_json::to_string_pretty(&DbtDependencyLocks { + manifest_digest, + profile_relation_root: Some(prepared.relation_root()), + engine: prepared.engine.engine.as_str().to_string(), + engine_version: prepared.engine.version.clone(), + adapter_version: prepared.engine.adapter_version.clone(), + package_lock_digest: prepared.package_lock_digest.clone(), + }) + .map_err(|e| Error::internal_err(format!("serializing the dbt lockfile: {e}"))) +} + +/// The `envs` of the script this dependency job is deploying, in the same shape +/// a run receives them. Empty when the version cannot be resolved — a raw +/// dependency job has no script row. +async fn script_envs( + db: &sqlx::Pool, + job_id: &Uuid, + w_id: &str, +) -> HashMap { + let Some(hash) = deploying_script_hash(db, job_id).await else { + return HashMap::new(); + }; + let envs = sqlx::query_scalar!( + "SELECT envs FROM script WHERE workspace_id = $1 AND hash = $2", + w_id, + hash + ) + .fetch_optional(db) + .await + .ok() + .flatten() + .flatten(); + crate::worker::build_envs(envs.as_ref()).unwrap_or_default() +} + +/// The script version this dependency job is deploying. `None` for a raw +/// dependency job (the CLI's lock generation), which has no script row. +async fn deploying_script_hash(db: &sqlx::Pool, job_id: &Uuid) -> Option { + sqlx::query_scalar!("SELECT runnable_id FROM v2_job WHERE id = $1", job_id) + .fetch_optional(db) + .await + .ok() + .flatten() + .flatten() +} + +/// Why a run re-ingests its graph instead of trusting the deployed version's — +/// and, since the reasons differ in WHOSE graph the result is, what becomes of +/// it. +/// +/// A model set this RUN decides — a caller's override, or a descriptor dynamic by +/// construction — is stored under the job id so the run page shows what it built, +/// leaving the script's ownership alone: those schemas and aliases would otherwise +/// stand as the script's until the next deploy. +/// +/// A moved PROFILE is the one that republishes, because every later run of this +/// version resolves there too. Publishing is also what ENDS a drift: the check +/// reads back the published root, so a run that saw a move and did not republish +/// leaves the next one seeing the same move. +#[derive(Clone, Copy, Default)] +pub struct GraphRefresh { + /// This run's models are not the deployed descriptor's: a `{{ }}` + /// placeholder in `vars` or a `$var:` in `env` (re-resolved every run), or + /// an invocation that overrode `vars`. Vars steer `enabled`, alias, schema, + /// database and materialization, so the deployed graph names another run's + /// relations. + per_run_models: bool, + /// The profile resolves somewhere other than where the published usages + /// point. The relations moved for the VERSION, not for one invocation. + profile_drift: bool, +} + +impl GraphRefresh { + /// Whether this run parses and ingests a graph of its own at all. + fn needed(&self) -> bool { + self.per_run_models || self.profile_drift + } + + /// The job to key this graph under, or `None` to write the version's own. + /// + /// Only a DRIFT alone writes the version's: the move is permanent, and + /// storing it per run would leave every later run — which no longer detects + /// a drift, because this one published the new root — reading the pre-move + /// rows. A run whose models are its own goes under the job id, in both + /// directions: written as the version's, a narrowing selection would drop + /// every model this invocation left out, and a widening one would add models + /// that version never had. + fn snapshot_job(&self, job_id: uuid::Uuid) -> Option { + self.per_run_models.then_some(job_id) + } + + /// Whether this ingest also becomes what the script owns. + /// + /// Exactly when it wrote the VERSION's graph and the caller scoped nothing. + /// The workspace graph takes an asset's relations from the `asset` rows and + /// its models, SQL, tests and `ref()` lineage from that version's + /// `dbt_node`/`dbt_edge`, so publishing relations the version's graph does + /// not name leaves those assets with no model behind them — a placeholder + /// that moves an alias would empty the current graph of everything dbt + /// contributes to it. + /// + /// So a run that stored a snapshot of its own publishes nothing, which + /// leaves two cases settled elsewhere and deliberately: an override's + /// relations are a one-off and are meant not to stand as the script's, and a + /// dynamic descriptor at a moved profile keeps its ownership at the deploy's + /// relations until a redeploy — every run of it still shows its own models, + /// and it re-parses regardless, so the undetected-forever drift costs it + /// nothing it was not already paying. + /// + /// The exact complement of `snapshot_job`, which is what lets the agent + /// worker's payload carry one `per_run` bit and no second flag: decouple the + /// two and that wire format stops describing this decision. + fn publishes_ownership(&self) -> bool { + !self.per_run_models + } + + /// Fold in what this invocation's own arguments say about its model set. + fn add_caller_args( + &mut self, + descriptor: &DbtDescriptor, + args: &HashMap>, + ) -> error::Result<()> { + if has_vars_override(args) { + self.per_run_models = true; + } + // A caller's selection is not necessarily a SUBSET of the deployed one: + // deployed `select: ["tag:nightly"]`, overridden with `["*"]`, builds models + // the deployed graph never had — and those are the ones whose progress, SQL + // and lineage the run page would otherwise have nothing to draw. + if selection_is_overridden(descriptor, args)? { + self.per_run_models = true; + } + Ok(()) + } +} + +pub struct PreparedProject { + pub project_dir: PathBuf, + pub profiles_dir: PathBuf, + pub engine: ProvisionedEngine, + /// Why this run's graph is its own rather than the deployed version's, if + /// it is. + pub graph_refresh: GraphRefresh, + /// Digest of the project's own files: the identity of the code that runs. + /// It keys the package cache (a `local:` dependency's content appears in no + /// manifest) and gates retry state, so a project edited between attempts + /// cannot resume the old one. + pub project_digest: String, + /// Resolution produced by `dbt deps` at deploy. The package cache is local + /// to one worker; this makes a cache miss on another worker fail closed if + /// an unlocked range or mutable Git revision has moved meanwhile. + pub package_lock_digest: Option, + /// The workspace warehouse's NAME, the `` component of every + /// `dbt://` asset this project produces. `None` when the project brings its + /// own `profiles.yml` and names no warehouse, in which case there is no + /// stable warehouse identity to key assets on. + pub warehouse: Option, + /// The descriptor's `profile.target`, passed as `--target` so it applies to + /// a project-owned `profiles.yml` as well as a rendered one. + pub target: Option, + /// The profile target's database. Nodes that override it qualify their + /// `dbt://` schema segment so two databases cannot collapse onto one node. + pub default_database: Option, + /// The profile target's schema, for the drift check against the lockfile. + pub default_schema: Option, + pub script_path: String, + pub env: Vec<(String, String)>, + /// The descriptor body, kept so an ingest can re-read its `# on` / `# mute` + /// annotations without threading the content through every caller. + pub descriptor_content: String, + /// The descriptor's `env`, resolved, in a stable order. Feeds run identity; + /// `env` itself is not usable there because it carries per-job values. + pub descriptor_env: std::collections::BTreeMap, + /// The invocation's own environment (the script's `envs`), in a stable + /// order. Every phase gets it, `dbt deps` included: `packages.yml` can + /// resolve a private package URL through `env_var()`, and a phase that saw + /// a different environment from the one the cache key was built on would + /// populate that key with the wrong tree. + pub invocation_env: Vec<(String, String)>, + /// Digest of the above. Keys the package cache alongside the descriptor's + /// environment; digested because the values are resolved secrets. + pub invocation_env_digest: u64, + /// Written nsjail profile for this job, when the worker sandboxes jobs. + /// `None` means the phases run unsandboxed, exactly as before. + pub sandbox_config: Option, + /// One-way digest of the rendered profile — the resolved connection, not + /// just the names it exposes. A resource repointed from one warehouse to + /// another that happens to use the same database and schema names is + /// invisible to `relation_root`, and a retry would then execute the saved + /// failures against a warehouse where the successful nodes do not exist. + pub profile_digest: String, +} + +impl PreparedProject { + /// Where this run's relations live: the resolved schema and database. Drift + /// here since the deploy means the stored graph names relations that no + /// longer exist. + fn relation_root(&self) -> String { + format!( + "{}|{}", + self.default_schema.as_deref().unwrap_or(""), + self.default_database.as_deref().unwrap_or(""), + ) + } + + /// The environment `dbt_project.yml`'s own `env_var()` calls render + /// against: the two the run gives dbt, in the order the child receives + /// them. `HOME` is left out on purpose — it is Windmill's, not the + /// project's, and it differs on every attempt. + fn template_env(&self) -> HashMap { + self.descriptor_env + .iter() + .map(|(k, v)| (k.clone(), v.clone())) + .chain(self.invocation_env.iter().cloned()) + .collect() + } + + /// A digest of the DESCRIPTOR's resolved environment, for `run_identity`. + /// Digested rather than listed: the values are resolved secrets. + /// + /// Only the descriptor's own entries. `env` additionally carries `HOME`, + /// set to this job's directory, which differs on every attempt — hashing it + /// would make a retry reject its own predecessor every time. + fn env_digest(&self) -> String { + stable_digest( + self.descriptor_env + .iter() + .flat_map(|(k, v)| [k.as_str(), v.as_str()]), + ) + } + + /// Everything that decides which relations a run produces, which is what a + /// retry has to match before it may resume a saved `run_results.json`: same + /// project files, same warehouse and target, same engine. Identity only, + /// never credentials — the profile is digested, and the digest is one-way. + /// + /// Anything omitted here is something a redeploy could change while a stale + /// `run_results.json` stays eligible, so `dbt retry` would resume one + /// project's failures inside another. The descriptor's resolved environment + /// is in it because `env_var()` can drive a model's schema, database, alias + /// or `enabled`. + fn run_identity(&self) -> String { + // The descriptor whole, not field by field, or the next field added to it + // is silently left out. And the RESOLVED engine and adapter versions: an + // unchanged project redeployed after a release resolves a newer dbt, whose + // retry would otherwise feed one version's `run_results.json` to another. + format!( + "{}|{}|{}|{}|{}|{}|{}|{}|{}", + self.project_digest, + self.package_lock_digest.as_deref().unwrap_or(""), + self.engine.engine.as_str(), + self.engine.version, + self.engine.adapter_version.as_deref().unwrap_or(""), + digest(&self.descriptor_content), + self.env_digest(), + self.relation_root(), + self.profile_digest, + ) + } +} + +#[allow(clippy::too_many_arguments)] +pub(crate) async fn prepare_project( + descriptor: &DbtDescriptor, + descriptor_content: &str, + locks: Option<&DbtDependencyLocks>, + job_dir: &str, + job_id: &Uuid, + w_id: &str, + // Keys the per-script retry-state cache. Passed in rather than patched onto + // the result afterwards: an empty value silently shares one state directory + // across every dbt script in the workspace, so a retry resumes another + // project's run_results.json. + script_path: &str, + // The version this job runs, which is the one whose stored graph the drift + // check below must read. `None` for a preview, which has no stored graph. + script_hash: Option, + conn: &Connection, + client: &AuthedClient, + ctx: &mut JobCtx<'_>, + // The invocation's own environment (script-level `envs`). Needed here + // because under a sandbox it must travel in the jail profile rather than + // on the process that execs nsjail. + invocation_env: &HashMap, + // The script's files: the dbt project itself. + modules: Option<&HashMap>, +) -> error::Result { + // The project IS this script's files. Nothing is fetched: the bundle is the + // project. A dependency job has no generic module-writing step, so it does + // the writing here; a run rewrites the same bytes, which costs nothing next + // to the dbt invocations that follow. + if let Some(modules) = modules { + write_module_files(job_dir, modules, None).await?; + } + let project_dir = PathBuf::from(job_dir); + if !project_dir.join("dbt_project.yml").exists() { + return Err(Error::BadRequest( + "this dbt script carries no project: `dbt_project.yml` was not found. Copy a dbt \ + project into its `` in + // any value (the app path comes from raw_app.yaml) would end the block. + const config = JSON.stringify({ + appPath: opts.appPath, + workspace: opts.workspace, + playerBaseUrl: opts.playerBaseUrl ?? null, + savePath: RECORDER_SAVE_PATH, + }).replace(/ + + + + + Windmill App Dev Recording + + + +
+ + Not recording + + + Passwords are masked. Mark sensitive elements with data-wm-no-record +
+ + + + + +`; +} diff --git a/cli/src/commands/app/devRecorderBundle.gen.ts b/cli/src/commands/app/devRecorderBundle.gen.ts new file mode 100644 index 0000000000..0e7f004b4b --- /dev/null +++ b/cli/src/commands/app/devRecorderBundle.gen.ts @@ -0,0 +1,15 @@ +// Generated by cli/generate-dev-recorder.ts. Do not edit. +// Run `bun run gen:dev-recorder` from cli/ to rebuild it from +// frontend/src/lib/components/recording/. + +/** Repo-relative sources bundled below. */ +export const DEV_RECORDER_SOURCES = [ + "frontend/src/lib/components/recording/rawAppRecording.svelte.ts", + "frontend/src/lib/components/recording/rawAppSnapshot.ts" +]; + +/** SHA-256 of those sources, as of this build. */ +export const DEV_RECORDER_SOURCE_HASH = "c93b8b23455528be0a03da303fb573536bd3f0b8386e6f5f1078a6b988f59082"; + +/** IIFE exposing `createRawAppRecording` on `window.__wmillRecorder`. */ +export const DEV_RECORDER_BUNDLE = "var __wmillRecorder=(()=>{var ne=Object.defineProperty;var Fe=Object.getOwnPropertyDescriptor;var Xe=Object.getOwnPropertyNames;var Be=Object.prototype.hasOwnProperty;var je=(t,o)=>{for(var n in o)ne(t,n,{get:o[n],enumerable:!0})},Ke=(t,o,n,s)=>{if(o&&typeof o==\"object\"||typeof o==\"function\")for(let l of Xe(o))!Be.call(t,l)&&l!==n&&ne(t,l,{get:()=>o[l],enumerable:!(s=Fe(o,l))||s.enumerable});return t};var We=t=>Ke(ne({},\"__esModule\",{value:!0}),t);var mt={};je(mt,{createRawAppRecording:()=>Ie});var G=\"data-wm-rec-target\",D=\"data-wm-no-record\";function Ye(t,o){let n=(u,c,f)=>{let d=f.trim();if(!d||/^(data:|blob:|about:|https?:|\\/\\/|#)/i.test(d))return u;try{return`url(${c}${new URL(d,o).href}${c})`}catch{return u}},s=\"\",l=0;for(;ln?String.fromCodePoint(parseInt(n,16)):s)}function Qe(t){let o=new Set,n=new Set;for(let s of Array.from(t.querySelectorAll(\"style\"))){if(_(s))continue;let l=s.textContent??\"\";for(let u of l.matchAll(Ge))o.add(Ee(u[1]));for(let u of l.matchAll(Je))n.add(Ee(u[1]))}return{classes:o,ids:n}}function Ze(t,o){let n=Qe(o),s=[...o.hasAttribute(D)?[o]:[],...Array.from(o.querySelectorAll(`[${D}]`))];for(let l of s){l.replaceChildren(t.createTextNode(\"\\u2022\\u2022\\u2022\")),l.setAttribute(D,\"\");for(let u of Array.from(l.attributes)){if(u.name===D)continue;let c=u.localName.toLowerCase();if(!ze.has(c))l.removeAttributeNode(u);else if(c===\"class\"){let f=u.value.split(/\\s+/).filter(d=>d&&n.classes.has(d));f.length?l.setAttribute(\"class\",f.join(\" \")):l.removeAttributeNode(u)}else c===\"id\"&&!n.ids.has(u.value)&&l.removeAttributeNode(u)}}}var et=4e6,tt=8e6;function nt(t,o){let n=t.querySelectorAll(\"canvas\"),s=o.querySelectorAll(\"canvas\");if(n.length!==s.length)return;let l=tt;for(let u=0;uet||f>l)continue;l-=f;let d;try{d=c.toDataURL(\"image/webp\",.85)}catch{continue}if(!d.startsWith(\"data:image/\"))continue;let m=c.getBoundingClientRect();if(!m.width||!m.height)continue;let y=t.defaultView?.getComputedStyle(c).display,v=!y||y===\"inline\"?\"inline-block\":y,F=s[u],X=F.getAttribute(\"style\");F.setAttribute(\"style\",`${X?X+\";\":\"\"}display:${v};box-sizing:border-box;width:${m.width}px;height:${m.height}px;background-image:url(\"${d}\");background-size:100% 100%;background-repeat:no-repeat`)}}function rt(t,o){let n=t.querySelectorAll(\"select\"),s=o.querySelectorAll(\"select\");if(n.length===s.length)for(let l=0;l_(f)))continue;let c=t.createElement(\"option\");c.setAttribute(\"selected\",\"\"),c.textContent=\"\\u2022\\u2022\\u2022\",s[l].replaceChildren(c)}}function it(t,o){let n=\"input, textarea, select\",s=t.querySelectorAll(n),l=o.querySelectorAll(n);if(s.length===l.length)for(let u=0;u{let n=o.styleSheet;if(!n)return o.cssText;try{let s=be(n.cssRules),l=n.media?.mediaText;return l?`@media ${l} {\n${s}\n}`:s}catch{return o.cssText}}).join(`\n`)}function ot(t,o){let n=be(o);t.href&&(n=Ye(n,t.href));let s=t.media?.mediaText;return s&&(n=`@media ${s} {\n${n}\n}`),n}function st(t,o,n){for(let s of Array.from(t.styleSheets)){let l=s.ownerNode;if(!C(l))continue;if(s.disabled){let m=re(o,l),y=m?ie(n,m):void 0;y&&(y.setAttribute(\"media\",\"not all\"),y.tagName===\"STYLE\"&&(y.textContent=\"\"));continue}if(_(l))continue;let u;try{let m=s.cssRules;if(!m)continue;u=m}catch{continue}let c=re(o,l);if(!c)continue;let f=ie(n,c);if(!f)continue;let d=ot(s,u);if(l.tagName===\"LINK\"){let m=t.createElement(\"style\");m.textContent=d,f.replaceWith(m)}else l.tagName===\"STYLE\"&&(f.textContent=d)}}function Se(t,o={}){let n=t.documentElement,s=n.cloneNode(!0);if(it(t,s),st(t,n,s),nt(t,s),rt(t,s),Ze(t,s),o.target){let d=re(n,o.target);(d?ie(s,d):void 0)?.setAttribute(G,\"\")}s.querySelectorAll(\"template, noscript\").forEach(d=>d.remove()),s.querySelectorAll(\"script\").forEach(d=>d.remove()),s.querySelectorAll('meta[http-equiv=\"refresh\" i]').forEach(d=>d.remove()),s.querySelectorAll(\"*\").forEach(d=>{for(let m of Array.from(d.attributes))m.name.toLowerCase().startsWith(\"on\")&&d.removeAttribute(m.name)});let l=t.defaultView,u=Math.round(l?.scrollY??t.documentElement.scrollTop??0),c=Math.round(l?.scrollX??t.documentElement.scrollLeft??0);if(u>0||c>0){let d=t.createElement(\"style\");d.textContent=`html { margin-top: -${u}px !important; margin-left: -${c}px !important; }`,s.querySelector(\"head\")?.appendChild(d)}let f=s.querySelector(\"head\");if(o.baseHref&&f&&!f.querySelector(\"base\")){let d=t.createElement(\"base\");d.setAttribute(\"href\",o.baseHref),f.prepend(d)}return`${s.outerHTML}`}var gt=`[${G}] {\n\toutline: 3px solid #ef4444 !important;\n\toutline-offset: 2px !important;\n\tbox-shadow: 0 0 0 6px rgba(239, 68, 68, 0.25) !important;\n}`;function oe(t){if(!t||_(t))return\"\";let o=t;return t.querySelector(`[${D}]`)&&(o=t.cloneNode(!0),o.querySelectorAll(`[${D}]`).forEach(n=>n.remove())),(o.textContent??\"\").replace(/\\s+/g,\" \").trim()}function ye(t,o=40){let n=oe(t);return n.length>o?`${n.slice(0,o)}\\u2026`:n}function we(t){let o=t.tagName.toLowerCase(),n=(t.getAttribute(\"type\")??\"text\").toLowerCase(),s=o===\"input\"?`input[${n}]`:o,l=t.labels?.[0],u=t.getAttribute(\"aria-label\")||(l&&!_(l)?ye(l):\"\")||(o===\"input\"&&[\"button\",\"submit\",\"reset\"].includes(n)?t.getAttribute(\"value\"):\"\")||t.getAttribute(\"placeholder\")||t.getAttribute(\"title\")||ye(t)||t.getAttribute(\"name\")||t.getAttribute(\"id\")||\"\";return u?`${s} \"${u}\"`:s}function Re(t){let o=[],n=t,s=0;for(;n&&s<5;){let l=n.tagName.toLowerCase();if(n.id){o.unshift(`#${n.id}`);break}let u=typeof n.className==\"string\"?n.className.trim().split(/\\s+/).filter(Boolean)[0]:void 0,c=n.parentElement,f=u?`${l}.${u}`:l;if(c){let d=Array.from(c.children).filter(m=>m.tagName===n.tagName);d.length>1&&(f+=`:nth-of-type(${d.indexOf(n)+1})`)}o.unshift(f),n=c,s++}return o.join(\" > \")}function Le(t,o,n){switch(t){case\"click\":return`Clicked ${o}`;case\"fill\":return`Filled ${o} with \"${n??\"\"}\"`;case\"select\":return`Selected \"${n??\"\"}\" in ${o}`;case\"toggle\":return n?`${n===\"checked\"?\"Checked\":\"Unchecked\"} ${o}`:`Toggled ${o}`;case\"submit\":return`Submitted ${o}`;case\"key\":return`Pressed ${n??\"key\"} in ${o}`;case\"navigate\":return n?`Navigated to ${n}`:\"Reloaded the app\"}}var V=400,ct=3e3,ve=6e4,Ce=800,_e=new Set([\"button\",\"submit\",\"reset\",\"image\"]),ut=new Set([\"range\",\"color\",\"date\",\"time\",\"datetime-local\",\"month\",\"week\"]),dt=new Set([\"\",\"text\",\"search\",\"url\",\"tel\",\"email\",\"password\",\"number\"]),J=200,ft=250,xe=500;function Ie(){let t=!1,o=0,n=0,s=\"\",l,u,c=[],f=[],d=new Map,m=0,y=!1,v=!1,F={width:0,height:0},X=\"\",z=[],A,M,O=0,L,b,h,q=new Set,Q,N=0,B,Z=!1;function ke(e){return new Promise(a=>{let i=()=>{if(N===0||Date.now()-e>=ve||!P()){a();return}setTimeout(i,V)};i()})}let Me=e=>new Promise(a=>setTimeout(a,e));function P(){try{return u?.contentDocument??void 0}catch{return}}function $(e){if(e===void 0)return;let a=d.get(e);if(a!==void 0)return a;if(m+e.length>41943040){y=!0,v=!0;return}let i=f.length;return f.push(e),d.set(e,i),m+=e.length,i}function S(e){if(v)return;let a=P();if(a)try{return Se(a,{target:e,baseHref:X})}catch(i){console.warn(\"raw app recorder: snapshot failed\",i);return}}function Ne(e){return e.replace(` ${G}=\"\"`,\"\")}function j(){h&&(h.observer.disconnect(),clearTimeout(h.timer),clearTimeout(h.cap),h=void 0)}function se(e){j();let a=P();if(!a)return;let i=()=>{if(N>0&&h&&Date.now()-h.startedAt{h&&(clearTimeout(h.timer),h.timer=setTimeout(i,V))});r.observe(a,{subtree:!0,childList:!0,attributes:!0,characterData:!0}),h={step:e,observer:r,startedAt:Date.now(),timer:setTimeout(i,V),cap:setTimeout(i,ct)}}function ae(e){if(!h)return;let a=h.step;j();let i=e!==void 0&&(L?.html===e||b?.html===e);a.after=$(i?Ne(e):S())}function x(e,a,i,r,w=!1){if(!t)return;let E=Date.now()-n,p=c[c.length-1],R=!!p&&!!a&&ue(M,a)&&p.kind===e&&(w||Ue(a)&&E-O=500&&!R){y=!0,v=!0;return}let T=!!a&&_(a),H=le(a?T?Ae(a):we(a):\"the app\")??\"the app\",W=r&&r.length>J?`${r.slice(0,J)}\\u2026`:r,k=!T||!W?W:e===\"toggle\"?void 0:Y(W),ge=Le(e,H,k);if(R&&p){p.value=k,p.label=ge,O=E,se(p);return}let he={t:E,kind:e,label:ge,target:H,selector:a&&!T?le(Re(a)):void 0,value:k,before:$(i??(e===\"key\"?S(a):void 0))};c.push(he),i!==void 0&&L?.html===i&&(L=void 0),i!==void 0&&b?.html===i&&(b=void 0),M=a,O=E,o=c.length,se(he)}function Pe(e){let i=e.closest(\"label\")?.control;return!i||e===i||i.contains(e)?!1:!e.closest(\"a, button, input, select, textarea\")}function $e(e){return e.ctrlKey||e.metaKey||e.altKey?!1:e.key.length===1||[\" \",\"Enter\",\"ArrowUp\",\"ArrowDown\",\"ArrowLeft\",\"ArrowRight\",\"Home\",\"End\"].includes(e.key)}function ee(e){if(g(e,\"SELECT\"))return!0;if(!g(e,\"INPUT\"))return!1;let a=e.type;return!K(e)&&!_e.has(a)}function He(e){return g(e,\"BUTTON\")?(e.type||\"submit\")===\"submit\":g(e,\"INPUT\")&&[\"submit\",\"image\"].includes(e.type)}function De(e){let a=c[c.length-1];return a?.kind===\"key\"&&a.value===\"Enter\"&&!!e&&!!M&&e.contains(M)&&Date.now()-n-OJ?`${e.slice(0,J)}\\u2026`:e}function K(e){return g(e,\"TEXTAREA\")||e.isContentEditable?!0:g(e,\"INPUT\")&&dt.has(e.type)}function ce(e){let a=g(e,\"INPUT\")||g(e,\"TEXTAREA\")?e.value:oe(e);return g(e,\"INPUT\")&&e.type===\"password\"||_(e)?Y(a):a}function U(e){let a=L?.el;if(!a)return;if(a===e||a.contains(e)||e.contains(a))return L?.html;let i=e.labels;if(i&&Array.from(i).some(r=>r===a||r.contains(a)))return L?.html}function ue(e,a){if(!e||!a)return!1;if(e===a)return!0;let i=e,r=a;return i.type===\"radio\"&&r.type===\"radio\"&&!!i.name&&i.name===r.name&&i.form===r.form}function te(e){if(b)return ue(b.el,e)?b.html:void 0}function I(){if(!A)return;let{el:e,before:a}=A;clearTimeout(A.timer),A=void 0,U(e)!==void 0&&(L=void 0),b?.el===e&&(b=void 0),x(\"fill\",e,a,ce(e))}function de(e){let a=(i,r)=>{e.addEventListener(i,r,!0),z.push(()=>e.removeEventListener(i,r,!0))};a(\"pointerdown\",i=>{let r=C(i.target)?i.target:void 0;r&&(L={el:r,html:S(r)})}),a(\"click\",i=>{let r=C(i.target)?i.target:void 0;r&&(A&&A.el!==r&&I(),!(K(r)||ee(r)||g(r,\"OPTION\"))&&(i.detail===0&&He(r)&&De(r.closest(\"form\"))||Pe(r)||x(\"click\",r,U(r)??S(r))))}),a(\"focusin\",i=>{let r=C(i.target)?i.target:void 0;L&&(!r||U(r)===void 0)&&(L=void 0)}),a(\"beforeinput\",i=>{let r=C(i.target)?i.target:void 0;!r||!K(r)||A?.el===r||U(r)===void 0&&(b={el:r,html:S(r),repeat:!1})}),a(\"input\",i=>{let r=C(i.target)?i.target:void 0;if(!(!r||!K(r)))if(A&&A.el!==r&&I(),A)clearTimeout(A.timer),A.timer=setTimeout(I,Ce);else{let w=U(r),E=te(r),p=w??E??S(r);ae(p),A={el:r,before:p,timer:setTimeout(I,Ce)}}}),a(\"change\",i=>{let r=C(i.target)?i.target:void 0;if(!r)return;if(K(r)){I();return}A&&I();let w=U(r),E=te(r),p=w??E,R=w===void 0&&E!==void 0&&!!b?.repeat;if(g(r,\"SELECT\")){let T=Array.from(r.selectedOptions),H=T.map(k=>k.label||k.value).join(\", \"),W=T.some(k=>_(k));x(\"select\",r,p,W?Y(H):H,R)}else if(g(r,\"INPUT\")){let T=r;[\"checkbox\",\"radio\"].includes(T.type)?x(\"toggle\",r,p,T.checked?\"checked\":\"unchecked\",R):T.type===\"file\"?x(\"fill\",r,p,Array.from(T.files??[]).map(H=>H.name).join(\", \")):x(\"fill\",r,p,ce(r),R)}}),a(\"submit\",i=>{let r=C(i.target)?i.target:void 0;I();let w=c[c.length-1];(w?.kind===\"click\"||w?.kind===\"key\"&&w.value===\"Enter\")&&M&&r&&r.contains(M)&&Date.now()-n-O{let r=C(i.target)?i.target:void 0;r&&ee(r)&&$e(i)&&(i.repeat&&b&&te(r)!==void 0?b.repeat=!0:b={el:r,html:S(r),repeat:i.repeat}),!(i.key!==\"Enter\"&&i.key!==\"Escape\")&&(i.key===\"Enter\"&&r&&(ee(r)||qe(r)||Oe(r))||(I(),x(\"key\",r,i.repeat?void 0:S(r),i.key,i.repeat)))})}function fe(){z.forEach(e=>e()),z=[]}function me(e){let a=e.contentWindow,i=E=>{let p=E.data;if(!p||typeof p!=\"object\"||E.source!==window)return;let{type:R,reqId:T}=p;typeof R!=\"string\"||!R.endsWith(\"Res\")||(q.delete(T),N=q.size)},r=()=>{let E=P();!E||E===Q||(a?.addEventListener(\"message\",i),Q=E)},w=E=>{let p=E.data;if(!p||typeof p!=\"object\"||E.source!==a)return;let{type:R,reqId:T}=p;typeof R!=\"string\"||R.endsWith(\"Res\")||T===void 0||(r(),q.add(T),N=q.size)};return window.addEventListener(\"message\",w),r(),()=>{window.removeEventListener(\"message\",w),a?.removeEventListener(\"message\",i),Q=void 0}}function pe(){fe();let e=h?.step;j(),A&&clearTimeout(A.timer),A=void 0,L=void 0,b=void 0;let a=P();if(!a)return;de(a),u&&(B?.(),B=me(u));let i=S();if(e&&(e.after=$(i)),f.length===0){$(i);return}x(\"navigate\",void 0,i,a.location?.hash||void 0)}return{get active(){return t},get stepCount(){return o},get stopping(){return Z},start(e,a){u=e;let i=P();return i?.documentElement?(t=!0,n=Date.now(),s=a.appPath,l=a.workspace,c=[],M=void 0,O=0,o=0,f=[],d=new Map,m=0,y=!1,v=!1,X=typeof window<\"u\"?window.location.origin:\"\",F={width:e.clientWidth||i.documentElement.clientWidth,height:e.clientHeight||i.documentElement.clientHeight},i.readyState===\"complete\"&&i.location?.href!==\"about:blank\"&&$(S()),de(i),e.addEventListener(\"load\",pe),q.clear(),N=0,B=me(e),!0):(u=void 0,!1)},async stop(){if(I(),fe(),t=!1,h&&N>0){Z=!0;let a=h.startedAt;await ke(a),P()&&await Me(V),Z=!1}if(h){let a=h.step;j(),a.after=$(S())}B?.(),B=void 0,q.clear(),N=0,u?.removeEventListener(\"load\",pe),L=void 0,b=void 0,u=void 0;let e={version:1,type:\"app\",recorded_at:new Date().toISOString(),app_path:s,workspace:l,total_duration_ms:Date.now()-n,viewport:F,frames:f,steps:c,truncated:y||void 0};return c=[],f=[],d=new Map,m=0,e},download(e){let a=new Blob([JSON.stringify(e)],{type:\"application/json\"}),i=URL.createObjectURL(a),r=document.createElement(\"a\");r.href=i,r.download=`app-recording-${(e.app_path||\"untitled\").replace(/\\//g,\"-\")}-${Date.now()}.json`,r.click(),URL.revokeObjectURL(i)}}}return We(mt);})();\n"; diff --git a/cli/src/commands/app/raw_apps.ts b/cli/src/commands/app/raw_apps.ts index fcfc68915e..9f83d0322a 100644 --- a/cli/src/commands/app/raw_apps.ts +++ b/cli/src/commands/app/raw_apps.ts @@ -1,5 +1,6 @@ import { requireLogin } from "../../core/auth.ts"; import { resolveWorkspace, validatePath } from "../../core/context.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; import { colors } from "@cliffy/ansi/colors"; import * as log from "../../core/log.ts"; import { sep as SEP } from "node:path"; @@ -16,7 +17,7 @@ import { deepEqual, readTextFile } from "../../utils/utils.ts"; import { replaceInlineScripts, repopulateFields } from "./app.ts"; import { createBundle, detectFrameworks } from "./bundle.ts"; -import { APP_BACKEND_FOLDER } from "./app_metadata.ts"; +import { APP_BACKEND_FOLDER, RECORDINGS_FOLDER } from "./app_metadata.ts"; import { writeIfChanged } from "../../utils/utils.ts"; import { yamlOptions } from "../sync/sync.ts"; import { applyExtraPermsDiff } from "../../core/extra_perms.ts"; @@ -143,7 +144,8 @@ function getRunnableIdFromCodeFile(fileName: string): string | undefined { * Returns an empty object if the backend folder doesn't exist. * * @param backendPath - Path to the backend folder - * @param defaultTs - Default TypeScript runtime ("bun" or "deno") + * @param defaultTs - TypeScript runtime a bare `.ts` denotes. Must match what + * newRawAppPathAssigner used to write the file, or the round-trip relabels it. */ export async function loadRunnablesFromBackend( backendPath: string, @@ -317,6 +319,12 @@ async function collectAppFiles( ) { continue; } + // Session recordings, which the dev server only ever writes at the app + // root. Matched there alone, so an app of its own with a `recordings/` + // component folder still ships it. + if (basePath === "/" && entry.name === RECORDINGS_FOLDER) { + continue; + } await readDirRecursive(fullPath + SEP, relativePath + "/"); } else if (entry.isFile()) { // Skip generated/metadata files that shouldn't be part of the app @@ -344,6 +352,7 @@ export async function pushRawApp( remotePath: string, localPath: string, message?: string, + defaultTs: "bun" | "deno" = "bun", ): Promise { if (alreadySynced.includes(localPath)) { return; @@ -377,7 +386,10 @@ export async function pushRawApp( // Load runnables from separate YAML files in the backend folder // Falls back to reading from raw_app.yaml if no separate files exist (backward compat) const backendPath = path.join(localPath, APP_BACKEND_FOLDER); - const runnablesFromBackend = await loadRunnablesFromBackend(backendPath); + const runnablesFromBackend = await loadRunnablesFromBackend( + backendPath, + defaultTs, + ); let runnables: Record; if (Object.keys(runnablesFromBackend).length > 0) { @@ -539,7 +551,14 @@ async function pushRawAppCommand( } const workspace = await resolveWorkspace(opts); await requireLogin(opts); + const merged = await mergeConfigWithConfigFile(opts); - await pushRawApp(workspace.workspaceId, remotePath, filePath); + await pushRawApp( + workspace.workspaceId, + remotePath, + filePath, + undefined, + merged.defaultTs, + ); log.info(colors.bold.underline.green("Raw app pushed")); } diff --git a/cli/src/commands/app/wmillTsDev.ts b/cli/src/commands/app/wmillTsDev.ts index cd77652f53..7cb2ea328b 100644 --- a/cli/src/commands/app/wmillTsDev.ts +++ b/cli/src/commands/app/wmillTsDev.ts @@ -68,11 +68,45 @@ function initWebSocket() { initWebSocket() +/** A runnable call leaves this page over the WebSocket without touching the DOM, + * so the session recorder of \`wmill app dev --recording\` (which frames the app) + * has nothing else to tell it a step is still waiting on the backend. Announcing + * the request and its answer to the shell mirrors what the deployed runner posts + * across the same boundary. */ +const framed = typeof window !== 'undefined' && window.parent !== window + +function notifyRecorder(type: string, reqId: string) { + if (framed) window.parent.postMessage({ type, reqId }, window.location.origin) +} + +// A reload takes the previous context and its WebSocket with it, so whatever it +// had in flight can never answer. Announcing a fresh module is how the shell +// learns those calls are dead: a message posted from the unloading document +// would be dropped with the realm that sent it, and this runs before any app +// code can issue a call of its own. +if (framed) { + window.parent.postMessage({ type: 'wmillDevReady' }, window.location.origin) +} + +function tracked(type: string, reqId: string, resolve: (v: any) => void, reject: (e: any) => void) { + notifyRecorder(type, reqId) + let settled = false + const done = () => { + if (settled) return + settled = true + notifyRecorder(type + 'Res', reqId) + } + return { + resolve: (v: any) => { done(); resolve(v) }, + reject: (e: any) => { done(); reject(e) } + } +} + async function doRequest(type: string, o: object) { await wsReady return new Promise((resolve, reject) => { const reqId = Math.random().toString(36) - reqs[reqId] = { resolve, reject } + reqs[reqId] = tracked(type, reqId, resolve, reject) ws?.send(JSON.stringify({ ...o, type, reqId })) }) } @@ -119,7 +153,7 @@ export function streamJob( return new Promise(async (resolve, reject) => { await wsReady const reqId = Math.random().toString(36) - reqs[reqId] = { resolve, reject, onUpdate } + reqs[reqId] = { ...tracked('streamJob', reqId, resolve, reject), onUpdate } ws?.send(JSON.stringify({ jobId, type: 'streamJob', reqId })) }) } diff --git a/cli/src/commands/dev/dev.ts b/cli/src/commands/dev/dev.ts index f588f5841d..00a6f72f0e 100644 --- a/cli/src/commands/dev/dev.ts +++ b/cli/src/commands/dev/dev.ts @@ -20,7 +20,19 @@ import { SyncOptions, mergeConfigWithConfigFile, } from "../../core/conf.ts"; -import { exts, removeExtensionToPath } from "../script/script.ts"; +import { + exts, + hasScriptExt, + readModulesFromDisk, + removeExtensionToPath, +} from "../script/script.ts"; +import type { ScriptModule } from "../../../gen/types.gen.ts"; +import { + DBT_DESCRIPTOR_NAME, + DBT_MODULE_SUFFIX, + getScriptBasePathFromModulePath, + isDbtModulePath, +} from "../../utils/resource_folders.ts"; import { inferContentTypeFromFilePath } from "../../utils/script_common.ts"; import { OpenFlow } from "../../../gen/types.gen.ts"; import { FlowFile } from "../flow/flow.ts"; @@ -108,7 +120,7 @@ function findFlowFolderPrefix(cpath: string): string | undefined { return undefined; } -async function listWorkspacePaths(): Promise { +export async function listWorkspacePaths(): Promise { // Walk first, capturing each item's metadata file path. Then read summaries in // parallel — one tree pass plus N file reads is faster than a serialized walk. const items: (WmPathItem & { _metaPath?: string })[] = []; @@ -136,6 +148,19 @@ async function listWorkspacePaths(): Promise { items.push({ path: stripFolderSuffix(childRel, APP_SUFFIXES), kind: "raw_app" }); continue; } + // A dbt script IS the project directory: its descriptor is optional, so + // there may be no file here to recognize it by. Not descended into + // either — the project's own `.sql` models would otherwise each be + // listed as a script of their own. + if (entry.name.endsWith(DBT_MODULE_SUFFIX)) { + const base = childRel.slice(0, -DBT_MODULE_SUFFIX.length); + items.push({ + path: base, + kind: "script", + _metaPath: childAbs.slice(0, -DBT_MODULE_SUFFIX.length) + ".script.yaml", + }); + continue; + } await walk(childAbs, childRel); } else if (entry.isFile()) { const matchedExt = exts.find((ext) => entry.name.endsWith(ext)); @@ -282,12 +307,40 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { const flowMetadataFile = getMetadataFileName("flow", "yaml"); async function loadPaths(pathsToLoad: string[]) { - const paths = pathsToLoad.filter((p) => - exts.some( - (ext) => p.endsWith(ext) - || p.endsWith(".flow/" + flowMetadataFile) - || p.endsWith("__flow/" + flowMetadataFile) - ) + // A change ANYWHERE inside a dbt project is a change to that script: the + // bundle is the project, so the whole thing is re-read and rebroadcast. + // Treating the file as a script of its own would drop the edit — a bare + // `.sql` has no language to infer, and a `.yml` or `.csv` is filtered out + // below — leaving the browser previewing the snapshot taken at startup. + const rest: string[] = []; + const dbtProjects = new Set(); + for (const raw of pathsToLoad) { + const rel = (await realpath(raw).catch(() => raw)) + .replace(base + SEP, "") + .replaceAll("\\", "/"); + const wmPath = isDbtModulePath(rel) + ? getScriptBasePathFromModulePath(rel) + : undefined; + // Every project in the batch, and each only once: a save-all or a + // `git checkout` touches many files at once, and returning on the first + // would drop both the other projects and whatever else changed with them. + if (wmPath) dbtProjects.add(wmPath); + else rest.push(raw); + } + for (const wmPath of dbtProjects) { + const edit = await loadWmPath(wmPath); + if (edit) { + log.info("Updated " + wmPath + " (dbt project)"); + broadcastChanges(edit); + } + } + if (rest.length === 0) return; + pathsToLoad = rest; + const paths = pathsToLoad.filter( + (p) => + hasScriptExt(p) || + p.endsWith(".flow/" + flowMetadataFile) || + p.endsWith("__flow/" + flowMetadataFile) ); if (paths.length == 0) { return; @@ -380,6 +433,10 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { tag?: string; lock?: string; temp_script_refs?: Record; + /** The bundle the dev page forwards to a preview run. A dbt project cannot + * run without it: the worker looks for `dbt_project.yml` in the bundle and + * refuses the job when it is not there. */ + modules?: Record; }; type LastEditFlow = { @@ -441,8 +498,23 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { for (const ext of exts) { const filePath = wmPath + ext; try { - await access(filePath); - const content = await readTextFile(filePath); + // A dbt project's descriptor is optional, so what says "this is a dbt + // script" is the project beside it. Requiring the descriptor to exist + // would list such a project in the picker and then refuse to load it. + const isAbsentDbtDescriptor = + ext === "__dbt/" + DBT_DESCRIPTOR_NAME && + !(await access(filePath).then( + () => true, + () => false + )) && + (await access(wmPath + DBT_MODULE_SUFFIX + "/dbt_project.yml").then( + () => true, + () => false + )); + if (!isAbsentDbtDescriptor) { + await access(filePath); + } + const content = isAbsentDbtDescriptor ? "" : await readTextFile(filePath); const lang = inferContentTypeFromFilePath(filePath, opts.defaultTs); const typed = (await parseMetadataFile(removeExtensionToPath(filePath), undefined))?.payload; const edit: LastEditScript = { @@ -453,6 +525,18 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { tag: typed?.tag, lock: typed?.lock, temp_script_refs: tempScriptRefs, + // Read VERBATIM for dbt, the way push does: the project's `.sql` and + // `.yml` files are dbt's, and inferring a language for each would + // drop the ones that are not Windmill scripts. + modules: + lang === "dbt" + ? await readModulesFromDisk( + wmPath + DBT_MODULE_SUFFIX, + opts.defaultTs, + true, + true + ) + : undefined, }; currentLastEdit = edit; return edit; diff --git a/cli/src/commands/flow/flow.ts b/cli/src/commands/flow/flow.ts index 17a5e1febb..6ace1eaf48 100644 --- a/cli/src/commands/flow/flow.ts +++ b/cli/src/commands/flow/flow.ts @@ -13,7 +13,12 @@ import { mkdirSync, writeFileSync } from "node:fs"; import { buildFolderPath, getMetadataFileName, loadNonDottedPathsSetting } from "../../utils/resource_folders.ts"; import { requireLogin } from "../../core/auth.ts"; -import { resolveWorkspace, validatePath } from "../../core/context.ts"; +import { + assertRemotePath, + resolveWorkspace, + toSyncRootRelativePath, + validatePath, +} from "../../core/context.ts"; import { resolve, track_job, pollForJobResult } from "../script/script.ts"; import { defaultFlowDefinition } from "../../../bootstrap/flow_bootstrap.ts"; import { SyncOptions, mergeConfigWithConfigFile } from "../../core/conf.ts"; @@ -232,11 +237,19 @@ export async function pushFlow( const hasOnBehalfOf = (localFlow as any).has_on_behalf_of ?? !!localFlow.on_behalf_of_email; delete (localFlow as any).has_on_behalf_of; + // The authorization half of the identity is never exported to the repo (the + // workspace tarball strips it); it only ever travels back from the remote row. + delete (localFlow as any).on_behalf_of; - const preserveFields: { on_behalf_of_email?: string; preserve_on_behalf_of?: boolean } = {}; + const preserveFields: { + on_behalf_of_email?: string; + on_behalf_of?: string; + preserve_on_behalf_of?: boolean; + } = {}; if (permissionedAsContext?.userIsAdminOrDeployer && hasOnBehalfOf) { if (flow && flow.on_behalf_of_email) { preserveFields.on_behalf_of_email = flow.on_behalf_of_email; + preserveFields.on_behalf_of = (flow as any).on_behalf_of; preserveFields.preserve_on_behalf_of = true; log.info(`Preserving ${flow.on_behalf_of_email} as on_behalf_of for flow ${remotePath}`); } @@ -597,6 +610,8 @@ async function preview( log.setSilent(true); } const useLocalPathScripts = !opts.remote; + // Captured before the config read, which chdirs to the wmill.yaml root. + const cwdBeforeConfig = process.cwd(); if (useLocalPathScripts) { opts = await mergeConfigWithConfigFile(opts); } @@ -604,6 +619,9 @@ async function preview( await requireLogin(opts); const codebases = useLocalPathScripts ? listSyncCodebases(opts) : []; + const argPath = flowPath; + flowPath = toSyncRootRelativePath(flowPath, cwdBeforeConfig); + // Normalize path - ensure it's a directory path to a .flow or __flow folder const isFlowDir = flowPath.endsWith(".flow") || flowPath.endsWith(".flow" + SEP) || flowPath.endsWith("__flow") || flowPath.endsWith("__flow" + SEP); @@ -625,6 +643,14 @@ async function preview( flowPath += SEP; } + // The flow's windmill path (e.g. "f/cli_smoke/myrelflow"). It is what the + // preview job runs under, and the anchor for relative-import resolution: + // inline scripts in this flow are treated as living at + // "/", so "./util" resolves to + // "/util" — matching the keys in temp_script_refs. + const flowWmPath = stripFlowSuffix(flowPath).replaceAll(SEP, "/"); + assertRemotePath(flowWmPath, argPath); + // Read and parse the flow definition const localFlow = (await yamlParseFile(flowPath + "flow.yaml")) as FlowFile; @@ -695,12 +721,6 @@ async function preview( // too — PathScript modules have already been rewritten to inline rawscript // when `useLocalPathScripts` is set, and tempScriptRefs covers relative // imports in inline scripts. - // Compute the flow's windmill path (e.g. "f/cli_smoke/myrelflow"). Used as - // the anchor for relative-import resolution: inline scripts in this flow are - // treated as living at "/", so "./util" resolves to - // "/util" — matching the keys in temp_script_refs. - const flowWmPath = stripFlowSuffix(flowPath).replaceAll(SEP, "/"); - if (opts.step) { await previewStep(opts.step, localFlow, flowWmPath, workspace, input, tempScriptRefs, opts.silent, opts.tag); return; @@ -1218,6 +1238,8 @@ const command = new Command() ...remote, path: flowPath, on_behalf_of_email: email, + // Derived server-side; see the script command for why. + on_behalf_of: undefined, preserve_on_behalf_of: true, // Preserve any user draft at this path (see backend skip_draft_deletion). skip_draft_deletion: true, diff --git a/cli/src/commands/generate-metadata/generate-metadata.ts b/cli/src/commands/generate-metadata/generate-metadata.ts index 7eab79b52d..9680c6abda 100644 --- a/cli/src/commands/generate-metadata/generate-metadata.ts +++ b/cli/src/commands/generate-metadata/generate-metadata.ts @@ -22,7 +22,7 @@ import { FSFSElement, ignoreF, } from "../sync/sync.ts"; -import { exts } from "../script/script.ts"; +import { hasScriptExt } from "../script/script.ts"; import { isFolderResourcePathAnyFormat, isScriptModulePath, isModuleEntryPoint, scriptPathToRemotePath } from "../../utils/resource_folders.ts"; import { listSyncCodebases, SyncCodebase } from "../../utils/codebase.ts"; import { @@ -51,7 +51,7 @@ async function walkLocalScripts( const elems = await elementsToMap( await FSFSElement(process.cwd(), codebases, false), (p, isD) => - (!isD && !exts.some((ext) => p.endsWith(ext))) || + (!isD && !hasScriptExt(p)) || ignore(p, isD) || isFolderResourcePathAnyFormat(p) || // Datatable migration `.sql` files aren't Windmill scripts. @@ -221,7 +221,7 @@ function categorizeLocalFiles( ) { appPaths.push(p); } else if ( - exts.some((ext) => p.endsWith(ext)) && + hasScriptExt(p) && !isFolderResourcePathAnyFormat(p) && // Datatable migration `.sql` files aren't Windmill scripts. !isDatatableMigrationPath(p) && diff --git a/cli/src/commands/gitsync-settings/converter.ts b/cli/src/commands/gitsync-settings/converter.ts index 6861603bb4..868fa1b330 100644 --- a/cli/src/commands/gitsync-settings/converter.ts +++ b/cli/src/commands/gitsync-settings/converter.ts @@ -34,6 +34,7 @@ export class GitSyncSettingsConverter { includeSettings: includeTypes.includes("settings"), includeKey: includeTypes.includes("key"), skipWorkspaceDependencies: !includeTypes.includes("workspacedependencies"), + skipDatatableMigrations: !includeTypes.includes("datatablemigration"), }; // Only include extraIncludes if it has content @@ -63,6 +64,7 @@ export class GitSyncSettingsConverter { if (opts.includeSettings) includeTypes.push("settings"); if (opts.includeKey) includeTypes.push("key"); if (!opts.skipWorkspaceDependencies) includeTypes.push("workspacedependencies"); + if (!opts.skipDatatableMigrations) includeTypes.push("datatablemigration"); const result: BackendGitSyncSettings = { include_path: opts.includes || [], @@ -102,6 +104,7 @@ export class GitSyncSettingsConverter { includeSettings: opts.includeSettings ?? false, includeKey: opts.includeKey ?? false, skipWorkspaceDependencies: opts.skipWorkspaceDependencies ?? false, + skipDatatableMigrations: opts.skipDatatableMigrations ?? false, }; } @@ -126,6 +129,7 @@ export class GitSyncSettingsConverter { includeSettings: opts.includeSettings, includeKey: opts.includeKey, skipWorkspaceDependencies: opts.skipWorkspaceDependencies, + skipDatatableMigrations: opts.skipDatatableMigrations, }; } diff --git a/cli/src/commands/gitsync-settings/types.ts b/cli/src/commands/gitsync-settings/types.ts index 7bebb1f1a1..2a74966ccf 100644 --- a/cli/src/commands/gitsync-settings/types.ts +++ b/cli/src/commands/gitsync-settings/types.ts @@ -38,6 +38,7 @@ export const GIT_SYNC_FIELDS = [ "includeSettings", "includeKey", "skipWorkspaceDependencies", + "skipDatatableMigrations", ] as const; export type GitSyncField = typeof GIT_SYNC_FIELDS[number]; @@ -59,6 +60,7 @@ export const INCLUDE_TYPE_MAPPINGS = { settings: "includeSettings", key: "includeKey", workspacedependencies: "skipWorkspaceDependencies", + datatablemigration: "skipDatatableMigrations", } as const; // Write mode for branch-based configuration diff --git a/cli/src/commands/init/template.ts b/cli/src/commands/init/template.ts index b0c83d7039..ef2bcad0a6 100644 --- a/cli/src/commands/init/template.ts +++ b/cli/src/commands/init/template.ts @@ -97,6 +97,7 @@ export const CONFIG_REFERENCE: ConfigOption[] = [ { name: "skipApps", type: "boolean", default: "false", description: "Skip syncing apps" }, { name: "skipFolders", type: "boolean", default: "false", description: "Skip syncing folders" }, { name: "skipWorkspaceDependencies", type: "boolean", default: "false", description: "Skip syncing workspace dependencies" }, + { name: "skipDatatableMigrations", type: "boolean", default: "false", description: "Skip syncing data table SQL migrations" }, { name: "includeSchedules", type: "boolean", default: "false", description: "Include schedules in sync", commented: true, templateValue: "true", groupNote: "Uncomment to include these (excluded by default):" }, diff --git a/cli/src/commands/pipeline/docs.ts b/cli/src/commands/pipeline/docs.ts index abee64cbf3..1ae38d5e71 100644 --- a/cli/src/commands/pipeline/docs.ts +++ b/cli/src/commands/pipeline/docs.ts @@ -17,11 +17,12 @@ import { colors } from "@cliffy/ansi/colors"; import { GlobalOptions } from "../../types.ts"; import { type AssetGraph, + hideDbtRunnables, buildLocalPipelineGraph, workspaceRoot, } from "./localGraph.ts"; -const ASSET_KINDS = "s3object,ducklake,datatable,volume"; +const ASSET_KINDS = "s3object,ducklake,datatable,volume,dbt"; function assetUri(kind: string, p: string): string { const prefix = kind === "s3object" ? "s3" : kind; @@ -39,7 +40,7 @@ async function fetchDeployedGraph( if (!res.ok) { throw new Error(`GET assets/graph -> ${res.status}: ${await res.text()}`); } - return (await res.json()) as AssetGraph; + return hideDbtRunnables((await res.json()) as AssetGraph); } // Render the pipeline graph as a markdown document. diff --git a/cli/src/commands/pipeline/localGraph.ts b/cli/src/commands/pipeline/localGraph.ts index bb51cd98eb..5f9e713f42 100644 --- a/cli/src/commands/pipeline/localGraph.ts +++ b/cli/src/commands/pipeline/localGraph.ts @@ -63,6 +63,10 @@ export type GraphRunnable = { // `buildMacroEdges` (the wasm asset parser emits neither the marker nor the // registry). Non-empty ⇒ definition-only node. macros?: { name: string; params?: string; is_table?: boolean }[]; + // Set by the deployed graph on a dbt script: it owns a whole project, so the + // node counts models rather than reading as a single-output script. Never set + // locally — a dbt descriptor has no asset parser here. + dbt?: { model_count: number }; }; export type GraphEdge = { runnable_kind: string; @@ -229,6 +233,7 @@ function commentPrefix(language: string): string { language === "ansible" || language === "ruby" || language === "rlang" || + language === "dbt" || language === "nu" || language === "powershell" ) @@ -342,7 +347,11 @@ function normalizeRetry(retry: ParseAssetsRaw["retry"]): ParseAssetsRaw["retry"] // Read-asset kinds whose read auto-derives a cascade trigger edge inside a // `// pipeline`. Mirror of backend `is_auto_trigger_kind` (windmill-common // assets.rs) / frontend `AUTO_TRIGGER_KINDS` (resolveGraph.ts) — ducklake -// tables and s3 objects only; resource/datatable/volume stay explicit-`// on`. +// tables and s3 objects; resource/datatable/volume/table stay explicit-`// on`. +// A local graph out of step with that set shows an edge the deploy will not +// cascade along, and the generated pipeline docs then describe the wrong DAG. +// `table` is excluded everywhere: a dbt run does not dispatch, and nothing else +// writes a warehouse relation. const AUTO_TRIGGER_KINDS = new Set(["ducklake", "s3object"]); // Asset-URI prefixes accepted by `// mute `, in lockstep with the @@ -550,6 +559,75 @@ export async function collectScripts( return out; } +// The structural minimum the dbt filter needs. Declared instead of taking +// `AssetGraph` so the bounded-cascade view of the same payload (`BCGraph`, a +// narrower shape over identical JSON) passes through without a cast. +type DbtFilterableGraph = { + runnables: { path: string; usage_kind: string; dbt?: unknown }[]; + edges: { runnable_kind: string; runnable_path: string }[]; + triggers: { runnable_kind: string; runnable_path: string }[]; + macro_edges?: { lib_path: string; consumer_path: string }[]; + test_edges?: { + producer_kind: string; + producer_path: string; + runnable_kind: string; + runnable_path: string; + }[]; +}; + +/** + * The pipeline's view of a deployed graph whose folder also holds a dbt project. + * + * `/assets/graph` is asset-usage driven, not membership driven: it lists every + * script that reads or writes a relation, so a dbt script appears there like any + * producer. That is right for the endpoint — the node is what attributes a + * relation to what builds it — but a dbt project is not a pipeline, so it must + * not be rendered as one of its scripts. Mirrors the frontend's + * `hideDbtRunnables`; the local builder drops these nodes at the source. + * + * Its relations stay: they are what a downstream pipeline script reads. + */ +export function hideDbtRunnables(graph: G): G { + // Keyed by `(usage_kind, path)`, the graph's identity for a runnable — a + // script and a flow may share a path, and keying on path alone would take the + // flow's node, edges and triggers down with the dbt script's. + const key = (usage_kind: string, path: string) => `${usage_kind}:${path}`; + const dbtKeys = new Set( + (graph.runnables ?? []) + .filter((r) => r.dbt) + .map((r) => key(r.usage_kind, r.path)), + ); + if (dbtKeys.size === 0) return graph; + const kept = (usage_kind: string, path: string) => + !dbtKeys.has(key(usage_kind, path)); + return { + ...graph, + runnables: graph.runnables.filter((r) => kept(r.usage_kind, r.path)), + edges: graph.edges.filter((e) => kept(e.runnable_kind, e.runnable_path)), + triggers: graph.triggers.filter((t) => + kept(t.runnable_kind, t.runnable_path), + ), + // A macro library is a script; a dbt script defines no macros, so neither + // end can be a flow. + ...(graph.macro_edges + ? { + macro_edges: graph.macro_edges.filter( + (m) => kept("script", m.lib_path) && kept("script", m.consumer_path), + ), + } + : {}), + ...(graph.test_edges + ? { + test_edges: graph.test_edges.filter( + (t) => + kept(t.producer_kind, t.producer_path) && + kept(t.runnable_kind, t.runnable_path), + ), + } + : {}), + }; +} + // Build the full pipeline asset-graph from local files in `f//`. // Only `// pipeline` scripts become graph nodes (pipeline membership), mirroring // the deployed graph endpoint. Returns the graph plus the in-pipeline scripts' @@ -617,6 +695,11 @@ export async function buildLocalPipelineGraph(args: { }); continue; } + // A dbt project is not a data pipeline: the deploy never marks a dbt script + // `auto_kind='pipeline'` and the pipeline canvas drops its node, so the + // local graph must not invent one. Its models don't belong here either — + // they come from the manifest the deploy derives, not from the descriptor. + if (s.language === "dbt") continue; if (!out.in_pipeline) continue; // not a pipeline member if (out.data_tests && out.data_tests.length > 0) { dataTestsByPath.set(s.path, out.data_tests); diff --git a/cli/src/commands/pipeline/pipeline.ts b/cli/src/commands/pipeline/pipeline.ts index 0738283fba..8d46119e14 100644 --- a/cli/src/commands/pipeline/pipeline.ts +++ b/cli/src/commands/pipeline/pipeline.ts @@ -24,6 +24,7 @@ import { } from "./boundedCascade.ts"; import { type AssetGraph, + hideDbtRunnables, type GraphTrigger, type LocalScript, buildLocalPipelineGraph, @@ -96,7 +97,7 @@ async function list(opts: GlobalOptions & { json?: boolean }) { } } -const ASSET_KINDS = "s3object,ducklake,datatable,volume"; +const ASSET_KINDS = "s3object,ducklake,datatable,volume,dbt"; function assetUri(kind: string, path: string): string { const prefix = kind === "s3object" ? "s3" : kind; @@ -132,8 +133,10 @@ async function show( } else { const workspace = await resolveWorkspace(opts); await requireLogin(opts); - graph = await apiGet( - `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + graph = hideDbtRunnables( + await apiGet( + `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + ), ); enrich = (nativeByScript, roots) => enrichRootMarkers(workspace.workspaceId, graph, nativeByScript, roots); } @@ -612,8 +615,10 @@ async function run( } } } else { - graph = await apiGet( - `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + graph = hideDbtRunnables( + await apiGet( + `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + ), ); // Recover marker-only `data_upload`/`webhook`/`email` triggers the graph // endpoint can't emit, so input-only entrypoints are cut here as they are diff --git a/cli/src/commands/script/script.ts b/cli/src/commands/script/script.ts index c699e3d58f..37000c9ff3 100644 --- a/cli/src/commands/script/script.ts +++ b/cli/src/commands/script/script.ts @@ -1,6 +1,11 @@ import { GlobalOptions } from "../../types.ts"; import { requireLogin } from "../../core/auth.ts"; -import { resolveWorkspace, validatePath } from "../../core/context.ts"; +import { + assertRemotePath, + resolveWorkspace, + toSyncRootRelativePath, + validatePath, +} from "../../core/context.ts"; import type { PermissionedAsContext } from "../../core/permissioned_as.ts"; import { applyExtraPermsDiff } from "../../core/extra_perms.ts"; import { writeFile, stat, mkdir } from "node:fs/promises"; @@ -71,10 +76,19 @@ import { isScriptModulePath, buildModuleFolderPath, getModuleFolderSuffix, + dbtGeneratedDirs, + isUnderGeneratedDir, + isLocalSecretFile, + moduleFileExclusion, + oversizedModuleFileError, + MAX_MODULE_BYTES, isModuleEntryPoint, getScriptBasePathFromModulePath, scriptPathToRemotePath, isRawAppPath, + DBT_DESCRIPTOR_NAME, + isDbtDescriptorPath, + isMissingDbtDescriptor, } from "../../utils/resource_folders.ts"; export interface ScriptFile { @@ -159,9 +173,18 @@ async function push(opts: PushOptions, filePath: string) { return; } - const fstat = await stat(filePath); - if (!fstat.isFile()) { - throw new Error("file path must refer to a file."); + // A dbt project's descriptor is optional, so the one content path a + // descriptor-less project has is deliberately not on disk. The project beside + // it is what says the script is real. + const absentDescriptor = await stat(filePath).then( + () => false, + (e) => isMissingDbtDescriptor(filePath, e) + ); + if (!absentDescriptor) { + const fstat = await stat(filePath); + if (!fstat.isFile()) { + throw new Error("file path must refer to a file."); + } } if (filePath.endsWith(".script.json") || filePath.endsWith(".script.yaml")) { @@ -174,7 +197,7 @@ async function push(opts: PushOptions, filePath: string) { // Warn about metadata state before pushing try { - const content = await readTextFile(filePath); + const content = await readScriptContent(filePath); const remotePath = removeExtensionToPath(filePath).replaceAll(SEP, "/"); const contentHash = await computePushMetadataHash(filePath, content); const conf = await readLockfile(); @@ -272,7 +295,7 @@ const MODULE_ENTRY_META_RE = /([\\/])script\.(yaml|json|lock)$/; * `__mod/`, so this only narrows it to the metadata extensions: a `script.yaml` * nested deeper in the module tree is a module file, not the script's metadata. */ -function isModuleEntryMetadata(p: string): boolean { +export function isModuleEntryMetadata(p: string): boolean { return isModuleEntryPoint(p) && MODULE_ENTRY_META_RE.test(p); } @@ -335,7 +358,7 @@ export async function handleFile( // standalone scripts — pushed via pushRawApp, not here. !isRawAppPath(path) && (!isScriptModulePath(path) || moduleEntryPoint) && - exts.some((exts) => path.endsWith(exts)) + hasScriptExt(path) ) { if (alreadySynced.includes(path)) { return true; @@ -345,6 +368,29 @@ export async function handleFile( alreadySynced.push(path); const remotePath = scriptPathToRemotePath(path); + // Before anything is written: `.py` and `__dbt/` deploy to ONE + // remote path, so whichever is pushed last replaces the other's script. + // Refused from either side — the descriptor is exempt only from finding its + // OWN project (it is that project's content file, so its base resolves to + // the same `dbt_project.yml`), never from an ordinary sibling. + // A folder-layout script is `__mod/script.ts`, so stripping its + // extension yields `__mod/script`, not the base both layouts deploy + // to. Wrong base, and the probe below looks in a directory that cannot + // exist — which is how a `__mod` script and a dbt project at one path were + // both pushed, each replacing the other. + const base = isScriptModulePath(path) + ? getScriptBasePathFromModulePath(path) ?? removeExtensionToPath(path) + : removeExtensionToPath(path); + const isDescriptor = isDbtDescriptorPath(path); + const other = isDescriptor + ? await collidingOrdinaryScript(base) + : await collidingDbtProject(base); + if (other) { + throw isDescriptor + ? dbtPathCollisionError(path, other) + : dbtPathCollisionError(other, path); + } + const language = inferContentTypeFromFilePath(path, opts?.defaultTs); const codebase = @@ -479,7 +525,7 @@ export async function handleFile( } catch { log.debug(`Script ${remotePath} does not exist on remote`); } - const content = await readTextFile(path); + const content = await readScriptContent(path); if (opts?.skipScriptsMetadata) { // if (codebase) { @@ -499,8 +545,14 @@ export async function handleFile( const scriptBasePath = moduleEntryPoint ? getScriptBasePathFromModulePath(path)! : path.substring(0, path.indexOf(".")); - const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(); - const modules = await readModulesFromDisk(moduleFolderPath, opts?.defaultTs, moduleEntryPoint); + const isDbt = language === "dbt"; + const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(language); + const modules = await readModulesFromDisk( + moduleFolderPath, + opts?.defaultTs, + moduleEntryPoint, + isDbt, + ); // A concurrent_limit of <= 0 means "concurrency disabled", not "zero slots" (which // would brick the runnable at the queue's concurrency gate). Emit it as omitted rather @@ -518,7 +570,11 @@ export async function handleFile( path: remotePath.replaceAll(SEP, "/"), summary: typed?.summary ?? "", kind: typed?.kind, - lock: typed?.lock, + // A dbt lock pins a resolved commit and engine versions that only a + // dependency job can determine, and that job is also what publishes the + // script's manifest graph. Sending one suppresses that job, so the push + // would deploy a stale lock AND leave the graph unpublished. + lock: language === "dbt" ? undefined : typed?.lock, schema: typed?.schema, tag: typed?.tag, ws_error_handler_muted: typed?.ws_error_handler_muted, @@ -544,10 +600,16 @@ export async function handleFile( const hasOnBehalfOf = (typed as any)?.has_on_behalf_of ?? !!typed?.on_behalf_of_email; delete (typed as any)?.has_on_behalf_of; + // The authorization half of the identity is never exported to the repo (the + // workspace tarball strips it); it only ever travels back from the remote row. + delete (typed as any)?.on_behalf_of; if (permissionedAsContext?.userIsAdminOrDeployer && hasOnBehalfOf) { if (remote && remote.on_behalf_of_email) { requestBodyCommon.on_behalf_of_email = remote.on_behalf_of_email; + (requestBodyCommon as any).on_behalf_of = ( + remote as any + ).on_behalf_of; (requestBodyCommon as any).preserve_on_behalf_of = true; log.info(`Preserving ${remote.on_behalf_of_email} as on_behalf_of for script ${remotePath}`); } @@ -679,6 +741,11 @@ export async function readModulesFromDisk( moduleFolderPath: string, defaultTs: "bun" | "deno" | undefined, folderLayout: boolean = false, + // A dbt project rides in its module folder as-is: `.sql` models (which the + // language inference below rejects as an ambiguous dialect), `.yml` schemas + // and `.csv` seeds are all part of the project and none is a Windmill script. + // Verbatim, or dbt receives a project missing exactly the files it needs. + verbatim: boolean = false, ): Promise | undefined> { if (!fs.existsSync(moduleFolderPath) || !fs.statSync(moduleFolderPath).isDirectory()) { return undefined; @@ -686,9 +753,18 @@ export async function readModulesFromDisk( const modules: Record = {}; + const skipDirs = verbatim + ? dbtGeneratedDirs(moduleFolderPath) + : new Set(); + // In folder layout mode, skip the entry point files (script.*, script.yaml, etc.) const isEntryPointFile = (name: string, isTopLevel: boolean) => { - if (!folderLayout || !isTopLevel) return false; + if (!isTopLevel) return false; + // A dbt project's descriptor is the script's CONTENT, so it must not also + // ride along as a module: the push would send the same text twice and dbt + // would find a stray file at its project root. + if (verbatim) return name === DBT_DESCRIPTOR_NAME; + if (!folderLayout) return false; return ( name.startsWith("script.") || name === "script.lock" || @@ -705,10 +781,63 @@ export async function readModulesFromDisk( const isTopLevel = relPrefix === ""; if (entry.isDirectory()) { + // A configured `target-path` may be nested (`build/target`), so the + // comparison is on the project-relative path, not the entry name. + if (skipDirs.size > 0 && isUnderGeneratedDir(relPath, skipDirs)) continue; readDir(fullPath, relPath); - } else if (entry.isFile() && !entry.name.endsWith(".lock") && !isEntryPointFile(entry.name, isTopLevel)) { - // Skip lock files — they're handled as the `lock` field on ScriptModule - if (exts.some((ext) => entry.name.endsWith(ext))) { + // `.lock` is the script's own lockfile in a `__mod` bundle (the `lock` + // field on ScriptModule) — but a dbt project's files are its author's, + // and one may legitimately be named `uv.lock`. Dropping it would break + // the unmodified-project round trip this bundle exists to keep. + } else if ( + entry.isFile() && + (verbatim || !entry.name.endsWith(".lock")) && + !isEntryPointFile(entry.name, isTopLevel) + ) { + if (verbatim) { + // Secrets stay on the machine that holds them. Skipped before the + // read, and loudly: a `.env` swept into the bundle is a credential + // stored in every version of the script and handed back on pull. + if (isLocalSecretFile(entry.name)) { + log.warn( + `Skipping ${relPath}: a local secrets file is not part of the dbt project — ` + + `dbt reads its values from the environment, so set them in the script's ` + + `environment variables or the descriptor's \`env\``, + ); + continue; + } + // A dbt project's authored files are text. A binary one -- an image + // under `docs/`, a `.DS_Store`, a parquet seed -- would be read as + // mojibake and, if it carries a NUL, rejected by Postgres with an + // opaque `unsupported Unicode escape sequence`, which the push then + // reports as success. Skip it, loudly: dbt does not read it either. + // + // Asked BEFORE reading: the predicate only stats the file and reads + // its first 8 KB, so a multi-gigabyte seed next to the project costs + // that rather than being loaded whole just to be rejected. + const exclusion = moduleFileExclusion(fullPath); + if (exclusion !== undefined) { + // Over the limit but readable as text — a large seed CSV is the + // realistic case — is refused rather than skipped: dbt WOULD have + // read it, so shipping the project without it deploys something that + // compiles here and fails at run time with a missing relation. + if (exclusion === "oversized") { + throw oversizedModuleFileError(relPath, fs.statSync(fullPath).size); + } + log.warn( + `Skipping ${relPath}: not a text file, so it is not part of the dbt project the ` + + `bundle carries — dbt does not read it either`, + ); + continue; + } + // `language` is a required field of the API type and is not used for + // these: the worker writes them to their relative path and dbt reads + // the tree. + modules[relPath] = { + content: fs.readFileSync(fullPath).toString("utf-8"), + language: "dbt" as ScriptModule["language"], + }; + } else if (exts.some((ext) => entry.name.endsWith(ext))) { const content = readTextFileSync(fullPath); const language = inferContentTypeFromFilePath(entry.name, defaultTs); @@ -889,6 +1018,90 @@ async function createScript( */ export class UnresolvableScriptContentFileError extends Error {} +/** + * A path claimed by both a dbt project and an ordinary script. + * + * Its own class because the module push tolerates "no parent found" and must + * NOT tolerate this: swallowed, the command reports success while deploying + * nothing. + */ +export class DbtPathCollisionError extends UnresolvableScriptContentFileError {} + +/** + * The dbt project a path would collide with, if there is one. + * + * `.py` and `__dbt/` deploy to the SAME remote path, so whichever + * is pushed last wins and replaces the other's script. The descriptor is + * optional, so `dbt_project.yml` — not the descriptor — is what says a project + * is there. Asked on BOTH push paths: an ordinary file goes straight to + * `handleFile`, a model reaches its parent through `findContentFile`, and a + * guard on one of them leaves the other silently overwriting. + */ +export async function collidingDbtProject( + basePath: string +): Promise { + const project = basePath + "__dbt/dbt_project.yml"; + return (await stat(project).then(() => true).catch(() => false)) + ? project + : undefined; +} + +/** + * The ordinary script file sharing a base with a dbt project, if there is one — + * the same collision as [`collidingDbtProject`], seen from the dbt side. + * + * Needed because a descriptor may be pushed DIRECTLY (`wmill script push + * __dbt/wm_dbt.yaml`), which never passes through the metadata resolution + * that would otherwise catch it. + */ +export async function collidingOrdinaryScript( + basePath: string +): Promise { + for (const ext of exts) { + if (ext === "__dbt/" + DBT_DESCRIPTOR_NAME) continue; + // Both layouts, because both deploy to `basePath`: the flat file, and the + // folder layout's entry point. + for (const candidate of [ + basePath + ext, + `${basePath}${getModuleFolderSuffix()}/script${ext}`, + ]) { + const isFile = await stat(candidate) + .then((s) => s.isFile()) + .catch(() => false); + if (isFile) return candidate; + } + } + return undefined; +} + +export function dbtPathCollisionError( + project: string, + other: string +): DbtPathCollisionError { + return new DbtPathCollisionError( + `${project} and ${other} deploy to the same path, so pushing either one ` + + `replaces the other's script. Keep one: move the dbt project to a path ` + + `of its own, or remove ${other}.` + ); +} + + +/** + * A script's content, tolerating the one content file that may not exist: a dbt + * project's descriptor is optional, and absent means an empty descriptor. + */ +async function readScriptContent(filePath: string): Promise { + try { + return await readTextFile(filePath); + } catch (e) { + // ONLY a missing file is an empty descriptor. A permission or I/O error on a + // descriptor that does exist would otherwise deploy the defaults — the + // `main` warehouse and the whole project — in place of what the file says. + if (isMissingDbtDescriptor(filePath, e)) return ""; + throw e; + } +} + export async function findContentFile(filePath: string) { // Folder layout: __mod/script.yaml -> __mod/script.ts const isModuleFolderMeta = isModuleEntryMetadata(filePath); @@ -924,6 +1137,20 @@ export async function findContentFile(filePath: string) { ) .filter((x) => x.file) .map((x) => x.path); + // A dbt project's descriptor is OPTIONAL, so `dbt_project.yml` is what says a + // dbt script lives at this path — the descriptor is often absent from the + // candidates above while the project is perfectly real. Asked BEFORE the + // counts below: a project beside an ordinary script is not "one candidate", + // it is two scripts claiming one remote path, and returning the ordinary one + // deploys it OVER the dbt script on the next push of any model. + const dbtCandidate = toCandidate("__dbt/" + DBT_DESCRIPTOR_NAME); + const dbtProject = await collidingDbtProject( + dbtCandidate.slice(0, -("__dbt/" + DBT_DESCRIPTOR_NAME).length), + ); + const nonDbtCandidates = validCandidates.filter((c) => c !== dbtCandidate); + if (dbtProject && nonDbtCandidates.length > 0) { + throw dbtPathCollisionError(dbtProject, nonDbtCandidates.join(", ")); + } if (validCandidates.length > 1) { throw new UnresolvableScriptContentFileError( `Multiple script files found next to ${filePath}: ${validCandidates.join(", ")} — ` + @@ -931,6 +1158,11 @@ export async function findContentFile(filePath: string) { ); } if (validCandidates.length < 1) { + // Resolving to the absent descriptor keeps one content path for every + // caller; reading it yields an empty descriptor. + if (dbtProject) { + return dbtCandidate; + } throw new UnresolvableScriptContentFileError( `No script file found next to ${filePath} — a script cannot be deployed from its metadata alone. ` + `Add the matching script file (e.g. ${toCandidate(".ts")} or ${toCandidate( @@ -999,6 +1231,10 @@ export function filePathExtensionFromContentType( return ".rb"; } else if (language === "rlang") { return ".r"; + } else if (language === "dbt") { + // Not an extension but a path suffix: a dbt script's content file lives + // inside the project folder, so ` + this` is where it belongs. + return "__dbt/" + DBT_DESCRIPTOR_NAME; // for related places search: ADD_NEW_LANG } else { throw new Error("Invalid language: " + language); @@ -1031,12 +1267,28 @@ export const exts = [ ".java", ".rb", ".r", + // Not an extension: a dbt script's content file is its descriptor, inside + // the project folder. `.script.yaml` -> `__dbt/wm_dbt.yaml`. + "__dbt/" + DBT_DESCRIPTOR_NAME, // for related places search: ADD_NEW_LANG ]; +/** + * Whether a path is a script's content file. + * + * Separators are normalized first: one "extension" is the path suffix + * `__dbt/wm_dbt.yaml`, which on Windows is spelled `__dbt\wm_dbt.yaml` and + * would match nothing — silently skipping every dbt project on that platform. + */ +export function hasScriptExt(p: string): boolean { + const norm = p.replaceAll("\\", "/"); + return exts.some((ext) => norm.endsWith(ext)); +} + export function removeExtensionToPath(path: string): string { + const norm = path.replaceAll("\\", "/"); for (const ext of exts) { - if (path.endsWith(ext)) { + if (norm.endsWith(ext)) { return path.substring(0, path.length - ext.length); } } @@ -1450,7 +1702,7 @@ export async function generateMetadata( await FSFSElement(process.cwd(), codebases, false), (p, isD) => { return ( - (!isD && !exts.some((ext) => p.endsWith(ext))) || + (!isD && !hasScriptExt(p)) || ignore(p, isD) || isFlowPath(p) || isAppPath(p) || @@ -1561,17 +1813,28 @@ async function preview( if (opts.silent) { log.setSilent(true); } + // Captured before the config read, which chdirs to the wmill.yaml root. + const cwdBeforeConfig = process.cwd(); opts = await mergeConfigWithConfigFile(opts); const workspace = await resolveWorkspace(opts); await requireLogin(opts); - if (!validatePath(filePath)) { - return; - } + const argPath = filePath; + filePath = toSyncRootRelativePath(filePath, cwdBeforeConfig); + const remotePath = scriptPathToRemotePath(filePath); + assertRemotePath(remotePath, argPath); - const fstat = await stat(filePath); - if (!fstat.isFile()) { - throw new Error("file path must refer to a file."); + // Same as push: a descriptor-less dbt project's content path is deliberately + // absent, and the project beside it is what says the script is real. + const absentDescriptor = await stat(filePath).then( + () => false, + (e) => isMissingDbtDescriptor(filePath, e) + ); + if (!absentDescriptor) { + const fstat = await stat(filePath); + if (!fstat.isFile()) { + throw new Error("file path must refer to a file."); + } } if (filePath.endsWith(".script.json") || filePath.endsWith(".script.yaml")) { @@ -1582,15 +1845,23 @@ async function preview( const codebases = await listSyncCodebases(opts); const language = inferContentTypeFromFilePath(filePath, opts?.defaultTs); - const content = await readTextFile(filePath); + const content = await readScriptContent(filePath); const input = opts.data ? await resolve(opts.data) : {}; - // Read modules from __mod/ folder if present + // Read modules from the bundle folder if present. Same suffix and same + // verbatim read as deploy: a dbt project lives in `__dbt/`, and parsing its + // files as scripts would drop the `dbt_project.yml` the executor looks for. const isFolderLayout = isModuleEntryPoint(filePath); + const isDbt = language === "dbt"; const moduleFolderPath = isFolderLayout ? path.dirname(filePath) - : filePath.substring(0, filePath.indexOf(".")) + getModuleFolderSuffix(); - const modules = await readModulesFromDisk(moduleFolderPath, opts?.defaultTs, isFolderLayout); + : filePath.substring(0, filePath.indexOf(".")) + getModuleFolderSuffix(language); + const modules = await readModulesFromDisk( + moduleFolderPath, + opts?.defaultTs, + isFolderLayout, + isDbt + ); // Check if this is a codebase script const codebase = @@ -1606,11 +1877,7 @@ async function preview( const { extractRelativeImports } = await import( "../../utils/relative_imports.ts" ); - const relImports = await extractRelativeImports( - content, - scriptPathToRemotePath(filePath), - language - ); + const relImports = await extractRelativeImports(content, remotePath, language); if (relImports.length > 0) { const { buildPreviewTempScriptRefs } = await import( "../generate-metadata/generate-metadata.ts" @@ -1713,7 +1980,7 @@ async function preview( const form = new FormData(); const previewPayload = { content: content, // Pass the original content (frontend does this too) - path: filePath.substring(0, filePath.indexOf(".")).replaceAll(SEP, "/"), + path: remotePath, args: input, language: language, tag: opts.tag, @@ -1783,7 +2050,7 @@ async function preview( workspace: workspace.workspaceId, requestBody: { content, - path: filePath.substring(0, filePath.indexOf(".")).replaceAll(SEP, "/"), + path: remotePath, args: input, language: language as any, tag: opts.tag, @@ -1872,6 +2139,10 @@ async function setPermissionedAs( lock: Array.isArray(remote.lock) ? remote.lock.join("\n") : remote.lock ?? undefined, parent_hash: remote.hash, on_behalf_of_email: email, + // The principal is derived server-side from the email, which resolves workspace + // members, groups and superadmins acting outside their workspaces alike — a + // client-side `usr` lookup would see only the first of those. + on_behalf_of: undefined, preserve_on_behalf_of: true, // Preserve any user draft at this path (see backend skip_draft_deletion). skip_draft_deletion: true, diff --git a/cli/src/commands/sync/pull.ts b/cli/src/commands/sync/pull.ts index 90a31f4919..2702e13714 100644 --- a/cli/src/commands/sync/pull.ts +++ b/cli/src/commands/sync/pull.ts @@ -84,7 +84,9 @@ export async function downloadZip( includeSettings?: boolean, includeKey?: boolean, skipWorkspaceDependencies?: boolean, - defaultTs?: "bun" | "deno" + skipDatatableMigrations?: boolean, + defaultTs?: "bun" | "deno", + syncBehavior?: string ): Promise { const requestHeaders = new Headers(); requestHeaders.set("Authorization", "Bearer " + workspace.token); @@ -98,6 +100,9 @@ export async function downloadZip( } const includeWorkspaceDependenciesValue = !(skipWorkspaceDependencies ?? false); + // `sync_behavior_version` lets the server skip work this client would only throw away: + // from v1 the on-behalf-of address is stripped below, so the tarball sends the + // `has_on_behalf_of` marker instead and never resolves an address. // `preserve_extra_perms=true` opts the tarball into surfacing granular ACLs // on flow / script / app rows. Default-off on the server protects cross- // workspace tarball imports from carrying ACLs that reference identities @@ -107,7 +112,7 @@ export async function downloadZip( }&skip_secrets=${skipSecrets ?? false}&include_schedules=${includeSchedules ?? false }&include_triggers=${includeTriggers ?? false}&include_users=${includeUsers ?? false }&include_groups=${includeGroups ?? false}&include_settings=${includeSettings ?? false - }&include_key=${includeKey ?? false}&include_workspace_dependencies=${includeWorkspaceDependenciesValue}&default_ts=${defaultTs ?? "bun"}&skip_resource_types=${skipResourceTypes ?? false}&settings_version=v2&preserve_extra_perms=true`; + }&include_key=${includeKey ?? false}&include_workspace_dependencies=${includeWorkspaceDependenciesValue}&skip_datatable_migrations=${skipDatatableMigrations ?? false}&default_ts=${defaultTs ?? "bun"}&skip_resource_types=${skipResourceTypes ?? false}&settings_version=v2&preserve_extra_perms=true&sync_behavior_version=${syncBehavior ?? "v0"}`; const baseUrl = workspace.remote + "api/w/" + workspace.workspaceId + "/workspaces/tarball?"; diff --git a/cli/src/commands/sync/sync.ts b/cli/src/commands/sync/sync.ts index 3e6dd464fc..a28dd0fbd5 100644 --- a/cli/src/commands/sync/sync.ts +++ b/cli/src/commands/sync/sync.ts @@ -1,13 +1,28 @@ import { requireLogin } from "../../core/auth.ts"; +import { markRequestsAsSyncOrigin } from "../../core/client.ts"; import { fetchVersion, resolveWorkspace } from "../../core/context.ts"; -import { writeFile, readdir, stat, rm, copyFile, mkdir } from "node:fs/promises"; +import { + writeFile, + readdir, + stat, + rm, + copyFile, + mkdir, +} from "node:fs/promises"; +import { existsSync } from "node:fs"; import { colors } from "@cliffy/ansi/colors"; import { Command } from "@cliffy/command"; import { Confirm } from "@cliffy/prompt/confirm"; import * as log from "../../core/log.ts"; import * as path from "node:path"; import { sep as SEP } from "node:path"; -import { stringify as yamlStringify, type DocumentOptions, type SchemaOptions, type CreateNodeOptions, type ToStringOptions } from "yaml"; +import { + stringify as yamlStringify, + type DocumentOptions, + type SchemaOptions, + type CreateNodeOptions, + type ToStringOptions, +} from "yaml"; import JSZip from "jszip"; import { minimatch } from "minimatch"; import { yamlParseContent } from "../../utils/yaml.ts"; @@ -39,13 +54,15 @@ import { exts, findContentFile, findResourceFile, + isModuleEntryMetadata, handleScriptMetadata, UnresolvableScriptContentFileError, removeExtensionToPath, filePathExtensionFromContentType, + hasScriptExt, } from "../script/script.ts"; -import { handleFile } from "../script/script.ts"; +import { DbtPathCollisionError, handleFile } from "../script/script.ts"; import { deepEqual, fetchRemoteVersion, @@ -86,7 +103,9 @@ import { gitSyncDeployPush, deriveGitSyncDeployIncludes, isForkWorkspace, + gitRecordedDatatableMigrationPaths, type GitSyncDeployItem, + type RecordedMigrationPaths, } from "../../utils/git.ts"; import { Workspace } from "../workspace/workspace.ts"; import { removePathPrefix } from "../../types.ts"; @@ -99,20 +118,31 @@ import { MalformedLockfileError, workspaceDependenciesPathToLanguageAndFilename, } from "../../utils/metadata.ts"; -import { DoubleLinkedDependencyTree, uploadScripts } from "../../utils/dependency_tree.ts"; -import { OpenFlow, NativeServiceName, ScriptModule } from "../../../gen/types.gen.ts"; +import { + DoubleLinkedDependencyTree, + uploadScripts, +} from "../../utils/dependency_tree.ts"; +import { + OpenFlow, + NativeServiceName, + ScriptModule, +} from "../../../gen/types.gen.ts"; import { pushResource } from "../resource/resource.ts"; import { newPathAssigner, newRawAppPathAssigner, PathAssigner, } from "../../../windmill-utils-internal/src/path-utils/path-assigner.ts"; -import { extractInlineScripts as extractInlineScriptsForFlows, extractCurrentMapping } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; +import { + extractInlineScripts as extractInlineScriptsForFlows, + extractCurrentMapping, +} from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; import { generateFlowLockInternal } from "../flow/flow_metadata.ts"; import { isExecutionModeAnonymous } from "../app/app.ts"; import { APP_BACKEND_FOLDER, generateAppLocksInternal, + RECORDINGS_FOLDER, } from "../app/app_metadata.ts"; import { isFlowPath, @@ -132,10 +162,15 @@ import { getFolderSuffixWithSep, getNonDottedPaths, isScriptModulePath, + oversizedDbtFileError, getModuleFolderSuffix, + isDbtModulePath, + isDbtGeneratedPath, isModuleEntryPoint, getScriptBasePathFromModulePath, hasWrongFormatSuffix, + DBT_DESCRIPTOR_NAME, + isDbtDescriptorPath, } from "../../utils/resource_folders.ts"; let branchDeprecationWarned = false; @@ -154,12 +189,12 @@ function configKeyForItemKind( return "resources"; case "variable": return "variables"; - // case "schedule": - // return "schedules"; - // default: - // return kind.endsWith("_trigger") ? "triggers" : null; - } - return null + // case "schedule": + // return "schedules"; + // default: + // return kind.endsWith("_trigger") ? "triggers" : null; + } + return null; } // Fetch ws_specific items from the server and merge their paths into specificItems. @@ -182,8 +217,11 @@ async function mergeWsSpecificFromServer( // 404 = endpoint not present on an older server: expected, log at debug. // Anything else (401/403/network) is a real failure that produces an // incomplete sync — surface it so the user notices. - const isApiError = err && typeof err === "object" && - "name" in err && (err as { name: unknown }).name === "ApiError"; + const isApiError = + err && + typeof err === "object" && + "name" in err && + (err as { name: unknown }).name === "ApiError"; const status = isApiError ? (err as { status?: number }).status : undefined; if (status === 404) { log.debug("listWsSpecific endpoint not available on server, skipping"); @@ -235,9 +273,7 @@ export function computeWsSpecificFlagOnlyPushes( ): Array<{ kind: string; serverPath: string; filePath: string }> { if (!localSpecificItems || serverItems === null) return []; - const serverSet = new Set( - serverItems.map((i) => `${i.item_kind}:${i.path}`), - ); + const serverSet = new Set(serverItems.map((i) => `${i.item_kind}:${i.path}`)); const out: Array<{ kind: string; serverPath: string; filePath: string }> = []; for (const filePath of Object.keys(localMap)) { @@ -261,7 +297,10 @@ export function computeWsSpecificFlagOnlyPushes( // Resolve workspace name from a --branch override (git branch → workspace name). // Falls back to using the branch value as-is (backward compat: old key = branch name). -function resolveWsNameFromBranch(opts: SyncOptions, branchName: string): string { +function resolveWsNameFromBranch( + opts: SyncOptions, + branchName: string, +): string { const match = findWorkspaceByGitBranch(opts.workspaces, branchName); return match ? match[0] : branchName; } @@ -288,17 +327,23 @@ export function resolveWsNameForConfigFromFlags( } // Warn if --workspace overrides auto-detected branch or if workspace not in config. -function warnWorkspaceOverride(opts: SyncOptions, wsNameForConfig: string | undefined): void { +function warnWorkspaceOverride( + opts: SyncOptions, + wsNameForConfig: string | undefined, +): void { if (!wsNameForConfig || !opts.workspaces) return; // Check if workspace exists in config - const wsEntry = (opts.workspaces as any)?.[wsNameForConfig] as WorkspaceEntryConfig | undefined; + const wsEntry = (opts.workspaces as any)?.[wsNameForConfig] as + WorkspaceEntryConfig | undefined; if (!wsEntry) { - const wsNames = Object.keys(opts.workspaces).filter((k) => k !== "commonSpecificItems"); + const wsNames = Object.keys(opts.workspaces).filter( + (k) => k !== "commonSpecificItems", + ); if (wsNames.length > 0) { log.warn( `⚠️ Workspace '${wsNameForConfig}' is not defined in the 'workspaces' section of wmill.yaml.\n` + - ` No workspace-specific overrides will be applied. Available workspaces: ${wsNames.join(", ")}` + ` No workspace-specific overrides will be applied. Available workspaces: ${wsNames.join(", ")}`, ); } return; @@ -308,11 +353,14 @@ function warnWorkspaceOverride(opts: SyncOptions, wsNameForConfig: string | unde if (isGitRepository()) { const currentBranch = getCurrentGitBranch(); if (currentBranch) { - const autoMatch = findWorkspaceByGitBranch(opts.workspaces, currentBranch); + const autoMatch = findWorkspaceByGitBranch( + opts.workspaces, + currentBranch, + ); if (autoMatch && autoMatch[0] !== wsNameForConfig) { log.info( `Current git branch '${currentBranch}' maps to workspace '${autoMatch[0]}', ` + - `but --workspace overrides to '${wsNameForConfig}'.` + `but --workspace overrides to '${wsNameForConfig}'.`, ); } } @@ -327,9 +375,14 @@ function resolveWsNameForFiles(_opts: SyncOptions, wsName: string): string { // After resolveWorkspace, infer the workspace config name from the resolved profile // by matching baseUrl + workspaceId against the workspaces config entries. -function inferWsNameFromProfile(opts: SyncOptions, profile: { remote: string; workspaceId: string }): string | undefined { +function inferWsNameFromProfile( + opts: SyncOptions, + profile: { remote: string; workspaceId: string }, +): string | undefined { if (!opts.workspaces) return undefined; - const wsNames = Object.keys(opts.workspaces).filter((k) => k !== "commonSpecificItems"); + const wsNames = Object.keys(opts.workspaces).filter( + (k) => k !== "commonSpecificItems", + ); for (const name of wsNames) { const entry = (opts.workspaces as any)[name] as WorkspaceEntryConfig; if (!entry?.baseUrl) continue; @@ -362,7 +415,13 @@ async function resolveEffectiveSyncOptions( promotion?: string, workspaceNameOverride?: string, ): Promise { - return await getEffectiveSettings(localConfig, promotion, false, false, workspaceNameOverride); + return await getEffectiveSettings( + localConfig, + promotion, + false, + false, + workspaceNameOverride, + ); } type DynFSElement = { @@ -462,6 +521,27 @@ async function addCodebaseDigestIfRelevant( return content; } +/** + * Whether a script's modules ARE a dbt project. + * + * Keyed on `dbt_project.yml` rather than on the descriptor: the descriptor is + * optional, so its absence says nothing, while a dbt project without + * `dbt_project.yml` is one dbt itself refuses to run. + * + * Its LANGUAGE decides, not its name. A dbt bundle is read verbatim and every + * file in it is stored as `dbt`; an ordinary modular script that happens to + * vendor a dbt project stores that same file as whatever its extension infers, + * and calling it dbt would lay the bundle out as `__dbt` and drop it on the + * next push. + */ +function isDbtModules(modules: unknown): boolean { + if (typeof modules !== "object" || modules === null) return false; + const marker = (modules as Record)[ + "dbt_project.yml" + ]; + return marker?.language === "dbt"; +} + export async function FSFSElement( p: string, codebases: SyncCodebase[], @@ -491,8 +571,14 @@ export async function FSFSElement( } }, async getContentText(): Promise { - const content = await readTextFile(localP); const itemPath = localP.substring(p.length + 1); + // BEFORE the read: an oversized dbt project file stays visible to the + // diff on purpose (so the push reports it rather than silently shipping + // an incomplete project), and buffering a multi-gigabyte seed to reach + // that error is what this refusal exists to avoid. + const oversized = oversizedDbtFileError(localP, itemPath); + if (oversized) throw oversized; + const content = await readTextFile(localP); const r = await addCodebaseDigestIfRelevant( itemPath, content, @@ -525,9 +611,14 @@ function prioritizeName(name: string): string { return name; } -export const yamlOptions: DocumentOptions & SchemaOptions & CreateNodeOptions & ToStringOptions = { +export const yamlOptions: DocumentOptions & + SchemaOptions & + CreateNodeOptions & + ToStringOptions = { sortMapEntries: (a, b) => { - return prioritizeName(String(a.key)).localeCompare(prioritizeName(String(b.key))); + return prioritizeName(String(a.key)).localeCompare( + prioritizeName(String(b.key)), + ); }, aliasDuplicateObjects: false, singleQuote: true, @@ -591,11 +682,15 @@ export function extractFieldsForRawApps(runnables: Record) { * References the raw-app skill for complete documentation and includes instance-specific * data configuration (datatable, schema, whitelisted tables). */ -export function generateAgentsDocumentation(data: { - tables?: string[]; - datatable?: string; - schema?: string; -} | undefined): string { +export function generateAgentsDocumentation( + data: + | { + tables?: string[]; + datatable?: string; + schema?: string; + } + | undefined, +): string { const tables = data?.tables ?? []; const defaultDatatable = data?.datatable; const defaultSchema = data?.schema; @@ -610,15 +705,19 @@ This file contains **app-specific configuration** for this raw app instance. ## Data Configuration -${defaultDatatable - ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ''}` - : '**No default datatable configured.** Set \`data.datatable\` in \`raw_app.yaml\` to enable database access.'} +${ + defaultDatatable + ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ""}` + : "**No default datatable configured.** Set \`data.datatable\` in \`raw_app.yaml\` to enable database access." +} ### Whitelisted Tables -${tables.length > 0 - ? `These tables are accessible to this app:\n\n${tables.map(t => `- \`${t}\``).join('\n')}` - : `**No tables whitelisted.** Add tables to \`data.tables\` in \`raw_app.yaml\`.`} +${ + tables.length > 0 + ? `These tables are accessible to this app:\n\n${tables.map((t) => `- \`${t}\``).join("\n")}` + : `**No tables whitelisted.** Add tables to \`data.tables\` in \`raw_app.yaml\`.` +} ### Adding a Table @@ -626,10 +725,10 @@ Edit \`raw_app.yaml\`: \`\`\`yaml data: - datatable: ${defaultDatatable || 'main'} - ${defaultSchema ? `schema: ${defaultSchema}\n ` : ''}tables: -${tables.length > 0 ? tables.map(t => ` - ${t}`).join('\n') : ' # Add tables here'} - - ${defaultDatatable || 'main'}/${defaultSchema ? defaultSchema + ':' : ''}new_table # ← Add like this + datatable: ${defaultDatatable || "main"} + ${defaultSchema ? `schema: ${defaultSchema}\n ` : ""}tables: +${tables.length > 0 ? tables.map((t) => ` - ${t}`).join("\n") : " # Add tables here"} + - ${defaultDatatable || "main"}/${defaultSchema ? defaultSchema + ":" : ""}new_table # ← Add like this \`\`\` **Table reference formats:** @@ -665,11 +764,15 @@ const rows = await sql\`SELECT * FROM table WHERE id = \${id}\`.fetch(); * Generates a simple DATATABLES.md with just the current configuration summary. * The detailed schema information is generated by generate_datatables.ts command. */ -export function generateDatatablesDocumentation(data: { - tables?: string[]; - datatable?: string; - schema?: string; -} | undefined): string { +export function generateDatatablesDocumentation( + data: + | { + tables?: string[]; + datatable?: string; + schema?: string; + } + | undefined, +): string { const tables = data?.tables ?? []; const defaultDatatable = data?.datatable; const defaultSchema = data?.schema; @@ -683,15 +786,19 @@ Run \`wmill app generate-agents\` to refresh with current workspace schemas. ## Current Configuration -${defaultDatatable - ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ''}` - : '**No default datatable configured.**'} +${ + defaultDatatable + ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ""}` + : "**No default datatable configured.**" +} ## Whitelisted Tables -${tables.length > 0 - ? `${tables.map(t => `- \`${t}\``).join('\n')}` - : `*No tables whitelisted. Add tables to \`data.tables\` in \`raw_app.yaml\`.*`} +${ + tables.length > 0 + ? `${tables.map((t) => `- \`${t}\``).join("\n")}` + : `*No tables whitelisted. Add tables to \`data.tables\` in \`raw_app.yaml\`.*` +} --- @@ -761,11 +868,17 @@ export function extractInlineScriptsForApps( return []; } -type FileResourceTypeInfo = { format_extension: string | null; is_fileset: boolean }; +type FileResourceTypeInfo = { + format_extension: string | null; + is_fileset: boolean; +}; function parseFileResourceTypeMap( raw: Record, -): { formatExtMap: Record; filesetMap: Record } { +): { + formatExtMap: Record; + filesetMap: Record; +} { const formatExtMap: Record = {}; const filesetMap: Record = {}; for (const [k, v] of Object.entries(raw)) { @@ -799,7 +912,9 @@ async function findFilesetResourceFile(changePath: string): Promise { // not found, try next } } - throw new Error(`No resource metadata file found for fileset resource: ${changePath}`); + throw new Error( + `No resource metadata file found for fileset resource: ${changePath}`, + ); } type FilesetPushResult = @@ -871,9 +986,13 @@ function ZipFSElement( const content = await zip.files[filename].async("text"); const parsed = JSON.parse(content); if (parsed.modules && Object.keys(parsed.modules).length > 0) { - _moduleScriptPaths.add( - filename.slice(0, -".script.json".length) - ); + const base = filename.slice(0, -".script.json".length); + // A dbt script's modules ARE its dbt project, so it keeps the flat + // layout: only the project goes in the folder, which is what + // `--project-dir` expects and what makes the import a plain copy. + if (!isDbtModules(parsed.modules)) { + _moduleScriptPaths.add(base); + } } } catch {} } @@ -926,7 +1045,7 @@ function ZipFSElement( let finalPath = transformPath(); // Redirect content files for scripts with modules into __mod/ folder - if (kind == "other" && exts.some((ext) => p.endsWith(ext))) { + if (kind == "other" && hasScriptExt(p)) { const normalizedP = p.replace(/^\.[\\/]/, ""); const moduleScripts = await getModuleScriptPaths(); for (const basePath of moduleScripts) { @@ -934,7 +1053,11 @@ function ZipFSElement( const ext = normalizedP.slice(basePath.length); // e.g., ".ts", ".py" const dir = path.dirname(finalPath); const base = path.basename(basePath); - finalPath = path.join(dir, base + getModuleFolderSuffix(), "script" + ext); + finalPath = path.join( + dir, + base + getModuleFolderSuffix(), + "script" + ext, + ); break; } } @@ -955,7 +1078,9 @@ function ZipFSElement( } let inlineScripts; try { - const assigner = newPathAssigner(defaultTs, { skipInlineScriptSuffix: getNonDottedPaths() }); + const assigner = newPathAssigner(defaultTs, { + skipInlineScriptSuffix: getNonDottedPaths(), + }); // Preserve original !inline filenames from the flow to avoid phantom renames const inlineMapping = extractCurrentMapping( flow.value.modules as any, @@ -969,27 +1094,40 @@ function ZipFSElement( SEP, defaultTs, assigner, - { skipInlineScriptSuffix: getNonDottedPaths(), failOnInlineDirective: true }, + { + skipInlineScriptSuffix: getNonDottedPaths(), + failOnInlineDirective: true, + }, ); if (flow.value.failure_module) { - inlineScripts.push(...extractInlineScriptsForFlows( - [flow.value.failure_module], - inlineMapping, - SEP, - defaultTs, - assigner, - { skipInlineScriptSuffix: getNonDottedPaths(), failOnInlineDirective: true }, - )); + inlineScripts.push( + ...extractInlineScriptsForFlows( + [flow.value.failure_module], + inlineMapping, + SEP, + defaultTs, + assigner, + { + skipInlineScriptSuffix: getNonDottedPaths(), + failOnInlineDirective: true, + }, + ), + ); } if (flow.value.preprocessor_module) { - inlineScripts.push(...extractInlineScriptsForFlows( - [flow.value.preprocessor_module], - inlineMapping, - SEP, - defaultTs, - assigner, - { skipInlineScriptSuffix: getNonDottedPaths(), failOnInlineDirective: true }, - )); + inlineScripts.push( + ...extractInlineScriptsForFlows( + [flow.value.preprocessor_module], + inlineMapping, + SEP, + defaultTs, + assigner, + { + skipInlineScriptSuffix: getNonDottedPaths(), + failOnInlineDirective: true, + }, + ), + ); } } catch (error) { log.error( @@ -1038,7 +1176,9 @@ function ZipFSElement( inlineScripts = extractInlineScriptsForApps( undefined, app?.["value"], - newPathAssigner(defaultTs, { skipInlineScriptSuffix: getNonDottedPaths() }), + newPathAssigner(defaultTs, { + skipInlineScriptSuffix: getNonDottedPaths(), + }), (_, val) => val["name"], false, ); @@ -1124,7 +1264,7 @@ function ZipFSElement( isDirectory: false, path: path.join(finalPath, filePath.substring(1)), async *getChildren() {}, - async getContentText() { + async getContentText() { if (typeof content !== "string") { throw new Error( `Content of raw app file ${filePath} is not a string`, @@ -1213,7 +1353,9 @@ function ZipFSElement( // Simplify fields for cleaner YAML output if (simplifiedRunnable.fields) { - simplifiedRunnable.fields = simplifyFields(simplifiedRunnable.fields); + simplifiedRunnable.fields = simplifyFields( + simplifiedRunnable.fields, + ); } yield { @@ -1272,17 +1414,28 @@ function ZipFSElement( log.error(`Failed to parse script.yaml at path: ${p}`); throw error; } - const hasModules = parsed["modules"] && Object.keys(parsed["modules"]).length > 0; + const hasModules = + parsed["modules"] && Object.keys(parsed["modules"]).length > 0; + // A dbt script's module folder holds its dbt project and dbt's own + // files, so its lock stays outside like a plain script's — only the + // descriptor lives in there. + const isDbtScript = isDbtModules(parsed["modules"]); if ( parsed["lock"] && parsed["lock"] != "" && parsed["codebase"] == undefined ) { - if (hasModules) { + if (hasModules && !isDbtScript) { // Lock lives inside __mod/ folder as script.lock - const scriptBase = removeSuffix(removeSuffix(p.replaceAll(SEP, "/"), ".json"), ".script"); + const scriptBase = removeSuffix( + removeSuffix(p.replaceAll(SEP, "/"), ".json"), + ".script", + ); parsed["lock"] = - "!inline " + scriptBase + getModuleFolderSuffix() + "/script.lock"; + "!inline " + + scriptBase + + getModuleFolderSuffix() + + "/script.lock"; } else { parsed["lock"] = "!inline " + @@ -1322,8 +1475,7 @@ function ZipFSElement( throw error; } const resourceType = parsed["resource_type"]; - const formatExtension = - resourceTypeToFormatExtension[resourceType]; + const formatExtension = resourceTypeToFormatExtension[resourceType]; const isFileset = resourceTypeToIsFileset[resourceType] ?? false; if (isFileset) { @@ -1388,18 +1540,24 @@ function ZipFSElement( throw error; } const lock = parsed["lock"]; - const scriptModules: Record | undefined = parsed["modules"]; + const scriptModules: Record | undefined = + parsed["modules"]; const hasModules = scriptModules && Object.keys(scriptModules).length > 0; + // A dbt script's module folder is its dbt project, so the metadata and + // lock stay beside it — the descriptor is the one Windmill file that goes + // in, because it is the script's content. + const isDbt = isDbtModules(scriptModules); // Compute base path and module folder const metaExt = useYaml ? ".yaml" : ".json"; const scriptBasePath = removeSuffix( removeSuffix(finalPath, metaExt), - ".script" + ".script", ); - const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(); + const moduleFolderPath = + scriptBasePath + getModuleFolderSuffix(isDbt ? "dbt" : undefined); - if (hasModules) { + if (hasModules && !isDbt) { // Redirect metadata into __mod/script.yaml r[0].path = path.join(moduleFolderPath, "script" + metaExt); } @@ -1407,9 +1565,10 @@ function ZipFSElement( if (lock && lock != "") { r.push({ isDirectory: false, - path: hasModules - ? path.join(moduleFolderPath, "script.lock") - : removeSuffix(finalPath, metaExt) + ".lock", + path: + hasModules && !isDbt + ? path.join(moduleFolderPath, "script.lock") + : removeSuffix(finalPath, metaExt) + ".lock", async *getChildren() {}, async getContentText() { return lock; @@ -1436,7 +1595,7 @@ function ZipFSElement( // Yield the module lock file if present if (mod.lock) { - const baseName = relPath.replace(/\.[^.]+$/, ''); + const baseName = relPath.replace(/\.[^.]+$/, ""); yield { isDirectory: false, path: path.join(moduleFolderPath, baseName + ".lock"), @@ -1464,11 +1623,14 @@ function ZipFSElement( throw error; } const resourceType = parsed["resource_type"]; - const formatExtension = - resourceTypeToFormatExtension[resourceType]; + const formatExtension = resourceTypeToFormatExtension[resourceType]; const isFileset = resourceTypeToIsFileset[resourceType] ?? false; - if (isFileset && typeof parsed["value"] === "object" && parsed["value"] !== null) { + if ( + isFileset && + typeof parsed["value"] === "object" && + parsed["value"] !== null + ) { const filesetBasePath = removeSuffix(finalPath, ".resource.json") + ".fileset"; // Push directory entry for the fileset @@ -1476,7 +1638,9 @@ function ZipFSElement( isDirectory: true, path: filesetBasePath, async *getChildren() { - for (const [relPath, fileContent] of Object.entries(parsed["value"])) { + for (const [relPath, fileContent] of Object.entries( + parsed["value"], + )) { if (typeof fileContent === "string") { yield { isDirectory: false, @@ -1652,9 +1816,15 @@ export async function elementsToMap( } const path = entry.path; // Include module files in the map so they're compared for changes, - // but they're pushed as part of their parent script via handleFile + // but they're pushed as part of their parent script via handleFile. + // `--skip-scripts` therefore covers them, and has to be applied here: the + // filters below are past this shortcut, so a changed module would push the + // parent script the flag asked to leave alone — every file of a dbt project + // is one of these. if (isScriptModulePath(path)) { - map[path] = await entry.getContentText(); + if (!skips.skipScripts) { + map[path] = await entry.getContentText(); + } continue; } if ( @@ -1663,6 +1833,10 @@ export async function elementsToMap( !isRawAppFile(path) && !isWorkspaceDependencies(path) ) { + // The metadata format decides which of the two metadata twins is read, + // and drops the other. A dbt descriptor is not metadata and is not + // reached here: it lives inside the project folder, so the module branch + // above already took it, in both modes. if (json && path.endsWith(".yaml")) continue; if (!json && path.endsWith(".json")) continue; @@ -1695,9 +1869,16 @@ export async function elementsToMap( } if (isRawAppFile(path)) { - const suffix = path.split(getFolderSuffix("raw_app") + SEP).pop(); + // FSFSElement builds paths with the platform separator, while the checks + // below are written with "/": without normalizing, none of them match on + // Windows and the push collector's own exclusions become perpetual diffs. + const suffix = path + .split(getFolderSuffix("raw_app") + SEP) + .pop() + ?.replaceAll(SEP, "/"); if ( suffix?.startsWith("dist/") || + suffix?.startsWith(RECORDINGS_FOLDER + "/") || suffix == "wmill.d.ts" || suffix == "package-lock.json" || suffix == "DATATABLES.md" @@ -1706,7 +1887,11 @@ export async function elementsToMap( } } - if (skips.skipResources && (isFileResource(path) || isFilesetResource(path))) continue; + if ( + skips.skipResources && + (isFileResource(path) || isFilesetResource(path)) + ) + continue; const ext = json ? ".json" : ".yaml"; if (!skips.includeSchedules && path.endsWith(".schedule" + ext)) continue; @@ -1749,6 +1934,8 @@ export async function elementsToMap( fileType === "workspace_dependencies" ) continue; + if (skips.skipDatatableMigrations && fileType === "datatable_migration") + continue; } catch { // If getTypeStrFromPath can't determine the type, continue processing the file } @@ -1827,17 +2014,38 @@ export async function elementsToMap( if (wrongFormatPaths.length > 0) { const isNonDotted = getNonDottedPaths(); - const foundFormat = isNonDotted ? ".flow/.app/.raw_app" : "__flow/__app/__raw_app"; - const expectedFormat = isNonDotted ? "__flow/__app/__raw_app" : ".flow/.app/.raw_app"; + const foundFormat = isNonDotted + ? ".flow/.app/.raw_app" + : "__flow/__app/__raw_app"; + const expectedFormat = isNonDotted + ? "__flow/__app/__raw_app" + : ".flow/.app/.raw_app"; const configHint = isNonDotted ? "Either remove 'nonDottedPaths: true' from wmill.yaml, or rename these directories to use __flow/__app/__raw_app format." : "Either add 'nonDottedPaths: true' to wmill.yaml, or rename these directories to use .flow/.app/.raw_app format."; const pathList = wrongFormatPaths.map((p) => ` ${p}`).join("\n"); throw new Error( - `Found ${wrongFormatPaths.length} directory(ies) using ${foundFormat} format, but wmill.yaml expects ${expectedFormat}:\n${pathList}\n${configHint}` + `Found ${wrongFormatPaths.length} directory(ies) using ${foundFormat} format, but wmill.yaml expects ${expectedFormat}:\n${pathList}\n${configHint}`, ); } + // A dbt project's descriptor is optional, and the two sides spell "absent" + // differently: nothing on disk, and nothing in the export (which omits an + // empty one so a project that never named a descriptor never grows one). + // Left alone that reads as an addition on every push and a deletion on every + // pull, forever. Both sides are given the empty descriptor the absence means, + // so a descriptor-less project reaches a clean sync state. + for (const key of Object.keys(map)) { + // Normalized first: the local map's keys are built with `path.join`, so on + // Windows this reads `__dbt\\dbt_project.yml` and an unnormalized match + // would synthesize nothing — leaving exactly the perpetual push/pull diff + // above unguarded, on that platform only. + if (!key.replaceAll("\\", "/").endsWith("__dbt/dbt_project.yml")) continue; + const descriptor = + key.slice(0, -"dbt_project.yml".length) + DBT_DESCRIPTOR_NAME; + if (!(descriptor in map)) map[descriptor] = ""; + } + return map; } @@ -1851,6 +2059,7 @@ export interface Skips { skipApps?: boolean | undefined; skipFolders?: boolean | undefined; skipWorkspaceDependencies?: boolean | undefined; + skipDatatableMigrations?: boolean | undefined; skipScriptsMetadata?: boolean | undefined; includeSchedules?: boolean | undefined; includeTriggers?: boolean | undefined; @@ -1967,7 +2176,11 @@ export function canonicalizeCaseInsensitiveKeys( const lk = seg.toLowerCase(); let entry = node.children.get(lk); if (!entry) { - entry = { canonical: seg, ambiguous: false, node: { children: new Map() } }; + entry = { + canonical: seg, + ambiguous: false, + node: { children: new Map() }, + }; node.children.set(lk, entry); } else if (entry.canonical !== seg) { entry.ambiguous = true; @@ -2141,7 +2354,9 @@ export function preservePendingScriptLocks( remoteParsed = isYaml ? yamlParseContent(metaKey, remote[metaKey]) : JSON.parse(remote[metaKey]); - localParsed = isYaml ? yamlParseContent(metaKey, localMeta) : JSON.parse(localMeta); + localParsed = isYaml + ? yamlParseContent(metaKey, localMeta) + : JSON.parse(localMeta); } catch { continue; } @@ -2155,7 +2370,8 @@ export function preservePendingScriptLocks( // The local side must reference an inline lock backed by a committed file. const localLock = localParsed["lock"]; - if (typeof localLock !== "string" || !localLock.startsWith("!inline ")) continue; + if (typeof localLock !== "string" || !localLock.startsWith("!inline ")) + continue; // Derive the lock-file key from the `!inline` reference itself, not from the // metadata path: a multi-module script keeps its lock at `…__mod/script.lock`, @@ -2188,10 +2404,37 @@ async function compareDynFSElement( ): Promise<{ changes: Change[]; localMap: Record }> { let [m1, m2] = els2 ? await Promise.all([ - elementsToMap(els1, ignore, json, skips, specificItems, branchOverride, isEls1Remote), - elementsToMap(els2, ignore, json, skips, specificItems, branchOverride, !isEls1Remote), + elementsToMap( + els1, + ignore, + json, + skips, + specificItems, + branchOverride, + isEls1Remote, + ), + elementsToMap( + els2, + ignore, + json, + skips, + specificItems, + branchOverride, + !isEls1Remote, + ), ]) - : [await elementsToMap(els1, ignore, json, skips, specificItems, branchOverride, isEls1Remote), {}]; + : [ + await elementsToMap( + els1, + ignore, + json, + skips, + specificItems, + branchOverride, + isEls1Remote, + ), + {}, + ]; // Reconcile letter-case differences between the local tree and the // authoritative server casing. Only meaningful for an actual two-sided diff @@ -2506,9 +2749,11 @@ const isNotWmillFile = (p: string, isDirectory: boolean) => { ); } - // Files inside __mod/ folders are script module files — always valid wmill files + // Files inside a module folder belong to their parent script, so they are + // always valid wmill files — except the ones dbt generates, which are not + // part of the bundle and must not surface as items of their own. if (isScriptModulePath(p)) { - return false; + return isDbtGeneratedPath(p); } try { @@ -2560,6 +2805,7 @@ export async function ignoreF(wmillconf: { extraIncludes?: string[]; skipResourceTypes?: boolean; skipWorkspaceDependencies?: boolean; + skipDatatableMigrations?: boolean; json?: boolean; includeUsers?: boolean; includeGroups?: boolean; @@ -2624,6 +2870,14 @@ export async function ignoreF(wmillconf: { ) { return false; // Don't ignore workspace dependencies (they are always included unless explicitly skipped) } + // `migrations/datatable/**` is outside the u/f/g namespaces the path + // filters are written against, so the skip flag is its only control. + if ( + !wmillconf.skipDatatableMigrations && + fileType === "datatable_migration" + ) { + return false; + } } catch { // If getTypeStrFromPath can't determine the type, fall through to normal logic } @@ -2637,6 +2891,46 @@ export async function ignoreF(wmillconf: { }; } +/** + * How many migration *records* a set of changed files covers. One migration is two + * files (`.up.sql` + optional `.down.sql`) for a single `(datatable, timestamp)`, + * so counting paths would overstate what a prompt is about to delete. + */ +export function countDatatableMigrationRecords( + changes: { path: string }[], +): number { + const records = new Set(); + for (const c of changes) { + const parsed = parseDatatableMigrationPath(c.path); + if (parsed) records.add(`${parsed.datatable}\0${parsed.timestamp}`); + } + return records.size; +} + +/** + * The `deleted` changes for data table migrations that a push cannot safely trust. + * + * Migrations bypass the repo's path filters (they live outside `f/`/`u/`), so a clone + * made before they were synced sees every server-side migration as remote-only — and + * `pushMigrationFromDisk` reads a locally absent `.up.sql` as an instruction to delete + * it. `recorded` is what this repository's own history says (see + * `gitRecordedDatatableMigrationPaths`): a recorded path was genuinely tracked, so its + * absence now is a real deletion; a path missing from a `known` set is one this branch + * has never had, and deleting it is a guess. `unknown` history is not evidence of + * anything, so nothing is trusted. The caller confirms whatever comes back explicitly + * and never deletes it unattended. + */ +export function untrackedDatatableMigrationDeletions< + T extends { name: string; path: string }, +>(changes: T[], recorded: RecordedMigrationPaths): T[] { + return changes.filter( + (c) => + c.name === "deleted" && + isDatatableMigrationPath(c.path) && + !(recorded.kind === "known" && recorded.paths.has(c.path)), + ); +} + interface ChangeTracker { scripts: string[]; flows: string[]; @@ -2644,13 +2938,78 @@ interface ChangeTracker { rawApps: string[]; } +/// The script a module file belongs to, added to the tracker so its top hash is +/// refreshed. Derived with `getScriptBasePathFromModulePath`, which normalizes +/// separators: searching the raw path for `__dbt/` found nothing on Windows, +/// where the folder is spelled `__dbt\`, so every model edit there was skipped. +async function addModuleParentToChanged(p: string, tracker: ChangeTracker) { + // A folder layout's METADATA — `__mod/script.yaml` — is an entry-point + // path too, and it is not a content file: pushed as one, the metadata pass + // asks `inferContentTypeFromFilePath` for the language of `.yaml` and aborts + // the whole command. It resolves to its content file like any other metadata. + if (isModuleEntryMetadata(p)) { + try { + const contentPath = await findContentFile(p); + if (contentPath && !tracker.scripts.includes(contentPath)) { + tracker.scripts.push(contentPath); + } + } catch { + // ignore — content file not found + } + return; + } + if (isModuleEntryPoint(p)) { + // Entry point (e.g. __mod/script.ts) IS the parent script content file. + if (!tracker.scripts.includes(p)) { + tracker.scripts.push(p); + } + return; + } + const scriptBasePath = getScriptBasePathFromModulePath(p); + if (scriptBasePath === undefined) { + return; + } + const push = async (candidate: string): Promise => { + try { + const contentPath = await findContentFile(candidate); + if (contentPath && !tracker.scripts.includes(contentPath)) { + tracker.scripts.push(contentPath); + } + return contentPath != undefined; + } catch { + return false; + } + }; + // A dbt script's metadata sits beside its folder; the descriptor is inside. + if (isDbtModulePath(p)) { + await push(scriptBasePath + ".script.yaml"); + return; + } + // Folder layout first (`__mod/script.{ext}`), then flat. + if (!(await push(scriptBasePath + getModuleFolderSuffix() + "/script.yaml"))) { + await push(scriptBasePath + ".script.yaml"); + } +} + async function addToChangedIfNotExists(p: string, tracker: ChangeTracker) { // Datatable migration .sql files are not scripts; they're synced via the // dedicated datatable_migration handler in the push loop. if (isDatatableMigrationPath(p)) { return; } - const isScript = exts.some((e) => p.endsWith(e)) && !isFileResource(p) && !isFilesetResource(p); + // Module files first, and whatever their extension: a dbt project authors + // `dbt_project.yml`, `packages.yml`, schema YAML and seed CSVs, none of which + // are Windmill script extensions — gated behind that test they never reached + // the tracker, so the top hash in `wmill-lock.yaml` (which covers the modules) + // stayed stale for exactly the files a dbt project is mostly made of. + if (isScriptModulePath(p)) { + await addModuleParentToChanged(p, tracker); + return; + } + const isScript = + hasScriptExt(p) && + !isFileResource(p) && + !isFilesetResource(p); if (isScript) { if (isFlowPath(p)) { const folder = extractFolderPath(p, "flow")!; @@ -2667,37 +3026,6 @@ async function addToChangedIfNotExists(p: string, tracker: ChangeTracker) { if (!tracker.rawApps.includes(folder)) { tracker.rawApps.push(folder); } - } else if (isScriptModulePath(p)) { - if (isModuleEntryPoint(p)) { - // Entry point (e.g. __mod/script.ts) IS the parent script content file - if (!tracker.scripts.includes(p)) { - tracker.scripts.push(p); - } - } else { - // Module file changed — find the parent script content file - const moduleSuffix = getModuleFolderSuffix() + "/"; - const idx = p.indexOf(moduleSuffix); - if (idx !== -1) { - const scriptBasePath = p.substring(0, idx); - // Try folder layout first: __mod/script.{ext} - try { - const contentPath = await findContentFile(scriptBasePath + getModuleFolderSuffix() + "/script.yaml"); - if (contentPath && !tracker.scripts.includes(contentPath)) { - tracker.scripts.push(contentPath); - } - } catch { - // Fall back to flat layout: scriptBasePath.script.yaml - try { - const contentPath = await findContentFile(scriptBasePath + ".script.yaml"); - if (contentPath && !tracker.scripts.includes(contentPath)) { - tracker.scripts.push(contentPath); - } - } catch { - // ignore — content file not found - } - } - } - } } else { if (!tracker.scripts.includes(p)) { tracker.scripts.push(p); @@ -2715,7 +3043,7 @@ async function addToChangedIfNotExists(p: string, tracker: ChangeTracker) { } } -async function buildTracker(changes: Change[]) { +export async function buildTracker(changes: Change[]) { const tracker: ChangeTracker = { scripts: [], flows: [], @@ -2734,7 +3062,7 @@ async function buildTracker(changes: Change[]) { * When a module file changes, find and push the parent script. * The parent script's handleFile will read the __mod/ folder and include all modules. */ -async function pushParentScriptForModule( +export async function pushParentScriptForModule( modulePath: string, workspace: Workspace, alreadySynced: string[], @@ -2743,11 +3071,82 @@ async function pushParentScriptForModule( rawWorkspaceDependencies: Record, codebases: SyncCodebase[], ): Promise { - const moduleSuffix = getModuleFolderSuffix() + "/"; - const idx = modulePath.indexOf(moduleSuffix); - if (idx === -1) return; - const scriptBasePath = modulePath.substring(0, idx); - const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(); + const isDbt = isDbtModulePath(modulePath); + // Via the shared helper, which normalizes separators: a Windows path spells + // the folder `__dbt\\`, and searching the raw path for `__dbt/` would find + // nothing and silently return without deploying the parent — while the caller + // still records the file as synced. + const scriptBasePath = getScriptBasePathFromModulePath(modulePath); + if (scriptBasePath === undefined) return; + const moduleFolderPath = + scriptBasePath + getModuleFolderSuffix(isDbt ? "dbt" : undefined); + + // A dbt project's descriptor sits INSIDE its folder (`__dbt/wm_dbt.yaml`) and + // is optional, so the project itself is what identifies the script. + if (isDbt) { + // Only the LOOKUP is tolerated: a module under no script's project is a + // stray file, not an error. Deploying it is not — swallowing that would let + // a module-only push report success while the remote project is unchanged. + // BEFORE the lookup, because the lookup succeeds whenever a descriptor is + // there: `dbt_project.yml` is what makes the bundle a project, and pushing + // without it replaces a healthy deployment with one whose dependency job + // fails for having no project at all. + const hasMetadata = + existsSync(scriptBasePath + ".script.yaml") || + existsSync(scriptBasePath + ".script.json"); + if (!existsSync(moduleFolderPath + "/dbt_project.yml")) { + if (hasMetadata) { + throw new Error( + `${moduleFolderPath} has no dbt_project.yml but ${scriptBasePath}.script.yaml ` + + `remains, so there is no dbt project left to push. Delete the metadata too to ` + + `archive the script, or restore the project.` + ); + } + // Nothing local claims this script any more — neither project nor + // metadata — so it is archived like any other locally deleted item. The + // deletions arrive one file at a time, hence `alreadySynced`. + const remote = scriptBasePath.replaceAll(SEP, "/"); + if (!alreadySynced.includes(remote)) { + alreadySynced.push(remote); + log.info(`Archiving script ${remote}`); + await wmill + .archiveScriptByPath({ workspace: workspace.workspaceId, path: remote }) + .catch((e: any) => { + // Only "already gone" is the state we wanted. An auth, network or + // server failure must fail the push: swallowing it reports success + // while the project stays deployed, which is the thing this branch + // exists to prevent. + if (e?.status !== 404) throw e; + log.debug(`${remote} was already gone remotely`); + }); + } + return; + } + let contentPath: string | undefined; + try { + contentPath = await findContentFile(scriptBasePath + ".script.yaml"); + } catch (e) { + // A path claimed by two scripts is not a parent that cannot be found: + // swallowed here, `wmill sync push` reports success on a model edit that + // deployed nothing, and the collision stays invisible until the ordinary + // script overwrites the project. + if (e instanceof DbtPathCollisionError) throw e; + log.debug(`Could not find parent script for dbt module: ${modulePath}`); + return; + } + if (contentPath) { + await handleFile( + contentPath, + workspace, + alreadySynced, + message, + opts, + rawWorkspaceDependencies, + codebases, + ); + } + return; + } // Try folder layout first: look for script.{ext} inside __mod/ try { @@ -2958,11 +3357,16 @@ export async function pull( let specificItems = getSpecificItemsForCurrentBranch(opts, wsNameForConfig); // Compute the workspace name for file naming (default to workspaceId) - let wsNameForFiles = wsNameForConfig ? resolveWsNameForFiles(opts, wsNameForConfig) : workspace.workspaceId; + let wsNameForFiles = wsNameForConfig + ? resolveWsNameForFiles(opts, wsNameForConfig) + : workspace.workspaceId; // Augment specificItems with server-side ws_specific entries const localSpecificItems = specificItems; - const wsSpecificMerge = await mergeWsSpecificFromServer(workspace.workspaceId, specificItems); + const wsSpecificMerge = await mergeWsSpecificFromServer( + workspace.workspaceId, + specificItems, + ); specificItems = wsSpecificMerge.merged; // Merge CLI flags with resolved settings (CLI flags take precedence only for explicit overrides) @@ -3003,7 +3407,9 @@ export async function pull( opts.includeSettings, opts.includeKey, opts.skipWorkspaceDependencies, + opts.skipDatatableMigrations, opts.defaultTs, + opts.syncBehavior, ); const remote = ZipFSElement( @@ -3108,7 +3514,8 @@ export async function pull( const conflicts = []; log.info(colors.gray(`Applying changes to files ...`)); - for await (const change of changes) { + for await (const rawChange of changes) { + const change: Change = rawChange; // Determine if this file should be written to a workspace-specific path let targetPath = change.path; if (specificItems && isSpecificItem(change.path, specificItems)) { @@ -3124,6 +3531,28 @@ export async function pull( const target = path.join(process.cwd(), targetPath); const stateTarget = path.join(process.cwd(), ".wmill", targetPath); + // An empty dbt descriptor is not a file: the remote spells "this project + // named no descriptor" as empty content, and writing that would put a + // Windmill file inside a project that has none. ABSENCE is the state to + // reach, so both copies are removed if present and their being missing — + // a project pulled for the first time — is the goal, not an error. The + // `.wmill` copy goes too, or the same change is reported on every pull. + // + // `force` covers the missing file and NOTHING else: a permission or + // read-only-filesystem failure has to surface, or the pull reports + // success while the old descriptor — its warehouse, its command, its + // arguments — is still what runs locally. + if ( + isDbtDescriptorPath(change.path) && + ((change.name === "added" && change.content === "") || + (change.name === "edited" && change.after === "")) + ) { + await rm(target, { force: true }); + if (opts.stateful) { + await rm(stateTarget, { force: true }); + } + continue; + } if (change.name === "edited") { if (opts.stateful) { try { @@ -3165,11 +3594,13 @@ export async function pull( // ignore } } - if (exts.some((e) => change.path.endsWith(e))) { + if (hasScriptExt(change.path)) { log.info( `Editing script content of ${targetPath}${ targetPath !== change.path - ? colors.gray(` (workspace-specific override for ${change.path})`) + ? colors.gray( + ` (workspace-specific override for ${change.path})`, + ) : "" }`, ); @@ -3180,7 +3611,9 @@ export async function pull( log.info( `Editing ${changeTypeLabel(change.path)}${targetPath}${ targetPath !== change.path - ? colors.gray(` (workspace-specific override for ${change.path})`) + ? colors.gray( + ` (workspace-specific override for ${change.path})`, + ) : "" }`, ); @@ -3198,7 +3631,9 @@ export async function pull( log.info( `Adding ${changeTypeLabel(change.path)}${targetPath}${ targetPath !== change.path - ? colors.gray(` (workspace-specific override for ${change.path})`) + ? colors.gray( + ` (workspace-specific override for ${change.path})`, + ) : "" }`, ); @@ -3215,18 +3650,15 @@ export async function pull( await copyFile(target, stateTarget); } } else if (change.name === "deleted") { - try { - log.info( - `Deleting ${changeTypeLabel(change.path)}${change.path}`, - ); - await rm(target); - if (opts.stateful) { - await rm(stateTarget); - } - } catch { - if (opts.stateful) { - await rm(stateTarget); - } + log.info(`Deleting ${changeTypeLabel(change.path)}${change.path}`); + // `force` on both: the goal is that neither copy exists, and a file + // already absent — a dbt project's optional descriptor is never written + // — is that goal, not an error. Anything else (permissions, a read-only + // mount) surfaces rather than leaving a file the sync believes is gone. + // The state copy goes too, or the same deletion replays on every sync. + await rm(target, { force: true }); + if (opts.stateful) { + await rm(stateTarget, { force: true }); } } } @@ -3362,7 +3794,8 @@ export async function pull( try { // Dynamic import to avoid a circular dep between sync.ts and // generate-metadata.ts. Don't "clean up" to a static import. - const { rehashOnly } = await import("../generate-metadata/generate-metadata.ts"); + const { rehashOnly } = + await import("../generate-metadata/generate-metadata.ts"); // Reuse the local-side file list from the change-tracker so we don't // re-walk the filesystem. Apply the just-applied changes to derive the // post-pull state: localMap is pre-pull, but auto-fill needs to see @@ -3381,7 +3814,7 @@ export async function pull( log.info( colors.gray( `Auto-filled ${total} missing lockfile entr${total === 1 ? "y" : "ies"} ` + - `(${filled.scripts} script, ${filled.flows} flow, ${filled.apps} app) from disk.`, + `(${filled.scripts} script, ${filled.flows} flow, ${filled.apps} app) from disk.`, ), ); } @@ -3469,17 +3902,14 @@ export async function gitDeploy( // the wm_deploy branch). Mirrors the hub script's `--promotion `. const promotion = useIndividualBranch && !opts.promotion - ? getCurrentGitBranch() ?? undefined + ? (getCurrentGitBranch() ?? undefined) : opts.promotion; await pull({ ...opts, yes: true, skipBranchValidation: true, - extraIncludes: [ - ...(opts.extraIncludes ?? []), - ...includes.extraIncludes, - ], + extraIncludes: [...(opts.extraIncludes ?? []), ...includes.extraIncludes], // Workspace-wide mode force-includes the deployed default-excluded kinds // (full mirror). Individual-branch/promotion mode forces nothing — these // keys stay ABSENT so pull resolves them from the promotion target's @@ -3527,7 +3957,9 @@ function prettyChanges( const folderNote = folderDefaultAnnotations?.get(change.path); const extraNote = folderNote - ? colors.cyan(` (will be permissioned as ${folderNote} via folder default)`) + ? colors.cyan( + ` (will be permissioned as ${folderNote} via folder default)`, + ) : ""; if (change.name === "added") { @@ -3672,14 +4104,12 @@ async function checkServerLockJobs( const pending = (queued as { script_path?: string }[]).filter((j) => belongsToPush(j.script_path), ).length; - const failed = ( - completed as { script_path?: string; result?: unknown }[] - ) + const failed = (completed as { script_path?: string; result?: unknown }[]) .filter((j) => belongsToPush(j.script_path)) .map((j) => ({ path: j.script_path!, - error: (j.result as { error?: { message?: string } } | undefined) - ?.error?.message, + error: (j.result as { error?: { message?: string } } | undefined)?.error + ?.message, })); return { pending, failed }; } catch { @@ -3689,9 +4119,15 @@ async function checkServerLockJobs( } export async function push( - opts: GlobalOptions & SyncOptions & { repository?: string; branch?: string; acceptOverridingPermissionedAsWithSelf?: boolean }, + opts: GlobalOptions & + SyncOptions & { + repository?: string; + branch?: string; + acceptOverridingPermissionedAsWithSelf?: boolean; + }, ) { if ((opts as any).jsonOutput) log.setSilent(true); + markRequestsAsSyncOrigin(); // Save original CLI options before merging with config file const originalCliOpts = { ...opts }; @@ -3752,7 +4188,9 @@ export async function push( let specificItems = getSpecificItemsForCurrentBranch(opts, wsNameForConfig); // Compute the workspace name for file naming (default to workspaceId) - let wsNameForFiles = wsNameForConfig ? resolveWsNameForFiles(opts, wsNameForConfig) : workspace.workspaceId; + let wsNameForFiles = wsNameForConfig + ? resolveWsNameForFiles(opts, wsNameForConfig) + : workspace.workspaceId; // Keep the pre-merge specificItems so we can detect entries that are // flagged locally but not yet ws_specific on the server (post-merge would @@ -3760,7 +4198,10 @@ export async function push( const localSpecificItems = specificItems; // Augment specificItems with server-side ws_specific entries - const wsSpecificMerge = await mergeWsSpecificFromServer(workspace.workspaceId, specificItems); + const wsSpecificMerge = await mergeWsSpecificFromServer( + workspace.workspaceId, + specificItems, + ); specificItems = wsSpecificMerge.merged; const serverWsSpecificItems = wsSpecificMerge.serverItems; @@ -3844,7 +4285,9 @@ export async function push( opts.includeSettings, opts.includeKey, opts.skipWorkspaceDependencies, + opts.skipDatatableMigrations, opts.defaultTs, + opts.syncBehavior, ))!, !opts.json, opts.defaultTs ?? "bun", @@ -3854,7 +4297,11 @@ export async function push( parseSyncBehavior(opts.syncBehavior) >= 1, ); - const local = await FSFSElement(path.join(process.cwd(), ""), codebases, false); + const local = await FSFSElement( + path.join(process.cwd(), ""), + codebases, + false, + ); const { changes, localMap } = await compareDynFSElement( local, remote, @@ -4120,14 +4567,20 @@ export async function push( let triggerCount = 0; for await (const entry of readDirRecursiveWithIgnore(() => false, local)) { if (entry.isDirectory) continue; - if (!opts.includeSchedules && entry.path.endsWith(".schedule.yaml")) scheduleCount++; - if (!opts.includeTriggers && entry.path.endsWith("_trigger.yaml")) triggerCount++; + if (!opts.includeSchedules && entry.path.endsWith(".schedule.yaml")) + scheduleCount++; + if (!opts.includeTriggers && entry.path.endsWith("_trigger.yaml")) + triggerCount++; } if (scheduleCount > 0) { - skippedWarnings.push(`Skipping ${scheduleCount} schedule file(s). Use --include-schedules or set includeSchedules: true in wmill.yaml`); + skippedWarnings.push( + `Skipping ${scheduleCount} schedule file(s). Use --include-schedules or set includeSchedules: true in wmill.yaml`, + ); } if (triggerCount > 0) { - skippedWarnings.push(`Skipping ${triggerCount} trigger file(s). Use --include-triggers or set includeTriggers: true in wmill.yaml`); + skippedWarnings.push( + `Skipping ${triggerCount} trigger file(s). Use --include-triggers or set includeTriggers: true in wmill.yaml`, + ); } for (const warning of skippedWarnings) { log.warn(warning); @@ -4137,6 +4590,44 @@ export async function push( await fetchRemoteVersion(workspace); + const recordedMigrationPaths: RecordedMigrationPaths = changes.some( + (c) => c.name === "deleted" && isDatatableMigrationPath(c.path), + ) + ? gitRecordedDatatableMigrationPaths() + : { kind: "known", paths: new Set() }; + const ambiguousMigrationDeletions = untrackedDatatableMigrationDeletions( + changes, + recordedMigrationPaths, + ); + const keepAmbiguousMigrationsOnRemote = () => { + log.info( + colors.yellow( + `Keeping ${countDatatableMigrationRecords(ambiguousMigrationDeletions)} data table migration(s) on the remote: ` + + (recordedMigrationPaths.kind === "known" + ? `this branch has never tracked them. Run 'wmill sync pull' to track them in git, or delete them from the workspace.` + : `${recordedMigrationPaths.reason}, so whether it ever tracked them cannot be established. ` + + `${recordedMigrationPaths.remedy} so a real deletion can be told apart, or delete them from the workspace.`), + ), + ); + const kept = changes.filter( + (c) => !ambiguousMigrationDeletions.includes(c), + ); + changes.length = 0; + changes.push(...kept); + }; + // An unattended run never resolves this ambiguity destructively, and a dry-run + // preview has to show what a push would really do — settle both before the + // change list is printed or serialized. A TTY push asks instead, after the + // user has seen the list. + let ambiguousMigrationsResolved = false; + if ( + ambiguousMigrationDeletions.length > 0 && + (opts.dryRun || opts.yes || !process.stdin.isTTY) + ) { + keepAmbiguousMigrationsOnRemote(); + ambiguousMigrationsResolved = true; + } + // Shared UI (the ui/ folder) is pushed out-of-band via pushSharedUi on apply // and is excluded from the file diff (isNotWmillFile), so surface its diff in // the dry-run preview. Without this the "Pull from repo" preview reads "no @@ -4218,7 +4709,18 @@ export async function push( `Run 'wmill folder add-missing' to create them locally, then push again.`; if (!userIsAdmin) { if (opts.jsonOutput) { - console.log(JSON.stringify({ success: false, error: "missing_folders", missing_folders: missingFolders, message: msg }, null, 2)); + console.log( + JSON.stringify( + { + success: false, + error: "missing_folders", + missing_folders: missingFolders, + message: msg, + }, + null, + 2, + ), + ); } else { log.error(msg); } @@ -4251,9 +4753,7 @@ export async function push( : {}), })), total: changes.length, - ...(changes.length > 0 - ? { warning: SYNC_PUSH_DESTRUCTIVE_WARNING } - : {}), + ...(changes.length > 0 ? { warning: SYNC_PUSH_DESTRUCTIVE_WARNING } : {}), }; console.log(JSON.stringify(result, null, 2)); return; @@ -4264,7 +4764,10 @@ export async function push( let folderDefaultAnnotations: Map | undefined; if (parseSyncBehavior(opts.syncBehavior) >= 1) { folderDefaultAnnotations = new Map(); - const folderRulesCache = new Map>(); + const folderRulesCache = new Map< + string, + Array<{ path_glob: string; permissioned_as: string }> + >(); for (const change of changes) { if (change.name !== "added") continue; const match = change.path.match(/^f\/([^/]+)\//); @@ -4272,14 +4775,26 @@ export async function push( const folderName = match[1]; if (!folderRulesCache.has(folderName)) { try { - const folder = await wmill.getFolder({ workspace: workspace.workspaceId, name: folderName }); - folderRulesCache.set(folderName, (folder as any).default_permissioned_as ?? []); + const folder = await wmill.getFolder({ + workspace: workspace.workspaceId, + name: folderName, + }); + folderRulesCache.set( + folderName, + (folder as any).default_permissioned_as ?? [], + ); } catch { folderRulesCache.set(folderName, []); } } const rules = folderRulesCache.get(folderName)!; - const remotePath = change.path.replace(/\.(script|schedule|http_trigger|websocket_trigger|kafka_trigger|nats_trigger|postgres_trigger|mqtt_trigger|amqp_trigger|sqs_trigger|gcp_trigger|azure_trigger|email_trigger)\.(yaml|json)$/, "").replace(/(\.flow|__flow)\/flow\.(yaml|json)$/, "").replace(/\.(app|raw_app)(\/app\.(yaml|json))?$/, ""); + const remotePath = change.path + .replace( + /\.(script|schedule|http_trigger|websocket_trigger|kafka_trigger|nats_trigger|postgres_trigger|mqtt_trigger|amqp_trigger|sqs_trigger|gcp_trigger|azure_trigger|email_trigger)\.(yaml|json)$/, + "", + ) + .replace(/(\.flow|__flow)\/flow\.(yaml|json)$/, "") + .replace(/\.(app|raw_app)(\/app\.(yaml|json))?$/, ""); const relative = remotePath.slice(`f/${folderName}/`.length); if (!relative) continue; for (const rule of rules) { @@ -4292,7 +4807,12 @@ export async function push( } if (!opts.jsonOutput) { - prettyChanges(changes, specificItems, wsNameForFiles, folderDefaultAnnotations); + prettyChanges( + changes, + specificItems, + wsNameForFiles, + folderDefaultAnnotations, + ); } if (opts.dryRun) { @@ -4306,7 +4826,9 @@ export async function push( const user = await wmill.whoami({ workspace: workspace.workspaceId }); const userIsAdminOrDeployer = user.is_admin || (user.groups ?? []).includes("wm_deployers"); - log.debug(`permissioned_as: user=${user.email}, is_admin=${user.is_admin}, groups=${JSON.stringify(user.groups)}, isAdminOrDeployer=${userIsAdminOrDeployer}`); + log.debug( + `permissioned_as: user=${user.email}, is_admin=${user.is_admin}, groups=${JSON.stringify(user.groups)}, isAdminOrDeployer=${userIsAdminOrDeployer}`, + ); permissionedAsContext = { userCache: new Map(), userIsAdminOrDeployer, @@ -4324,10 +4846,12 @@ export async function push( !!process.stdin.isTTY, ); } else if (folderDefaultAnnotations && folderDefaultAnnotations.size > 0) { - log.warn(colors.yellow( - `This workspace has folder default_permissioned_as rules that affect ${folderDefaultAnnotations.size} item(s) being pushed, ` + - `but syncBehavior is not set in wmill.yaml. Add 'syncBehavior: v1' to enable ownership preservation on update and on_behalf_of stripping on pull.` - )); + log.warn( + colors.yellow( + `This workspace has folder default_permissioned_as rules that affect ${folderDefaultAnnotations.size} item(s) being pushed, ` + + `but syncBehavior is not set in wmill.yaml. Add 'syncBehavior: v1' to enable ownership preservation on update and on_behalf_of stripping on pull.`, + ), + ); } // Reject malformed datatable migrations (duplicate timestamps, orphan downs) @@ -4358,6 +4882,18 @@ export async function push( return; } + if (ambiguousMigrationDeletions.length > 0 && !ambiguousMigrationsResolved) { + const deleteThem = await Confirm.prompt({ + message: + `Nothing in this repository's history accounts for ${countDatatableMigrationRecords(ambiguousMigrationDeletions)} migration definition(s), so it may simply never have synced them. ` + + `Delete them from the workspace anyway?`, + default: false, + }); + if (!deleteThem) { + keepAmbiguousMigrationsOnRemote(); + } + } + const start = performance.now(); const pushStartedAt = new Date().toISOString(); log.info(colors.gray(`Applying changes to files ...`)); @@ -4374,7 +4910,14 @@ export async function push( // Group changes by base path (before first dot) const groupedChanges = new Map(); for (const change of changes) { - const basePath = change.path.split(".")[0]; + // A module file is pushed by pushing its parent script, so it belongs in + // that script's group. Left in a group of its own it gets its own + // `alreadySynced`, and a push touching several files of one bundle then + // deploys the script once per file: several versions in a row, of which + // only the last is the one the asset graph ends up describing. + const basePath = + getScriptBasePathFromModulePath(change.path) ?? + change.path.split(".")[0]; if (!groupedChanges.has(basePath)) { groupedChanges.set(basePath, []); } @@ -4422,7 +4965,8 @@ export async function push( const effectiveParallelism = () => folderPhaseRemaining > 0 ? 1 : parallelizationFactor; // Cache git branch at the start to avoid repeated execSync calls per change - const cachedWsNameForPush = wsNameForFiles || (isGitRepository() ? getCurrentGitBranch() : null); + const cachedWsNameForPush = + wsNameForFiles || (isGitRepository() ? getCurrentGitBranch() : null); // Datatable migrations are two files (.up.sql/.down.sql) for one record, so // dedupe upsert/delete by (datatable, version) across the whole push. @@ -4461,6 +5005,10 @@ export async function push( if (deleteRawApp) { changes = [deleteRawApp]; } else { + // The app is one bundle, so a single change re-pushes all of it. + // That leaves the loop exactly one change: any skip it takes for + // a raw-app path drops the whole app from the push, and nothing + // downstream records that as a failure. changes.splice(1, changes.length - 1); } } @@ -4562,13 +5110,19 @@ export async function push( const currentBranch = cachedWsNameForPush; let isFileResWsSpecific = false; - if (currentBranch && isWorkspaceSpecificFile(resourceFilePath)) { + if ( + currentBranch && + isWorkspaceSpecificFile(resourceFilePath) + ) { serverPath = fromWorkspaceSpecificPath( resourceFilePath, currentBranch, ); isFileResWsSpecific = true; - } else if (specificItems && isSpecificItem(change.path, specificItems)) { + } else if ( + specificItems && + isSpecificItem(change.path, specificItems) + ) { isFileResWsSpecific = true; } @@ -4612,7 +5166,8 @@ export async function push( // Check if this is a branch-specific item and get the original workspace-specific path let originalWorkspaceSpecificPath: string | undefined; - const isWsSpecific = specificItems && isSpecificItem(change.path, specificItems); + const isWsSpecific = + specificItems && isSpecificItem(change.path, specificItems); if (isWsSpecific) { originalWorkspaceSpecificPath = getWorkspaceSpecificPath( change.path, @@ -4628,13 +5183,17 @@ export async function push( newObj, opts.plainSecrets ?? false, alreadySynced, - opts.message, - originalWorkspaceSpecificPath, - permissionedAsContext, - isWsSpecific ? true : undefined, { - noninteractive: (opts.yes ?? false) || !process.stdin.isTTY, - skipReencrypt: opts.skipReencryptOnKeyChange, + message: opts.message, + originalLocalPath: originalWorkspaceSpecificPath, + permissionedAsContext, + wsSpecific: isWsSpecific ? true : undefined, + keyPushOpts: { + noninteractive: + (opts.yes ?? false) || !process.stdin.isTTY, + skipReencrypt: opts.skipReencryptOnKeyChange, + }, + defaultTs: opts.defaultTs, }, ); @@ -4695,7 +5254,11 @@ export async function push( !isRawAppFile(change.path) && (change.path.endsWith(".script.json") || change.path.endsWith(".script.yaml") || - change.path.endsWith(".lock") || + // A `.lock` is the script's generated lockfile — except inside + // a dbt bundle, where the project may author one (`uv.lock`). + // Skipping it there would report the add on every push and + // never apply it, because no state file is written either. + (change.path.endsWith(".lock") && !isDbtModulePath(change.path)) || isFileResource(change.path)) ) { continue; @@ -4735,7 +5298,8 @@ export async function push( // Determine the actual local file path for this change // For branch-specific items, we read from workspace-specific files but push to base server paths let localFilePath = change.path; - const isAddedWsSpecific = specificItems && isSpecificItem(change.path, specificItems); + const isAddedWsSpecific = + specificItems && isSpecificItem(change.path, specificItems); if (isAddedWsSpecific) { const workspaceSpecificPath = getWorkspaceSpecificPath( change.path, @@ -4754,13 +5318,17 @@ export async function push( obj, opts.plainSecrets ?? false, [], - opts.message, - localFilePath, // Pass the actual local file path - permissionedAsContext, - isAddedWsSpecific ? true : undefined, { - noninteractive: (opts.yes ?? false) || !process.stdin.isTTY, - skipReencrypt: opts.skipReencryptOnKeyChange, + message: opts.message, + originalLocalPath: localFilePath, + permissionedAsContext, + wsSpecific: isAddedWsSpecific ? true : undefined, + keyPushOpts: { + noninteractive: + (opts.yes ?? false) || !process.stdin.isTTY, + skipReencrypt: opts.skipReencryptOnKeyChange, + }, + defaultTs: opts.defaultTs, }, ); @@ -4768,7 +5336,15 @@ export async function push( await writeFile(stateTarget, change.content, "utf-8"); } } else if (change.name === "deleted") { - if (change.path.endsWith(".lock")) { + // Same as the added branch: a dbt project's own `.lock` is one of + // its files, so deleting it has to reach the parent script. A raw + // app's `.lock` is part of its bundle, and a raw-app group is + // collapsed to one change, so skipping it drops the whole app. + if ( + !isRawAppFile(change.path) && + change.path.endsWith(".lock") && + !isDbtModulePath(change.path) + ) { continue; } if (isScriptModulePath(change.path)) { @@ -4828,8 +5404,13 @@ export async function push( path: resourcePath, }); } catch (e: any) { - if (e?.status === 404 && deletedVarsResPaths.includes(resourcePath)) { - log.debug(`Resource ${resourcePath} already deleted by linked variable`); + if ( + e?.status === 404 && + deletedVarsResPaths.includes(resourcePath) + ) { + log.debug( + `Resource ${resourcePath} already deleted by linked variable`, + ); } else { throw e; } @@ -4848,7 +5429,10 @@ export async function push( // Metadata file deleted — delete the entire flow await wmill.deleteFlowByPath({ workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("flow", "json")), + path: removeSuffix( + target, + getDeleteSuffix("flow", "json"), + ), }); } else { // Inline script file deleted within flow folder @@ -4871,7 +5455,7 @@ export async function push( undefined, opts.plainSecrets ?? false, alreadySynced, - opts.message, + { message: opts.message }, ); } else { // Flow folder doesn't exist locally — delete on server @@ -4890,7 +5474,10 @@ export async function push( // Metadata file deleted — delete the entire app await wmill.deleteApp({ workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("app", "json")), + path: removeSuffix( + target, + getDeleteSuffix("app", "json"), + ), }); } else { // Inline script file deleted within app folder @@ -4913,7 +5500,7 @@ export async function push( undefined, opts.plainSecrets ?? false, alreadySynced, - opts.message, + { message: opts.message }, ); } else { // App folder doesn't exist locally — delete on server @@ -4932,7 +5519,10 @@ export async function push( // Delete the entire raw app await wmill.deleteApp({ workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("raw_app", "json")), + path: removeSuffix( + target, + getDeleteSuffix("raw_app", "json"), + ), }); } else { const rawAppFolder = extractFolderPath(target, "raw_app"); @@ -4956,7 +5546,7 @@ export async function push( undefined, opts.plainSecrets ?? false, alreadySynced, - opts.message, + { message: opts.message, defaultTs: opts.defaultTs }, ); } else { // The entire raw app folder was deleted locally, @@ -5047,7 +5637,7 @@ export async function push( const triggerInfo = extractNativeTriggerInfo(change.path); if (!triggerInfo) { throw new Error( - `Invalid native trigger path: ${change.path}` + `Invalid native trigger path: ${change.path}`, ); } await wmill.deleteNativeTrigger({ @@ -5065,8 +5655,13 @@ export async function push( path: variablePath, }); } catch (e: any) { - if (e?.status === 404 && deletedVarsResPaths.includes(variablePath)) { - log.debug(`Variable ${variablePath} already deleted by linked resource`); + if ( + e?.status === 404 && + deletedVarsResPaths.includes(variablePath) + ) { + log.debug( + `Variable ${variablePath} already deleted by linked resource`, + ); } else { throw e; } @@ -5177,10 +5772,14 @@ export async function push( log.warn(`Failed to push shared UI folder: ${e}`); } try { - await offerToRunNewMigrations(workspace.workspaceId, newDatatableMigrations, { - yes: opts.yes, - jsonOutput: opts.jsonOutput, - }); + await offerToRunNewMigrations( + workspace.workspaceId, + newDatatableMigrations, + { + yes: opts.yes, + jsonOutput: opts.jsonOutput, + }, + ); } catch (e: any) { log.warn( `Failed to run new datatable migrations: ${e?.body ?? e?.message ?? e}`, @@ -5314,7 +5913,10 @@ const command = new Command() .option("--json", "Use JSON instead of YAML") .option("--skip-variables", "Skip syncing variables (including secrets)") .option("--skip-secrets", "Skip syncing only secrets variables") - .option("--include-secrets", "Include secrets in sync (overrides skipSecrets in wmill.yaml)") + .option( + "--include-secrets", + "Include secrets in sync (overrides skipSecrets in wmill.yaml)", + ) .option("--skip-resources", "Skip syncing resources") .option("--skip-resource-types", "Skip syncing resource types") .option("--skip-scripts", "Skip syncing scripts") @@ -5370,7 +5972,10 @@ const command = new Command() .option("--json", "Use JSON instead of YAML") .option("--skip-variables", "Skip syncing variables (including secrets)") .option("--skip-secrets", "Skip syncing only secrets variables") - .option("--include-secrets", "Include secrets in sync (overrides skipSecrets in wmill.yaml)") + .option( + "--include-secrets", + "Include secrets in sync (overrides skipSecrets in wmill.yaml)", + ) .option("--skip-resources", "Skip syncing resources") .option("--skip-resource-types", "Skip syncing resource types") .option("--skip-scripts", "Skip syncing scripts") @@ -5424,7 +6029,10 @@ const command = new Command() "--locks-required", "Fail if scripts or flow inline scripts that need locks have no locks", ) - .option("--auto-metadata", "Automatically regenerate stale metadata (locks and schemas) before pushing") + .option( + "--auto-metadata", + "Automatically regenerate stale metadata (locks and schemas) before pushing", + ) .option( "--accept-overriding-permissioned-as-with-self", "Accept that items with a different permissioned_as will be updated with your own user", @@ -5463,7 +6071,7 @@ const command = new Command() ) .option( "--dev-workspace-label ", - "Environment label of a dev workspace (dev/staging); its deploys go to that branch", + "Environment label of a dev workspace (dev, staging, uat, ...); its deploys go to that branch", ) .option( "--parent-dev-workspace-label ", diff --git a/cli/src/commands/workspace/merge.ts b/cli/src/commands/workspace/merge.ts index 585772a9d0..9f39bec52c 100644 --- a/cli/src/commands/workspace/merge.ts +++ b/cli/src/commands/workspace/merge.ts @@ -2,7 +2,7 @@ import { GlobalOptions } from "../../types.ts"; import { colors } from "@cliffy/ansi/colors"; import { Table } from "@cliffy/table"; import * as log from "../../core/log.ts"; -import { setClient } from "../../core/client.ts"; +import { markRequestsAsSyncOrigin, setClient } from "../../core/client.ts"; import { tryResolveBranchWorkspace } from "../../core/context.ts"; import * as wmill from "../../../gen/services.gen.ts"; import { OpenAPI } from "../../../gen/core/OpenAPI.ts"; @@ -534,6 +534,12 @@ async function mergeWorkspaces( direction === "to-parent" ? parentWorkspaceId : forkWorkspaceId; // 10. Deploy + // Updating the fork copies the parent's state in, so nothing it writes there — + // least of all a deletion — may be read as the fork's own decision. Merging the + // other way stays authored: someone chose those changes for the target. + if (direction === "to-fork") { + markRequestsAsSyncOrigin(); + } let successCount = 0; let failCount = 0; // Datatable migrations deployed (not deleted) into the target. Deploying a diff --git a/cli/src/core/client.ts b/cli/src/core/client.ts index 4b0b98e30f..f7073fe501 100644 --- a/cli/src/core/client.ts +++ b/cli/src/core/client.ts @@ -1,5 +1,19 @@ import { OpenAPI } from "../../gen/index.ts"; +/** + * Mark every subsequent request as applying a state computed elsewhere rather + * than authoring one in the target workspace (the fork tally reads the header to + * decide whether a removal was deliberate). + * + * Belongs to the commands that apply and nothing that authors: `sync push` + * (including the git-sync auto-pull, which runs it inside a job) and the + * parent-to-fork half of `workspace merge`. + */ +export function markRequestsAsSyncOrigin() { + const existing = typeof OpenAPI.HEADERS === "object" ? OpenAPI.HEADERS : {}; + OpenAPI.HEADERS = { ...existing, "X-Windmill-Deploy-Origin": "sync" }; +} + export function setClient(token?: string, baseUrl?: string) { if (baseUrl === undefined) { baseUrl = process.env["BASE_INTERNAL_URL"] ?? diff --git a/cli/src/core/conf.ts b/cli/src/core/conf.ts index 0129a4fd18..ebad7bc51c 100644 --- a/cli/src/core/conf.ts +++ b/cli/src/core/conf.ts @@ -78,6 +78,7 @@ export interface SyncOptions { skipResourceTypes?: boolean; skipSecrets?: boolean; skipWorkspaceDependencies?: boolean; + skipDatatableMigrations?: boolean; skipScripts?: boolean; skipFlows?: boolean; skipApps?: boolean; @@ -343,6 +344,7 @@ export const DEFAULT_SYNC_OPTIONS: Readonly< | "includeSchedules" | "includeTriggers" | "skipWorkspaceDependencies" + | "skipDatatableMigrations" | "skipScripts" | "skipFlows" | "skipApps" @@ -375,6 +377,7 @@ export const DEFAULT_SYNC_OPTIONS: Readonly< includeSettings: false, includeKey: false, skipWorkspaceDependencies: false, + skipDatatableMigrations: false, nonDottedPaths: false, syncBehavior: "v1", } as const; diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index 59de21e2bf..ccfc78dbef 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.775.2"; +export const VERSION = "1.783.0"; diff --git a/cli/src/core/context.ts b/cli/src/core/context.ts index 6440213d5c..502157f0a7 100644 --- a/cli/src/core/context.ts +++ b/cli/src/core/context.ts @@ -22,8 +22,11 @@ import { readConfigFile, findWorkspaceByGitBranch, getEffectiveWorkspaceId, + getWmillYamlPath, WorkspaceEntryConfig, } from "./conf.ts"; +import { existsSync, realpathSync } from "node:fs"; +import { basename, dirname, isAbsolute, join, relative, resolve } from "node:path"; import { getCurrentGitBranch, getOriginalBranchForWorkspaceForks, @@ -743,6 +746,92 @@ export async function tryResolveVersion( } } +/** + * Directory the local tree mirrors the workspace from: the one holding + * wmill.yaml. Not `process.cwd()` — that only lands there once a config read + * has chdir'd into it, which `--remote` and fully-flagged invocations skip. + */ +function syncRoot(): string { + const wmillYaml = getWmillYamlPath(); + return wmillYaml ? dirname(wmillYaml) : process.cwd(); +} + +/** + * Re-express a user-supplied file or folder argument as a path relative to the + * sync root, so the Windmill path derived from it is the same whatever shape + * the argument had (`./f/a/b.ts`, `/abs/repo/f/a/b.ts`, `b.ts` from inside + * `f/a`). + * + * `cwdBeforeConfig` must be the working directory as it was *before* the + * command read wmill.yaml: reading it chdirs into the directory holding it, and + * a relative argument was written against the directory the user was in. + * Arguments are resolved against that directory first and the sync root second, + * so both readings work. + * + * Resolving the sync root leaves the process in it, which is what makes the + * returned path readable by the caller — keep that in step if this ever stops + * going through `getWmillYamlPath`. + */ +export function toSyncRootRelativePath( + arg: string, + cwdBeforeConfig: string +): string { + const root = syncRoot(); + const candidates = isAbsolute(arg) + ? [arg] + : [resolve(cwdBeforeConfig, arg), resolve(root, arg)]; + // A descriptor-less dbt project is named by a file that is deliberately not + // there, so an argument existing under neither reading is still a real one if + // the directory holding it is; only a path whose directory is missing too + // falls through to the sync-root reading for the caller to reject. + const abs = + candidates.find((c) => existsSync(c)) ?? + candidates.find((c) => existsSync(dirname(c))) ?? + candidates.at(-1)!; + const rel = relative(root, abs); + if (rel === "") return "."; + if (!rel.startsWith("..")) return rel; + // `relative` is purely lexical, so a root reached through a symlink (macOS' + // /var -> /private/var, a symlinked checkout) makes an absolute argument look + // like it escapes the tree. Resolve links only then, so a symlinked file + // *inside* the tree keeps the path it is filed under. + try { + const resolved = relative(realpathSync(root), realpathOfNamed(abs)); + return resolved === "" ? "." : resolved; + } catch { + return rel; + } +} + +/** + * `realpathSync` needs its target to exist, and a dbt descriptor deliberately + * does not. The directory naming it does, so resolve that and reattach. + */ +function realpathOfNamed(p: string): string { + return existsSync(p) + ? realpathSync(p) + : join(realpathSync(dirname(p)), basename(p)); +} + +/** Windmill workspace path: `u|f|g` followed by at least a folder and a name. */ +const REMOTE_PATH_RE = /^[ufg](\/[^/]+){2,}$/; + +/** + * Guard the Windmill path a preview run is pushed under. A preview job carries + * no runnable of its own, so this path is the only identity it has: it is what + * `WM_JOB_PATH` reports, what the runs page links to, and what relative imports + * inside the previewed code resolve against. + */ +export function assertRemotePath(remotePath: string, arg: string): void { + if (REMOTE_PATH_RE.test(remotePath)) return; + throw new Error( + `Cannot derive a Windmill path from '${arg}'` + + (remotePath ? ` (it maps to '${remotePath}')` : "") + + `: a preview runs under the path of the file it previews, which must sit inside the ` + + `wmill.yaml root and be of the form //.` + ); +} + export function validatePath(path: string): boolean { if (!(path.startsWith("g") || path.startsWith("u") || path.startsWith("f"))) { log.infoStderr( diff --git a/cli/src/guidance/skills.gen.ts b/cli/src/guidance/skills.gen.ts index 50ac0abc9f..71047bbec5 100644 --- a/cli/src/guidance/skills.gen.ts +++ b/cli/src/guidance/skills.gen.ts @@ -547,6 +547,11 @@ const result: wmill.S3Object = await wmill.writeS3File( Import: import * as wmill from 'windmill-client' +To know who is running the script, read the contextual variables rather than calling the API: +\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when +the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL +is the user the job is permissioned as. WM_USERNAME is the matching username. + workerHasInternalServer(): boolean /** @@ -577,11 +582,6 @@ async getResource(path?: string, undefinedIfEmpty?: boolean): Promise */ async getRootJobId(jobId?: string): Promise -/** - * @deprecated Use runScriptByPath or runScriptByHash instead - */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise - /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill @@ -646,11 +646,6 @@ async getResult(jobId: string): Promise */ async getResultMaybe(jobId: string): Promise -/** - * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead - */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise - /** * Run a script asynchronously by its path * @param path - Script path in Windmill @@ -703,13 +698,6 @@ getStatePath(): string */ async setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise -/** - * Set the state - * @param state state to set - * @deprecated use setState instead - */ -async setInternalState(state: any): Promise - /** * Set the state * @param state state to set @@ -745,12 +733,6 @@ async setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): P */ async getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise -/** - * Get the internal state - * @deprecated use getState instead - */ -async getInternalState(): Promise - /** * Get the state shared across executions * @param path Optional state resource path override. Defaults to \`getStatePath()\`. @@ -886,15 +868,6 @@ async getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ cancel: string; }> -/** - * @deprecated use getResumeUrls instead - */ -getResumeEndpoints(approver?: string): Promise<{ - approvalPage: string; - resume: string; - cancel: string; -}> - /** * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only) * @param audience audience of the token @@ -917,15 +890,6 @@ base64ToUint8Array(data: string): Uint8Array */ uint8ArrayToBase64(arrayBuffer: Uint8Array): string -/** - * Get email from workspace username - * This method is particularly useful for apps that require the email address of the viewer. - * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. - * @param username - * @returns email address - */ -async usernameToEmail(username: string): Promise - /** * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields. * @@ -1347,6 +1311,11 @@ const result: wmill.S3Object = await wmill.writeS3File( Import: import * as wmill from 'windmill-client' +To know who is running the script, read the contextual variables rather than calling the API: +\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when +the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL +is the user the job is permissioned as. WM_USERNAME is the matching username. + workerHasInternalServer(): boolean /** @@ -1377,11 +1346,6 @@ async getResource(path?: string, undefinedIfEmpty?: boolean): Promise */ async getRootJobId(jobId?: string): Promise -/** - * @deprecated Use runScriptByPath or runScriptByHash instead - */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise - /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill @@ -1446,11 +1410,6 @@ async getResult(jobId: string): Promise */ async getResultMaybe(jobId: string): Promise -/** - * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead - */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise - /** * Run a script asynchronously by its path * @param path - Script path in Windmill @@ -1503,13 +1462,6 @@ getStatePath(): string */ async setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise -/** - * Set the state - * @param state state to set - * @deprecated use setState instead - */ -async setInternalState(state: any): Promise - /** * Set the state * @param state state to set @@ -1545,12 +1497,6 @@ async setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): P */ async getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise -/** - * Get the internal state - * @deprecated use getState instead - */ -async getInternalState(): Promise - /** * Get the state shared across executions * @param path Optional state resource path override. Defaults to \`getStatePath()\`. @@ -1686,15 +1632,6 @@ async getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ cancel: string; }> -/** - * @deprecated use getResumeUrls instead - */ -getResumeEndpoints(approver?: string): Promise<{ - approvalPage: string; - resume: string; - cancel: string; -}> - /** * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only) * @param audience audience of the token @@ -1717,15 +1654,6 @@ base64ToUint8Array(data: string): Uint8Array */ uint8ArrayToBase64(arrayBuffer: Uint8Array): string -/** - * Get email from workspace username - * This method is particularly useful for apps that require the email address of the viewer. - * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. - * @param username - * @returns email address - */ -async usernameToEmail(username: string): Promise - /** * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields. * @@ -2241,6 +2169,11 @@ const result: wmill.S3Object = await wmill.writeS3File( Import: import * as wmill from 'windmill-client' +To know who is running the script, read the contextual variables rather than calling the API: +\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when +the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL +is the user the job is permissioned as. WM_USERNAME is the matching username. + workerHasInternalServer(): boolean /** @@ -2271,11 +2204,6 @@ async getResource(path?: string, undefinedIfEmpty?: boolean): Promise */ async getRootJobId(jobId?: string): Promise -/** - * @deprecated Use runScriptByPath or runScriptByHash instead - */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise - /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill @@ -2340,11 +2268,6 @@ async getResult(jobId: string): Promise */ async getResultMaybe(jobId: string): Promise -/** - * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead - */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise - /** * Run a script asynchronously by its path * @param path - Script path in Windmill @@ -2397,13 +2320,6 @@ getStatePath(): string */ async setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise -/** - * Set the state - * @param state state to set - * @deprecated use setState instead - */ -async setInternalState(state: any): Promise - /** * Set the state * @param state state to set @@ -2439,12 +2355,6 @@ async setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): P */ async getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise -/** - * Get the internal state - * @deprecated use getState instead - */ -async getInternalState(): Promise - /** * Get the state shared across executions * @param path Optional state resource path override. Defaults to \`getStatePath()\`. @@ -2580,15 +2490,6 @@ async getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ cancel: string; }> -/** - * @deprecated use getResumeUrls instead - */ -getResumeEndpoints(approver?: string): Promise<{ - approvalPage: string; - resume: string; - cancel: string; -}> - /** * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only) * @param audience audience of the token @@ -2611,15 +2512,6 @@ base64ToUint8Array(data: string): Uint8Array */ uint8ArrayToBase64(arrayBuffer: Uint8Array): string -/** - * Get email from workspace username - * This method is particularly useful for apps that require the email address of the viewer. - * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. - * @param username - * @returns email address - */ -async usernameToEmail(username: string): Promise - /** * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields. * @@ -3965,6 +3857,11 @@ result: S3Object = wmill.write_s3_file( Import: import wmill +To know who is running the script, read the contextual variables rather than calling the API: +\`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")\`. WM_END_USER_EMAIL is the app +viewer when the run was triggered from an app and empty otherwise (both variables are always +defined), WM_EMAIL is the user the job is permissioned as. WM_USERNAME is the matching username. + def worker_has_internal_server() -> bool def get_mocked_api() -> Optional[dict] @@ -4006,11 +3903,6 @@ def post(endpoint, raise_for_status = True, **kwargs) -> httpx.Response # New authentication token string def create_token(duration = dt.timedelta(days=1)) -> str -# Create a script job and return its job id. -# -# .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str - # Create a script job by path and return its job id. def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str @@ -4020,11 +3912,6 @@ def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: i # Create a flow job and return its job id. def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str -# Run script synchronously and return its result. -# -# .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any - # Run script by path synchronously and return its result. def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any @@ -4411,11 +4298,6 @@ def get_approval_urls(step_key: str = 'approval', approver: str = None) -> dict # - The function checks for required environment variables (\`WM_FLOW_JOB_ID\`, \`WM_FLOW_STEP_ID\`) to ensure it is run in the appropriate context. def request_interactive_slack_approval(slack_resource_path: str, channel_id: str, message: str = None, approver: str = None, default_args_json: dict = None, dynamic_enums_json: dict = None) -> None -# Get email from workspace username -# This method is particularly useful for apps that require the email address of the viewer. -# Indeed, in the viewer context WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. -def username_to_email(username: str) -> str - # Send a message to a Microsoft Teams conversation with conversation_id, where success is used to style the message def send_teams_message(conversation_id: str, text: str, success: bool = True, card_block: dict = None) @@ -4449,6 +4331,18 @@ def get_workspace() -> str def get_version() -> str +# Create a script job and return its job ID. +# +# Args: +# hash_or_path: Script hash or path (determined by presence of '/') +# args: Script arguments +# scheduled_in_secs: Delay before execution in seconds +# tag: Override the worker tag the job runs on +# +# Returns: +# Job ID string +def run_script_async(hash_or_path: str, args: Dict[str, Any] = None, scheduled_in_secs: int = None, tag: str = None) -> str + # Run a script synchronously by hash and return its result. # # Args: @@ -5486,7 +5380,7 @@ Reference a specific resource using \`$res:\` prefix: ## OpenFlow Schema -{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable \`ai_agent\` resource (hybrid linking). When set, the agent brain\\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\\nthat resource; the module's input_transforms then only carry the flow-local inputs\\n(user_message/user_attachments).\\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\\nshared resource; overlaid onto the tools' input_transforms at runtime \\u2014 including when\\n\`agent\` is unset, since a step forked for editing keeps these overrides until it is saved\\nback or unlinked.\\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, +{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"Address of the account the flow runs on behalf of. Derived from on_behalf_of on read; accepted on write, where it is resolved to the account it names."},"on_behalf_of":{"type":"string","description":"The flow runs with the permissions of this identity: u/{username}, g/{group}, or a bare email when the username is itself email-shaped. The only stored half of the identity; on_behalf_of_email is derived from it. Omit it when writing and it is resolved from that address instead."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume","dbt"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable \`ai_agent\` resource (hybrid linking). When set, the agent brain\\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\\nthat resource; the module's input_transforms then only carry the flow-local inputs\\n(user_message/user_attachments).\\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\\nshared resource; overlaid onto the tools' input_transforms at runtime \\u2014 including when\\n\`agent\` is unset, since a step forked for editing keeps these overrides until it is saved\\nback or unlinked.\\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, "raw-app": `--- name: raw-app description: MUST use when creating raw apps. @@ -6840,8 +6734,12 @@ app related commands - \`--host \` - Host to bind the dev server to - \`--entry \` - Entry point file (default: index.ts for Svelte/Vue, index.tsx otherwise) - \`--no-open\` - Don't automatically open the browser + - \`--recording\` - Frame the app in a shell with a Record button, to capture a replayable session recording of the app under development - \`app lint [app_folder:string]\` - Lint a raw app folder to validate structure and buildability - \`--fix\` - Attempt to fix common issues (not implemented yet) +- \`app bundle [app_folder:string]\` - Bundle a raw app folder to js/css without deploying it + - \`--out \` - Directory to write bundle.js and bundle.css into (default: /dist) + - \`--no-minify\` - Skip minification - \`app new\` - create a new raw app from a template - \`--summary \` - App summary (short description). Skips the prompt when provided. Triggers non-interactive mode. - \`--path \` - App path (e.g., f/folder/my_app or u/username/my_app). Skips the prompt when provided. Triggers non-interactive mode. diff --git a/cli/src/types.ts b/cli/src/types.ts index f0cd58abbb..6dd2e6ff77 100644 --- a/cli/src/types.ts +++ b/cli/src/types.ts @@ -5,6 +5,7 @@ import * as path from "node:path"; import { sep as SEP } from "node:path"; import { stringify as yamlStringify } from "yaml"; import { yamlParseContent } from "./utils/yaml.ts"; +import { isDbtDescriptorPath } from "./utils/resource_folders.ts"; import { pushApp } from "./commands/app/app.ts"; import { pushFolder } from "./commands/folder/folder.ts"; import { pushFlow } from "./commands/flow/flow.ts"; @@ -174,6 +175,21 @@ function redactString(s: string): string { return s.slice(0, 5) + "*".repeat(s.length - 5); } +export interface PushObjOptions { + /** Optional commit/update message */ + message?: string; + /** The original local file path (used for branch-specific resource file resolution) */ + originalLocalPath?: string; + /** Identity to attribute the push to, for the types that carry one */ + permissionedAsContext?: PermissionedAsContext; + /** Whether the item is workspace-specific */ + wsSpecific?: boolean; + /** encryption_key push: non-interactive flag and explicit re-encryption choice */ + keyPushOpts?: PushWorkspaceKeyOptions; + /** TypeScript runtime a bare `.ts` denotes, for raw-app runnables */ + defaultTs?: "bun" | "deno"; +} + /** * Pushes an object to the workspace server based on its type * @param workspace - The workspace ID to push to @@ -182,9 +198,7 @@ function redactString(s: string): string { * @param newObj - The new object state to push * @param plainSecrets - Whether to store secrets in plain text * @param alreadySynced - Array to track already synced items - * @param message - Optional commit/update message - * @param originalLocalPath - The original local file path (used for branch-specific resource file resolution) - * @param keyPushOpts - Options for the encryption_key push: non-interactive flag and explicit re-encryption choice + * @param opts - Per-type extras; see PushObjOptions */ export async function pushObj( workspace: string, @@ -193,12 +207,16 @@ export async function pushObj( newObj: any, plainSecrets: boolean, alreadySynced: string[], - message?: string, - originalLocalPath?: string, - permissionedAsContext?: PermissionedAsContext, - wsSpecific?: boolean, - keyPushOpts?: PushWorkspaceKeyOptions, + opts: PushObjOptions = {}, ) { + const { + message, + originalLocalPath, + permissionedAsContext, + wsSpecific, + keyPushOpts, + defaultTs, + } = opts; const typeEnding = getTypeStrFromPath(p); if (typeEnding === "app") { @@ -212,7 +230,7 @@ export async function pushObj( if (!rawAppName) { throw new Error(`Could not extract raw app name from path: ${p}`); } - await pushRawApp(workspace, rawAppName, buildFolderPath(rawAppName, "raw_app"), message); + await pushRawApp(workspace, rawAppName, buildFolderPath(rawAppName, "raw_app"), message, defaultTs); } else if (typeEnding === "folder") { await pushFolder(workspace, p, befObj, newObj); } else if (typeEnding === "variable") { @@ -388,7 +406,11 @@ export function getTypeStrFromPath( parsed.ext == ".rb" || parsed.ext == ".r" || // for related places search: ADD_NEW_LANG - (parsed.ext == ".yml" && parsed.name.split(".").pop() == "playbook") + (parsed.ext == ".yml" && parsed.name.split(".").pop() == "playbook") || + // A dbt descriptor is `__dbt/wm_dbt.yaml`. Without this it reads + // as one of the CLI's own `.yaml` metadata files and a pull writes the + // script's metadata and lock but never its content. + isDbtDescriptorPath(p) ) { return "script"; } diff --git a/cli/src/utils/git.ts b/cli/src/utils/git.ts index da1ac14414..5ca06f427a 100644 --- a/cli/src/utils/git.ts +++ b/cli/src/utils/git.ts @@ -75,6 +75,114 @@ export function getWorkspaceIdForWorkspaceForkFromBranchName(branchName: string) return `${WM_FORK_PREFIX}-${branchName.slice(start)}` } +/** + * Whether this checkout can vouch for what it once held under `migrations/datatable/**`. + * + * `known` lists every such path recorded on the current branch (paths stay listed after + * the commit that removed them, so a real deletion is still recognisable). `unknown` + * means a file's absence from the working tree proves nothing, either because the + * history can't be read — no repository, a shallow clone's truncated history, an + * unresolvable root, a failing git — or because the working tree deliberately doesn't + * mirror it, as in a sparse checkout. Both shapes carry the same obligation on the + * caller: trust nothing, rather than read absence as evidence. + */ +export type RecordedMigrationPaths = + | { kind: "known"; paths: Set } + | { kind: "unknown"; reason: string; remedy: string }; + +/** + * Resolve [`RecordedMigrationPaths`] for the checkout at the current directory. + * + * This is the durable answer to "did this checkout ever track that migration?", which + * the working tree cannot give: an absent `migrations/datatable/` is equally a clone + * that never pulled migrations and one where the last was deleted, and creating a + * migration locally (`wmill datatable migrate new`) makes the directory appear without + * anything having been tracked. + * + * Scoped to `HEAD`, not `--all`: a migration that only ever existed on some other + * branch is not evidence that *this* branch ever tracked it, and using it as such would + * authorize deleting it. Output paths are repo-root-relative, so the `--show-prefix` of + * the working directory is stripped to match the cwd-relative paths a sync diff uses. + */ +export function gitRecordedDatatableMigrationPaths(): RecordedMigrationPaths { + if (!isGitRepository()) { + return { + kind: "unknown", + reason: "this directory is not a git repository", + remedy: "Run the push from a git checkout of the synced repository", + }; + } + const shallow = spawnSync("git", ["rev-parse", "--is-shallow-repository"], { + encoding: "utf8", + stdio: "pipe", + }); + if ((shallow.stdout ?? "").trim() === "true") { + return { + kind: "unknown", + reason: "this is a shallow clone, so its history is truncated", + remedy: + "Fetch the full history (for actions/checkout, fetch-depth: 0)", + }; + } + // A sparse checkout can record migrations in history while never materialising + // them in the working tree, so their absence there says nothing about whether the + // user deleted them. Over-protective for a sparse cone that does include + // migrations/, which the interactive prompt can still override. + // `--type=bool` normalises git's booleans (1, yes, on, …) to true/false; a raw + // `--get` would let `core.sparseCheckout = 1` walk straight past this. + const sparse = spawnSync( + "git", + ["config", "--type=bool", "--get", "core.sparseCheckout"], + { encoding: "utf8", stdio: "pipe" }, + ); + if ((sparse.stdout ?? "").trim() === "true") { + return { + kind: "unknown", + reason: + "this is a sparse checkout, so its working tree may not hold every tracked file", + remedy: "Run the push from a full (non-sparse) checkout", + }; + } + const prefixOut = spawnSync("git", ["rev-parse", "--show-prefix"], { + encoding: "utf8", + stdio: "pipe", + }); + if ((prefixOut.status ?? 1) !== 0) { + return { + kind: "unknown", + reason: "the repository root could not be resolved", + remedy: "Check that git runs correctly in this directory", + }; + } + const prefix = (prefixOut.stdout ?? "").trim(); + + const r = spawnSync( + "git", + ["log", "HEAD", "--format=", "--name-only", "--", "migrations/datatable"], + { encoding: "utf8", stdio: "pipe", maxBuffer: 64 * 1024 * 1024 }, + ); + if ((r.status ?? 1) !== 0) { + log.debug(`Could not read git history for migrations: ${r.stderr ?? ""}`); + return { + kind: "unknown", + reason: "its history could not be read", + remedy: "Check that git runs correctly in this directory", + }; + } + const paths = new Set(); + for (const line of (r.stdout ?? "").split("\n")) { + const p = line.trim(); + if (p.length === 0) continue; + if (prefix.length > 0) { + if (!p.startsWith(prefix)) continue; + paths.add(p.slice(prefix.length)); + } else { + paths.add(p); + } + } + return { kind: "known", paths }; +} + export function isGitRepository(): boolean { try { execSync("git rev-parse --git-dir", { @@ -319,6 +427,10 @@ export function gitSyncIncludePattern( return `${path}.azure_trigger.*`; case "emailtrigger": return `${path}.email_trigger.*`; + case "datatable_migration": + // One migration is two files under `migrations/datatable/
/`; the + // backend already sends the repo-relative base path. + return `${path}.up.sql,${path}.down.sql`; default: // Scripts: `${path}.*` matches the dotted layout // (`${path}.script.yaml` etc.), `${path}__mod/**` matches the folder diff --git a/cli/src/utils/metadata.ts b/cli/src/utils/metadata.ts index ebb35e9d33..1905121c3a 100644 --- a/cli/src/utils/metadata.ts +++ b/cli/src/utils/metadata.ts @@ -19,9 +19,10 @@ import { languageNeedsLock, } from "./script_common.ts"; import { inferContentTypeFromFilePath } from "./script_common.ts"; -import { getModuleFolderSuffix, isModuleEntryPoint, scriptPathToRemotePath } from "./resource_folders.ts"; +import { dbtGeneratedDirs, isUnderGeneratedDir, isBundledModuleFile, getModuleFolderSuffix, isModuleEntryPoint, scriptPathToRemotePath } from "./resource_folders.ts"; import { findCodebase, yamlOptions } from "../commands/sync/sync.ts"; import { generateHash, readInlinePathSync, getHeaders, readTextFile, readTextFileSync } from "./utils.ts"; +import { DBT_DESCRIPTOR_NAME, isMissingDbtDescriptor } from "./resource_folders.ts"; import { detectAuthGatewayChallenge } from "./http_guards.ts"; import { SyncCodebase } from "./codebase.ts"; @@ -218,6 +219,13 @@ export async function generateScriptMetadataInternal( const language = inferContentTypeFromFilePath(scriptPath, opts.defaultTs); + // Whether the metadata and lock live INSIDE that folder. They do for a `__mod` + // bundle, whose folder is Windmill's. A dbt project's folder is dbt's, taken + // verbatim, so its companions stay beside it — writing them in would leave + // stray `script.yaml`/`script.lock` files in the deployed project and leave + // the metadata sync actually reads untouched. + const metadataInFolder = isFolderLayout && language !== "dbt"; + // For folder layout, parseMetadataFile is called with remotePath which // will find __mod/script.yaml via the folder layout fallback const metadataWithType = await parseMetadataFile( @@ -225,8 +233,12 @@ export async function generateScriptMetadataInternal( undefined, ); - // read script content - const scriptContent = await readTextFile(scriptPath); + // read script content — a dbt project's descriptor is optional, and absent + // means an empty descriptor rather than a script that cannot be pushed. + const scriptContent = await readTextFile(scriptPath).catch((e) => { + if (isMissingDbtDescriptor(scriptPath, e)) return ""; + throw e; + }); const metadataContent = await readTextFile(metadataWithType.path); const filteredRawWorkspaceDependencies = filterWorkspaceDependencies( @@ -238,7 +250,8 @@ export async function generateScriptMetadataInternal( // Compute the module folder path early so we can include module hashes in stale check const moduleFolderPath = isFolderLayout ? path.dirname(scriptPath) - : scriptPath.substring(0, scriptPath.indexOf(".")) + getModuleFolderSuffix(); + : scriptPath.substring(0, scriptPath.indexOf(".")) + + getModuleFolderSuffix(language); const hasModules = existsSync(moduleFolderPath) && statSync(moduleFolderPath).isDirectory(); @@ -250,7 +263,8 @@ export async function generateScriptMetadataInternal( let moduleHashes: Record = {}; if (hasModules) { moduleHashes = await computeModuleHashes( - moduleFolderPath, opts.defaultTs, tree ? {} : rawWorkspaceDependencies, isFolderLayout + moduleFolderPath, opts.defaultTs, tree ? {} : rawWorkspaceDependencies, isFolderLayout, + language === "dbt", ); } const hasModuleHashes = Object.keys(moduleHashes).length > 0; @@ -349,7 +363,7 @@ export async function generateScriptMetadataInternal( if (!hasCodebase) { const tempScriptRefs = tree?.getTempScriptRefs(remotePath); - const lockPathOverride = isFolderLayout + const lockPathOverride = metadataInFolder ? path.dirname(scriptPath) + "/script.lock" : undefined; await updateScriptLock( @@ -366,8 +380,13 @@ export async function generateScriptMetadataInternal( metadataParsedContent.lock = ""; } - // Generate locks for modules in __mod/ folder - if (hasModules) { + // Generate locks for modules in __mod/ folder. + // + // Never for a dbt bundle: its files are the project's own SQL and YAML, none + // of which is a Windmill script needing a lockfile, and writing `foo.lock` + // beside `foo.sql` puts our artifacts inside a tree we promise to round-trip + // byte-for-byte. + if (hasModules && language !== "dbt") { // Identify which modules changed by comparing per-module hashes let changedModules: string[] | undefined; if (hasModuleHashes) { @@ -387,7 +406,7 @@ export async function generateScriptMetadataInternal( ); } } else { - if (isFolderLayout) { + if (metadataInFolder) { metadataParsedContent.lock = "!inline " + remotePath.replaceAll(SEP, "/") + getModuleFolderSuffix() + "/script.lock"; } else { @@ -399,7 +418,7 @@ export async function generateScriptMetadataInternal( // Write metadata back to the correct path let metaPath: string; let newMetadataContent: string; - if (isFolderLayout) { + if (metadataInFolder) { if (metadataWithType.isJson) { metaPath = path.dirname(scriptPath) + "/script.json"; newMetadataContent = JSON.stringify(metadataParsedContent); @@ -765,15 +784,18 @@ async function updateScriptLock( tempScriptRefs?: Record, lockPathOverride?: string, ): Promise { - if ( - !( - (workspaceDependenciesLanguages.some((l) => l.language == language) && - language !== "powershell") || - language == "deno" || - language == "rust" || - language == "ansible" - ) - ) { + if (!languageNeedsLock(language)) { + // A dbt lock is written by the dependency job on a worker, from a real + // `dbt deps`/`dbt parse`, so there is nothing to generate here. Restore the + // reference to the file `wmill sync pull` wrote: the caller has already + // resolved it into the metadata, and leaving it resolved inlines the lock + // into the yaml on every run. + if (language === "dbt") { + const lockPath = lockPathOverride ?? remotePath + ".script.lock"; + if (existsSync(lockPath)) { + metadataContent.lock = "!inline " + lockPath.replaceAll(SEP, "/"); + } + } return; } @@ -995,6 +1017,9 @@ export async function inferSchema( } else if (language === "rlang") { const { parse_r } = await loadParser("windmill-parser-wasm-r"); inferedSchema = JSON.parse(parse_r(content)); + } else if (language === "dbt") { + const { parse_dbt } = await loadParser("windmill-parser-wasm-yaml"); + inferedSchema = JSON.parse(parse_dbt(content)); // for related places search: ADD_NEW_LANG } else { throw new Error("Invalid language: " + language); @@ -1368,8 +1393,15 @@ async function computeModuleHashes( defaultTs: "bun" | "deno" | undefined, rawWorkspaceDependencies: Record, isFolderLayout: boolean, + // A dbt project's files are taken verbatim, so hash them the same way the + // push reads them: a `.sql` model has no inferable language, and dropping it + // here would leave an edited model looking up to date. + verbatim: boolean = false, ): Promise> { const hashes: Record = {}; + const skipDirs = verbatim + ? dbtGeneratedDirs(moduleFolderPath) + : new Set(); async function readDir(dirPath: string, relPrefix: string) { const entries = readdirSync(dirPath, { withFileTypes: true }); @@ -1379,15 +1411,31 @@ async function computeModuleHashes( const isTopLevel = relPrefix === ""; if (entry.isDirectory()) { + // A configured `target-path` may be nested (`build/target`), so the + // comparison is on the project-relative path, not the entry name. + if (skipDirs.size > 0 && isUnderGeneratedDir(relPath, skipDirs)) continue; await readDir(fullPath, relPath); + // See the bundle builder: a verbatim (dbt) bundle carries a `.lock` the + // project authored, so the hash has to see it or a change to it would + // never be detected as a change. } else if ( entry.isFile() && - !entry.name.endsWith(".lock") && - !(isFolderLayout && isTopLevel && entry.name.startsWith("script.")) + (verbatim || !entry.name.endsWith(".lock")) && + !(isFolderLayout && isTopLevel && entry.name.startsWith("script.")) && + // The descriptor is the script's CONTENT, hashed as such: counting it + // here too would make one edit look like two changes. + !(verbatim && isTopLevel && entry.name === DBT_DESCRIPTOR_NAME) ) { - try { - inferContentTypeFromFilePath(entry.name, defaultTs); - } catch { + if (!verbatim) { + try { + inferContentTypeFromFilePath(entry.name, defaultTs); + } catch { + continue; + } + } else if (!isBundledModuleFile(fullPath)) { + // Hash only what the push actually sends. Hashing a file the bundle + // drops would make the script permanently stale: every check would + // see a change no push can ever resolve. continue; } const content = readTextFileSync(fullPath); diff --git a/cli/src/utils/resource_folders.ts b/cli/src/utils/resource_folders.ts index 653c443935..04ed260f07 100644 --- a/cli/src/utils/resource_folders.ts +++ b/cli/src/utils/resource_folders.ts @@ -8,6 +8,7 @@ * (.flow, .app, .raw_app) or dunder-prefixed names (__flow, __app, __raw_app). */ +import { existsSync } from "node:fs"; import * as log from "../core/log.ts"; import { sep as SEP } from "node:path"; import { yamlParseFile } from "./yaml.ts"; @@ -503,28 +504,323 @@ export function isFlowFolderMetadataFile(p: string): boolean { * to avoid confusion with file extensions. */ const MODULE_SUFFIX = "__mod"; +/** dbt scripts carry a whole dbt project, not helper code. The folder says so, + * and it is what a dbt developer points `--project-dir` at. */ +export const DBT_MODULE_SUFFIX = "__dbt"; +const MODULE_SUFFIXES = [MODULE_SUFFIX, DBT_MODULE_SUFFIX]; + +/** A dbt project's descriptor, inside the project it configures and OPTIONAL: + * an unmodified dbt project is already a complete Windmill script, and this + * file only appears when one needs something Windmill-specific (run arguments, + * a named warehouse, an engine pin). Its absence is an empty descriptor, never + * a missing script. */ +export const DBT_DESCRIPTOR_NAME = "wm_dbt.yaml"; + +/** Where a dbt script's descriptor lives, given its base path. */ +export function dbtDescriptorPath(scriptBasePath: string): string { + return scriptBasePath + DBT_MODULE_SUFFIX + "/" + DBT_DESCRIPTOR_NAME; +} + +/** Whether an error is a dbt descriptor that simply is not there. */ +export function isMissingDbtDescriptor(filePath: string, e: unknown): boolean { + if ((e as { code?: string })?.code !== "ENOENT") return false; + const norm = filePath.replaceAll("\\", "/"); + if (!isDbtDescriptorPath(norm)) return false; + // And the PROJECT is there. Absent both, this is not a descriptor-less + // project but a path that does not exist — a typo, or a project someone + // deleted — and treating it as an empty descriptor pushes `modules: + // undefined` over a deployed bundle, dropping it while reporting success. + return existsSync( + norm.slice(0, -DBT_DESCRIPTOR_NAME.length) + "dbt_project.yml" + ); +} + +/** Whether a path is a dbt descriptor, i.e. a dbt script's content file. */ +export function isDbtDescriptorPath(p: string): boolean { + const norm = normalizeSep(p); + const base = getScriptBasePathFromModulePath(norm); + return base !== undefined && norm === dbtDescriptorPath(base); +} /** - * Get the module folder suffix (always "__mod") + * Module folder suffix for a script: `__dbt` for a dbt project, `__mod` + * otherwise. */ -export function getModuleFolderSuffix(): string { - return MODULE_SUFFIX; +export function getModuleFolderSuffix(language?: string): string { + return language === "dbt" ? DBT_MODULE_SUFFIX : MODULE_SUFFIX; } /** * Check if a path is inside a script module folder. - * Matches patterns like: .../my_script__mod/... + * Matches patterns like: .../my_script__mod/... or .../my_project__dbt/... */ export function isScriptModulePath(p: string): boolean { - return normalizeSep(p).includes(MODULE_SUFFIX + "/"); + const n = normalizeSep(p); + return MODULE_SUFFIXES.some((suffix) => n.includes(suffix + "/")); +} + +/** Per-file ceiling for a dbt project's bundle. Real dbt code is small (about + * 500 bytes median, 1.9 KB at p90 measured across dbt_utils), so this only + * ever catches a committed dataset, which belongs in the warehouse rather than + * in every version of the script. */ +export const MAX_MODULE_BYTES = 5 * 1024 * 1024; + +/** + * Whether a dbt project file is one the bundle carries. + * + * A dbt project's authored files are text. A binary one -- an image under + * `docs/`, a `.DS_Store`, a parquet seed -- would be read as mojibake and, if + * it carries a NUL, rejected by Postgres with an opaque `unsupported Unicode + * escape sequence`. Binary is detected the way `git` does it, by a NUL in the + * first 8000 bytes, rather than by extension, which `docs/` and stray dotfiles + * do not follow. + * + * The push, the staleness hash and the sync diff all ask this same question: a + * file one drops and another keeps is a change no push can ever resolve. + */ +export function isBundledModuleFile(fullPath: string): boolean { + return moduleFileExclusion(fullPath) === undefined; +} + +/** + * WHY the bundle does not carry a file, when it does not. + * + * The two reasons are not interchangeable. `binary` is dbt's own leftovers and + * stray archives: nothing to say, so sync hides them. `oversized` is a file the + * project authored and dbt WOULD read — a large seed CSV — so it has to stay + * visible in the diff, or the push that reports the actionable size error never + * runs and the remote project is silently left incomplete. + */ +export function moduleFileExclusion( + fullPath: string, +): "binary" | "oversized" | undefined { + // Size from `stat` and only the first 8 KB read: a project may sit next to a + // multi-gigabyte parquet seed or a stray archive, and reading one whole just + // to classify it would stall the sync or exhaust the CLI. + let size: number; + let fd: number; + try { + size = fs.statSync(fullPath).size; + fd = fs.openSync(fullPath, "r"); + } catch { + // Unreadable is not the same as excluded. A pull asks this about files that + // do not exist locally yet, and answering "not carried" there would make + // sync ignore the whole incoming project and write nothing. + return undefined; + } + let binary: boolean; + try { + const head = Buffer.alloc(8000); + const read = fs.readSync(fd, head, 0, 8000, 0); + binary = head.subarray(0, read).includes(0); + } catch { + return undefined; + } finally { + fs.closeSync(fd); + } + if (binary) return "binary"; + return size > MAX_MODULE_BYTES ? "oversized" : undefined; +} + +/** + * The refusal an oversized dbt project file earns, raised WITHOUT reading it. + * + * dbt would have read the file, so deploying the project without it ships + * something that compiles here and fails at run time with a missing relation — + * hence an error rather than a skip. Every path that would otherwise load the + * body (the sync map, the push) asks first: a multi-gigabyte seed must not be + * buffered just to be refused. + */ +export function oversizedModuleFileError(relPath: string, size: number): Error { + return new Error( + `${relPath} is ${Math.ceil(size / 1024 / 1024)} MB, over the ` + + `${MAX_MODULE_BYTES / 1024 / 1024} MB per-file limit for a dbt project file. ` + + `Deploying without it would leave the project incomplete — shrink the file, or ` + + `keep it out of the project folder.`, + ); +} + +/** + * Refuse an oversized dbt project file before its content is read. `undefined` + * for everything else, including binary files the bundle merely drops. + */ +export function oversizedDbtFileError( + fullPath: string, + relPath: string, +): Error | undefined { + if (!isDbtModulePath(relPath)) return undefined; + if (moduleFileExclusion(fullPath) !== "oversized") return undefined; + let size = 0; + try { + size = fs.statSync(fullPath).size; + } catch { + return undefined; + } + return oversizedModuleFileError(relPath, size); +} + +/** Whether a path is inside a dbt project's module folder specifically: those + * files are taken verbatim, with no language inference. */ +export function isDbtModulePath(p: string): boolean { + // The OUTERMOST boundary decides, like `getScriptBasePathFromModulePath`. + // Scanning anywhere in the path would call `foo__mod/vendor/x__dbt/a.ts` a dbt + // project file, and the push would then look for `foo.script.yaml` instead of + // the ordinary module entry point and skip the edit. + const norm = normalizeSep(p); + const base = getScriptBasePathFromModulePath(norm); + return base !== undefined && norm.startsWith(base + DBT_MODULE_SUFFIX + "/"); +} + +/** dbt writes these; a project authors them nowhere. Importing a stale + * `target/` would ship a manifest this runtime then reads as the graph, and + * `dbt_packages/` is a vendored copy the worker restores from its own cache. */ +const DBT_GENERATED_DIRS = ["target", "dbt_packages", "logs", ".git", ".venv", "__pycache__"]; + +const dbtGeneratedDirsCache = new Map< + string, + { stamp: string; dirs: Set } +>(); + +/** `{{ env_var('NAME') }}` / `{{ env_var("NAME", "default") }}`. */ +const DBT_ENV_VAR_CALL = + /\{\{\s*env_var\(\s*['"]([^'"]+)['"]\s*(?:,\s*['"]([^'"]*)['"]\s*)?\)\s*\}\}/g; + +/** + * Render `dbt_project.yml`'s own `env_var()` calls, which dbt allows there too. + * A directory setting left as its template names no directory on disk, so the + * generated tree it points at would be bundled as project source. + * + * Against `process.env`, because the CLI runs where the project was built: that + * is the environment dbt used to produce the tree being read. + */ +export function renderDbtEnvVars(value: string): string { + return value.replace( + DBT_ENV_VAR_CALL, + (whole, name: string, fallback: string | undefined) => + process.env[name] ?? fallback ?? whole, + ); +} + +/** + * Files that keep a project's secrets next to it rather than in it: dbt reads + * none of them (`env_var()` takes the process environment), and the documented + * way into a bundle is `cp -r my-project/.`, which copies whatever the checkout + * holds — including the `.env` a `.gitignore` was keeping out of the repo. + */ +export function isLocalSecretFile(name: string): boolean { + return name === ".env" || name.startsWith(".env.") || name === ".envrc"; +} + +/** + * Directories to leave out of a dbt project's module bundle, as project-relative + * paths — `target-path` and friends may be nested (`build/target`). + * + * `target-path`, `packages-install-path` and `clean-targets` are configurable, + * so they are read from the project rather than assumed. Cached per project + * folder: this is called once per file of a sync. + */ +export function dbtGeneratedDirs(moduleFolderPath: string): Set { + const projectFile = path.join(moduleFolderPath, "dbt_project.yml"); + // Cached against the project file's identity, not merely its folder: `wmill + // dev` is a long-running process, so a `target-path` edited mid-session would + // otherwise keep excluding the old directory and start bundling the new one + // as project source. One entry per folder, replaced when the file changes. + let stamp = ""; + try { + const st = fs.statSync(projectFile); + stamp = `${st.mtimeMs}:${st.size}`; + } catch { + // No project file yet: the defaults apply, and "absent" is its own stamp. + } + const cached = dbtGeneratedDirsCache.get(moduleFolderPath); + if (cached && cached.stamp === stamp) return cached.dirs; + const dirs = new Set(DBT_GENERATED_DIRS); + const add = (raw: string) => { + const v = normalizeSep(renderDbtEnvVars(raw).trim().replace(/^["']|["']$/g, "")) + .replace(/^\.\//, "") + .replace(/\/+$/, ""); + // A configured path that escapes the project is dbt's problem, not ours; + // ignoring it here just means those files stay in the bundle. + if (v && !v.startsWith("/") && !v.split("/").includes("..")) dirs.add(v); + }; + try { + const projectYml = fs.readFileSync(projectFile, "utf-8"); + for (const m of projectYml.matchAll( + /^\s*(?:target-path|packages-install-path)\s*:\s*([^\n#]+)/gm, + )) { + add(m[1]); + } + // `clean-targets` in either of dbt's two spellings: inline `[a, b]`, and the + // block form, whose entries are on the lines that follow. + const lines = projectYml.split(/\r?\n/); + for (let i = 0; i < lines.length; i++) { + const head = lines[i].match(/^\s*clean-targets\s*:\s*(.*)$/); + if (!head) continue; + const inline = head[1].match(/^\[([^\]]*)\]/); + if (inline) { + inline[1].split(",").forEach(add); + continue; + } + for (let j = i + 1; j < lines.length; j++) { + const item = lines[j].match(/^\s+-\s*([^\n#]+)$/); + if (!item) break; + add(item[1]); + } + } + } catch { + // No dbt_project.yml yet (a descriptor pushed before its project): the + // defaults still apply. + } + dbtGeneratedDirsCache.set(moduleFolderPath, { stamp, dirs }); + return dirs; +} + +/** + * Whether a project-relative path sits inside one of `dirs`. Compared segment + * by segment: `targetx/a` must not match a configured `target`. + */ +export function isUnderGeneratedDir(rel: string, dirs: Set): boolean { + const n = normalizeSep(rel); + for (const d of dirs) { + if (n === d || n.startsWith(d + "/")) return true; + } + return false; +} + +/** + * Whether a path under a `__dbt/` folder is one dbt generated rather than one + * the project authors. Those never belong to the bundle, so sync must not offer + * them as items of their own either. + */ +export function isDbtGeneratedPath(p: string): boolean { + const n = normalizeSep(p); + // Anchored on the outermost boundary, like every other helper here. Matching + // `__dbt/` anywhere would call `foo__mod/vendor/x__dbt/target/a.ts` generated + // dbt output, and `ignoreF` would then exclude an ordinary module file so a + // module-only edit never deploys its parent script. + if (!isDbtModulePath(n)) return false; + const base = getScriptBasePathFromModulePath(n)!; + const projectRoot = base + DBT_MODULE_SUFFIX; + const rel = n.slice(projectRoot.length + 1); + if (isUnderGeneratedDir(rel, dbtGeneratedDirs(projectRoot))) { + return true; + } + // Not generated, but not carried either: the bundle drops it, so the diff + // must not keep offering it as a pending change. An OVERSIZED one is the + // exception — see `moduleFileExclusion`: hiding it here is what would make an + // edit to a large seed report no change at all. + return ( + isLocalSecretFile(n.slice(n.lastIndexOf("/") + 1)) || + moduleFileExclusion(p) === "binary" + ); } /** * Build the module folder path from a script's base path (without extension). - * e.g., "f/my_script" -> "f/my_script__mod" + * e.g., "f/my_script" -> "f/my_script__mod", or "__dbt" for a dbt project. */ -export function buildModuleFolderPath(scriptBasePath: string): string { - return scriptBasePath + MODULE_SUFFIX; +export function buildModuleFolderPath(scriptBasePath: string, language?: string): string { + return scriptBasePath + getModuleFolderSuffix(language); } /** @@ -533,10 +829,21 @@ export function buildModuleFolderPath(scriptBasePath: string): string { */ export function isModuleEntryPoint(p: string): boolean { const norm = normalizeSep(p); + // Anchored on the OUTERMOST module boundary, like + // `getScriptBasePathFromModulePath`. Scanning for `__mod/` alone would match a + // `legacy__mod/` directory nested inside a dbt project — dbt owns those names + // verbatim — and call its `script.ts` this script's entry point. + const base = getScriptBasePathFromModulePath(norm); + if (base === undefined) return false; + // A dbt project's entry point is its descriptor, which sits INSIDE the + // project so that an author writes nothing outside the directory dbt itself + // reads. + if (norm.startsWith(base + DBT_MODULE_SUFFIX + "/")) { + return norm === dbtDescriptorPath(base); + } const suffix = MODULE_SUFFIX + "/"; - const idx = norm.indexOf(suffix); - if (idx === -1) return false; - const rest = norm.slice(idx + suffix.length); + if (!norm.startsWith(base + suffix)) return false; + const rest = norm.slice(base.length + suffix.length); return rest.startsWith("script.") && !rest.includes("/"); } @@ -544,13 +851,21 @@ export function isModuleEntryPoint(p: string): boolean { * Extract the script base path from a module folder entry. * e.g., "u/admin/my_script__mod/script.ts" -> "u/admin/my_script" * e.g., "u/admin/my_script__mod/helper.ts" -> "u/admin/my_script" + * e.g., "f/x/proj__dbt/models/a.sql" -> "f/x/proj" */ export function getScriptBasePathFromModulePath(p: string): string | undefined { const norm = normalizeSep(p); - const suffix = MODULE_SUFFIX + "/"; - const idx = norm.indexOf(suffix); - if (idx === -1) return undefined; - return norm.slice(0, idx); + // The OUTERMOST boundary, not the first suffix that happens to match. A dbt + // project's directories are the author's verbatim, so `foo__dbt/models/ + // legacy__mod/a.sql` is legal — taking `__mod` first would call + // `foo__dbt/models/legacy` the script and look for a descriptor that is not + // there, silently skipping the deploy. + let best: number | undefined; + for (const suffix of MODULE_SUFFIXES) { + const idx = norm.indexOf(suffix + "/"); + if (idx !== -1 && (best === undefined || idx < best)) best = idx; + } + return best === undefined ? undefined : norm.slice(0, best); } /** diff --git a/cli/src/utils/script_common.ts b/cli/src/utils/script_common.ts index a713e41782..318f84b7c9 100644 --- a/cli/src/utils/script_common.ts +++ b/cli/src/utils/script_common.ts @@ -1,3 +1,4 @@ +import { isDbtDescriptorPath } from "./resource_folders.ts"; export type ScriptLanguage = | "python3" | "deno" @@ -21,6 +22,7 @@ export type ScriptLanguage = | "ansible" | "ruby" | "rlang" + | "dbt" | "java"; // for related places search: ADD_NEW_LANG @@ -106,6 +108,8 @@ export function inferContentTypeFromFilePath( return "java"; } else if (contentPath.endsWith(".rb")) { return "ruby"; + } else if (isDbtDescriptorPath(contentPath)) { + return "dbt"; } else if (contentPath.endsWith(".r")) { return "rlang"; // for related places search: ADD_NEW_LANG @@ -119,7 +123,7 @@ export function inferContentTypeFromFilePath( throw new Error( `Cannot infer script language from extension '${ext}' (file ${contentPath}).` + hint + - "\nSupported extensions: .ts (bun/deno), .py, .go, .sh, .ps1, .php, .rs, .cs, .nu, .java, .rb, .r, .gql, .playbook.yml, .pg.sql, .my.sql, .bq.sql, .sf.sql, .ms.sql, .odb.sql, .duckdb.sql" + "\nSupported extensions: .ts (bun/deno), .py, .go, .sh, .ps1, .php, .rs, .cs, .nu, .java, .rb, .r, .gql, .playbook.yml, .pg.sql, .my.sql, .bq.sql, .sf.sql, .ms.sql, .odb.sql, .duckdb.sql, and a dbt project folder `__dbt/`" ); } } diff --git a/cli/test/datatable_migrations_unit.test.ts b/cli/test/datatable_migrations_unit.test.ts index 09af49f890..2db9371629 100644 --- a/cli/test/datatable_migrations_unit.test.ts +++ b/cli/test/datatable_migrations_unit.test.ts @@ -15,6 +15,7 @@ import * as path from "node:path"; import * as os from "node:os"; import { parseDatatableMigrationPath } from "../src/types.ts"; import { validateLocalMigrations } from "../src/commands/datatable_migrations.ts"; +import { untrackedDatatableMigrationDeletions } from "../src/commands/sync/sync.ts"; describe("parseDatatableMigrationPath", () => { test("parses up and down files of the new layout", () => { @@ -120,3 +121,61 @@ describe("validateLocalMigrations", () => { expect(validateLocalMigrations()).toEqual([]); }); }); + +// ============================================================================= +// untrackedDatatableMigrationDeletions — the push-side safety net. +// +// Migrations bypass the repo's path filters, so a clone made before they were +// synced sees every server-side migration as remote-only, and +// `pushMigrationFromDisk` reads a missing `.up.sql` as "delete it". What the repo +// has ever committed under migrations/datatable/ is the durable answer to "did we +// track this?" — the working tree is not, because creating a migration locally +// makes the directory appear without anything having been tracked. +// ============================================================================= + +describe("untrackedDatatableMigrationDeletions", () => { + const A_UP = "migrations/datatable/mydb/20260101000000_a.up.sql"; + const A_DOWN = "migrations/datatable/mydb/20260101000000_a.down.sql"; + const changes = [ + { name: "deleted", path: A_UP }, + { name: "deleted", path: A_DOWN }, + { name: "deleted", path: "f/foo/bar.script.yaml" }, + { name: "added", path: "migrations/datatable/mydb/20260102000000_b.up.sql" }, + ]; + + test("trusts a deletion the repository has committed before", () => { + expect( + untrackedDatatableMigrationDeletions(changes, { kind: "known", paths: new Set([A_UP, A_DOWN]) }), + ).toEqual([]); + }); + + test("flags migrations this repository has never recorded", () => { + expect( + untrackedDatatableMigrationDeletions(changes, { kind: "known", paths: new Set() }).map((c) => c.path), + ).toEqual([A_UP, A_DOWN]); + }); + + test("a locally created migration does not vouch for unrelated ones", () => { + // `wmill datatable migrate new` makes migrations/datatable/ exist without the + // checkout having tracked anything, so only the recorded paths count. + const recorded = { + kind: "known" as const, + paths: new Set(["migrations/datatable/mydb/20260102000000_b.up.sql"]), + }; + expect( + untrackedDatatableMigrationDeletions(changes, recorded).map((c) => c.path), + ).toEqual([A_UP, A_DOWN]); + }); + + test("trusts nothing when the history cannot be consulted", () => { + // A shallow clone or a non-repository can't prove a path was never tracked, + // so absence is not read as permission to delete. + expect( + untrackedDatatableMigrationDeletions(changes, { + kind: "unknown", + reason: "this is a shallow clone, so its history is truncated", + remedy: "Fetch the full history (for actions/checkout, fetch-depth: 0)", + }).map((c) => c.path), + ).toEqual([A_UP, A_DOWN]); + }); +}); diff --git a/cli/test/dbt_module_tracker_unit.test.ts b/cli/test/dbt_module_tracker_unit.test.ts new file mode 100644 index 0000000000..90abee18ae --- /dev/null +++ b/cli/test/dbt_module_tracker_unit.test.ts @@ -0,0 +1,145 @@ +/** + * `buildTracker` decides whose top hash `wmill-lock.yaml` refreshes. A dbt + * project is mostly files that are not Windmill script extensions — the project + * file, `packages.yml`, schema YAML, seed CSVs — and its folder is spelled + * `__dbt\` on Windows, so both the extension gate and a raw-path search for + * `__dbt/` left the descriptor untracked and its hash stale. + */ +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as os from "node:os"; +import { buildTracker, elementsToMap } from "../src/commands/sync/sync.ts"; +import { isDbtGeneratedPath } from "../src/utils/resource_folders.ts"; +import { readModulesFromDisk } from "../src/commands/script/script.ts"; + +describe("buildTracker with a dbt project", () => { + let dir: string; + let cwd: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), "wmill-dbt-tracker-")); + cwd = process.cwd(); + process.chdir(dir); + fs.mkdirSync(path.join(dir, "f/analytics/analytics__dbt/models"), { + recursive: true, + }); + fs.mkdirSync(path.join(dir, "f/analytics/analytics__dbt/seeds"), { + recursive: true, + }); + // The descriptor sits INSIDE the project folder, and `findContentFile` + // resolves the metadata path to it. + fs.writeFileSync(path.join(dir, "f/analytics/analytics.script.yaml"), "{}"); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/dbt_project.yml"), + "name: analytics\n", + ); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + }); + + afterEach(() => { + process.chdir(cwd); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + const tracked = async (p: string) => + (await buildTracker([{ name: "edited", path: p, before: "", after: "" }])) + .scripts; + + test("a model edit selects the descriptor", async () => { + expect( + await tracked("f/analytics/analytics__dbt/models/stg_orders.sql"), + ).toEqual(["f/analytics/analytics__dbt/wm_dbt.yaml"]); + }); + + test("so do the files that are not script extensions", async () => { + for (const p of [ + "f/analytics/analytics__dbt/dbt_project.yml", + "f/analytics/analytics__dbt/packages.yml", + "f/analytics/analytics__dbt/models/_models.yml", + "f/analytics/analytics__dbt/seeds/country_codes.csv", + ]) { + expect(await tracked(p)).toEqual( + ["f/analytics/analytics__dbt/wm_dbt.yaml"], + `${p} left the descriptor untracked`, + ); + } + }); + + // Regression: hoisting the module check above the extension gate made + // `__mod/script.yaml` — a folder-layout script's METADATA, which is an + // entry-point path — look like its own content file. Pushed as one, the + // metadata pass asks for the language of `.yaml` and aborts the command. Not a + // dbt shape at all; reached by editing the summary of any modular script. + test("a modular script's own metadata resolves to its content file", async () => { + fs.mkdirSync(path.join(dir, "f/helpers/util__mod"), { recursive: true }); + fs.writeFileSync(path.join(dir, "f/helpers/util__mod/script.yaml"), "{}"); + fs.writeFileSync( + path.join(dir, "f/helpers/util__mod/script.ts"), + "export function main() {}\n", + ); + expect(await tracked("f/helpers/util__mod/script.yaml")).toEqual([ + "f/helpers/util__mod/script.ts", + ]); + }); + + test("and a Windows-separated path", async () => { + expect( + await tracked("f\\analytics\\analytics__dbt\\models\\stg_orders.sql"), + ).toEqual(["f/analytics/analytics__dbt/wm_dbt.yaml"]); + }); + + // A dbt descriptor is the script's CONTENT and is a `.yaml` inside the + // project folder. `--json` drops every metadata `.yaml` as the twin it does + // not read — dropping this one too leaves a workspace whose dbt scripts have + // metadata, a lock and a project bundle, but nothing to run. + test("--json keeps the descriptor while dropping metadata yaml", async () => { + const file = (path: string) => ({ + path, + isDirectory: false, + getChildren: async function* () {}, + getContentText: async () => "x", + }); + const root = { + path: "", + isDirectory: true, + getChildren: async function* () { + yield file("f/analytics/analytics__dbt/wm_dbt.yaml"); + yield file("f/analytics/analytics.script.yaml"); + yield file("f/analytics/analytics.script.json"); + }, + getContentText: async () => "", + }; + const map = await elementsToMap(root as any, () => false, true, {}); + expect(Object.keys(map).sort()).toEqual([ + "f/analytics/analytics.script.json", + "f/analytics/analytics__dbt/wm_dbt.yaml", + ]); + }); + + // `cp -r my-dbt-project/.` copies whatever the checkout holds, and what a + // `.gitignore` was keeping out of the repo is exactly the file that must not + // become a script version. Both halves: bundled, it is uploaded; offered by + // the diff, every push asks to upload it again. + test("a local .env is neither bundled nor offered as a change", async () => { + const project = path.join(dir, "f/analytics/analytics__dbt"); + fs.writeFileSync(path.join(project, ".env"), "DBT_PASSWORD=hunter2\n"); + fs.writeFileSync(path.join(project, "models/stg.sql"), "select 1"); + + const modules = await readModulesFromDisk(project, undefined, false, true); + // Sorted: the bundle is a set of paths, and the walk follows `readdirSync`, + // whose order is the filesystem's. + expect(Object.keys(modules ?? {}).sort()).toEqual([ + "dbt_project.yml", + "models/stg.sql", + ]); + + // The predicate the sync's ignore filter asks, so the file is not offered + // as an item of its own either. + expect(isDbtGeneratedPath("f/analytics/analytics__dbt/.env")).toBe(true); + expect(isDbtGeneratedPath("f/analytics/analytics__dbt/models/stg.sql")).toBe(false); + }); +}); diff --git a/cli/test/dbt_optional_descriptor_unit.test.ts b/cli/test/dbt_optional_descriptor_unit.test.ts new file mode 100644 index 0000000000..f44bdc6103 --- /dev/null +++ b/cli/test/dbt_optional_descriptor_unit.test.ts @@ -0,0 +1,323 @@ +/** + * An unmodified dbt project is already a complete Windmill script: the + * descriptor is optional, and a project that never names one must push, diff + * and pull without ever growing a Windmill file inside it. + */ +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as os from "node:os"; +import { FSFSElement, elementsToMap } from "../src/commands/sync/sync.ts"; +import { listWorkspacePaths } from "../src/commands/dev/dev.ts"; +import { + DbtPathCollisionError, + findContentFile, + handleFile, + hasScriptExt, + removeExtensionToPath, +} from "../src/commands/script/script.ts"; +import { pushParentScriptForModule } from "../src/commands/sync/sync.ts"; + +/** The local map's keys are the walk's own — `path.join`, so `__dbt\\` on + * Windows — while a remote's are the API's. The synthesized descriptor follows + * the spelling of the `dbt_project.yml` it was derived from, like every other + * key in that map, so an assertion on one platform's separator tests the + * platform and not the synthesis. */ +const normalized = (m: Record) => + Object.fromEntries( + Object.entries(m).map(([k, v]) => [k.replaceAll("\\", "/"), v]), + ); + +describe("a dbt project without a descriptor", () => { + let dir: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), "wmill-dbt-nodesc-")); + fs.mkdirSync(path.join(dir, "f/analytics/analytics__dbt/models"), { + recursive: true, + }); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/dbt_project.yml"), + "name: analytics\n", + ); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/models/stg_orders.sql"), + "select 1", + ); + fs.writeFileSync(path.join(dir, "f/analytics/analytics.script.yaml"), "{}"); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + // Without this the project has no content file, so nothing identifies it as a + // script and the whole project silently never deploys. + test("is still discovered, as an empty descriptor", async () => { + const root = await FSFSElement(dir, [], true); + const map = await elementsToMap(root, () => false, false, {}); + expect(normalized(map)["f/analytics/analytics__dbt/wm_dbt.yaml"]).toBe(""); + }); + + // The metadata has to resolve to a content path that is not on disk, or every + // caller that goes metadata -> content aborts the push. + test("resolves from its metadata to the absent descriptor", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + expect(await findContentFile("f/analytics/analytics.script.yaml")).toBe( + "f/analytics/analytics__dbt/wm_dbt.yaml", + ); + } finally { + process.chdir(cwd); + } + }); + + // The descriptor is the one "extension" that contains a separator, so on + // Windows it is spelled `__dbt\wm_dbt.yaml` and a forward-slash suffix test + // matches nothing — every dbt project skipped, with no error. + test("is recognized when the path is spelled with backslashes", () => { + const win = "f\\analytics\\analytics__dbt\\wm_dbt.yaml"; + expect(hasScriptExt(win)).toBe(true); + expect(removeExtensionToPath(win)).toBe("f\\analytics\\analytics"); + }); + + // Both `.py` and `__dbt/` deploy to the SAME remote path, and the + // descriptor is optional — so the project is invisible to the candidate list + // while being perfectly real. Resolved to the ordinary file, a model edit + // deploys the Python script over the dbt one. + test("refuses to resolve when an ordinary script shares its path", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + fs.writeFileSync(path.join(dir, "f/analytics/analytics.py"), "def main(): ..."); + const err = await findContentFile("f/analytics/analytics.script.yaml").then( + () => undefined, + (e) => e as Error, + ); + expect(err?.message).toContain("f/analytics/analytics__dbt/dbt_project.yml"); + expect(err?.message).toContain("f/analytics/analytics.py"); + } finally { + process.chdir(cwd); + } + }); + + // The guard has to sit on the push paths themselves, not only on the + // metadata->content resolution: an ordinary file goes straight to + // `handleFile`, and a module edit reaches its parent through a call whose + // errors were swallowed — so each path could still overwrite the other's + // script while reporting success. + test("both push paths refuse the collision", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + fs.writeFileSync(path.join(dir, "f/analytics/analytics.py"), "def main(): ..."); + const ordinary = await handleFile( + "f/analytics/analytics.py", + { workspaceId: "w", remote: "http://localhost", name: "w", token: "t" } as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(ordinary).toBeInstanceOf(DbtPathCollisionError); + expect(ordinary?.message).toContain("f/analytics/analytics.py"); + + const model = await pushParentScriptForModule( + "f/analytics/analytics__dbt/models/stg_orders.sql", + { workspaceId: "w", remote: "http://localhost", name: "w", token: "t" } as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(model).toBeInstanceOf(DbtPathCollisionError); + } finally { + process.chdir(cwd); + } + }); + + // The guard above must not fire on the project's OWN descriptor: that file is + // the dbt script's content, and its base resolves to the same + // `__dbt/dbt_project.yml` — so a naive check finds the project + // colliding with itself and every dbt push fails before deploying anything. + test("a project does not collide with itself", async () => { + const cwd = process.cwd(); + process.chdir(dir); + const remote = { workspaceId: "w", remote: "http://127.0.0.1:1", name: "w", token: "t" }; + const push = (p: string) => + handleFile(p, remote as any, [], undefined, undefined, {}, []).then( + () => undefined, + (e) => e as Error, + ); + try { + // Descriptor-less: the fixture's project, pushed through the module path. + const nodesc = await pushParentScriptForModule( + "f/analytics/analytics__dbt/models/stg_orders.sql", + remote as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(nodesc).not.toBeInstanceOf(DbtPathCollisionError); + + // Descriptor present, pushed directly. Both reach the network — which is + // unreachable here on purpose — so anything BUT the collision is a pass. + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + expect(await push("f/analytics/analytics__dbt/wm_dbt.yaml")).not.toBeInstanceOf( + DbtPathCollisionError, + ); + } finally { + process.chdir(cwd); + } + }); + + // The exemption above is only for the project's OWN marker. A descriptor + // pushed DIRECTLY never passes through the metadata resolution that catches + // the collision, so without this it would deploy over the ordinary script + // sitting at the same remote path. + test("but a descriptor still refuses an ordinary script at its path", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + fs.writeFileSync(path.join(dir, "f/analytics/analytics.py"), "def main(): ..."); + const err = await handleFile( + "f/analytics/analytics__dbt/wm_dbt.yaml", + { workspaceId: "w", remote: "http://127.0.0.1:1", name: "w", token: "t" } as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(err).toBeInstanceOf(DbtPathCollisionError); + expect(err?.message).toContain("f/analytics/analytics.py"); + } finally { + process.chdir(cwd); + } + }); + + // Both layouts deploy to the same remote path, and the folder layout is the + // one whose base is NOT its filename: `__mod/script.ts` deploys to + // ``, exactly where the dbt project goes. + test("the collision holds for a folder-layout script too", async () => { + const cwd = process.cwd(); + process.chdir(dir); + const remote = { workspaceId: "w", remote: "http://127.0.0.1:1", name: "w", token: "t" }; + const push = (p: string) => + handleFile(p, remote as any, [], undefined, undefined, {}, []).then( + () => undefined, + (e) => e as Error, + ); + try { + fs.mkdirSync(path.join(dir, "f/analytics/analytics__mod"), { recursive: true }); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__mod/script.ts"), + "export async function main() {}", + ); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + + // From the dbt side: the descriptor must find the `__mod` entry point. + const fromDbt = await push("f/analytics/analytics__dbt/wm_dbt.yaml"); + expect(fromDbt).toBeInstanceOf(DbtPathCollisionError); + expect(fromDbt?.message).toContain("analytics__mod/script.ts"); + + // And from the ordinary side, whose base is not its filename. + const fromMod = await push("f/analytics/analytics__mod/script.ts"); + expect(fromMod).toBeInstanceOf(DbtPathCollisionError); + expect(fromMod?.message).toContain("analytics__dbt/dbt_project.yml"); + } finally { + process.chdir(cwd); + } + }); + + // The two sides spell "absent" differently — nothing on disk, nothing in the + // export — so without one normalization a descriptor-less project reads as an + // addition on every push and a deletion on every pull, forever. + test("compares equal to a remote that carries no descriptor either", async () => { + const remote = { + path: "", + isDirectory: true, + getChildren: async function* () { + for (const p of [ + "f/analytics/analytics__dbt/dbt_project.yml", + "f/analytics/analytics__dbt/models/stg_orders.sql", + "f/analytics/analytics.script.yaml", + ]) { + yield { + path: p, + isDirectory: false, + getChildren: async function* () {}, + getContentText: async () => "x", + }; + } + }, + getContentText: async () => "", + }; + const local = await elementsToMap( + await FSFSElement(dir, [], true), + () => false, + false, + {}, + ); + const remoteMap = await elementsToMap(remote as any, () => false, false, {}); + const key = "f/analytics/analytics__dbt/wm_dbt.yaml"; + expect(normalized(local)[key]).toBe(""); + expect(normalized(remoteMap)[key]).toBe(""); + }); +}); + +// `wmill dev` walks basenames, and a dbt script is a DIRECTORY whose descriptor +// may not exist — so it is recognized by the project folder or not at all. The +// walk must also stop there: the project's own `.sql` models match the script +// extensions and would each be listed as a script of their own. +describe("dev-mode discovery of a dbt project", () => { + let dir: string + let cwd: string + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'wmill-dbt-dev-')) + fs.mkdirSync(path.join(dir, 'f/analytics/analytics__dbt/models'), { recursive: true }) + fs.writeFileSync(path.join(dir, 'f/analytics/analytics__dbt/dbt_project.yml'), 'name: a\n') + fs.writeFileSync(path.join(dir, 'f/analytics/analytics__dbt/models/stg.sql'), 'select 1') + cwd = process.cwd() + process.chdir(dir) + }) + + afterEach(() => { + process.chdir(cwd) + fs.rmSync(dir, { recursive: true, force: true }) + }) + + test('lists the project itself and nothing inside it', async () => { + const items = await listWorkspacePaths() + const paths = items.map((i) => i.path).sort() + expect(paths).toEqual(['f/analytics/analytics']) + }) +}) diff --git a/cli/test/deploy_on_behalf_of_unit.test.ts b/cli/test/deploy_on_behalf_of_unit.test.ts new file mode 100644 index 0000000000..2214c0dc37 --- /dev/null +++ b/cli/test/deploy_on_behalf_of_unit.test.ts @@ -0,0 +1,83 @@ +import { expect, test } from "bun:test"; +import { deployItem } from "../windmill-utils-internal/src/deploy.ts"; + +// `deployItem` spreads the source item into the request body, and a script's/flow's +// on_behalf_of names a username that only exists in the source +// workspace. Sending it to the target pairs one workspace's principal with the other's +// email, which the backend rejects. Deleting the spread is an easy regression, so pin +// that the key never reaches the wire. +function recordingProvider(captured: [string, any][], flowExists: boolean) { + const source = { + on_behalf_of_email: "alice@corp", + on_behalf_of: "u/alice", + }; + return { + existsFlowByPath: async () => flowExists, + existsScriptByPath: async () => true, + getFlowByPath: async () => ({ + path: "f/x/f", + summary: "", + value: { modules: [] }, + ...source, + }), + createFlow: async (p: any) => void captured.push(["createFlow", p.requestBody]), + updateFlow: async (p: any) => void captured.push(["updateFlow", p.requestBody]), + getScriptByPath: async () => ({ + path: "f/x/s", + summary: "", + content: "x", + language: "bun", + hash: "abc", + ...source, + }), + createScript: async (p: any) => + void captured.push(["createScript", p.requestBody]), + } as any; +} + +test("deployItem: never sends the source workspace's on_behalf_of", async () => { + const captured: [string, any][] = []; + + // The clear is written out once per branch, so exercise all three: a flow that + // does not exist in the target (create), one that does (update — the branch + // `wmill workspace merge` takes for anything already deployed), and a script. + await deployItem( + recordingProvider(captured, false), + "flow" as any, + "f/x/f", + "src", + "dst", + "alice@corp", + ); + await deployItem( + recordingProvider(captured, true), + "flow" as any, + "f/x/f", + "src", + "dst", + "alice@corp", + ); + await deployItem( + recordingProvider(captured, false), + "script" as any, + "f/x/s", + "src", + "dst", + "alice@corp", + ); + + expect(captured.map(([fn]) => fn)).toEqual([ + "createFlow", + "updateFlow", + "createScript", + ]); + for (const [, body] of captured) { + // The email is still overridden with the caller's choice... + expect(body.on_behalf_of_email).toBe("alice@corp"); + expect(body.preserve_on_behalf_of).toBe(true); + // ...while the principal is dropped, so the backend derives the target's own. + expect( + "on_behalf_of" in JSON.parse(JSON.stringify(body)), + ).toBe(false); + } +}); diff --git a/cli/test/dev_recorder_bundle_unit.test.ts b/cli/test/dev_recorder_bundle_unit.test.ts new file mode 100644 index 0000000000..2c7a35f696 --- /dev/null +++ b/cli/test/dev_recorder_bundle_unit.test.ts @@ -0,0 +1,42 @@ +/** + * The recorder `wmill app dev --recording` serves is generated from the + * frontend's raw-app recorder, not written here, so it can silently ship a stale + * event model after that recorder changes. The committed bundle records the + * sources it was built from and their hash; this fails when they no longer + * agree. + * + * Fix a failure with `bun run gen:dev-recorder` from cli/. + */ + +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import { hashRecorderSources } from "../generate-dev-recorder.ts"; +import { + DEV_RECORDER_BUNDLE, + DEV_RECORDER_SOURCE_HASH, + DEV_RECORDER_SOURCES, +} from "../src/commands/app/devRecorderBundle.gen.ts"; + +const REPO_ROOT = path.join(import.meta.dir, "..", ".."); + +describe("dev recorder bundle", () => { + test("exposes the recorder factory as a global", () => { + expect(DEV_RECORDER_BUNDLE).toContain("__wmillRecorder"); + expect(DEV_RECORDER_BUNDLE).toContain("createRawAppRecording"); + // Runes are stripped at generation; one left in would throw at load time. + expect(DEV_RECORDER_BUNDLE).not.toContain("$state"); + }); + + test("is up to date with the frontend recorder", () => { + expect(DEV_RECORDER_SOURCES.length).toBeGreaterThan(0); + const present = DEV_RECORDER_SOURCES.every((f) => + fs.existsSync(path.join(REPO_ROOT, f)) + ); + // The published CLI package ships without the frontend sources. + if (!present) return; + expect(hashRecorderSources(DEV_RECORDER_SOURCES, REPO_ROOT)).toBe( + DEV_RECORDER_SOURCE_HASH, + ); + }); +}); diff --git a/cli/test/dev_recorder_routes_unit.test.ts b/cli/test/dev_recorder_routes_unit.test.ts new file mode 100644 index 0000000000..29ad71c983 --- /dev/null +++ b/cli/test/dev_recorder_routes_unit.test.ts @@ -0,0 +1,43 @@ +/** + * Guards on the routes `wmill app dev --recording` adds: what may write a + * recording, what a recording may be named, and that two saves never collide. + */ + +import { expect, test } from "bun:test"; +import { + isOwnOrigin, + isRecordingFileName, + recordingFileName, +} from "../src/commands/app/devRecorder.ts"; + +test("only the shell's own origin may save a recording", () => { + expect(isOwnOrigin("http://localhost:4000", "localhost:4000")).toBe(true); + expect(isOwnOrigin("http://127.0.0.1:4000", "127.0.0.1:4000")).toBe(true); + // A cross-site POST carrying JSON under a simple content type needs no + // preflight, so a foreign origin sharing the port must still be refused. + expect(isOwnOrigin("http://attacker.example:4000", "localhost:4000")).toBe( + false, + ); + expect(isOwnOrigin("null", "localhost:4000")).toBe(false); + // No Origin at all is a non-browser client, not a cross-site page. + expect(isOwnOrigin(undefined, "localhost:4000")).toBe(true); +}); + +test("recording names stay inside the recordings folder", () => { + expect(isRecordingFileName("recording-2026-01-01-00-00-00-000.json")).toBe( + true, + ); + expect(isRecordingFileName("../../../etc/passwd")).toBe(false); + expect(isRecordingFileName("..%2Fx.json")).toBe(false); + expect(isRecordingFileName("sub/dir.json")).toBe(false); + expect(isRecordingFileName("recording.txt")).toBe(false); +}); + +test("two saves in the same millisecond get distinct names", () => { + const now = new Date("2026-01-01T00:00:00.123Z"); + const first = recordingFileName(now, 0); + const second = recordingFileName(now, 1); + expect(first).toBe("recording-2026-01-01-00-00-00-123.json"); + expect(second).not.toBe(first); + expect(isRecordingFileName(second)).toBe(true); +}); diff --git a/cli/test/elements_to_map_branch_specific_unit.test.ts b/cli/test/elements_to_map_branch_specific_unit.test.ts index b9abaa8dd1..ebab7ef929 100644 --- a/cli/test/elements_to_map_branch_specific_unit.test.ts +++ b/cli/test/elements_to_map_branch_specific_unit.test.ts @@ -423,3 +423,45 @@ test("elementsToMap: isRemote undefined behaves like local (backward compatible) // Base file should be skipped (same behavior as isRemote=false) expect(Object.keys(result).includes("f/test.variable.yaml")).toEqual(false); }); + +// ============================================================================= +// REGRESSION TEST: --skip-scripts covers a script's module files +// A module is deployed as part of its parent script, and the module shortcut +// runs before every skip filter — so a changed model under a dbt project's +// `__dbt/` folder pushed the script `--skip-scripts` asked to leave alone. +// ============================================================================= + +test("elementsToMap: script modules are excluded when skipScripts is set", async () => { + const files: MockFile[] = [ + { path: "f/analytics/analytics.dbt.yaml", content: "profile: {}\n" }, + { + path: "f/analytics/analytics__dbt/models/stg_orders.sql", + content: "select 1", + }, + { path: "f/Shared/Variable/TestVar.variable.yaml", content: "value: test" }, + ]; + + const kept = await elementsToMap( + createMockDynFSElement(files), + noIgnore, + false, + defaultSkips, + ); + expect( + Object.keys(kept).includes("f/analytics/analytics__dbt/models/stg_orders.sql"), + ).toEqual(true); + + const skipped = await elementsToMap( + createMockDynFSElement(files), + noIgnore, + false, + { skipScripts: true }, + ); + expect( + Object.keys(skipped).includes("f/analytics/analytics__dbt/models/stg_orders.sql"), + ).toEqual(false); + // Everything else is unaffected. + expect( + Object.keys(skipped).includes("f/Shared/Variable/TestVar.variable.yaml"), + ).toEqual(true); +}); diff --git a/cli/test/git_unit.test.ts b/cli/test/git_unit.test.ts index 9665b8581a..92d44e09d9 100644 --- a/cli/test/git_unit.test.ts +++ b/cli/test/git_unit.test.ts @@ -182,6 +182,30 @@ describe("computeGitSyncDeployBranch", () => { ).toBe("wm_deploy/prod/f__team_a"); }); + test("datatable migration branches off its repo-relative migrations/ path", () => { + const items = [ + { + path_type: "datatable_migration", + path: "migrations/datatable/mydb/20260101000000_add_users", + }, + ]; + expect( + computeGitSyncDeployBranch({ ...base, useIndividualBranch: true, items }) + ).toBe( + "wm_deploy/prod/datatable_migration/migrations__datatable__mydb__20260101000000_add_users" + ); + // group_by_folder collapses every data table's migrations onto one branch — + // the backend's debounce key takes the same two segments. + expect( + computeGitSyncDeployBranch({ + ...base, + useIndividualBranch: true, + groupByFolder: true, + items, + }) + ).toBe("wm_deploy/prod/migrations__datatable"); + }); + test("falls back to parent_path when path is absent", () => { expect( computeGitSyncDeployBranch({ @@ -357,6 +381,17 @@ describe("gitSyncIncludePattern", () => { "f/t.amqp_trigger.*" ); }); + test("datatable migration expands to its two repo-relative .sql files", () => { + expect( + gitSyncIncludePattern( + "datatable_migration", + "migrations/datatable/mydb/20260101000000_add_users" + ) + ).toBe( + "migrations/datatable/mydb/20260101000000_add_users.up.sql," + + "migrations/datatable/mydb/20260101000000_add_users.down.sql" + ); + }); }); describe("deriveGitSyncDeployIncludes", () => { diff --git a/cli/test/gitsync_converter_unit.test.ts b/cli/test/gitsync_converter_unit.test.ts index e84c2ffa11..628b6dd161 100644 --- a/cli/test/gitsync_converter_unit.test.ts +++ b/cli/test/gitsync_converter_unit.test.ts @@ -21,6 +21,24 @@ describe("GitSyncSettingsConverter.fromBackendFormat", () => { expect(result.skipWorkspaceDependencies).toBe(false); }); + test("converts datatablemigration in include_type to skipDatatableMigrations: false", () => { + const backend = { + include_path: ["f/**"], + include_type: ["script", "flow", "datatablemigration"], + }; + const result = GitSyncSettingsConverter.fromBackendFormat(backend); + expect(result.skipDatatableMigrations).toBe(false); + }); + + test("sets skipDatatableMigrations: true when datatablemigration is absent", () => { + const backend = { + include_path: ["f/**"], + include_type: ["script", "flow"], + }; + const result = GitSyncSettingsConverter.fromBackendFormat(backend); + expect(result.skipDatatableMigrations).toBe(true); + }); + test("sets skipWorkspaceDependencies: true when workspacedependencies is absent", () => { const backend = { include_path: ["f/**"], @@ -71,6 +89,21 @@ describe("GitSyncSettingsConverter.fromBackendFormat", () => { // ============================================================================= describe("GitSyncSettingsConverter.toBackendFormat", () => { + test("adds datatablemigration when skipDatatableMigrations is false", () => { + expect( + GitSyncSettingsConverter.toBackendFormat({ + includes: ["f/**"], + skipDatatableMigrations: false, + }).include_type, + ).toContain("datatablemigration"); + expect( + GitSyncSettingsConverter.toBackendFormat({ + includes: ["f/**"], + skipDatatableMigrations: true, + }).include_type, + ).not.toContain("datatablemigration"); + }); + test("adds workspacedependencies when skipWorkspaceDependencies is false", () => { const opts = { includes: ["f/**"], diff --git a/cli/test/pipeline_local_graph_unit.test.ts b/cli/test/pipeline_local_graph_unit.test.ts index 7b8b8b0dc5..29d016857c 100644 --- a/cli/test/pipeline_local_graph_unit.test.ts +++ b/cli/test/pipeline_local_graph_unit.test.ts @@ -5,6 +5,7 @@ import { join } from "node:path"; import { buildLocalPipelineGraph, + hideDbtRunnables, parseMuteAnnotations, } from "../src/commands/pipeline/localGraph.ts"; @@ -915,3 +916,96 @@ test("pipeline docs renders a `Macro libraries` section (call + `// use`)", asyn }, ); }); + +test("a dbt script is not a pipeline node — a dbt-only folder has no graph", async () => { + await withFolder( + { + "warehouse.dbt.yaml": `engine: dbt-core-1x\nprofile:\n resource: $res:u/admin/wh\n`, + }, + async (root, folder) => { + const { graph, scripts } = await buildLocalPipelineGraph({ + root, + folder, + defaultTs: "bun", + }); + // A dbt project is authored and run as itself, never as a pipeline member; + // the deploy leaves its `auto_kind` unset, so the local graph must agree. + expect(graph.runnables).toEqual([]); + expect(scripts).toEqual([]); + }, + ); +}); + +test("a folder mixing dbt and a pipeline keeps only the pipeline member", async () => { + await withFolder( + { + "warehouse.dbt.yaml": `engine: dbt-core-1x\nprofile:\n resource: $res:u/admin/wh\n`, + "report.bun.ts": `// pipeline\nexport async function main() {}\n`, + }, + async (root, folder) => { + const { graph } = await buildLocalPipelineGraph({ + root, + folder, + defaultTs: "bun", + }); + expect(graph.runnables.map((r) => r.path)).toEqual(["f/mypipe/report"]); + }, + ); +}); + +test("hideDbtRunnables drops the dbt node from a deployed graph, keeping its relations", () => { + // `/assets/graph` is asset-usage driven, so it returns the dbt script as a + // producer. The CLI's deployed views must not render it as a pipeline script. + const deployed = { + runnables: [ + { path: "f/x/dbtproj", usage_kind: "script" as const, dbt: { model_count: 2 } }, + { path: "f/x/report", usage_kind: "script" as const, in_pipeline: true }, + ], + assets: [ + { kind: "table", path: "u/a/wh/s/stg_orders" }, + { kind: "table", path: "u/a/wh/s/fct_orders" }, + ], + edges: [ + { runnable_kind: "script", runnable_path: "f/x/dbtproj", asset_kind: "table", asset_path: "u/a/wh/s/stg_orders", access_type: "w" as const }, + { runnable_kind: "script", runnable_path: "f/x/dbtproj", asset_kind: "table", asset_path: "u/a/wh/s/fct_orders", access_type: "w" as const }, + { runnable_kind: "script", runnable_path: "f/x/report", asset_kind: "table", asset_path: "u/a/wh/s/fct_orders", access_type: "r" as const }, + ], + triggers: [ + { trigger_kind: "asset" as const, asset_kind: "table", asset_path: "u/a/wh/s/fct_orders", runnable_kind: "script", runnable_path: "f/x/report" }, + ], + }; + const g = hideDbtRunnables(deployed); + expect(g.runnables.map((r) => r.path)).toEqual(["f/x/report"]); + expect(g.edges.map((e) => e.runnable_path)).toEqual(["f/x/report"]); + // The relations stay: they are what the downstream pipeline script reads. + expect(g.assets).toHaveLength(2); + expect(g.triggers).toHaveLength(1); +}); + +test("hideDbtRunnables keeps a flow sharing a path with the dbt script", () => { + // Runnable identity in the graph is `(usage_kind, path)`; a script and a flow + // may share a path, so only the dbt script may be removed. + const g = hideDbtRunnables({ + runnables: [ + { path: "f/x/proj", usage_kind: "script", dbt: { model_count: 1 } }, + { path: "f/x/proj", usage_kind: "flow" }, + ], + edges: [ + { runnable_kind: "flow", runnable_path: "f/x/proj" }, + { runnable_kind: "script", runnable_path: "f/x/proj" }, + ], + triggers: [{ runnable_kind: "flow", runnable_path: "f/x/proj" }], + }); + expect(g.runnables.map((r) => r.usage_kind)).toEqual(["flow"]); + expect(g.edges.map((e) => e.runnable_kind)).toEqual(["flow"]); + expect(g.triggers).toHaveLength(1); +}); + +test("hideDbtRunnables is a no-op when the folder has no dbt project", () => { + const g = { + runnables: [{ path: "f/x/a", usage_kind: "script" }], + edges: [], + triggers: [], + }; + expect(hideDbtRunnables(g)).toBe(g); +}); diff --git a/cli/test/preview.test.ts b/cli/test/preview.test.ts index 4ead0a06c9..ffa900c8f7 100644 --- a/cli/test/preview.test.ts +++ b/cli/test/preview.test.ts @@ -185,6 +185,60 @@ test("script preview: regular script (non-codebase)", async () => { }); }); +test("script preview: job path is the script's Windmill path for every argument shape", async () => { + await withTestBackend(async (backend, tempDir) => { + await createWmillConfig(tempDir, { defaultTs: "bun" }); + await createScript( + tempDir, + "f/test/job_path_script.ts", + `export function main() { + return process.env.WM_JOB_PATH; +}` + ); + + const invocations: Array<[string, string]> = [ + ["f/test/job_path_script.ts", tempDir], + ["./f/test/job_path_script.ts", tempDir], + [`${tempDir}/f/test/job_path_script.ts`, tempDir], + ["job_path_script.ts", `${tempDir}/f/test`], + ]; + + for (const [arg, workingDir] of invocations) { + const result = await backend.runCLICommand( + ["script", "preview", arg, "--silent"], + workingDir + ); + + expect(result.code).toEqual(0); + expect(result.stdout.trim()).toEqual(`"f/test/job_path_script"`); + } + + // Folder layout: the job runs under the script's path, not the entry file's. + await createScript( + tempDir, + "f/test/job_path_module__mod/script.ts", + `export function main() { + return process.env.WM_JOB_PATH; +}` + ); + const moduleResult = await backend.runCLICommand( + ["script", "preview", "f/test/job_path_module__mod/script.ts", "--silent"], + tempDir + ); + expect(moduleResult.code).toEqual(0); + expect(moduleResult.stdout.trim()).toEqual(`"f/test/job_path_module"`); + + // A file outside the workspace tree has no Windmill path to run under, so + // the run is refused rather than pushed with a made-up one. + await writeFile(`${tempDir}/stray.ts`, `export function main() {}`, "utf-8"); + const strayResult = await backend.runCLICommand( + ["script", "preview", "stray.ts", "--silent"], + tempDir + ); + expect(strayResult.code).toEqual(1); + }); +}); + test("script preview: codebase script (CJS)", async () => { await withTestBackend(async (backend, tempDir) => { await createWmillConfig(tempDir, { @@ -529,6 +583,37 @@ test("flow preview: simple flow", async () => { }); }); +test("flow preview: step job path is anchored on the flow's Windmill path", async () => { + await withTestBackend(async (backend, tempDir) => { + await createWmillConfig(tempDir, { defaultTs: "bun" }); + await createFlow(tempDir, "f/test/job_path_flow.flow", { + summary: "Test flow", + scriptContent: `export function main() { return process.env.WM_JOB_PATH; }`, + }); + + // The last one is `--remote` from a subdirectory: that combination reads + // no config of its own, so it is the one shape where nothing but the path + // resolution can put the process in the sync root. + const invocations: Array<[string[], string]> = [ + [["f/test/job_path_flow.flow"], tempDir], + [["./f/test/job_path_flow.flow"], tempDir], + [[`${tempDir}/f/test/job_path_flow.flow`], tempDir], + [["job_path_flow.flow"], `${tempDir}/f/test`], + [["--remote", "job_path_flow.flow"], `${tempDir}/f/test`], + ]; + + for (const [args, workingDir] of invocations) { + const result = await backend.runCLICommand( + ["flow", "preview", ...args, "--silent"], + workingDir + ); + + expect(result.code).toEqual(0); + expect(result.stdout.trim()).toEqual(`"f/test/job_path_flow/a"`); + } + }); +}); + test("flow preview: uses local PathScript by default and remote PathScript with --remote", async () => { await withTestBackend(async (backend, tempDir) => { await createWmillConfig(tempDir, { defaultTs: "bun" }); diff --git a/cli/test/preview_path_unit.test.ts b/cli/test/preview_path_unit.test.ts new file mode 100644 index 0000000000..4bf9e8632c --- /dev/null +++ b/cli/test/preview_path_unit.test.ts @@ -0,0 +1,112 @@ +/** + * A preview job carries no runnable, so the path derived from the file + * argument is the whole of its identity — it has to come out the same + * whatever shape the argument had, and be refused rather than guessed when + * the file has no place in the workspace tree. + */ +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as os from "node:os"; +import { + assertRemotePath, + toSyncRootRelativePath, +} from "../src/core/context.ts"; + +describe("toSyncRootRelativePath", () => { + let root: string; + let previousCwd: string; + let temps: string[]; + + beforeEach(() => { + previousCwd = process.cwd(); + temps = []; + // realpath: macOS' tmpdir is /var -> /private/var, and the assertions + // compare against what the process reports as its own directory. + root = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), "wmill_preview_path_")), + ); + temps.push(root); + fs.writeFileSync(path.join(root, "wmill.yaml"), "defaultTs: bun\n"); + fs.mkdirSync(path.join(root, "f", "test"), { recursive: true }); + fs.writeFileSync(path.join(root, "f", "test", "script.ts"), ""); + process.chdir(root); + }); + + afterEach(() => { + process.chdir(previousCwd); + for (const dir of temps) fs.rmSync(dir, { recursive: true, force: true }); + }); + + const normalized = (p: string) => p.replaceAll("\\", "/"); + + /** A dbt project whose descriptor is deliberately not written. */ + function dbtProject(): string { + const project = path.join(root, "f", "test", "proj__dbt"); + fs.mkdirSync(project, { recursive: true }); + fs.writeFileSync(path.join(project, "dbt_project.yml"), "name: proj\n"); + return project; + } + + test("every spelling of the same file lands on the same path", () => { + const fromRoot = ["f/test/script.ts", "./f/test/script.ts"].map((arg) => + normalized(toSyncRootRelativePath(arg, root)), + ); + const absolute = normalized( + toSyncRootRelativePath(path.join(root, "f", "test", "script.ts"), root), + ); + // As typed from the directory the file is in: the config read has already + // moved the process to the root by the time the argument is resolved. + const fromSubdir = normalized( + toSyncRootRelativePath("script.ts", path.join(root, "f", "test")), + ); + + expect(fromRoot).toEqual(["f/test/script.ts", "f/test/script.ts"]); + expect(absolute).toEqual("f/test/script.ts"); + expect(fromSubdir).toEqual("f/test/script.ts"); + }); + + test("a file that is deliberately absent keeps the directory it was named in", () => { + // A dbt project's descriptor is optional; `wmill script preview + // wm_dbt.yaml` from inside the project must still resolve to the project. + const project = dbtProject(); + + expect(normalized(toSyncRootRelativePath("wm_dbt.yaml", project))).toEqual( + "f/test/proj__dbt/wm_dbt.yaml", + ); + }); + + // Windows only creates symlinks for a privileged process. + test.skipIf(process.platform === "win32")( + "an absent file reached through a symlinked root still lands in the tree", + () => { + dbtProject(); + const aliasDir = fs.mkdtempSync(path.join(os.tmpdir(), "wmill_alias_")); + temps.push(aliasDir); + const alias = path.join(aliasDir, "link"); + fs.symlinkSync(root, alias, "dir"); + + const arg = path.join(alias, "f", "test", "proj__dbt", "wm_dbt.yaml"); + expect(normalized(toSyncRootRelativePath(arg, root))).toEqual( + "f/test/proj__dbt/wm_dbt.yaml", + ); + }, + ); + + test("a file outside the tree stays outside it", () => { + const outside = path.join(root, "..", "elsewhere.ts"); + expect(toSyncRootRelativePath(outside, root).startsWith("..")).toBe(true); + }); +}); + +describe("assertRemotePath", () => { + test("accepts a workspace path and refuses anything else", () => { + expect(() => assertRemotePath("f/test/script", "f/test/script.ts")).not.toThrow(); + expect(() => assertRemotePath("u/admin/script", "script.ts")).not.toThrow(); + for (const bad of ["", "script", "f/script", "../elsewhere"]) { + expect(() => assertRemotePath(bad, "arg.ts")).toThrow( + /Cannot derive a Windmill path/, + ); + } + }); +}); diff --git a/cli/test/raw_app_recordings_skip_unit.test.ts b/cli/test/raw_app_recordings_skip_unit.test.ts new file mode 100644 index 0000000000..408671cfe3 --- /dev/null +++ b/cli/test/raw_app_recordings_skip_unit.test.ts @@ -0,0 +1,63 @@ +/** + * `wmill app dev --recording` writes multi-MB session recordings into + * `.raw_app/recordings/`. They are local artifacts: the sync differ must + * not offer them as app source (the push itself drops them in + * `collectAppFiles`, so a differ that still sees them reports a change that + * pushing can never settle). + */ + +import { expect, test } from "bun:test"; +import { sep as SEP } from "node:path"; +import { elementsToMap } from "../src/commands/sync/sync.ts"; + +type MockFile = { path: string; content: string }; + +// FSFSElement joins with the platform separator, and the exclusion has to hold +// on Windows too. +const p = (...parts: string[]) => parts.join(SEP); + +function mockElement(files: MockFile[]) { + return { + isDirectory: true, + path: "", + async getContentText() { + return ""; + }, + async *getChildren() { + for (const file of files) { + yield { + isDirectory: false, + path: file.path, + async getContentText() { + return file.content; + }, + async *getChildren() {}, + }; + } + }, + }; +} + +test("elementsToMap skips recordings/ at the root of a raw app folder only", async () => { + const app = p("f", "demo", "myapp.raw_app"); + const files: MockFile[] = [ + { path: p(app, "index.tsx"), content: "export {}" }, + { + path: p(app, "recordings", "recording-2026-01-01-00-00-00.json"), + content: '{"version":1}', + }, + // The dev server never writes here, so this is the app's own source. + { path: p(app, "src", "recordings", "fixture.json"), content: "{}" }, + ]; + + const result = await elementsToMap( + mockElement(files) as any, + () => false, + false, + {}, + ); + + expect(Object.keys(result).sort()).toEqual( + [p(app, "index.tsx"), p(app, "src", "recordings", "fixture.json")].sort(), + ); +}); diff --git a/cli/test/raw_app_sync.test.ts b/cli/test/raw_app_sync.test.ts index 3cc7026430..315b274d80 100644 --- a/cli/test/raw_app_sync.test.ts +++ b/cli/test/raw_app_sync.test.ts @@ -464,6 +464,74 @@ excludes: []`, "utf-8"); }); }); +test("Raw App: deleted .lock sorting first does not short-circuit the app push", async () => { + // Regression: raw-app changes collapse to one representative change and + // deletes sort first, so removing a backend runnable makes its `.lock` the + // representative. Skipping a `.lock` there drops the whole app while the + // push still reports success. + await withTestBackend(async (backend, tempDir) => { + const testWorkspace = { + remote: backend.baseUrl, + workspaceId: backend.workspace, + name: "raw_app_lock_first_test", + token: backend.token + }; + await addWorkspace(testWorkspace, { force: true, configDir: backend.testConfigDir }); + + await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun +includes: + - "**" +excludes: []`, "utf-8"); + + const appDir = path.join(tempDir, "f", "test", "lock_first_app.raw_app"); + await mkdir(path.join(tempDir, "f", "test"), { recursive: true }); + await createRawAppOnDisk(appDir, true); + + // A backend runnable's lockfile. Among the app's files it sorts first, + // so once deleted it becomes changes[0] for the whole group. + const queryLockPath = path.join(appDir, "backend", "query.lock"); + await writeFile(queryLockPath, INLINE_SCRIPT_A_LOCK, "utf-8"); + + const pushResult1 = await backend.runCLICommand( + ['sync', 'push', '--yes'], + tempDir, "raw_app_lock_first_test" + ); + expect(pushResult1.code).toEqual(0); + await waitForDeploymentJobs(backend); + + // Remove the backend runnable (lock included) and edit a frontend file. + await rm(queryLockPath); + await rm(path.join(appDir, "backend", "query.ts")); + await rm(path.join(appDir, "backend", "query.yaml")); + const appTsxPath = path.join(appDir, "App.tsx"); + const appTsxContent = await readFileContent(appTsxPath); + await writeFile( + appTsxPath, + appTsxContent.replace("hello world", "REGRESSION MARKER"), + "utf-8" + ); + + const pushResult2 = await backend.runCLICommand( + ['sync', 'push', '--yes'], + tempDir, "raw_app_lock_first_test" + ); + expect(pushResult2.code).toEqual(0); + await waitForDeploymentJobs(backend); + + // The edit must have reached the remote bundle, and the removed runnable + // must be gone from it. + const appResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/apps/get/p/f/test/lock_first_app` + ); + expect(appResp.status).toEqual(200); + const appJson = await appResp.json(); + const files = appJson?.value?.files ?? {}; + expect(files["/App.tsx"]).toContain("REGRESSION MARKER"); + // Backend runnables live in value.runnables, not in the bundled files. + expect(appJson?.value?.runnables?.query).toBeUndefined(); + }); +}); + test("Raw App: delete file and push", async () => { await withTestBackend(async (backend, tempDir) => { // Set up workspace diff --git a/cli/test/raw_app_workspace_deps_unit.test.ts b/cli/test/raw_app_workspace_deps_unit.test.ts new file mode 100644 index 0000000000..a76064e8b9 --- /dev/null +++ b/cli/test/raw_app_workspace_deps_unit.test.ts @@ -0,0 +1,84 @@ +/** + * Raw app workspace dependencies + * + * A raw app keeps its runnables in `backend/`, not in `raw_app.yaml`. The + * workspace dependency filtering must resolve those files, otherwise the + * default `dependencies/package.json` is dropped and locks are regenerated + * against unpinned versions. + * + * Exercised through the legacy (tree-less) path: tree mode sources its deps + * from `getMismatchedWorkspaceDeps()`, which is only populated by an + * `uploadScripts` round-trip, so it cannot run offline. Both paths filter the + * same `appValue`, so resolving it correctly is what this pins. + */ + +import { expect, test } from "bun:test"; +import * as path from "node:path"; +import os from "node:os"; +import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { generateAppLocksInternal } from "../src/commands/app/app_metadata.ts"; +import { Workspace } from "../src/commands/workspace/workspace.ts"; + +const stubWorkspace: Workspace = { + remote: "http://localhost:0/", + workspaceId: "test", + name: "test", + token: "test", +}; + +const APP_FOLDER = path.join("f", "example.raw_app"); + +async function withTempDir(fn: (tempDir: string) => Promise): Promise { + const tempDir = await mkdtemp(path.join(os.tmpdir(), "wmill_raw_app_deps_")); + const originalCwd = process.cwd(); + try { + process.chdir(tempDir); + await fn(tempDir); + } finally { + process.chdir(originalCwd); + await rm(tempDir, { recursive: true, force: true }); + } +} + +test("raw app: default workspace deps are picked up from backend runnables", async () => { + await withTempDir(async () => { + await mkdir(path.join(APP_FOLDER, "backend"), { recursive: true }); + await writeFile( + path.join(APP_FOLDER, "raw_app.yaml"), + `summary: "example raw app"\npolicy:\n execution_mode: publisher\n triggerables: {}\n`, + "utf-8", + ); + await writeFile( + path.join(APP_FOLDER, "backend", "test.ts"), + `import * as wmill from "windmill-client"\n\nexport async function main() {\n return wmill.getVariable("example")\n}\n`, + "utf-8", + ); + + await generateAppLocksInternal( + APP_FOLDER, + true, + false, + stubWorkspace, + { defaultTs: "bun" }, + true, // justUpdateMetadataLock — no backend round-trip + true, + ); + + expect( + await generateAppLocksInternal(APP_FOLDER, true, true, stubWorkspace, { defaultTs: "bun" }, false, true), + ).toBeUndefined(); + + // The runnable has no `package_json` annotation, so it uses the default + // manifest — adding it must invalidate the app. + await mkdir("dependencies", { recursive: true }); + await writeFile( + path.join("dependencies", "package.json"), + `{"dependencies": {"windmill-client": "1.742.0"}}`, + "utf-8", + ); + + expect( + await generateAppLocksInternal(APP_FOLDER, true, true, stubWorkspace, { defaultTs: "bun" }, false, true), + ).toEqual("f/example.raw_app"); + }); +}); diff --git a/cli/test/resource_folders_unit.test.ts b/cli/test/resource_folders_unit.test.ts index 0d07812124..de3daa18cd 100644 --- a/cli/test/resource_folders_unit.test.ts +++ b/cli/test/resource_folders_unit.test.ts @@ -3,7 +3,7 @@ * Tests both dotted (.flow, .app, .raw_app) and non-dotted (__flow, __app, __raw_app) modes. */ -import { expect, test, describe, beforeEach } from "bun:test"; +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; import { setNonDottedPaths, getNonDottedPaths, @@ -37,6 +37,13 @@ import { transformJsonPathToDir, isModuleEntryPoint, getScriptBasePathFromModulePath, + isDbtGeneratedPath, + dbtGeneratedDirs, + isUnderGeneratedDir, + isDbtModulePath, + isBundledModuleFile, + moduleFileExclusion, + MAX_MODULE_BYTES, } from "../src/utils/resource_folders.ts"; import { removeWorkerPrefix } from "../src/commands/worker-groups/worker-groups.ts"; @@ -657,3 +664,194 @@ describe("removeWorkerPrefix", () => { expect(removeWorkerPrefix("worker__")).toBe(""); }); }); + +// A dbt project's generated directories never belong to the bundle: hashing and +// uploading a local `target/` would make every local `dbt run` look like a +// project change, and a stale manifest in it is what the runtime reads as the +// graph. They are configurable, so they are read from the project. +describe("dbtGeneratedDirs", () => { + const fs = require("node:fs"); + const os = require("node:os"); + const nodePath = require("node:path"); + let dir: string; + // Every temp directory this block makes, so none outlives the run: the helper + // below mints one per call by design. + let made: string[] = []; + + beforeEach(() => { + dir = fs.mkdtempSync(nodePath.join(os.tmpdir(), "dbtgen-")); + made = [dir]; + }); + + afterEach(() => { + for (const d of made) fs.rmSync(d, { recursive: true, force: true }); + }); + + // A fresh directory per call: `dbtGeneratedDirs` memoizes per project folder, + // since within one sync the project file does not change under it. + const write = (yml: string) => { + const d = fs.mkdtempSync(nodePath.join(os.tmpdir(), "dbtgen-")); + made.push(d); + fs.writeFileSync(nodePath.join(d, "dbt_project.yml"), yml); + return dbtGeneratedDirs(d); + }; + + test("defaults apply with no project file", () => { + const dirs = dbtGeneratedDirs(dir); + expect(dirs.has("target")).toBe(true); + expect(dirs.has("dbt_packages")).toBe(true); + }); + + test("reads nested target-path and packages-install-path", () => { + const dirs = write( + 'name: p\ntarget-path: "build/target"\npackages-install-path: ./vendor/pkgs\n', + ); + expect(dirs.has("build/target")).toBe(true); + expect(dirs.has("vendor/pkgs")).toBe(true); + }); + + test("reads clean-targets in both of dbt's spellings", () => { + expect(write('name: p\nclean-targets: ["a", b]\n').has("a")).toBe(true); + const block = write("name: p\nclean-targets:\n - out/one\n - two\nmodels: {}\n"); + expect(block.has("out/one")).toBe(true); + expect(block.has("two")).toBe(true); + expect(block.has("models")).toBe(false); + }); + + test("ignores a configured path that escapes the project", () => { + const dirs = write('name: p\ntarget-path: "../../etc"\n'); + expect([...dirs].some((d) => d.includes(".."))).toBe(false); + }); +}); + +describe("isUnderGeneratedDir", () => { + const dirs = new Set(["target", "build/target"]); + + test("matches the directory and everything under it", () => { + expect(isUnderGeneratedDir("target", dirs)).toBe(true); + expect(isUnderGeneratedDir("target/manifest.json", dirs)).toBe(true); + expect(isUnderGeneratedDir("build/target/run_results.json", dirs)).toBe(true); + }); + + test("does not match a sibling sharing the prefix", () => { + expect(isUnderGeneratedDir("targetx/a.sql", dirs)).toBe(false); + expect(isUnderGeneratedDir("models/target_helper.sql", dirs)).toBe(false); + expect(isUnderGeneratedDir("build/targeted/a.sql", dirs)).toBe(false); + }); +}); + +// A Windows path spells the folder `__dbt\\`. A lookup that searched the raw +// path for `__dbt/` would find nothing, so a module-only edit would return +// without deploying its parent while the file was still recorded as synced. +describe("dbt module paths on either separator", () => { + test("isDbtModulePath and getScriptBasePathFromModulePath accept backslashes", () => { + expect(isDbtModulePath("f\\x\\proj__dbt\\models\\a.sql")).toBe(true); + expect(getScriptBasePathFromModulePath("f\\x\\proj__dbt\\models\\a.sql")).toBe( + "f/x/proj", + ); + expect(getScriptBasePathFromModulePath("f/x/proj__dbt/models/a.sql")).toBe( + "f/x/proj", + ); + }); + + test("a path with no module folder has no base path", () => { + expect(getScriptBasePathFromModulePath("f/x/proj.dbt.yaml")).toBeUndefined(); + }); +}); + +// The push, the staleness hash and the sync diff all ask this question. They +// must agree: a file one drops and another keeps is a change no push resolves. +describe("isBundledModuleFile", () => { + const fs = require("node:fs"); + const os = require("node:os"); + const nodePath = require("node:path"); + let dir: string; + + beforeEach(() => { + dir = fs.mkdtempSync(nodePath.join(os.tmpdir(), "bundled-")); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + const write = (name: string, data: Buffer | string) => { + const p = nodePath.join(dir, name); + fs.writeFileSync(p, data); + return p; + }; + + test("keeps text, including empty and unicode", () => { + expect(isBundledModuleFile(write("a.sql", "select 1\n"))).toBe(true); + expect(isBundledModuleFile(write("empty.sql", ""))).toBe(true); + expect(isBundledModuleFile(write("u.sql", "select 'café'\n"))).toBe(true); + }); + + test("drops a binary file, which a NUL identifies", () => { + expect(isBundledModuleFile(write("x.png", Buffer.from([0x89, 0x50, 0x00, 0x1a])))).toBe( + false, + ); + }); + + test("drops a file over the per-file limit", () => { + expect(isBundledModuleFile(write("huge.csv", "x".repeat(MAX_MODULE_BYTES + 1)))).toBe( + false, + ); + }); + + // The two reasons a file is not carried are not interchangeable: sync hides + // dbt's binary leftovers, but an oversized seed the project authored has to + // stay in the diff, or the push that reports the size error never runs and the + // remote project is left incomplete without saying so. + test("says WHY a file is not carried", () => { + expect(moduleFileExclusion(write("a.sql", "select 1\n"))).toBe(undefined); + expect(moduleFileExclusion(write("x.png", Buffer.from([0x89, 0x50, 0x00, 0x1a])))).toBe( + "binary", + ); + expect(moduleFileExclusion(write("seed.csv", "x".repeat(MAX_MODULE_BYTES + 1)))).toBe( + "oversized", + ); + }); + + // A pull asks this about files that do not exist locally yet. Answering "not + // carried" there made sync ignore the whole incoming project and write + // nothing — the bundle silently vanished on every fresh checkout. + test("treats a missing file as carried, not as excluded", () => { + expect(isBundledModuleFile(nodePath.join(dir, "does-not-exist.sql"))).toBe(true); + }); +}); + +test("a nested __mod inside a dbt project does not steal the script boundary", () => { + // dbt owns its directory names verbatim, so a folder ending `__mod` is legal + // inside a project. The script is the OUTER module boundary. + expect( + getScriptBasePathFromModulePath("f/x/proj__dbt/models/legacy__mod/a.sql"), + ).toBe("f/x/proj"); + expect(getScriptBasePathFromModulePath("f/x/proj__dbt/models/a.sql")).toBe( + "f/x/proj", + ); + expect(getScriptBasePathFromModulePath("f/x/s__mod/inner.ts")).toBe("f/x/s"); +}); + +test("a nested __mod inside a dbt project is not a module entry point", () => { + expect(isModuleEntryPoint("f/x/s__mod/script.ts")).toBe(true); + // `legacy__mod` is a legal dbt directory; its script.ts belongs to the dbt + // project, not to a module folder of its own. + expect(isModuleEntryPoint("f/x/proj__dbt/models/legacy__mod/script.ts")).toBe( + false, + ); + expect(isModuleEntryPoint("f/x/proj__dbt/models/a.sql")).toBe(false); +}); + +test("a __dbt directory nested inside an ordinary module is not a dbt project file", () => { + expect(isDbtModulePath("f/x/proj__dbt/models/a.sql")).toBe(true); + // `vendor/x__dbt/` belongs to the `foo__mod` script, not to a dbt project. + expect(isDbtModulePath("f/x/foo__mod/vendor/x__dbt/a.ts")).toBe(false); + expect(isDbtModulePath("f/x/foo__mod/helper.ts")).toBe(false); +}); + +test("a __dbt/target nested in an ordinary module is not generated dbt output", () => { + expect(isDbtGeneratedPath("f/x/proj__dbt/target/manifest.json")).toBe(true); + // Belongs to the `foo__mod` script; excluding it would drop a real edit. + expect(isDbtGeneratedPath("f/x/foo__mod/vendor/x__dbt/target/a.ts")).toBe(false); +}); diff --git a/cli/windmill-utils-internal/src/deploy.ts b/cli/windmill-utils-internal/src/deploy.ts index 42874fda97..047d50ee78 100644 --- a/cli/windmill-utils-internal/src/deploy.ts +++ b/cli/windmill-utils-internal/src/deploy.ts @@ -460,6 +460,10 @@ export async function deployItem( ...flow, preserve_on_behalf_of: preserveOnBehalfOf, on_behalf_of_email: onBehalfOf, + // Usernames are per-workspace, so the source's principal names nobody in + // the target (or names a different person). Clearing it lets the backend + // derive the target's own principal from the email above. + on_behalf_of: undefined, }, }); } else { @@ -469,6 +473,10 @@ export async function deployItem( ...flow, preserve_on_behalf_of: preserveOnBehalfOf, on_behalf_of_email: onBehalfOf, + // Usernames are per-workspace, so the source's principal names nobody in + // the target (or names a different person). Clearing it lets the backend + // derive the target's own principal from the email above. + on_behalf_of: undefined, }, }); } @@ -493,6 +501,8 @@ export async function deployItem( parent_hash: parentHash, preserve_on_behalf_of: preserveOnBehalfOf, on_behalf_of_email: onBehalfOf, + // See the flow branch: a source-workspace principal is never valid here. + on_behalf_of: undefined, }, }); } else if (kind === "app" || kind === "raw_app") { diff --git a/cli/windmill-utils-internal/src/path-utils/path-assigner.ts b/cli/windmill-utils-internal/src/path-utils/path-assigner.ts index 94d9085b94..6aaeea4db1 100644 --- a/cli/windmill-utils-internal/src/path-utils/path-assigner.ts +++ b/cli/windmill-utils-internal/src/path-utils/path-assigner.ts @@ -36,6 +36,7 @@ export const LANGUAGE_EXTENSIONS: Record = { bunnative: "ts", ruby: "rb", rlang: "r", + dbt: "dbt.yaml", // for related places search: ADD_NEW_LANG }; @@ -84,6 +85,7 @@ export const EXTENSION_TO_LANGUAGE: Record = { "playbook.yml": "ansible", "java": "java", "duckdb.sql": "duckdb", + "dbt.yaml": "dbt", "rb": "ruby", // Plain .ts defaults to bun (will be overridden by defaultTs setting) "ts": "bun", diff --git a/debugger/Dockerfile b/debugger/Dockerfile index 51e993746a..a80643e823 100644 --- a/debugger/Dockerfile +++ b/debugger/Dockerfile @@ -46,7 +46,9 @@ WORKDIR /app # Copy the debug service files COPY dap_debug_service.ts . COPY dap_websocket_server_bun.ts . +COPY env_passthrough.ts . COPY dap_websocket_server.py . +COPY registry_config.ts . # Expose the default port EXPOSE 5679 diff --git a/debugger/README.md b/debugger/README.md index dbeb213351..1dba49d977 100644 --- a/debugger/README.md +++ b/debugger/README.md @@ -75,6 +75,122 @@ Options: | `DAP_NSJAIL_PATH` | nsjail binary path | nsjail | | `DAP_NSJAIL_CONFIG` | nsjail config file path | - | +### Dependency preparation + +Before debugging a script, its imports are installed through `windmill prepare-deps`, which runs +`uv` (Python) or `bun install` (TypeScript) without a database connection. The install runs in the +service rather than in the session because the registry configuration usually embeds credentials +and a debug server executes the submitted script inside a process the script can read; the Python +server is handed only the resulting venv, with `--venv-path`, and a Bun session only the resulting +`node_modules`. + +`DAP_PREPARE_DEPS_TIMEOUT_MS` bounds the install (default 120000); past it the session starts +without its dependencies. When the install fails, the CLI answers `success: false` and carries the +installer's stderr in both `error` and `install_stderr`; the service reports it to the client as an +`output` event, so the reason (unreachable mirror, untrusted certificate, unknown package) reaches +the user instead of a bare `ModuleNotFoundError` at the first import. + +### Registry configuration + +Because `prepare-deps` has no database, the service reads the instance settings for it from +`GET /api/debug/registry_config` on `WINDMILL_BASE_URL` and passes them down over the CLI's stdin +request. It is authorized by the launch token of the session being started, and serves only the +settings that session's own installer runs on, so a TypeScript session's token cannot be used to +read the Python index credentials. + +The token also reaches the browser, so what it can fetch is what a workspace member can fetch. +Sessions started by an operator are refused outright, since an operator cannot run a preview job +either; for a member who can, the npm settings are already exposed by a preview (a worker leaves +the same `.npmrc` / `bunfig.toml` in the directory the previewed script runs in), while the Python +index URL, which otherwise only appears as uv's argv, becomes readable where it was not before. + +These settings are Enterprise-only, exactly as they are for jobs, and a CE instance reports that in +the session's output rather than applying them: + +| Setting | Applies to | +|---------|------------| +| `npm_config_registry` | `bun install` registry and its `:_authToken=` | +| `npmrc` | written verbatim as `.npmrc`, taking precedence over `npm_config_registry` | +| `bunfig_install_scopes` | `[install.scopes]` in the generated `bunfig.toml` | +| `pip_index_url` | `uv --index-url` | +| `pip_extra_index_url` | `uv --extra-index-url`, comma-separated | + +`uv_index_strategy` is served on any edition, like it is to a worker. An index URL holding the +`EPHEMERAL_TOKEN` placeholder is not served at all: only a worker can run the command that +substitutes it. + +The credential-bearing files (`.npmrc`, `bunfig.toml`) are written under +`/var/tmp/windmill-debug-registry`, not into the directory the install runs in, and are deleted +when the install ends. That directory is not private to the install: a session resolves its +`node_modules` symlink back into it, and `nsjail.debug.config.proto` bind-mounts the whole of +`/tmp` into every session, so credentials left there would be readable by a concurrent session. +`/var/tmp` is a tmpfs in that same config, one instance per jail, so a session sees an empty one +and a jailed install's credentials go away with the jail even when it is killed (the service kills +an install with SIGKILL, which no cleanup in the installer can survive). An install running +unjailed writes to the host's `/var/tmp` instead, where a directory a kill left behind is removed +by the next install; a session running unjailed is unconfined anyway and sees the whole filesystem, +as it already does the rest of the service's state. + +The rest of the registry configuration has no instance setting and is read from the environment of +the debug service. Where two names are listed the first wins; a worker reads the same names: + +| Variable | Description | Default | +|----------|-------------|---------| +| `PY_TRUSTED_HOST` / `PIP_TRUSTED_HOST` | Hosts to trust, whitespace-separated (`--trusted-host`) | - | +| `PY_INDEX_CERT` / `PIP_INDEX_CERT` | CA bundle for the index, passed to uv as `SSL_CERT_FILE`. Falls back to `SSL_CERT_FILE`, then `REQUESTS_CA_BUNDLE`, then `CURL_CA_BUNDLE`, so a host that configures its CA under any of those names is picked up. Whichever is used **replaces** uv's own roots rather than adding to them, so it has to be a complete bundle: one holding only a private CA leaves every public index untrusted. `bun install` gets the same bundle as `NODE_EXTRA_CA_CERTS`, the only spelling Bun reads | - | +| `SSL_CERT_DIR` | Directory of certificates, forwarded to uv as-is. Replaces uv's roots the same way the bundle does, so a directory holding only a private CA leaves public indexes untrusted | - | +| `PY_NATIVE_CERT` / `UV_NATIVE_TLS` | `true` to also trust the platform certificate store (`--native-tls`) | false | +| `UV_HTTP_TIMEOUT` | uv HTTP request timeout, in seconds | uv's own default | +| `DAP_REGISTRY_CONFIG_TIMEOUT_MS` | How long to wait on the settings fetch before installing without it | 10000 | + +`PY_INDEX_URL` / `PIP_INDEX_URL` and `PY_EXTRA_INDEX_URL` / `PIP_EXTRA_INDEX_URL`, along with +`UV_INDEX_STRATEGY`, are still read from the same environment whenever the fetch yields no index: +because the instance has none set, because this is a CE instance, or because the session was not +allowed the settings. A Python debug service configured that way therefore keeps working, but setting +them is an instance-wide decision to install Python dependencies from that index, independent of who +opened the session; leave them unset to let the instance settings alone decide. The npm settings have +no such fallback: the instance settings are the only source. + +Proxy variables (`HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`, in either case) are forwarded from the +service into each session, since the debugged script needs them for its own outbound calls, exactly +as a job's script does on a worker. When a proxy is set without a bypass list, `NO_PROXY` defaults +to `localhost,127.0.0.1` so calls to `BASE_INTERNAL_URL` are not proxied. + +Trust roots are forwarded alongside them: `SSL_CERT_FILE`, `SSL_CERT_DIR`, `REQUESTS_CA_BUNDLE`, +`CURL_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS`. Behind a TLS-intercepting proxy these are what let the +debugged script's own HTTPS calls verify, and installing the CA in the container's system store is +not enough on its own, since `requests` carries its own bundle and Node reads only +`NODE_EXTRA_CA_CERTS`. Registry settings are deliberately not forwarded: they carry credentials and +only the service needs them. + +Registering that CA in the container's system store happens on its own: mount it into +`/usr/local/share/ca-certificates/` **named `*.crt`**, the only extension `update-ca-certificates` +reads, and `windmill_extra` runs it before starting any service. `RUN_UPDATE_CA_CERTIFICATE_AT_START=true` forces the same thing whether or not +certificates are mounted there, and `RUN_UPDATE_CA_CERTIFICATE_PATH` overrides the tool, matching +the server and worker. Both are best-effort: a UID that cannot write `/etc/ssl/certs` logs a warning +and the container still boots. `INIT_SCRIPT` remains the hook for anything more involved, and unlike +the CA update it aborts startup when it fails. + +Note what the system store does *not* cover, which is most of what a debug session installs with: +uv trusts its own bundled roots unless `PY_NATIVE_CERT`/`UV_NATIVE_TLS` is `true`, Bun and Node read +only `NODE_EXTRA_CA_CERTS`, and `requests` carries certifi. Registering the CA fixes Python's stdlib +`ssl`, `curl` and `git`; the rest still needs the variables above. + +Keeping the settings out of the session's environment only bounds what the debugged script can read +from itself. An unsandboxed session runs under the same user as the service and can still read the +service's environment through `/proc`, the same way a job can read a worker's when the worker runs +unsandboxed. Isolating sessions from the service takes `--nsjail --nsjail-config +nsjail.debug.config.proto`: it is that config's PID namespace and `mount_proc` that put the service +out of reach, not the flag on its own. + +The installer is jailed on the same terms, in both languages: `uv pip install` builds source +distributions and `bun install` runs postinstall scripts, so a package's own code executes there +too. It keeps the service's environment across that boundary — the config sets `keep_env`, which is +how the settings above reach it — so replacing that with an allowlist would have to carry the +registry and CA variables in explicitly. It also runs in its own process group, because `uv` and +`bun` are grandchildren: signalling only the installer reparents them to init and they keep +downloading, which would make the timeout and the cancel-on-disconnect half-measures. + ### Frontend Integration ```svelte diff --git a/debugger/dap_debug_service.ts b/debugger/dap_debug_service.ts index 2b9ce46272..11616f7b11 100644 --- a/debugger/dap_debug_service.ts +++ b/debugger/dap_debug_service.ts @@ -40,7 +40,14 @@ import { tmpdir } from 'node:os' import { join } from 'node:path' // Import the working Bun debug session from the standalone server -import { DebugSession as BunDebugSessionWorking, type NsjailConfig } from './dap_websocket_server_bun' +import { + DebugSession as BunDebugSessionWorking, + killProcessTree, + nsjailWrap, + type NsjailConfig +} from './dap_websocket_server_bun' +import { sessionEnv } from './env_passthrough' +import { fetchRegistryConfig, type RegistryConfig } from './registry_config' // ============================================================================ // Configuration @@ -348,39 +355,35 @@ interface SpawnOptions { cmd: string[] cwd?: string env?: Record + stdin?: Blob + /** + * Hand the child this process's whole environment rather than the minimal set below. Only + * the dependency installer wants it: the registry credentials and CA settings it reads are + * precisely what the minimal set exists to keep away from a debugged script. + */ + inheritEnv?: boolean + /** + * Start the child in its own process group, so killProcessTree can signal what it spawns. + */ + detached?: boolean stdout?: 'pipe' | 'inherit' stderr?: 'pipe' | 'inherit' } +/** + * How long `windmill prepare-deps` may take before the session gives up on it and starts without + * the dependencies. Raise it for slow private mirrors, where a large install can outlast the default. + */ +const PREPARE_DEPS_TIMEOUT_MS = Number(process.env.DAP_PREPARE_DEPS_TIMEOUT_MS) || 120_000 + /** * Spawn a process, optionally wrapped with nsjail. * This is the key function for sandboxed execution. */ function spawnProcess(options: SpawnOptions): Subprocess { - let cmd = options.cmd + const cmd = nsjailWrap(options.cmd, config.nsjail, options.cwd) if (config.nsjail.enabled) { - // Build nsjail command - const nsjailCmd = [config.nsjail.binaryPath] - - // Add config file if specified - if (config.nsjail.configPath) { - nsjailCmd.push('--config', config.nsjail.configPath) - } - - // Add any extra nsjail arguments - nsjailCmd.push(...config.nsjail.extraArgs) - - // Add working directory if specified - if (options.cwd) { - nsjailCmd.push('--cwd', options.cwd) - } - - // Separator and actual command - nsjailCmd.push('--') - nsjailCmd.push(...cmd) - - cmd = nsjailCmd logger.info(`Spawning with nsjail: ${cmd.join(' ')}`) } else { logger.info(`Spawning: ${cmd.join(' ')}`) @@ -391,9 +394,12 @@ function spawnProcess(options: SpawnOptions): Subprocess { return spawn({ cmd, cwd: options.cwd || process.cwd(), + ...(options.stdin ? { stdin: options.stdin } : {}), + ...(options.detached ? { detached: true } : {}), stdout: options.stdout || 'pipe', stderr: options.stderr || 'pipe', env: { + ...(options.inheritEnv ? process.env : {}), // Essential system vars PATH: process.env.PATH || '/usr/bin:/bin', HOME: process.env.HOME, @@ -489,6 +495,15 @@ abstract class BaseDebugSession { // Python Debug Session // ============================================================================ +const DEFAULT_DEBUGPY_TIMEOUT_MS = 10_000 + +// `launch` waits on dependency preparation in the Python server, which allows `windmill +// prepare-deps` up to 120s; anything shorter here reports a timeout while the install is +// still legitimately running. +const DEBUGPY_TIMEOUT_MS_BY_COMMAND: Record = { + launch: 180_000 +} + class PythonDebugSession extends BaseDebugSession { private debugpyWs: WebSocket | null = null private debugpySeq = 1 @@ -502,6 +517,9 @@ class PythonDebugSession extends BaseDebugSession { private scriptResult: unknown = undefined private envVars: Record = {} private windmillPath?: string + private venvPath?: string + private prepareDepsProcess: Subprocess | null = null + private disposed = false private debugMode: boolean constructor(ws: { send: (data: string) => void; close: () => void }, windmillPath?: string, debugMode = false) { @@ -531,11 +549,13 @@ class PythonDebugSession extends BaseDebugSession { arguments: args } + const timeoutMs = DEBUGPY_TIMEOUT_MS_BY_COMMAND[command] ?? DEFAULT_DEBUGPY_TIMEOUT_MS + return new Promise((resolve, reject) => { const timeout = setTimeout(() => { this.pendingDebugpyRequests.delete(seq) - reject(new Error(`Debugpy command timeout: ${command}`)) - }, 10000) + reject(new Error(`Debugpy command timeout: ${command} (after ${timeoutMs}ms)`)) + }, timeoutMs) this.pendingDebugpyRequests.set(seq, { resolve: (value) => { @@ -627,6 +647,110 @@ class PythonDebugSession extends BaseDebugSession { } } + /** + * Install the script's imports through `windmill prepare-deps` and return the venv to add to + * the debugged script's sys.path. + * + * This runs here rather than in the Python server because the registry settings the CLI is + * given routinely embed private-registry credentials, and the Python server executes the + * submitted script inside its own interpreter: anything in that process is recoverable by the + * script. The service never executes user code, so the credentials stop here. + * + * It still goes through spawnProcess so nsjail confines it on the same terms as the debuggee: + * `uv pip install` builds source distributions, which executes their build backend's arbitrary + * Python. Those same credentials are what `inheritEnv` is for — the jail config keeps the + * environment across the boundary, so nothing else has to carry them in. + */ + private async prepareDependencies(code: string, registry: RegistryConfig): Promise { + if (!this.windmillPath) { + logger.info('No windmill binary path configured, skipping dependency preparation') + return null + } + + const warn = (reason: string): null => { + // cleanup() kills the installer, which ends the read with nothing to parse. Reporting + // that as an install failure blames the user for their own disconnect, on a websocket + // that is being torn down anyway. + if (this.disposed) { + return null + } + logger.error(`prepare-deps failed: ${reason}`) + this.sendEvent('output', { + category: 'stderr', + output: `Failed to prepare dependencies: ${reason}\n` + }) + return null + } + + try { + const proc = spawnProcess({ + cmd: [this.windmillPath, 'prepare-deps'], + // The venv has to be built against the interpreter that will run the script: its + // site-packages goes on that interpreter's sys.path, and uv otherwise picks its + // own, which silently leaves compiled extensions unimportable. + stdin: new Blob([ + JSON.stringify({ + code, + language: 'python3', + python_path: config.pythonPath, + registry + }) + '\n' + ]), + inheritEnv: true, + detached: true + }) + this.prepareDepsProcess = proc + + // The launch response is already sent, so an install that never returns would leave the + // client waiting on a session that never starts, with nothing on screen. The deadline + // races the read rather than only killing the child: a grandchild holding the pipe open + // keeps the read pending long after the child itself is gone. + let timer: ReturnType | undefined + // spawnProcess's return type does not carry the piped stdio through + const read = (async () => ({ + output: await new Response(proc.stdout as ReadableStream).text(), + stderr: await new Response(proc.stderr as ReadableStream).text() + }))() + const result = await Promise.race([ + read, + new Promise((resolve) => { + timer = setTimeout(() => resolve(null), PREPARE_DEPS_TIMEOUT_MS) + }) + ]) + clearTimeout(timer) + this.prepareDepsProcess = null + + if (!result) { + killProcessTree(proc) + return warn( + `dependency installation timed out after ${PREPARE_DEPS_TIMEOUT_MS / 1000}s` + ) + } + const { output, stderr } = result + + const lastLine = output.trim().split('\n').pop() || '' + if (!lastLine.startsWith('{')) { + return warn(stderr.trim() || 'windmill binary produced no response') + } + + const response = JSON.parse(lastLine) + if (!response.success) { + // install_stderr is the installer's raw output; `error` already contains it, so + // prefer whichever the CLI version at hand provides. + return warn(response.install_stderr || response.error || 'unknown error') + } + + if (response.venv_path) { + logger.info(`Dependencies installed at: ${response.venv_path}`) + } else { + logger.info('No external dependencies to install') + } + return response.venv_path || null + } catch (error) { + return warn(String(error)) + } + } + private async startPythonProcess(cwd: string): Promise { if (!this.scriptPath) { throw new Error('No script path') @@ -648,10 +772,11 @@ class PythonDebugSession extends BaseDebugSession { '--host', '127.0.0.1' ] - // Pass windmill path for dependency auto-installation if configured - if (this.windmillPath) { - cmd.push('--windmill', this.windmillPath) - logger.info(`Python session: autoinstall enabled with windmill at ${this.windmillPath}`) + // Dependencies are installed by the service (see prepareDependencies), so the server is + // handed the resulting venv instead of the windmill binary it would install with. + if (this.venvPath) { + cmd.push('--venv-path', this.venvPath) + logger.info(`Python session: using dependencies at ${this.venvPath}`) } // Pass debug flag to Python subprocess @@ -662,7 +787,7 @@ class PythonDebugSession extends BaseDebugSession { this.process = spawnProcess({ cmd, cwd, - env: { PYTHONUNBUFFERED: '1', ...this.envVars } + env: { PYTHONUNBUFFERED: '1', ...sessionEnv(), ...this.envVars } }) // Read stderr to capture startup messages @@ -794,6 +919,13 @@ class PythonDebugSession extends BaseDebugSession { this.debugpyWs.onclose = () => { logger.info('Debugpy WebSocket closed') this.debugpyWs = null + // A Python server that dies mid-request must fail it now; otherwise the caller + // waits out the command timeout, which for `launch` is minutes. + const aborted = Array.from(this.pendingDebugpyRequests.values()) + this.pendingDebugpyRequests.clear() + for (const pending of aborted) { + pending.reject(new Error('Debugpy connection closed')) + } } }) } @@ -905,6 +1037,10 @@ class PythonDebugSession extends BaseDebugSession { } private async handleLaunch(request: DAPMessage): Promise { + // Per launch, not per session: cleanup() also runs when a program finishes normally, and + // the flag must only mean "torn down while this launch was still preparing". + this.disposed = false + const args = request.arguments || {} let code = args.code as string | undefined this.scriptPath = args.program as string | undefined @@ -912,6 +1048,8 @@ class PythonDebugSession extends BaseDebugSession { this.callMain = (args.callMain as boolean) || false this.mainArgs = (args.args as Record) || {} this.envVars = (args.env as Record) || {} + // Also what authorizes the registry configuration fetch below. + const token = args.token as string | undefined // Enforce signing on every launch. The token is passed in the launch // arguments and is verified against the inline `code` (see windmill-api-debug). @@ -925,7 +1063,6 @@ class PythonDebugSession extends BaseDebugSession { return } - const token = args.token as string | undefined if (!token) { logger.error('No debug token provided but signed requests are required') this.sendResponse(request, false, {}, 'Debug token required. Ensure the debug session was signed by the backend.') @@ -987,6 +1124,32 @@ sys.stdout.flush() this.sendResponse(request) try { + if (code) { + const registry = await fetchRegistryConfig(token, logger) + // A round trip of its own, during which the client can give up: the installer runs + // a source distribution's build backend, so starting one for a session that is + // already gone executes package code nobody is waiting for. + if (this.disposed) { + logger.info('Session torn down during the registry configuration fetch, not installing') + await this.cleanup() + return + } + if (registry.message) { + this.sendEvent('output', { category: 'console', output: `${registry.message}\n` }) + } + this.venvPath = (await this.prepareDependencies(code, registry)) ?? undefined + } + + // Installing takes long enough for the client to give up meanwhile, and cleanup() has + // then already run: starting the debuggee now would leave a process nothing owns + // executing the script for a session that is gone. Clean up again on the way out, + // since a teardown that landed before the script was written left it behind. + if (this.disposed) { + logger.info('Session torn down during dependency preparation, not starting Python') + await this.cleanup() + return + } + await this.startPythonProcess(cwd) // Re-apply breakpoints to the Python server using the actual script path @@ -1012,7 +1175,13 @@ sys.stdout.flush() }) } catch (error) { this.sendEvent('output', { category: 'stderr', output: `Failed to start Python: ${error}\n` }) + // Claim the terminated event before cleanup kills the process, otherwise the + // `exited` handler sends a second one whose empty body erases this error. + this.terminatedSent = true this.sendEvent('terminated', { error: String(error) }) + // A Python server that refused the launch stays in its connection loop, so + // nothing else ever reaps it, its websocket or the temp dir. + await this.cleanup() } } @@ -1036,6 +1205,14 @@ sys.stdout.flush() } async cleanup(): Promise { + this.disposed = true + + // A client that gives up mid-install must not leave the package manager running + if (this.prepareDepsProcess) { + killProcessTree(this.prepareDepsProcess) + this.prepareDepsProcess = null + } + if (this.debugpyWs) { this.debugpyWs.close() this.debugpyWs = null diff --git a/debugger/dap_websocket_server.py b/debugger/dap_websocket_server.py index 624b339894..33937d9617 100644 --- a/debugger/dap_websocket_server.py +++ b/debugger/dap_websocket_server.py @@ -277,12 +277,55 @@ class WindmillDebugger(bdb.Bdb): return {} +PREPARE_DEPS_TIMEOUT_SECONDS = 120 +PREPARE_DEPS_PROGRESS_INTERVAL_SECONDS = 5 + + +@dataclass +class PrepareResult: + """ + Outcome of dependency preparation. + + `error` holds anything worth telling the user, including a problem reported by an + otherwise successful preparation. Only `fatal` means the packages are known to be + missing: failing to reach the CLI at all says nothing about the script's imports and + must not block a session that would otherwise run. + """ + + venv_path: str | None = None + error: str | None = None + fatal: bool = False + + +def _prepare_error_detail(response: dict) -> str: + """ + Build the failure reason from a prepare-deps response. + + `error` is the installer's own output prefixed with the step that failed, and + `install_stderr` is that same output unprefixed, so take one rather than both: joining + them prints the installer's output twice, and the prefix is what tells a reader whether + the venv or the install was what went wrong. + """ + for key in ("error", "install_stderr"): + detail = str(response.get(key) or "").strip() + if detail: + return detail + return "unknown error" + + +def _first_line(detail: str, limit: int = 300) -> str: + """Condense a multi-line failure into the single line a DAP response message allows.""" + line = next((s.strip() for s in detail.splitlines() if s.strip()), detail.strip()) + return line[:limit] + + class DebugSession: """Manages a single debug session.""" - def __init__(self, websocket, windmill_path: str | None = None): + def __init__(self, websocket, windmill_path: str | None = None, prepared_venv_path: str | None = None): self.websocket = websocket self.windmill_path = windmill_path + self._prepared_venv_path = prepared_venv_path self.seq = 1 self.initialized = False self.configured = False @@ -304,14 +347,22 @@ class DebugSession: self.seq += 1 return seq - def prepare_dependencies(self, code: str) -> str | None: + def prepare_dependencies(self, code: str) -> PrepareResult: """ Prepare Python dependencies by calling the windmill CLI. - Returns the path to the venv's site-packages directory, or None if no dependencies needed. + + Blocks for as long as the install takes, so it must run off the event loop; use + `_prepare_dependencies_with_progress` instead of calling this directly. """ + if self._prepared_venv_path: + # The debug service installs dependencies itself so that the registry credentials + # the CLI needs never enter this interpreter, which executes the debugged script. + logger.info(f"Using dependencies prepared by the debug service: {self._prepared_venv_path}") + return PrepareResult(venv_path=self._prepared_venv_path) + if not self.windmill_path: logger.info("No windmill binary path configured, skipping dependency preparation") - return None + return PrepareResult() logger.info(f"Preparing dependencies using {self.windmill_path}") @@ -328,7 +379,7 @@ class DebugSession: input=input_data, capture_output=True, text=True, - timeout=120, # 2 minute timeout for dependency installation + timeout=PREPARE_DEPS_TIMEOUT_SECONDS, ) elapsed = time.time() - start_time @@ -337,7 +388,11 @@ class DebugSession: if result.returncode != 0: logger.error(f"prepare-deps failed (stderr): {result.stderr}") logger.error(f"prepare-deps failed (stdout): {result.stdout}") - return None + detail = (result.stderr or "").strip() or (result.stdout or "").strip() + return PrepareResult( + error=detail or f"windmill prepare-deps exited with code {result.returncode}", + fatal=True, + ) # Log raw output for debugging logger.debug(f"prepare-deps stdout: {result.stdout[:500] if result.stdout else '(empty)'}") @@ -350,15 +405,18 @@ class DebugSession: json_start = output.find('{') if json_start == -1: logger.error(f"No JSON in prepare-deps output: {output}") - return None + return PrepareResult( + error=f"No JSON in prepare-deps output: {output[:500] or '(empty)'}" + ) json_str = output[json_start:] response = json.loads(json_str) logger.debug(f"prepare-deps response: {response}") if not response.get("success"): - logger.error(f"prepare-deps error: {response.get('error')}") - return None + detail = _prepare_error_detail(response) + logger.error(f"prepare-deps error: {detail}") + return PrepareResult(error=detail, fatal=True) venv_path = response.get("venv_path") cached = response.get("cached", False) @@ -371,18 +429,50 @@ class DebugSession: else: logger.info("No external dependencies detected in code") - return venv_path + return PrepareResult(venv_path=venv_path) except subprocess.TimeoutExpired: - logger.error("prepare-deps timed out after 120s") - return None + message = f"prepare-deps timed out after {PREPARE_DEPS_TIMEOUT_SECONDS}s" + logger.error(message) + return PrepareResult(error=message, fatal=True) except json.JSONDecodeError as e: + raw = output[:500] if 'output' in dir() else '(not available)' logger.error(f"Failed to parse prepare-deps JSON output: {e}") - logger.error(f"Raw output was: {output[:500] if 'output' in dir() else '(not available)'}") - return None + logger.error(f"Raw output was: {raw}") + return PrepareResult(error=f"Failed to parse prepare-deps output: {e}\n{raw}") except Exception as e: logger.exception(f"Error preparing dependencies: {e}") - return None + return PrepareResult(error=f"Error preparing dependencies: {e}") + + async def _prepare_dependencies_with_progress(self, code: str) -> PrepareResult: + """ + Run dependency preparation on a worker thread, reporting progress while it runs. + + The install can take minutes on a cold cache; on the event loop it would stall + websocket keepalive until it returns and block the progress events below. + """ + await self.send_event( + "output", {"category": "stdout", "output": "Preparing dependencies...\n"} + ) + + task = asyncio.create_task(asyncio.to_thread(self.prepare_dependencies, code)) + waited = 0 + while True: + done, _ = await asyncio.wait( + {task}, timeout=PREPARE_DEPS_PROGRESS_INTERVAL_SECONDS + ) + if done: + break + waited += PREPARE_DEPS_PROGRESS_INTERVAL_SECONDS + await self.send_event( + "output", + { + "category": "stdout", + "output": f"Still preparing dependencies... ({waited}s)\n", + }, + ) + + return task.result() def _next_var_ref(self) -> int: ref = self._variables_ref_counter @@ -521,7 +611,25 @@ class DebugSession: # Prepare dependencies before modifying the code if code: - self._venv_path = self.prepare_dependencies(code) + prepared = await self._prepare_dependencies_with_progress(code) + if prepared.error: + prefix = ( + "Failed to prepare dependencies" + if prepared.fatal + else "Warning: dependency preparation reported a problem, running anyway" + ) + await self.send_event( + "output", + {"category": "stderr", "output": f"{prefix}:\n{prepared.error}\n"}, + ) + if prepared.fatal: + await self.send_response( + request, + success=False, + message=f"Failed to prepare dependencies: {_first_line(prepared.error)}", + ) + return + self._venv_path = prepared.venv_path # If callMain is True, append a call to main() with the provided args if self._call_main and code: @@ -894,13 +1002,17 @@ class DebugSession: ) -# Module-level variable to store windmill binary path +# Module-level variables to store the windmill binary path and, when the debug service +# already installed the script's dependencies, the venv to use instead of installing here. _windmill_path: str | None = None +_prepared_venv_path: str | None = None async def handle_connection(websocket) -> None: """Handle a WebSocket connection.""" - session = DebugSession(websocket, windmill_path=_windmill_path) + session = DebugSession( + websocket, windmill_path=_windmill_path, prepared_venv_path=_prepared_venv_path + ) logger.info(f"New connection from {websocket.remote_address}") try: @@ -924,13 +1036,21 @@ async def handle_connection(websocket) -> None: session._cleanup_temp_file() -async def main(host: str = "localhost", port: int = 5679, windmill_path: str | None = None) -> None: +async def main( + host: str = "localhost", + port: int = 5679, + windmill_path: str | None = None, + prepared_venv_path: str | None = None, +) -> None: """Start the DAP WebSocket server.""" - global _windmill_path + global _windmill_path, _prepared_venv_path _windmill_path = windmill_path + _prepared_venv_path = prepared_venv_path if windmill_path: logger.info(f"Windmill binary path: {windmill_path}") + if prepared_venv_path: + logger.info(f"Dependencies prepared by the debug service: {prepared_venv_path}") logger.info(f"Starting DAP WebSocket server on ws://{host}:{port}") async with serve(handle_connection, host, port): @@ -944,6 +1064,7 @@ if __name__ == "__main__": parser.add_argument("--host", default="localhost", help="Host to bind to") parser.add_argument("--port", type=int, default=5679, help="Port to listen on") parser.add_argument("--windmill", help="Path to windmill binary for dependency preparation (or set WINDMILL_PATH env var)") + parser.add_argument("--venv-path", help="Site-packages directory of a venv the caller already prepared; skips dependency installation") parser.add_argument("--debug", action="store_true", help="Enable debug logging") args = parser.parse_args() @@ -957,6 +1078,6 @@ if __name__ == "__main__": windmill_path = args.windmill or os.environ.get("WINDMILL_PATH") try: - asyncio.run(main(args.host, args.port, windmill_path)) + asyncio.run(main(args.host, args.port, windmill_path, args.venv_path)) except KeyboardInterrupt: logger.info("Server stopped") diff --git a/debugger/dap_websocket_server_bun.ts b/debugger/dap_websocket_server_bun.ts index d6ed8967ad..3b5072b1e6 100644 --- a/debugger/dap_websocket_server_bun.ts +++ b/debugger/dap_websocket_server_bun.ts @@ -22,9 +22,12 @@ */ import { spawn, type Subprocess } from 'bun' +import { readFileSync } from 'node:fs' import { mkdtemp, writeFile, unlink, rmdir, symlink } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' +import { sessionEnv } from './env_passthrough' +import { fetchRegistryConfig, type RegistryConfig } from './registry_config' // Types for V8 Inspector Protocol interface V8Message { @@ -222,6 +225,8 @@ function generateMainCallArgs(code: string, args: Record): stri const WINDMILL_BASE_URL = process.env.WINDMILL_BASE_URL || process.env.BASE_INTERNAL_URL // e.g., http://localhost:8000 const REQUIRE_SIGNED_REQUESTS = process.env.REQUIRE_SIGNED_DEBUG_REQUESTS !== 'false' +const PREPARE_DEPS_TIMEOUT_MS = 120_000 + // Opt-in cross-origin protection (CSWSH defense-in-depth); see // dap_debug_service.ts for the rationale. Only enforced for this file's // standalone Bun.serve entrypoint (the windmill-extra runtime imports the @@ -554,6 +559,63 @@ export interface NsjailConfig { extraArgs?: string[] } +/** + * Wrap a command so nsjail runs it, or return it unchanged when sandboxing is off. + * The environment is not filtered here: the config sets `keep_env`, so the jailed process + * receives whatever the spawning call gives it. + */ +export function nsjailWrap(cmd: string[], nsjail: NsjailConfig | undefined, cwd?: string): string[] { + if (!nsjail?.enabled) { + return cmd + } + const wrapped = [nsjail.binaryPath] + if (nsjail.configPath) { + wrapped.push('--config', nsjail.configPath) + } + if (nsjail.extraArgs) { + wrapped.push(...nsjail.extraArgs) + } + if (cwd) { + wrapped.push('--cwd', cwd) + } + wrapped.push('--', ...cmd) + return wrapped +} + +/** + * SIGKILL a subprocess along with everything it spawned. + * + * SIGKILL because `windmill prepare-deps` does not act on SIGTERM while uv is running. The + * whole group because uv is a grandchild: signalling the child alone reparents uv to init and + * it keeps downloading. The group id is read back from /proc instead of assumed, since a group + * kill aimed at this service's own group would take down every service in the container; a + * child spawned without `detached` therefore only gets the plain kill. + */ +export function killProcessTree(proc: Subprocess): void { + if (proc.exitCode !== null || proc.signalCode !== null) { + // Nothing left to signal, and the pid may already have been handed to someone else + return + } + + let ownsGroup = false + try { + const stat = readFileSync(`/proc/${proc.pid}/stat`, 'utf8') + // The comm field can hold spaces and parentheses, so read the fields after its closing one + ownsGroup = Number(stat.slice(stat.lastIndexOf(')') + 2).split(' ')[2]) === proc.pid + } catch { + // Already reaped, or not Linux: fall back to killing the process alone + } + try { + if (ownsGroup) { + process.kill(-proc.pid, 'SIGKILL') + } else { + proc.kill('SIGKILL') + } + } catch (error) { + logger.error('Failed to kill process:', error) + } +} + /** * VLQ (Variable-Length Quantity) decoder for source maps. * Returns array of decoded integers from VLQ string. @@ -751,6 +813,11 @@ export class DebugSession { // Path to installed node_modules (set after prepare-deps runs) private nodeModulesPath?: string + // Running dependency installer, so a teardown mid-install can stop it + private prepareDepsProcess: Subprocess | null = null + + private disposed = false + constructor(ws: WebSocket, options?: { nsjailConfig?: NsjailConfig; bunPath?: string; windmillPath?: string }) { this.ws = ws this.nsjailConfig = options?.nsjailConfig @@ -1435,6 +1502,10 @@ export class DebugSession { * Handle the 'launch' request. */ async handleLaunch(request: DAPMessage): Promise { + // Per launch, not per session: cleanup() also runs when a program finishes normally, and + // the flag must only mean "torn down while this launch was still preparing". + this.disposed = false + const args = request.arguments || {} let code = args.code as string | undefined this.scriptPath = args.program as string | undefined @@ -1442,6 +1513,8 @@ export class DebugSession { this.callMain = (args.callMain as boolean) || false this.mainArgs = (args.args as Record) || {} this.envVars = (args.env as Record) || {} + // Also what authorizes the registry configuration fetch below. + const token = args.token as string | undefined // Enforce signing on every launch. The token is passed in the launch // arguments and is verified against the inline `code` (see windmill-api-debug). @@ -1455,7 +1528,6 @@ export class DebugSession { return } - const token = args.token as string | undefined if (!token) { logger.error('No debug token provided but signed requests are required') this.sendResponse(request, false, {}, 'Debug token required. Ensure the debug session was signed by the backend.') @@ -1487,7 +1559,29 @@ export class DebugSession { // Prepare dependencies using the original code (before any modifications) // This analyzes imports and installs required npm packages if (code) { - this.nodeModulesPath = await this.prepareDependencies(code) || undefined + const registry = await fetchRegistryConfig(token, logger) + // A round trip of its own, during which the client can give up: the installer runs the + // packages' postinstall scripts, so starting one for a session that is already gone + // executes package code nobody is waiting for. + if (this.disposed) { + logger.info('Session was torn down during the registry configuration fetch, not installing') + this.sendResponse(request, false, {}, 'Session terminated during dependency preparation') + return + } + if (registry.message) { + this.sendEvent('output', { category: 'console', output: `${registry.message}\n` }) + } + this.nodeModulesPath = await this.prepareDependencies(code, registry) || undefined + + // Installing takes long enough for the client to give up meanwhile, and cleanup() has + // then already run: starting the debuggee now would leak a process nothing owns. + // The response still goes out, since a client that terminated without closing the + // socket is otherwise left waiting out its own launch timeout. + if (this.disposed) { + logger.info('Session was torn down during dependency preparation, not starting Bun') + this.sendResponse(request, false, {}, 'Session terminated during dependency preparation') + return + } // Remove version specifiers from imports (e.g., "lodash@4" -> "lodash") // This must happen AFTER prepareDependencies (which needs the versions) @@ -1570,8 +1664,21 @@ export class DebugSession { * Prepare dependencies by calling the windmill CLI's prepare-deps command. * This analyzes imports in the code and installs required npm packages. * Returns the path to node_modules if any were installed. + * + * Jailed on the same terms as the debuggee: `bun install` runs the packages' postinstall + * scripts, which is user-supplied code executing next to the other services in the container. + * Its environment is inherited rather than filtered, which is what carries the CA settings + * into the installer (the jail keeps the environment across the boundary). + * + * The CLI has no database, so `registry` carries the instance's registry settings down to it + * instead. They configure `bun install` and nothing else: the debugged script never gets + * them, since it could read them back out of the process it runs in. */ - private async prepareDependencies(code: string, language: string = 'bun'): Promise { + private async prepareDependencies( + code: string, + registry: RegistryConfig, + language: string = 'bun' + ): Promise { if (!this.windmillPath) { logger.info('No windmill binary path configured, skipping dependency preparation') return null @@ -1579,21 +1686,67 @@ export class DebugSession { logger.info(`Preparing dependencies using ${this.windmillPath}`) + // The launch response is only sent once this returns, so without progress a cold + // cache looks like a frozen debugger for as long as the install takes. + this.sendEvent('output', { category: 'console', output: 'Preparing dependencies...\n' }) + let waited = 0 + const progress = setInterval(() => { + waited += 5 + this.sendEvent('output', { + category: 'console', + output: `Still preparing dependencies... (${waited}s)\n` + }) + }, 5000) + let killTimer: ReturnType | undefined + let timedOut = false + try { - const input = JSON.stringify({ code, language }) + '\n' + const input = JSON.stringify({ code, language, registry }) + '\n' logger.info(`prepare-deps input length: ${input.length}`) - // Spawn the windmill binary with prepare-deps command + // Spawn the windmill binary with prepare-deps command. Its environment is inherited + // rather than filtered, which is what gives prepare-deps the container's index and + // certificate settings; the allowlist above is what keeps them from the debugged + // script, and the jail keeps them across its own boundary. + const cmd = nsjailWrap([this.windmillPath, 'prepare-deps'], this.nsjailConfig) + logger.info(`Spawning${this.nsjailConfig?.enabled ? ' with nsjail' : ''}: ${cmd.join(' ')}`) const proc = spawn({ - cmd: [this.windmillPath, 'prepare-deps'], + cmd, stdin: new Blob([input]), // Use Blob for complete stdin data stdout: 'pipe', - stderr: 'pipe' + stderr: 'pipe', + // So the installer and the bun it spawns can be killed as one group + detached: true }) + this.prepareDepsProcess = proc + + // Bound the wait: the only other ceiling is the DAP client's launch timeout, + // which is minutes, so a wedged installer would hang the session that long. + killTimer = setTimeout(() => { + timedOut = true + logger.error(`prepare-deps timed out after ${PREPARE_DEPS_TIMEOUT_MS}ms`) + killProcessTree(proc) + }, PREPARE_DEPS_TIMEOUT_MS) // Wait for completion const output = await new Response(proc.stdout).text() const stderr = await new Response(proc.stderr).text() + + // The read also ends when cleanup() kills the installer, which leaves no output to + // parse. Reporting that as an install failure blames the user for their own Stop. + if (this.disposed) { + return null + } + + if (timedOut) { + const errorMsg = `prepare-deps timed out after ${PREPARE_DEPS_TIMEOUT_MS / 1000}s` + this.sendEvent('output', { + category: 'console', + output: `Warning: Failed to prepare dependencies: ${errorMsg}\n` + }) + return null + } + logger.info(`prepare-deps output: ${output.substring(0, 200)}`) logger.info(`prepare-deps stderr: ${stderr.substring(0, 200)}`) @@ -1642,12 +1795,19 @@ export class DebugSession { logger.info('No external dependencies to install') return null } catch (error) { + if (this.disposed) { + return null + } logger.error(`Failed to prepare dependencies: ${error}`) this.sendEvent('output', { category: 'console', output: `Warning: Failed to prepare dependencies: ${error}\n` }) return null + } finally { + clearInterval(progress) + clearTimeout(killTimer) + this.prepareDepsProcess = null } } @@ -1663,24 +1823,13 @@ export class DebugSession { const inspectUrl = `127.0.0.1:${inspectPort}` // Build the command - optionally wrapped with nsjail - let cmd: string[] = [this.bunPath, `--inspect-wait=${inspectUrl}`, this.scriptPath] + const cmd = nsjailWrap( + [this.bunPath, `--inspect-wait=${inspectUrl}`, this.scriptPath], + this.nsjailConfig, + cwd + ) if (this.nsjailConfig?.enabled) { - const nsjailCmd = [this.nsjailConfig.binaryPath] - - if (this.nsjailConfig.configPath) { - nsjailCmd.push('--config', this.nsjailConfig.configPath) - } - - if (this.nsjailConfig.extraArgs) { - nsjailCmd.push(...this.nsjailConfig.extraArgs) - } - - nsjailCmd.push('--cwd', cwd) - nsjailCmd.push('--') - nsjailCmd.push(...cmd) - - cmd = nsjailCmd logger.info(`Starting Bun with nsjail: ${cmd.join(' ')}`) } else { logger.info(`Starting Bun with --inspect-wait=${inspectUrl}`) @@ -1698,12 +1847,16 @@ export class DebugSession { }, 10000) }) - // Only include essential env vars + client-provided ones + // Only include essential env vars + the network-config allowlist + client-provided ones. // Don't inherit all of process.env to keep debugger environment clean const envVars: Record = { // Essential system vars PATH: process.env.PATH || '/usr/bin:/bin', HOME: process.env.HOME, + // Proxy / TLS settings inherited from the container, before the client's env so an + // explicit override still wins. Package-index settings are deliberately absent: this + // runs user-supplied code and index URLs carry registry credentials. + ...sessionEnv(), // Client-provided env vars (WM_WORKSPACE, WM_TOKEN, etc.) // Note: WM_BASE_URL is already overridden by BASE_INTERNAL_URL if set ...this.envVars @@ -2453,15 +2606,23 @@ export class DebugSession { } /** - * Clean up resources. + * Clean up resources. Public because both servers call it when a client goes away. */ - private async cleanup(): Promise { + async cleanup(): Promise { + this.disposed = true + // Close inspector connection if (this.inspectorWs) { this.inspectorWs.close() this.inspectorWs = null } + // A disconnect during dependency installation must not leave bun install running + if (this.prepareDepsProcess) { + killProcessTree(this.prepareDepsProcess) + this.prepareDepsProcess = null + } + // Kill process if (this.process) { this.process.kill() @@ -2601,9 +2762,14 @@ if (import.meta.main) { logger.error('Error handling message:', error) } }, - close(ws) { + async close(ws) { logger.info('Client disconnected') - sessions.delete(ws) + const session = sessions.get(ws) + if (session) { + // Dropping the session without this leaves its installer and debuggee running + await session.cleanup() + sessions.delete(ws) + } } } }) diff --git a/debugger/env_passthrough.ts b/debugger/env_passthrough.ts new file mode 100644 index 0000000000..f713483d05 --- /dev/null +++ b/debugger/env_passthrough.ts @@ -0,0 +1,43 @@ +/** + * Container network configuration forwarded to a debug session, matching what a worker gives a + * job's script. The session environment is built from an allowlist rather than inherited, so an + * outbound proxy or a private CA is unreachable from a session unless these are passed + * explicitly. Registry settings are deliberately absent: they carry credentials and are consumed + * by the service itself (see PythonDebugSession.prepareDependencies). + * + * Lives in its own module because both session kinds build their own environment, and + * dap_debug_service.ts already imports from dap_websocket_server_bun.ts. + */ +export const SESSION_ENV_VARS = [ + 'HTTP_PROXY', + 'HTTPS_PROXY', + 'NO_PROXY', + // The lowercase spellings take precedence in the worker, so forward both. + 'http_proxy', + 'https_proxy', + 'no_proxy', + // Trust roots for a TLS-intercepting proxy. Installing the CA in the container's system + // store is not enough on its own: requests carries its own bundle and Node reads only + // NODE_EXTRA_CA_CERTS, so a debugged script's own HTTPS calls fail without these. + 'SSL_CERT_FILE', + 'SSL_CERT_DIR', + 'REQUESTS_CA_BUNDLE', + 'CURL_CA_BUNDLE', + 'NODE_EXTRA_CA_CERTS' +] + +export function sessionEnv(): Record { + const env: Record = {} + for (const key of SESSION_ENV_VARS) { + const value = process.env[key] + if (value) { + env[key] = value + } + } + // A proxy without a bypass list would send the script's calls to BASE_INTERNAL_URL through it; + // the worker defaults the same way (PROXY_ENVS in windmill-worker). + if (!env.NO_PROXY && !env.no_proxy && (env.HTTP_PROXY || env.http_proxy || env.HTTPS_PROXY || env.https_proxy)) { + env.NO_PROXY = 'localhost,127.0.0.1' + } + return env +} diff --git a/debugger/nsjail.debug.config.proto b/debugger/nsjail.debug.config.proto index 5e538e6002..65ea3456cf 100644 --- a/debugger/nsjail.debug.config.proto +++ b/debugger/nsjail.debug.config.proto @@ -63,6 +63,15 @@ mount { rw: true } +# Private scratch, one instance per jail. `windmill prepare-deps` writes the registry +# credentials here rather than into its install directory under the shared /tmp above, so no +# other session can read them, and they go away with the jail even when it is killed. +mount { + dst: "/var/tmp" + fstype: "tmpfs" + rw: true +} + # Debugger scripts directory (for Python debugger server) mount { src: "/debugger" diff --git a/debugger/registry_config.ts b/debugger/registry_config.ts new file mode 100644 index 0000000000..b7cb0451e5 --- /dev/null +++ b/debugger/registry_config.ts @@ -0,0 +1,84 @@ +/** + * Dependency-registry settings for a debug session's install. + * + * `windmill prepare-deps` installs a session's imports with no database connection, so the + * instance settings that point at a private npm or pip registry cannot be read there. They + * are fetched here instead, from the backend that signed the session's launch token, and + * passed down to the CLI over its stdin request. + * + * They stop at the installer. A registry URL usually embeds credentials and a debugged + * script can read whatever the process running it holds, so none of these values are ever + * put in a session's environment (see README.md, "Registry configuration"). + */ + +export interface RegistryConfig { + npm_config_registry?: string + npmrc?: string + bunfig_install_scopes?: string + pip_index_url?: string + pip_extra_index_url?: string + uv_index_strategy?: string + /** Why the instance's settings are not in this response, for the user to see. */ + message?: string +} + +const WINDMILL_BASE_URL = process.env.WINDMILL_BASE_URL || process.env.BASE_INTERNAL_URL + +/** + * Bounds how long a launch waits on the backend. The session can still start without the + * settings, it just installs from the public registries, so an unreachable backend must + * not hold it up for longer than the install itself would take. + */ +const FETCH_TIMEOUT_MS = Number(process.env.DAP_REGISTRY_CONFIG_TIMEOUT_MS) || 10_000 + +/** + * Fetch the registry settings for a session, authorized by its launch token. + * + * Never throws and never blocks a launch: on any failure it returns a config carrying only + * a `message`, so the session starts against the public registries and the user is told why + * instead of being left with an unexplained "package not found". + */ +export async function fetchRegistryConfig( + token: string | undefined, + logger: { info: (...args: unknown[]) => void; warn: (...args: unknown[]) => void } +): Promise { + if (!token || !WINDMILL_BASE_URL) { + return {} + } + + const url = `${WINDMILL_BASE_URL.replace(/\/$/, '')}/api/debug/registry_config` + try { + const response = await fetch(url, { + headers: { authorization: `Bearer ${token}` }, + signal: AbortSignal.timeout(FETCH_TIMEOUT_MS) + }) + if (response.status === 401 || response.status === 403 || response.status === 404) { + // Expected answers, not something the user can act on: a session that may not read + // the settings (an operator's) is refused, and a backend older than this image has + // no such route at all. Both install from the public registries. + logger.info(`Registry configuration not served for this session (${response.status})`) + return {} + } + if (!response.ok) { + const detail = (await response.text().catch(() => '')).trim() + return { + message: `Could not read the registry configuration (${response.status}): ${detail || response.statusText}` + } + } + + const config: RegistryConfig = await response.json() + // The values carry registry credentials, so only their names are logged. + const configured = Object.entries(config) + .filter(([key, value]) => key !== 'message' && value) + .map(([key]) => key) + logger.info( + configured.length > 0 + ? `Registry configuration from instance settings: ${configured.join(', ')}` + : 'No registry configuration set on the instance' + ) + return config + } catch (error) { + logger.warn(`Failed to fetch registry configuration: ${error}`) + return { message: `Could not read the registry configuration: ${error}` } + } +} diff --git a/debugger/test_dap_server.py b/debugger/test_dap_server.py index 45e138f2a3..17430ea25f 100644 --- a/debugger/test_dap_server.py +++ b/debugger/test_dap_server.py @@ -45,6 +45,10 @@ def main(x: str, count: int = 1): # Breakpoints for the main() test: lines 3 and 4 (inside main function) MAIN_BREAKPOINT_LINES = [3, 4] +# `launch` waits on dependency installation, so the import test below needs far more than +# the default budget on a cold cache. +REQUEST_TIMEOUTS = {"launch": 180.0} + class DAPTestClient: def __init__(self, url: str = "ws://localhost:5679"): @@ -103,7 +107,7 @@ class DAPTestClient: # Wait for response with timeout try: - response = await asyncio.wait_for(future, timeout=10.0) + response = await asyncio.wait_for(future, timeout=REQUEST_TIMEOUTS.get(command, 10.0)) return response except asyncio.TimeoutError: print(f"Timeout waiting for response to {command}") diff --git a/docker-compose.yml b/docker-compose.yml index 93fbf28e8d..a801a0ce7c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -187,6 +187,10 @@ services: # - DEBUG_ALLOWED_ORIGINS=https://your-windmill-host # Optional CSWSH hardening: comma-separated allowlist of browser Origins permitted to open debug WebSockets volumes: - lsp_cache:/pyls/.cache + # Behind a TLS-intercepting proxy, mount its CA here (as .crt) and it is registered in the + # system trust store before any service starts. That alone does not cover dependency + # installation — see debugger/README.md for the variables it also needs + # - ./corp-ca.crt:/usr/local/share/ca-certificates/corp-ca.crt:ro logging: *default-logging caddy: diff --git a/docker/DockerfileExtra b/docker/DockerfileExtra index 808bc6ed2a..9eb2a987f2 100644 --- a/docker/DockerfileExtra +++ b/docker/DockerfileExtra @@ -94,7 +94,9 @@ WORKDIR /debugger # Copy debugger files COPY debugger/dap_debug_service.ts . COPY debugger/dap_websocket_server_bun.ts . +COPY debugger/env_passthrough.ts . COPY debugger/dap_websocket_server.py . +COPY debugger/registry_config.ts . COPY debugger/nsjail.debug.config.proto . # Install Python debugger dependencies using uv diff --git a/docker/DockerfileFull b/docker/DockerfileFull index 6e3f218fd6..b827208c1d 100644 --- a/docker/DockerfileFull +++ b/docker/DockerfileFull @@ -45,6 +45,19 @@ RUN apt-get install -y ruby ruby-bundler RUN apt-get install -y r-base-dev \ && Rscript -e 'install.packages("renv", lib="/usr/lib/R/library", repos="https://cloud.r-project.org")' +# dbt +# NO dbt engine is baked in. Fusion may not be: its license grants only a +# non-transferable, non-sublicensable redistribution right. dbt-core 1.x cannot +# be, because its adapter is a Python package chosen per project. dbt-core 2.x +# could be — one adapter-agnostic binary — but shipping a pre-release nobody is +# defaulted onto costs a layer in every image and a version pinned in two places +# that nothing keeps in step. The worker fetches whichever engine a project asks +# for on first use and caches it (docs/dbt-runtime.md, decision 1). +# +# An operator who wants one pre-staged — an air-gapped instance, or a fleet that +# should not fetch per worker — populates `DBT_BUNDLED_DIR` (default +# /usr/local/dbt) with `core2x-/dbt-sa-cli` in their own image layer. + # Fix UV cache permissions for non-root user support (uid 1000, etc.) # The uv tool install ansible command populates the UV cache with root-owned files RUN chmod -R a+rw /tmp/windmill/cache/uv && \ diff --git a/docker/DockerfileFullEe b/docker/DockerfileFullEe index 14bfceca50..5b5347558a 100644 --- a/docker/DockerfileFullEe +++ b/docker/DockerfileFullEe @@ -78,6 +78,19 @@ RUN apt-get install -y iptables # Kerberos runtime RUN apt-get install -y libsasl2-modules-gssapi-mit krb5-user +# dbt +# NO dbt engine is baked in. Fusion may not be: its license grants only a +# non-transferable, non-sublicensable redistribution right. dbt-core 1.x cannot +# be, because its adapter is a Python package chosen per project. dbt-core 2.x +# could be — one adapter-agnostic binary — but shipping a pre-release nobody is +# defaulted onto costs a layer in every image and a version pinned in two places +# that nothing keeps in step. The worker fetches whichever engine a project asks +# for on first use and caches it (docs/dbt-runtime.md, decision 1). +# +# An operator who wants one pre-staged — an air-gapped instance, or a fleet that +# should not fetch per worker — populates `DBT_BUNDLED_DIR` (default +# /usr/local/dbt) with `core2x-/dbt-sa-cli` in their own image layer. + # Fix UV cache permissions for non-root user support (uid 1000, etc.) # The uv tool install ansible command populates the UV cache with root-owned files RUN chmod -R a+rw /tmp/windmill/cache/uv && \ diff --git a/docker/entrypoint-extra.sh b/docker/entrypoint-extra.sh index a06901e25c..b32ad2dd94 100644 --- a/docker/entrypoint-extra.sh +++ b/docker/entrypoint-extra.sh @@ -33,6 +33,58 @@ if [ ! -w "$HOME" ]; then fi export HOME +# Register CA certificates mounted into the image before anything opens a TLS connection. +# Best-effort on purpose, unlike INIT_SCRIPT below: a non-root UID cannot write /etc/ssl/certs, and +# a deployment that never needed a custom CA must still boot. Env var names and the default-off +# behavior match the server/worker binary, so one setting covers every container. What the system +# trust store does and does not reach is documented in debugger/README.md. +CA_CERT_DIR=/usr/local/share/ca-certificates + +update_ca_certificates() { + local reason="$1" + local tool="${RUN_UPDATE_CA_CERTIFICATE_PATH:-/usr/sbin/update-ca-certificates}" + local output + if [ ! -x "$tool" ]; then + echo "[entrypoint] $reason but $tool is not executable, skipping CA update" + return + fi + echo "[entrypoint] $reason, running $tool" + if output=$("$tool" 2>&1); then + echo "[entrypoint] CA certificates updated" + else + # Carry the tool's own message: the usual cause is an unwritable /etc/ssl/certs under a + # non-root UID, but guessing that in place of the real error hides everything else. + echo "[entrypoint] WARNING: $tool failed (UID $(id -u)): ${output:-no output}; continuing" >&2 + fi +} + +if [ "$(echo "${RUN_UPDATE_CA_CERTIFICATE_AT_START:-false}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then + update_ca_certificates "RUN_UPDATE_CA_CERTIFICATE_AT_START=true" +elif [ -n "$(find -L "$CA_CERT_DIR" -type f -name '*.crt' -print -quit 2>/dev/null)" ]; then + # Certificates mounted there are unambiguous intent, and they do nothing until registered, so + # take the same action without making the operator also find the env var. + update_ca_certificates "Found certificates in $CA_CERT_DIR" +elif [ -n "$(ls -A "$CA_CERT_DIR" 2>/dev/null)" ]; then + # Reporting success over a mount update-ca-certificates ignores would be worse than saying + # nothing: .pem is the spelling people reach for, and only .crt is read. + echo "[entrypoint] WARNING: $CA_CERT_DIR has files but none named *.crt, the only extension" \ + "update-ca-certificates reads; they will be ignored" >&2 +fi + +# INIT_SCRIPT is the documented hook for preparing the host before anything reaches the network +# (CA certificates, proxies, mounts), matching the worker's INIT_SCRIPT. It must therefore complete +# before any service starts, and a failure has to abort: services that come up with an unprepared +# trust store fail every TLS handshake instead, which is far harder to diagnose. +if [ -n "$INIT_SCRIPT" ]; then + echo "[entrypoint] Running INIT_SCRIPT..." + bash -c "$INIT_SCRIPT" || { + code=$? + echo "[entrypoint] ERROR: INIT_SCRIPT failed with exit code $code, aborting" >&2 + exit "$code" + } + echo "[entrypoint] INIT_SCRIPT completed" +fi + # Setup NETRC if provided (for LSP) if [ -n "$NETRC" ]; then echo "$NETRC" > "$HOME/.netrc" diff --git a/docs/agent-worker-e2e.md b/docs/agent-worker-e2e.md new file mode 100644 index 0000000000..e1fd122b98 --- /dev/null +++ b/docs/agent-worker-e2e.md @@ -0,0 +1,131 @@ +# Running an agent worker locally, for e2e + +An agent worker reaches the database only through the API, so whole code paths +(`Connection::Http`) are never taken by a normal `cargo run`. Exercising them +needs a real one. Every step below has a failure mode that looks like something +else; they are listed with the error each produces. + +## 1. Build with the right features + +Four features, and the agent's own mode gate is the one that is easy to miss: + +```bash +cd backend +cargo build --features quickjs,private,enterprise,license,agent_worker_server +``` + +- `agent_worker_server` mounts `/api/agent_workers/*` on the SERVER. Without it, + `create_agent_token` returns **404** with an empty body. +- `enterprise` + `license` compile the agent MODE into the binary. Without them + the worker exits immediately with `Agent mode is only available in the EE`, + even though the server side works and mints tokens happily. + +Verify before spending time on the handshake — the panic string must be absent: + +```bash +strings target/debug/windmill | grep -c "Agent mode is only available in the EE" # want 0 +``` + +**Pin this feature set for the whole session.** `target/debug/windmill` is one +path shared by every feature combination, and cargo swaps the cached artifact in +and out as the set changes — a `cargo build --features quickjs` (or any build +with a different set) in another pane silently replaces the binary the server and +agent are about to run, and the swap back "completes" in under a second, so it +does not look like a rebuild happened. The symptom is the agent 401ing again +after it had been working, or the EE panic reappearing. Re-run the `strings` +check above whenever anything unexpected regresses, and start the server and the +agent from the SAME build. + +## 2. Run the server without a local worker + +`MODE=server` so nothing else drains the queue and the agent is provably the one +that ran the job: + +```bash +DATABASE_URL= PORT=8420 MODE=server ./target/debug/windmill +``` + +## 3. Mint a token — with an expiry, and unquoted + +```bash +TOK=$(curl -s -X POST localhost:8420/api/auth/login -H 'Content-Type: application/json' \ + -d '{"email":"admin@windmill.dev","password":"changeme"}') + +AT=$(curl -s -X POST localhost:8420/api/agent_workers/create_agent_token \ + -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \ + -d '{"worker_group":"agentgrp","tags":["dbt"],"exp":1900000000}' | tr -d '"') +``` + +Two traps, both of which surface as a bare `401` on the agent and a decoded +reason only in the SERVER log: + +- **`exp` must be a real timestamp.** `"exp": null` mints a token the validator + rejects with `Missing required claim: exp`. +- **The response is JSON, so it arrives quoted.** Keeping the `"` gives + `Base64 error: Invalid byte 34, offset 0` — hence the `tr -d '"'`. + +Pass the token **exactly as minted**. It looks like `jwt_agent_` and the +client appends its own hostname-derived suffix to form `jwt_agent__`, +which is what the server splits on. Adding a suffix yourself yields +`Base64 error: Encoded text cannot have a 6-bit remainder`. + +## 4. Start the agent + +`WORKER_TAGS` must contain the tag the JOBS carry, not a tag you invent — a job +whose tag nothing serves sits in `v2_job_queue` forever and looks like a hang. +dbt scripts default to the `dbt` tag. + +```bash +AGENT_TOKEN="$AT" BASE_INTERNAL_URL=http://localhost:8420 MODE=agent \ + WINDMILL_DIR=/home/$USER/wmagent \ + WORKER_GROUP=agentgrp WORKER_TAGS=dbt PORT=8499 ./target/debug/windmill +``` + +`WINDMILL_DIR` off `/tmp` matters on a dev box. Jobs fail with `IoErr: Disk quota +exceeded (os error 122)` while writing the project's files, and `df` looks +healthy — free space and free inodes both. `/tmp` is a tmpfs and Linux supports +per-user quotas on it, so the limit is the user's, not the filesystem's; several +agent sessions' caches under `/tmp` are enough to reach it. Point the worker at a +real disk instead of trying to clean up under the quota. + +Confirm it registered rather than trusting a quiet log: + +```sql +SELECT worker FROM worker_ping + WHERE worker_group = 'agentgrp' AND ping_at > now() - interval '2 min'; +-- ag-agentgrp-- +``` + +## Reading failures + +The agent only ever prints `Agent worker cannot connect to server. Please check +AGENT_TOKEN and BASE_INTERNAL_URL`. The actual reason is in the server log, from +`windmill-api-agent-workers/src/ee.rs` — grep it for `JWT_AGENT auth error`. + +## Confirming the agent is what ran the job + +`worker` on the completed job starts with `ag-`: + +```bash +curl -s -H "Authorization: Bearer $TOK" \ + "localhost:8420/api/w//jobs_u/completed/get/" | jq -r .worker +``` + +## What dbt does on an agent worker + +Runs, retries, and publishes its graph — including a per-run snapshot for a +dynamic descriptor, which it POSTs to `/api/agent_workers/dbt_graph/{workspace}` +rather than writing itself. + +What it does not get is LIVE progress: the reporter tails a JSON event log and +needs a SQL connection, so per-model state is settled from `run_results.json` +when the run ends. Retry state lives only in the worker-local generation, since +there is no database row to arbitrate against — which is why `state_dir` is keyed +by principal. + +Confirming a run really exercised that path: + +```sql +SELECT job_id, count(*) FROM dbt_node WHERE script_path = '' GROUP BY job_id; +-- a row keyed to the JOB id (not the zero UUID) means the agent published a snapshot +``` diff --git a/docs/dbt-runtime.md b/docs/dbt-runtime.md new file mode 100644 index 0000000000..dc06a22fa0 --- /dev/null +++ b/docs/dbt-runtime.md @@ -0,0 +1,1232 @@ +# Windmill as a dbt runtime + +Implementation spec for running an existing dbt project on Windmill with no +changes to the project itself. Companion to [`pipelines-vs-dbt.md`](./pipelines-vs-dbt.md), +which covers the opposite direction (native pipeline features that replace dbt). +The two are complementary: this is the adoption ramp, that is the long game. + +Benchmark to beat is Airflow + [astronomer-cosmos](https://astronomer.github.io/astronomer-cosmos/), +the dominant way dbt is orchestrated today. + +## Scope + +- **In**: run an unmodified dbt project synced into Windmill, one Windmill job per + invocation, live per-model observability, dbt models as first-class assets in + the existing asset graph. +- **Out**: one Windmill job per dbt model, `state:modified` / slim CI, + `dbt docs` hosting, semantic layer, dbt platform integration. +- **CE**: the runtime, the manifest ingest, the asset graph and every piece of + UI ship in CE, as do all adapters except two. Only the `mssql` and `oracle` + adapters are EE, mirroring the native `ScriptLang` boundary (decision 21). + +## Decision log + +| # | Decision | Resolution | +|---|---|---| +| 1 | dbt engine | Three-way toggle (`dbt-core-1x` \| `dbt-core-2x` \| `fusion`); shipped default `dbt-core-1x`, instance-configurable. See below | +| 2 | Artifact shape | `ScriptLang::Dbt` | +| 3 | Graph in v0 | Yes, both runtime and graph | +| 4 | Execution granularity | One job per invocation | +| 5 | Project storage | The project is the script's module bundle; nothing is cloned. See "Where the dbt project lives" | +| 6 | Multiple run configs | Per-run `select` on one script; N scripts means N projects | +| 7 | Run-time `select` | Descriptor default plus run-arg override | +| 8 | Credentials | Workspace warehouses, plus `profiles.yml` passthrough. A descriptor never names a resource. See below | +| 9 | Adapter mappings | postgres, redshift, mysql, snowflake, bigquery, databricks translate from their Windmill resource; **every** adapter dbt has is reachable from a `dbt_profile` resource, or the project's own `profiles.yml` | +| 10 | Private repo auth | Not applicable: the project is synced, not fetched | +| 11 | Asset kind | `dbt:////` — keyed on the relation, not on dbt's node id. See below | +| 12 | Graph refresh | Deploy-time, re-ingested per run only when the descriptor is dynamic, plus an explicit `parse` of the editor's buffer. See below | +| 13 | Manifest storage | Sidecar table for nodes/edges. Full manifest **not** stored — see below | +| 14 | Metadata depth | Tests, strategy, tags, freshness, column descriptions. Column **lineage** is not in the manifest — see below | +| 15 | Node rendering | Asset nodes per model plus one runnable node for the script | +| 16 | Progress | Live, from the JSON event stream | +| 17 | Test failures | Honor dbt's own `severity` | +| 18 | Retry | Automatic node-level retry in-job, plus `dbt retry` as a run argument. See below | +| 19 | Caching | Worker-local global cache, keyed by the project digest and the resolution the deploy pinned | +| 20 | Images | Full images only | +| 21 | Licensing | CE except the `mssql` / `oracle` adapters. See below | +| 22 | Naming | Match Cosmos field names; importer deferred | +| 23 | Descriptor | `wm_dbt.yaml` inside the project, OPTIONAL. See below | +| 24 | Warehouse | Configured on the workspace by name, `main` by default. See below | + +## Decision 1: engine toggle, and why the shipped default is not Fusion yet + +`engine: dbt-core-1x | dbt-core-2x | fusion` in the descriptor. Omitted, it is +`dbt-core-1x`, which runs today's projects untouched. + +No engine is baked into any image. Each is fetched or built on first use and +cached, for a different reason in each case. + +| Engine | Distribution | Cold start | License | +|---|---|---|---| +| `dbt-core-1x` (default) | A uv venv resolved per adapter on first use, then cached. **Cannot** be baked: the adapter is a Python package chosen per project | One venv build per (core range, adapter) | Apache 2.0 | +| `dbt-core-2x` | One adapter-agnostic Rust binary, fetched from GitHub releases on first use, cached | One download | Apache 2.0 | +| `fusion` | **Never bundled.** Fetched from dbt Labs on first use, cached | One download (~290MB) | dbt Fusion engine license agreement | + +2.x is the one that *could* be baked, and deliberately is not: it is a +pre-release (`2.0.0-alpha.5`) that nothing is defaulted onto, so baking it costs +a layer in every image and a version pinned in two places with nothing keeping +them in step. An operator who wants an engine pre-staged — an air-gapped +instance, or a fleet that should not fetch per worker — populates +`DBT_BUNDLED_DIR` (default `/usr/local/dbt`) with `core2x-/dbt-sa-cli` +in a derived image; the worker prefers it over its own cache. + +Two things to know before choosing 2.x: it is a pre-release, and it does not +emit the per-node events the run page animates (see "Live per-model progress"), +so a run on it reports its models only at the end. + +The 1.x venv resolves `dbt-core>=1.8,<2.0.0` *together with* the adapter rather +than pinning a core version, because several adapters cap below the newest core +(dbt-oracle and dbt-databricks below 1.12) and an independent pin makes those +projects unprovisionable. The lockfile records whichever version the resolver +actually chose. Both bounds and each engine version are env-overridable +(`DBT_CORE_1X_FLOOR`, `DBT_CORE_1X_CEILING`, `DBT_CORE_2X_VERSION`). + +Fusion is the fastest option and the toggle exists so users can choose it. Two +things block making it the *shipped* default, both verifiable rather than matters +of taste: + +1. **Redistribution terms.** The Fusion license grants only a "limited, + non-exclusive, non-transferable, non-sublicensable" redistribution right, and + 4.1 forbids introducing "obstacles or delays that have the effect of hampering + or interfering with (a) communication between Provider and End User, (b) + User's ability to view, access, or use the Product and/or any Account + Features." A sandboxed non-interactive job runner sits squarely in that + clause's path, and "may not share, pool, or relay its own login credentials to + any End User" reads directly onto putting one dbt platform token in a + workspace secret. That needs counsel, not an engineering judgment. + **Fetch-at-runtime is the mitigation**: the user's own instance pulls the + binary from dbt Labs directly, so Windmill never redistributes and never + interposes. Do not bake Fusion into any image. +2. **Fusion is v2 semantics, and v2 drops all deprecated functionality.** Every + deprecation warning, including historic ones and those added in 1.10, must be + resolved before a project runs on it. An arbitrary existing dbt 1.x project + therefore may not run unchanged, which is this feature's entire premise. dbt + ships an autofix tool and Fusion/Core interoperate side by side, so it is a + migration users can do, but not one Windmill should silently require of them. + +Consequence: ship with `dbt-core-1x`, which runs today's projects untouched, and +flip the instance default to `fusion` once counsel clears the runtime-fetch model +and a real project is verified end to end on it. Both dbt-core engines are +exercised by the e2e suite, so the flip is a config change, not a port. + +## Decision 21: mirror the native warehouse boundary, do not invent one + +Everything structural is CE: the executor, all three engines, the manifest +ingest, the `dbt://` asset graph, live progress, the editor. The only gate is +on two adapters, and it is not a dbt-specific policy — it is the same boundary +the native script languages already draw. Since `bigquery` and `snowflake` +became CE, the only warehouse `ScriptLang`s still behind a license are `mssql` +and `oracledb`, so those two dbt adapters are EE and every other one (postgres, +mysql, duckdb, snowflake, bigquery, databricks, redshift, clickhouse, +salesforce) is CE. Gating any of the others would make reaching a warehouse +through dbt stricter than reaching it natively, which is backwards. + +Those two are *recognized* (for the gate and for the pip package the 1.x +engine's venv needs), but no Windmill connection resource translates into them: +an `oracledb` resource is `{user, password, database}` with no +host/protocol/service, and dbt-sqlserver needs an ODBC `driver` the images do not +install. They reach their warehouse through a `dbt_profile` resource or the +project's own `profiles.yml`, which is also how duckdb, clickhouse and salesforce +work. + +Recognition is what the gate keys on, and it survives the open adapter set: a +`dbt_profile` stating `sqlserver`, `mssql` or `oracle` resolves to the same +`KnownAdapter` a resource type would, so it is gated identically. An adapter +Windmill has never heard of is never enterprise — the boundary mirrors the two +native warehouse languages, and an adapter with no Windmill runtime behind it is +not one of them. + +The gate almost never fires in practice: `dbt-core-2x` supports neither adapter, +so it can only apply to `dbt-core-1x` with one of those two. + +**The mechanism differs from the native languages.** They gate at compile time, +so a CE binary simply lacks the executor. That is not available here: there is +one dbt executor and the adapter is only known once the profile resolves. So it +is a runtime check on the resolved adapter, at both deploy and run, and it must +say what is wrong — a silent degradation that surfaces later as a connection +error is worse than no gate at all. + +One trap: `ee_oss::LICENSE_KEY_VALID` is initialized to `true` in the OSS +variant, so reading it alone passes on a CE build. The check is +`cfg!(feature = "enterprise") && LICENSE_KEY_VALID`, which rejects both a CE +build and an enterprise build whose key did not verify. + +## Decision 11: `dbt://`, keyed on the relation and not on the dbt node + +`dbt:////`, one `AssetKind`, where `` is the +workspace warehouse's NAME, so two scripts running against the same warehouse +agree on identity. + +The SCHEME names the producer, because dbt is the only thing that creates one of +these: no other language derives warehouse relations, `// materialize` takes +DuckLake targets only, and a dbt run does not dispatch. Calling the kind +something generic promised a parity with native Snowflake and BigQuery scripts +that does not exist. + +The PATH is the physical relation, and that is the load-bearing half. dbt-core +has no cross-project `ref()`: two projects meet when one materializes a mart and +the next declares it a `source`. Their dbt identities differ there — +`model.a_pkg.orders` against `source.b_pkg.analytics.orders` — while the relation +does not, so keying on `unique_id` would make every project an island and turn +the handoff into two unconnected nodes. `unique_id` also embeds the package name +from `dbt_project.yml`, which two unrelated projects may both call `analytics`, +collapsing two different tables onto one node. The relation cannot collide that +way. It is also what a DuckDB, Python, TS or Ansible script can name in a +`// on dbt://…` annotation to join the lineage — those four are the languages +with a body-asset parser; the native SQL ones cannot declare assets at all. + +A dbt run does **not** trigger those readers. See "no cascade from dbt" below. + +An ephemeral model (an inlined CTE, never written), an exposure, or a source that +is not separately modelled has no physical relation and therefore no place in +this namespace. If those ever prove worth rendering they need a key of their own +— `unique_id` suits them, precisely because nothing else can refer to them. + +Two traps, both of which quietly defeat the point if handled wrong. + +**Identifier canonicalization.** `manifest.json` gives `relation_name` +pre-quoted (`"windmill"."Analytics"."Orders"`), an annotation is written by hand, +and the warehouses disagree on case: Snowflake folds unquoted identifiers up, +Postgres folds them down, DuckDB compares case-insensitively. Two spellings of +one table produce two nodes, no edge, and nothing looks broken in isolation. So +one rule is applied in exactly one place — `parse_asset_syntax`, the single +point where an asset URI becomes a graph key: strip the quote characters +(`"`, backtick, `[`/`]`) from the schema and name, then ASCII-lowercase them, +matching the case-insensitive identifier comparison the DuckDB paths already +use. The warehouse-name prefix is spelled as the workspace configures it and +stays case-sensitive. + +**Warehouse identity is the workspace warehouse's name**, exactly as +`ducklake://main.orders` keys on the workspace lake's name — never the host, +account or database. A descriptor cannot name a resource at all (Decision 24), so +there is exactly one spelling per warehouse and the ambiguity a per-project +resource would create does not arise. The warehouse names the default database +too, so it stays out of the key; a model that *overrides* +its database (Snowflake `database`, BigQuery `project`) is genuinely elsewhere and +qualifies its schema segment as `.`, so two same-named relations +in different databases cannot collapse onto one node. A project that brings its own +`profiles.yml` reports its target's database from that file, read with the same +keys the renderer writes, so it spells a relation exactly as a workspace-warehouse +project does and the two meet on one node. Only where the target leaves its +database implicit does every relation qualify, because assuming they share one +database is exactly what would collapse them. + +Three call sites derive this key: the manifest ingest that creates the node, and +the live-progress and end-of-run paths that record status against it. They share +one function, because a site that derives it differently records progress against +a path no node has — the run still succeeds and the graph simply never moves. The same +warehouse is reachable under several hostnames, and credential material has no +business in an asset key. Accepted limitation, worth knowing before it is +filed as a bug: **two workspace warehouses pointing at the same physical +warehouse do not unify**, so assets under one will not share edges with assets +under the other. Point both projects at one warehouse to link them. + +## Decision 24: the warehouse is a workspace setting, named, and the only one + +A descriptor names a warehouse by NAME (`profile.warehouse`, `main` when it names +none) and cannot name a resource. Admins configure the warehouses under Settings +→ dbt, where each entry points at a resource, exactly as `large_file_storage` +points at the object-storage resource and a DuckLake names its catalog. + +**What a warehouse may point at.** Either a Windmill connection resource whose +type `render_profile` translates (`postgresql`, `redshift`, `mysql`, `snowflake`, +`snowflake_oauth`, `bigquery`, `gcp_service_account`, `databricks`), or a +**`dbt_profile`** resource, whose VALUE IS one entry of that file's `outputs` +map — `type` included, nothing lifted out or renamed. A block is copied from a +working `profiles.yml` and pasted in, which is the whole point: a type that asked +the user to restructure their block first would be doing the translation this +exists to avoid. Its schema declares no properties, so the resource form renders +one JSON editor over the value (`ResourceForm.svelte`). The picker is +constrained to exactly these (`WAREHOUSE_RESOURCE_TYPES`); anything else has no +way to become a target at all, which is why an unconstrained picker was a trap: +it offered slack and github resources for a field that can only be a warehouse. + +The two exist for different reasons. A Windmill resource is the ergonomic path +and is shared with everything else that connects to that warehouse, but it is +*not* a dbt target: each adapter arm translates the fields Windmill's resource +happens to carry into the keys dbt reads, so only what an arm covers can be +expressed, and an adapter with no arm cannot be reached from one at all. +`dbt_profile` inverts that — nothing is translated, so any adapter and any key it +documents works. + +Which of the two a value is cannot be read off the value: both are objects with a +`type`, and Windmill's bigquery resource is a service-account JSON that says +`type: service_account`. So the warehouse carries its resource's TYPE +(`DbtWarehouseConnection.resource_type`), and that is also what finally makes +decision 9's "the resource type name is the authority" true at runtime rather +than aspirational — the translated path resolved its adapter by sniffing +connection fields until it had the name. + +**`dbt_profile` is open, deliberately.** Its `type` is not checked against a list: +`DbtAdapter` carries an optional `KnownAdapter` beside the name, so the eleven +adapters Windmill has facts about (a field mapping, a pip package, the license +gate) keep them, and every other adapter dbt has — `trino`, `athena`, `spark`, +whatever ships next — is carried by name and rendered, licensed and identified +without Windmill knowing anything about it. A closed list would have made +"whatever dbt supports" mean "whatever this enum lists", and each new adapter a +Windmill release. The name is constrained to `[a-z0-9_-]` starting alphanumeric +*because* it is open: it reaches a pip requirement and a venv path on the host, +where a leading `-` is a flag and a `/` is a path segment. + +**Installing one is a separate question from using one.** `dbt-core-1x` fetches +`dbt-` from PyPI, `dbt-` is not a reserved prefix there, and that install +runs through `run_tool` — outside the nsjail ordinary Python dependency +installation uses, with uv executing a source distribution's PEP 517 backend. An +unbounded name would therefore let a script author publish `dbt-` and run code +as the worker, on the one dependency path that is not sandboxed. So +`ensure_adapter_installable` gates that install on `PUBLISHED_ADAPTERS` plus +whatever an operator lists in `DBT_EXTRA_ADAPTERS`: the author chooses which +adapter to use, the admin decides which packages this instance trusts. Nothing +else is gated — a profile still renders for any adapter, and `dbt-core-2x` and +`fusion` carry their adapters in the binary, install nothing, and take any +`type` at all. + +Two keys are not passed through: `type` (Windmill writes the adapter's own dbt +spelling) and `root_certificate_pem`, which is a PEM body rather than the path +dbt hands the driver — it is written beside `profiles.yml` and pointed at by +`sslrootcert`, as it is for a translated postgres resource. `profile.schema` and +`threads` from the descriptor override their block keys rather than joining them. + +Three things follow, and they are the reason for the rule rather than +consequences to work around. + +**A dbt project carries no connection.** The same project runs locally against a +developer's own `~/.dbt/profiles.yml` and on Windmill against the workspace +warehouse, with no Windmill-specific file in between and nothing to strip before +committing it to a repository. This is what makes Decision 23 possible at all: if +a project had to name its own resource, the descriptor could never be optional. + +**Asset identity has exactly one spelling.** Keying on a name is only sound +because a name is all there is. Had both `profile.resource` and +`profile.warehouse` existed, one physical warehouse would be reachable under two +spellings and two projects on it would silently fail to share nodes — the exact +failure Decision 11 exists to prevent. + +**dbt is unpermissioned, and the blast radius is bounded by construction +instead.** The warehouse resource is read with NO permission check on the runner, +exactly as `s3://` reaches the workspace bucket without the caller being granted +the storage resource: configuring a warehouse is what makes it available, and +anyone who may run a dbt script may build with it and read its models. What is +reachable stays bounded because only an admin writes the setting and a descriptor +cannot name a resource, only one of the names an admin configured. + +Per-relation rules were considered and rejected. `s3://` can enforce a path glob +because Windmill mediates every object operation through its proxy; dbt has no +such chokepoint — Windmill renders `profiles.yml` and dbt opens its own +connection. A rule could only be a pre-run check against the manifest, and a +`pre-hook`, a macro or `dbt run-operation` issues arbitrary SQL on the same +connection, so it would stop the ordinary case while implying a guarantee it +cannot keep. + +A project that brings its own `profiles.yml` still connects with it, and then +names a warehouse only to say where its assets belong. The name must still match +a configured warehouse — a typo is not identity, it strands the project's models +on a node nothing else reaches — but it grants nothing, since nothing here is +granted. It gets no identity by default, because defaulting to `main` would key a +self-hosted profile's tables onto a workspace warehouse it never connected to. + +That label is worth having only because such a project spells its relations the +same way: Windmill reads the target's database out of the project's own file +(Decision 11), so a mart it builds and a workspace-warehouse project's `source` +on the same relation land on ONE node. Without that the label would name a +namespace and still share nothing, which is the failure it exists to prevent. + +An agent worker cannot read the database, so it resolves the name through a +job-scoped API route. That route returns the resolved connection, which is why it +requires a job token: a running job already holds those credentials in its +rendered `profiles.yml`, and a browsable route would hand them to anyone. The +same worker posts its per-model outcomes to a second job-scoped route, since the +live reporter tails a log straight into the database and cannot run there. An +agent's run page therefore fills in when the run ends rather than during it. +Both routes are posted with the JOB's token: an agent's own credential +authenticates only against the agent surface. + +## Decision 23: the descriptor is optional, and lives inside the project + +` +``` + +Find the chunk by a preserved string literal — minification keeps them, e.g. zod's +`Error converting schema to JSON.` locates the chunk holding `JSONSchemaGenerator`. diff --git a/docs/reusable-ai-agents.md b/docs/reusable-ai-agents.md index 728b883109..8fa9facd74 100644 --- a/docs/reusable-ai-agents.md +++ b/docs/reusable-ai-agents.md @@ -50,6 +50,26 @@ tool starting can run the changed definition, or fail if the tool was removed. P carrying the resolved definition into the child job rather than its id. Inline (unlinked) agents are unaffected: their tools live in the flow value, which is snapshotted with the run. +## Version history + +Editing a resource appends a row to `resource_version` (all types except `state` and `cache`, +which the platform rewrites on every job), so an agent's prompt, model and tool set can be +diffed and restored from the resource editor's History drawer. Restoring writes the old value +forward as a new version rather than rewinding, keeping the history append-only. + +History captures the resource, not its transitive closure. A `$var:`/`$res:` reference is stored +as the reference, so two versions can be byte-identical while the agent behaves differently +because the referenced variable changed underneath them. Anything comparing agent runs across +versions has to account for that. + +A superseded value is retained for up to 100 versions. Values written through the UI keep their +secrets in linked variables, but one pushed by `wmill` or written by `setResource` can hold an +inline credential, and overwriting it no longer removes it from the database — anyone who can +read the resource can read it from the history. Rotating such a credential therefore does not +erase the old one on its own — follow the rotation with **Clear past versions** in the History +drawer, which drops every version but the current value for that one resource. Secret *variables* +are deliberately not versioned at all for the same reason. + ## Dependencies and locks A linked step carries `tools: []`, and no dependency job ever visits the `ai_agent` resource, so the diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 4ab6621c30..7113b74147 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.775.2", + "version": "1.783.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.775.2", + "version": "1.783.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { @@ -94,7 +94,7 @@ "windmill-parser-wasm-rust": "1.647.1", "windmill-parser-wasm-ts": "1.695.0", "windmill-parser-wasm-wac": "1.668.6", - "windmill-parser-wasm-yaml": "1.593.0", + "windmill-parser-wasm-yaml": "1.770.0", "windmill-sql-datatype-parser-wasm": "1.512.0", "windmill-utils-internal": "1.8.2", "xterm": "^5.3.0", @@ -157,13 +157,12 @@ "tar": "^7.5.4", "tslib": "^2.6.1", "typescript": "^5.5.0", - "vite": "^8.0.13", + "vite": "^8.2.0", "vite-plugin-mkcert": "^2.0.0", "vitest": "^4.1.0", "vitest-browser-svelte": "^2.0.1" }, "optionalDependencies": { - "@rollup/rollup-linux-x64-gnu": "^4.35.0", "fsevents": "^2.3.3" }, "peerDependencies": { @@ -1098,40 +1097,6 @@ "@csstools/css-tokenizer": "^2.4.1" } }, - "node_modules/@emnapi/core": { - "version": "1.11.2", - "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.2.tgz", - "integrity": "sha512-TC8MkTuZUtcTSiFeuC0ksCh9QIJ5+F21MvZ4Wn4ORfYaFJ/0dsiudv5tVkejgwZlwQ39jL9WWDe2lz8x0WglOA==", - "dev": true, - "license": "MIT", - "optional": true, - "dependencies": { - "@emnapi/wasi-threads": "1.2.2", - "tslib": "^2.4.0" - } - }, - "node_modules/@emnapi/runtime": { - "version": "1.11.2", - "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.2.tgz", - "integrity": "sha512-kyOl3X0DuTiT1h2ft8r2fYO8JYtU9a9Xis/zBSiGArNaagCOWx90N1k2wxp18czFDH+OgcWGb5ZP/XMt3dcyPA==", - "dev": true, - "license": "MIT", - "optional": true, - "dependencies": { - "tslib": "^2.4.0" - } - }, - "node_modules/@emnapi/wasi-threads": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.2.tgz", - "integrity": "sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==", - "dev": true, - "license": "MIT", - "optional": true, - "dependencies": { - "tslib": "^2.4.0" - } - }, "node_modules/@eslint-community/eslint-utils": { "version": "4.9.0", "resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.9.0.tgz", @@ -1655,25 +1620,6 @@ "@chevrotain/types": "~11.1.1" } }, - "node_modules/@napi-rs/wasm-runtime": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.4.tgz", - "integrity": "sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==", - "dev": true, - "license": "MIT", - "optional": true, - "dependencies": { - "@tybys/wasm-util": "^0.10.1" - }, - "funding": { - "type": "github", - "url": "https://github.com/sponsors/Brooooooklyn" - }, - "peerDependencies": { - "@emnapi/core": "^1.7.1", - "@emnapi/runtime": "^1.7.1" - } - }, "node_modules/@noble/hashes": { "version": "1.8.0", "resolved": "https://registry.npmjs.org/@noble/hashes/-/hashes-1.8.0.tgz", @@ -1726,9 +1672,9 @@ } }, "node_modules/@oxc-project/types": { - "version": "0.130.0", - "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.130.0.tgz", - "integrity": "sha512-ibD2usx9JRu7f5pu2tMKMI4cpA4NgXJQoYRP4pQ7Pxmn1l6k/53qWtQWZayhYy3X4QZkt90Ot+mJEaeXouio6Q==", + "version": "0.142.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.142.0.tgz", + "integrity": "sha512-7W+2q5AKQVU36fkaryontrHn3YDt1RyUYXatw9i5H8ocYe2sPKSFB6eS8WNPeRKiN1qAWWZUPm7gwFzJGrccqQ==", "devOptional": true, "license": "MIT", "funding": { @@ -1802,9 +1748,9 @@ "license": "SEE LICENSE IN LICENSE" }, "node_modules/@rolldown/binding-android-arm64": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.1.tgz", - "integrity": "sha512-fJI3I0r3C3Oj/zdBCpaCmBRZYf07xpaq4yCfDDoSFm+beWNzbIl26puW8RraUdugoJw/95zerNOn6jasAhzSmg==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.2.2.tgz", + "integrity": "sha512-l7x215OGvo1s52JWmR8U/DAVzEDWBCIbTm28aeJV/WDTSHgcKXaZTuBT0hJMs5NggilfJTW3clZVvd24yfKJxA==", "cpu": [ "arm64" ], @@ -1819,9 +1765,9 @@ } }, "node_modules/@rolldown/binding-darwin-arm64": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.1.tgz", - "integrity": "sha512-cKnAhWEsV7TPcA/5EAteDp6KcJZBQ2G+BqE7zayMMi7kMvwRsbv7WT9aOnn0WNl4SKEIf43vjS31iUPu80nzXg==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.2.2.tgz", + "integrity": "sha512-9u9Xv6c1AJZT0FfwH5vrMG5Jjcwhc1MlyrPu0XfTqkzsmqfks2M6W/o5XwAJgVVN/jHpqqngC1WevHKKTIUtIA==", "cpu": [ "arm64" ], @@ -1836,9 +1782,9 @@ } }, "node_modules/@rolldown/binding-darwin-x64": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.1.tgz", - "integrity": "sha512-YKrVwQjIRBPo+5G/u03wGjbdy4q7pyzCe93DK9VJ7zkVmeg8LJ7GbgsiHWdR4xSoe4CAXRD7Bcjgbtr64bkXNg==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.2.2.tgz", + "integrity": "sha512-9W1mbGZAfW3oqd85bhBkmpyHCCzL1TeG/zFFP3vg7b0rlly8cxOcre5nXwz+LHazCwac2MNWgPdPCHndABjpWQ==", "cpu": [ "x64" ], @@ -1853,9 +1799,9 @@ } }, "node_modules/@rolldown/binding-freebsd-x64": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.1.tgz", - "integrity": "sha512-z/oBsREo46SsFqBwYtFe0kpJeBijAT48O/WXLI4suiCLBkr03RTtTJMCzSdDd2znlh8VJizL09XVkQgk8IZonw==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.2.2.tgz", + "integrity": "sha512-0p1lhiCSCyaerFwtrdZQUx7NqGk6LQnaRKWX7tFQqwQgvX0rjM15cIkm3pax1UpEakK14C4mOxx/jSqCBdBRqQ==", "cpu": [ "x64" ], @@ -1870,9 +1816,9 @@ } }, "node_modules/@rolldown/binding-linux-arm-gnueabihf": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.1.tgz", - "integrity": "sha512-ik8q7GM11zxvYxFc2PeDcT6TBvhCQMaUxfph/M5l9sKuTs/Sjg3L+Byw0F7w0ZVLBZmx30P+gG0ECzzN+MFcmQ==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.2.2.tgz", + "integrity": "sha512-e+cOJXrJ2L3zx6YzqPg+f6Wbk3V1cKB8bOhbaYdVYN3DdquzNdRAmrbETz1qnt5yp/c7JNlNjmITiA2cVneQ7w==", "cpu": [ "arm" ], @@ -1887,9 +1833,9 @@ } }, "node_modules/@rolldown/binding-linux-arm64-gnu": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.1.tgz", - "integrity": "sha512-QoSx2EkyrrdZ6kcyE8stqZ62t0Yra8Fs5ia9lOxJrh6TMQJK7gQKmscdTHf7pOXKREKrVwOtJcQG3qVSfc866A==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.2.2.tgz", + "integrity": "sha512-JsSMsj6sNat/MuhG5fnBD7QgbtpHKVe30x5/bAVirDHdhoQRXJkF6xc0Jqk8O4fiCUQAzMOoH9wZi3m60c8wtg==", "cpu": [ "arm64" ], @@ -1904,9 +1850,9 @@ } }, "node_modules/@rolldown/binding-linux-arm64-musl": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.1.tgz", - "integrity": "sha512-uwNwFpwKeNiZawfAWBgg0VIztPTV3ihhh1vV334h9ivnNLorxnQMU6Fz8wG1Zb4Qh9LC1/MkcyT3YlDXG3Rsgg==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.2.2.tgz", + "integrity": "sha512-B5G/zJdHaoJn9vD50eGHWkiWfmq8Uhi3IiLPJTzmZTrAalk1bztUikSXo0qga18ibE0IXboyeMUnhPjhAJ45wQ==", "cpu": [ "arm64" ], @@ -1921,9 +1867,9 @@ } }, "node_modules/@rolldown/binding-linux-ppc64-gnu": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.1.tgz", - "integrity": "sha512-zY1bul7OWr7DFBiJ++wofXvnr8B45ce3QsQUhKrIhXsygAh7bTkwyeM1bi1a2g5C/yC/N8TZyGDEoMfm/l9mpg==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.2.2.tgz", + "integrity": "sha512-6mC/awzKka8W6EoekjegpfGkjz8jXWDX63pqu/HYVpyKtZfu65Jsh4QAH3Kej3CAv/c1oGX7psTmFEbr0mDxLA==", "cpu": [ "ppc64" ], @@ -1938,9 +1884,9 @@ } }, "node_modules/@rolldown/binding-linux-s390x-gnu": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.1.tgz", - "integrity": "sha512-0frlsT/f4Ft6I7SMESTKnF3cZsdicQn1dCMkF/jT9wDLE+gGoiQfv1nmT9e+s7s/fekvvy6tZM2jHvI2tkbJDQ==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.2.2.tgz", + "integrity": "sha512-412MX9fJLdA1IK28EZnc8jYv2HRTleOZgfLQumJ5zy7OeJLZlg/CETwFaXjNmGVxG51cFHpKLqb5LKvBC+HsHA==", "cpu": [ "s390x" ], @@ -1955,9 +1901,9 @@ } }, "node_modules/@rolldown/binding-linux-x64-gnu": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.1.tgz", - "integrity": "sha512-XABVmGp9Tg0WspTVvwduTc4fpqy6JnAUrSQe6OuyqD/03nI7r0O9OWUkMIwFrjKAIqolvqoA4ZrJppgwE0Gxmw==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.2.2.tgz", + "integrity": "sha512-Q/+HI/ToJafZ1iCqGgVQXUEkIjufHCTF0gBQ2a5o3cg7GJ2h0qyq3nvvSmU+bGda2/7ygXpTY4TM6gO9OhQ0ZA==", "cpu": [ "x64" ], @@ -1972,9 +1918,9 @@ } }, "node_modules/@rolldown/binding-linux-x64-musl": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.1.tgz", - "integrity": "sha512-bV4fzswuzVcKD90o/VM6QqKxnxlDq0g2BISDLNVmxrnhpv1DDbyPhCIjYfvzYLV+MvkKKnQt2Q6AO86SEBULUQ==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.2.2.tgz", + "integrity": "sha512-ZKp/w41n6wCvxzxQHtQSbuphfX3Y4cCvbjkKHusrLx4lh+JWLTU7StSltO/DKARISzbj368d+qUaCjI8K2wzXw==", "cpu": [ "x64" ], @@ -1989,9 +1935,9 @@ } }, "node_modules/@rolldown/binding-openharmony-arm64": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.1.tgz", - "integrity": "sha512-/Mh0Zhq3OP7fVs0kcQHZP6lZEthMGTaSf8UBQYSFEZDWGXXlEC+nJ6EqenaK2t4LBXMe3A+K/G2BVXXdtOr4PQ==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.2.2.tgz", + "integrity": "sha512-pxE6xD4KS3eAROkKK5yrhB9/3+vhlhVGMvlQLbdpzrBGDbKrnzx3RLwPaHvasLo6jgaiBLn7e04Df9C4tYhjmA==", "cpu": [ "arm64" ], @@ -2005,29 +1951,10 @@ "node": "^20.19.0 || >=22.12.0" } }, - "node_modules/@rolldown/binding-wasm32-wasi": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.1.tgz", - "integrity": "sha512-+1xc9X45l8ufsBAm6Gjvx2qDRIY9lTVt0cgWNcJ+1gdhXvkbxePA60yRTwSTuXL09CMhyJmjpV7E3NoyxbqFQQ==", - "cpu": [ - "wasm32" - ], - "dev": true, - "license": "MIT", - "optional": true, - "dependencies": { - "@emnapi/core": "1.11.2", - "@emnapi/runtime": "1.11.2", - "@napi-rs/wasm-runtime": "^1.1.4" - }, - "engines": { - "node": "^20.19.0 || >=22.12.0" - } - }, "node_modules/@rolldown/binding-win32-arm64-msvc": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.1.tgz", - "integrity": "sha512-1D+UqZdfnuR+Jy1GgMJwi85bD40H21uNmOPRWQhw4oRSuolZ/B5rixZ45DK2KXOTCvmVCecauWgEhbw8bI7tOw==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.2.2.tgz", + "integrity": "sha512-4MqEue5re+xIZzAWsB8sj0P1kqZySWqIuN4t6QaIO/YA6SFwySOLruvWQFzfmqk8LBK2P30KCSJwf6mJCZZ5/A==", "cpu": [ "arm64" ], @@ -2042,9 +1969,9 @@ } }, "node_modules/@rolldown/binding-win32-x64-msvc": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.1.tgz", - "integrity": "sha512-INAycaWuhlOK3wk4mRHGsdgwYWmd9cChdPdE9bwWmy6rn9VqVNYNFGhOdXrofXUxwHIncSiPNb8tNm8knDVIeQ==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.2.2.tgz", + "integrity": "sha512-NweNxxD0Nf9t8v7kodun45Ijp3EIwYY+uydPP6qBEYvfBqhIjN6dZMzlQja3tqX/aLs3F3Uz+AxDpKgRhpOZQg==", "cpu": [ "x64" ], @@ -2065,19 +1992,6 @@ "devOptional": true, "license": "MIT" }, - "node_modules/@rollup/rollup-linux-x64-gnu": { - "version": "4.54.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.54.0.tgz", - "integrity": "sha512-WGt5J8Ij/rvyqpFexxk3ffKqqbLf9AqrTBbWDk7ApGUzaIs6V+s2s84kAxklFwmMF/vBNGrVdYgbblCOFFezMQ==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] - }, "node_modules/@scalar/openapi-parser": { "version": "0.15.0", "resolved": "https://registry.npmjs.org/@scalar/openapi-parser/-/openapi-parser-0.15.0.tgz", @@ -2350,17 +2264,6 @@ "svelte": "^5.0.0" } }, - "node_modules/@tybys/wasm-util": { - "version": "0.10.2", - "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.2.tgz", - "integrity": "sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==", - "dev": true, - "license": "MIT", - "optional": true, - "dependencies": { - "tslib": "^2.4.0" - } - }, "node_modules/@types/chai": { "version": "5.2.3", "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz", @@ -8339,9 +8242,9 @@ "license": "MIT" }, "node_modules/lightningcss": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.32.0.tgz", - "integrity": "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.33.0.tgz", + "integrity": "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==", "devOptional": true, "license": "MPL-2.0", "dependencies": { @@ -8355,23 +8258,23 @@ "url": "https://opencollective.com/parcel" }, "optionalDependencies": { - "lightningcss-android-arm64": "1.32.0", - "lightningcss-darwin-arm64": "1.32.0", - "lightningcss-darwin-x64": "1.32.0", - "lightningcss-freebsd-x64": "1.32.0", - "lightningcss-linux-arm-gnueabihf": "1.32.0", - "lightningcss-linux-arm64-gnu": "1.32.0", - "lightningcss-linux-arm64-musl": "1.32.0", - "lightningcss-linux-x64-gnu": "1.32.0", - "lightningcss-linux-x64-musl": "1.32.0", - "lightningcss-win32-arm64-msvc": "1.32.0", - "lightningcss-win32-x64-msvc": "1.32.0" + "lightningcss-android-arm64": "1.33.0", + "lightningcss-darwin-arm64": "1.33.0", + "lightningcss-darwin-x64": "1.33.0", + "lightningcss-freebsd-x64": "1.33.0", + "lightningcss-linux-arm-gnueabihf": "1.33.0", + "lightningcss-linux-arm64-gnu": "1.33.0", + "lightningcss-linux-arm64-musl": "1.33.0", + "lightningcss-linux-x64-gnu": "1.33.0", + "lightningcss-linux-x64-musl": "1.33.0", + "lightningcss-win32-arm64-msvc": "1.33.0", + "lightningcss-win32-x64-msvc": "1.33.0" } }, "node_modules/lightningcss-android-arm64": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.32.0.tgz", - "integrity": "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.33.0.tgz", + "integrity": "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==", "cpu": [ "arm64" ], @@ -8390,9 +8293,9 @@ } }, "node_modules/lightningcss-darwin-arm64": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.32.0.tgz", - "integrity": "sha512-RzeG9Ju5bag2Bv1/lwlVJvBE3q6TtXskdZLLCyfg5pt+HLz9BqlICO7LZM7VHNTTn/5PRhHFBSjk5lc4cmscPQ==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.33.0.tgz", + "integrity": "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==", "cpu": [ "arm64" ], @@ -8411,9 +8314,9 @@ } }, "node_modules/lightningcss-darwin-x64": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.32.0.tgz", - "integrity": "sha512-U+QsBp2m/s2wqpUYT/6wnlagdZbtZdndSmut/NJqlCcMLTWp5muCrID+K5UJ6jqD2BFshejCYXniPDbNh73V8w==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.33.0.tgz", + "integrity": "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==", "cpu": [ "x64" ], @@ -8432,9 +8335,9 @@ } }, "node_modules/lightningcss-freebsd-x64": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.32.0.tgz", - "integrity": "sha512-JCTigedEksZk3tHTTthnMdVfGf61Fky8Ji2E4YjUTEQX14xiy/lTzXnu1vwiZe3bYe0q+SpsSH/CTeDXK6WHig==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.33.0.tgz", + "integrity": "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==", "cpu": [ "x64" ], @@ -8453,9 +8356,9 @@ } }, "node_modules/lightningcss-linux-arm-gnueabihf": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.32.0.tgz", - "integrity": "sha512-x6rnnpRa2GL0zQOkt6rts3YDPzduLpWvwAF6EMhXFVZXD4tPrBkEFqzGowzCsIWsPjqSK+tyNEODUBXeeVHSkw==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.33.0.tgz", + "integrity": "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==", "cpu": [ "arm" ], @@ -8474,9 +8377,9 @@ } }, "node_modules/lightningcss-linux-arm64-gnu": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.32.0.tgz", - "integrity": "sha512-0nnMyoyOLRJXfbMOilaSRcLH3Jw5z9HDNGfT/gwCPgaDjnx0i8w7vBzFLFR1f6CMLKF8gVbebmkUN3fa/kQJpQ==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.33.0.tgz", + "integrity": "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==", "cpu": [ "arm64" ], @@ -8495,9 +8398,9 @@ } }, "node_modules/lightningcss-linux-arm64-musl": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.32.0.tgz", - "integrity": "sha512-UpQkoenr4UJEzgVIYpI80lDFvRmPVg6oqboNHfoH4CQIfNA+HOrZ7Mo7KZP02dC6LjghPQJeBsvXhJod/wnIBg==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.33.0.tgz", + "integrity": "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==", "cpu": [ "arm64" ], @@ -8516,9 +8419,9 @@ } }, "node_modules/lightningcss-linux-x64-gnu": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.32.0.tgz", - "integrity": "sha512-V7Qr52IhZmdKPVr+Vtw8o+WLsQJYCTd8loIfpDaMRWGUZfBOYEJeyJIkqGIDMZPwPx24pUMfwSxxI8phr/MbOA==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.33.0.tgz", + "integrity": "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==", "cpu": [ "x64" ], @@ -8537,9 +8440,9 @@ } }, "node_modules/lightningcss-linux-x64-musl": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.32.0.tgz", - "integrity": "sha512-bYcLp+Vb0awsiXg/80uCRezCYHNg1/l3mt0gzHnWV9XP1W5sKa5/TCdGWaR/zBM2PeF/HbsQv/j2URNOiVuxWg==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.33.0.tgz", + "integrity": "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==", "cpu": [ "x64" ], @@ -8558,9 +8461,9 @@ } }, "node_modules/lightningcss-win32-arm64-msvc": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.32.0.tgz", - "integrity": "sha512-8SbC8BR40pS6baCM8sbtYDSwEVQd4JlFTOlaD3gWGHfThTcABnNDBda6eTZeqbofalIJhFx0qKzgHJmcPTnGdw==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.33.0.tgz", + "integrity": "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==", "cpu": [ "arm64" ], @@ -8579,9 +8482,9 @@ } }, "node_modules/lightningcss-win32-x64-msvc": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.32.0.tgz", - "integrity": "sha512-Amq9B/SoZYdDi1kFrojnoqPLxYhQ4Wo5XiL8EVJrVsB8ARoC1PWW6VGtT0WKCemjy8aC+louJnjS7U18x3b06Q==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.33.0.tgz", + "integrity": "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==", "cpu": [ "x64" ], @@ -10785,9 +10688,9 @@ } }, "node_modules/postcss": { - "version": "8.5.14", - "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz", - "integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==", + "version": "8.5.25", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.25.tgz", + "integrity": "sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw==", "devOptional": true, "funding": [ { @@ -10805,7 +10708,7 @@ ], "license": "MIT", "dependencies": { - "nanoid": "^3.3.11", + "nanoid": "^3.3.16", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" }, @@ -11497,9 +11400,9 @@ "license": "MIT" }, "node_modules/postcss/node_modules/nanoid": { - "version": "3.3.11", - "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz", - "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==", + "version": "3.3.16", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.16.tgz", + "integrity": "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q==", "devOptional": true, "funding": [ { @@ -12195,13 +12098,13 @@ "license": "Unlicense" }, "node_modules/rolldown": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.1.tgz", - "integrity": "sha512-X0KQHljNnEkWNqqiz9zJrGunh1B0HgOxLXvnFpCOcadzcy5qohZ3tqMEUg00vncoRovXuK3ZqCT9KnnKzoInFQ==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.2.2.tgz", + "integrity": "sha512-opwpo1tQBAcpSUJDt94B7hhLNGOKjCdE//XXjeLrnx9b83bjnw45tXdg1b09yEw/VLFBJGZpwRULMmOZo7ol+A==", "devOptional": true, "license": "MIT", "dependencies": { - "@oxc-project/types": "=0.130.0", + "@oxc-project/types": "=0.142.0", "@rolldown/pluginutils": "^1.0.0" }, "bin": { @@ -12211,21 +12114,20 @@ "node": "^20.19.0 || >=22.12.0" }, "optionalDependencies": { - "@rolldown/binding-android-arm64": "1.0.1", - "@rolldown/binding-darwin-arm64": "1.0.1", - "@rolldown/binding-darwin-x64": "1.0.1", - "@rolldown/binding-freebsd-x64": "1.0.1", - "@rolldown/binding-linux-arm-gnueabihf": "1.0.1", - "@rolldown/binding-linux-arm64-gnu": "1.0.1", - "@rolldown/binding-linux-arm64-musl": "1.0.1", - "@rolldown/binding-linux-ppc64-gnu": "1.0.1", - "@rolldown/binding-linux-s390x-gnu": "1.0.1", - "@rolldown/binding-linux-x64-gnu": "1.0.1", - "@rolldown/binding-linux-x64-musl": "1.0.1", - "@rolldown/binding-openharmony-arm64": "1.0.1", - "@rolldown/binding-wasm32-wasi": "1.0.1", - "@rolldown/binding-win32-arm64-msvc": "1.0.1", - "@rolldown/binding-win32-x64-msvc": "1.0.1" + "@rolldown/binding-android-arm64": "1.2.2", + "@rolldown/binding-darwin-arm64": "1.2.2", + "@rolldown/binding-darwin-x64": "1.2.2", + "@rolldown/binding-freebsd-x64": "1.2.2", + "@rolldown/binding-linux-arm-gnueabihf": "1.2.2", + "@rolldown/binding-linux-arm64-gnu": "1.2.2", + "@rolldown/binding-linux-arm64-musl": "1.2.2", + "@rolldown/binding-linux-ppc64-gnu": "1.2.2", + "@rolldown/binding-linux-s390x-gnu": "1.2.2", + "@rolldown/binding-linux-x64-gnu": "1.2.2", + "@rolldown/binding-linux-x64-musl": "1.2.2", + "@rolldown/binding-openharmony-arm64": "1.2.2", + "@rolldown/binding-win32-arm64-msvc": "1.2.2", + "@rolldown/binding-win32-x64-msvc": "1.2.2" } }, "node_modules/roughjs": { @@ -13835,9 +13737,9 @@ "license": "MIT" }, "node_modules/tinyglobby": { - "version": "0.2.16", - "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", - "integrity": "sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg==", + "version": "0.2.17", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", + "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", "devOptional": true, "license": "MIT", "dependencies": { @@ -14373,17 +14275,17 @@ } }, "node_modules/vite": { - "version": "8.0.13", - "resolved": "https://registry.npmjs.org/vite/-/vite-8.0.13.tgz", - "integrity": "sha512-MFtjBYgzmSxmgA4RAfjIyXWpGe1oALnjgUTzzV7QLx/TKxCzjtMH6Fd9/eVK+5Fg1qNoz5VAwsmMs/NofrmJvw==", + "version": "8.2.0", + "resolved": "https://registry.npmjs.org/vite/-/vite-8.2.0.tgz", + "integrity": "sha512-pn+CFpM0lwDeKwmOq1ZaBK/9sjorZcgqxki6MbY/jPEVd9vichIlmlD4HmQ5wdP5EgqQCFRaACBxMC7uEGc6lQ==", "devOptional": true, "license": "MIT", "dependencies": { - "lightningcss": "^1.32.0", - "picomatch": "^4.0.4", - "postcss": "^8.5.14", - "rolldown": "1.0.1", - "tinyglobby": "^0.2.16" + "lightningcss": "^1.33.0", + "picomatch": "^4.0.5", + "postcss": "^8.5.23", + "rolldown": "~1.2.0", + "tinyglobby": "^0.2.17" }, "bin": { "vite": "bin/vite.js" @@ -14399,7 +14301,7 @@ }, "peerDependencies": { "@types/node": "^20.19.0 || >=22.12.0", - "@vitejs/devtools": "^0.1.18", + "@vitejs/devtools": "^0.4.0", "esbuild": "^0.27.0 || ^0.28.0", "jiti": ">=1.21.0", "less": "^4.0.0", @@ -14960,9 +14862,9 @@ "integrity": "sha512-/ovcLWlIO+TQMrnWwcWoXquJI6hTZ+Zpo4POhV8z6e+pb4cVModGysASI9mERn2e9uj8/xSRmYqk54TLG8oUbQ==" }, "node_modules/windmill-parser-wasm-yaml": { - "version": "1.593.0", - "resolved": "https://registry.npmjs.org/windmill-parser-wasm-yaml/-/windmill-parser-wasm-yaml-1.593.0.tgz", - "integrity": "sha512-Gyx4aR2jsJYuDrD3mCNTmz7LWOQQXPw5yKNCC1xRgUOPfjsD/tINAFfsBLwVOSmlQQcFZO+wHm4KtDtXOcnGVw==" + "version": "1.770.0", + "resolved": "https://registry.npmjs.org/windmill-parser-wasm-yaml/-/windmill-parser-wasm-yaml-1.770.0.tgz", + "integrity": "sha512-RngxbcKYYFt3MHYqdYTzbtH6EMPfqJtY4/mqRqfLUie4cE9mmOBa9X9M3VEFVo53CD1dCRa9wJKZmeJzE5We5Q==" }, "node_modules/windmill-sql-datatype-parser-wasm": { "version": "1.512.0", diff --git a/frontend/package.json b/frontend/package.json index 01e68d2acf..ff080f353b 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.775.2", + "version": "1.783.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", @@ -73,7 +73,7 @@ "tar": "^7.5.4", "tslib": "^2.6.1", "typescript": "^5.5.0", - "vite": "^8.0.13", + "vite": "^8.2.0", "vite-plugin-mkcert": "^2.0.0", "vitest": "^4.1.0", "vitest-browser-svelte": "^2.0.1" @@ -170,7 +170,7 @@ "windmill-parser-wasm-rust": "1.647.1", "windmill-parser-wasm-ts": "1.695.0", "windmill-parser-wasm-wac": "1.668.6", - "windmill-parser-wasm-yaml": "1.593.0", + "windmill-parser-wasm-yaml": "1.770.0", "windmill-sql-datatype-parser-wasm": "1.512.0", "windmill-utils-internal": "1.8.2", "xterm": "^5.3.0", @@ -647,7 +647,6 @@ } }, "optionalDependencies": { - "@rollup/rollup-linux-x64-gnu": "^4.35.0", "fsevents": "^2.3.3" } } diff --git a/frontend/src/lib/aiStore.ts b/frontend/src/lib/aiStore.ts index bd03f1baa4..1e8fb6ac61 100644 --- a/frontend/src/lib/aiStore.ts +++ b/frontend/src/lib/aiStore.ts @@ -1,6 +1,9 @@ +// Keep this module a leaf: it holds AI model *state* only, and must not import the AI +// client (`components/copilot/lib`) or anything under `components/copilot/chat` — those +// import aiStore back, and such a cycle crashes the app once the bundler splits it across +// chunks (docs/frontend-import-cycles.md; the build fails on the chunk cycle, not on this). import { writable, get } from 'svelte/store' -import { workspaceAIClients } from './components/copilot/lib' -import { type AIProviderModel, type AIProvider, WorkspaceService, type AIConfig } from './gen' +import { type AIProviderModel, type AIProvider, type AIConfig } from './gen' import { aiUserDisabled, COPILOT_SESSION_MODEL_SETTING_NAME, @@ -77,31 +80,10 @@ function dedupeModels(models: AIProviderModel[]): AIProviderModel[] { }) } -// copilotInfo/copilotSessionModel are global, so concurrent loads (e.g. a fast -// session switch between workspaces) race: an earlier call resolving last would -// clobber the active workspace's config. Apply only the most recent call's -// result via a monotonic token — last invocation wins regardless of resolution -// order. init() is synchronous so its ordering already matches. -let loadCopilotToken = 0 // The workspace copilotInfo currently reflects. A session send awaits this // matching its committed workspace so getCurrentModel() can't read the previous // workspace's provider/model while the scoped load is still in flight. export const copilotWorkspace = writable(undefined) -export async function loadCopilot(workspace: string) { - const token = ++loadCopilotToken - workspaceAIClients.init(workspace) - try { - const info = await WorkspaceService.getCopilotInfo({ workspace }) - if (token !== loadCopilotToken) return - setCopilotInfo(info) - copilotWorkspace.set(workspace) - } catch (err) { - if (token !== loadCopilotToken) return - setCopilotInfo({}) - copilotWorkspace.set(workspace) - console.error('Could not get copilot info', err) - } -} export function setCopilotInfo(aiConfig: AIConfig) { if (Object.keys(aiConfig.providers ?? {}).length > 0) { diff --git a/frontend/src/lib/assets/app.css b/frontend/src/lib/assets/app.css index 23bf45c7fa..760f4c0261 100644 --- a/frontend/src/lib/assets/app.css +++ b/frontend/src/lib/assets/app.css @@ -60,6 +60,155 @@ font-display: swap; } + /* Firefox rasterizes the macOS bitmap emoji font (Apple Color Emoji, sbix) at a size + that ignores an ancestor's `transform: scale()`, so emoji keep their unzoomed size + inside the flow graph canvas. Noto's COLR glyphs are vectors and scale like any + other text. unicode-range split verbatim from Google Fonts so pages fetch only the + chunks they render — a range edited out of step with its file mis-gates that. */ + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.0.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: U+1f1e6-1f1ff; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.1.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+200d, U+2620, U+26a7, U+fe0f, U+1f308, U+1f38c, U+1f3c1, U+1f3f3-1f3f4, U+1f6a9, + U+e0062-e0063, U+e0065, U+e0067, U+e006c, U+e006e, U+e0073-e0074, U+e0077, U+e007f; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.2.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+23, U+2a, U+30-39, U+a9, U+ae, U+200d, U+203c, U+2049, U+20e3, U+2122, U+2139, U+2194-2199, + U+21a9-21aa, U+23cf, U+23e9-23ef, U+23f8-23fa, U+24c2, U+25aa-25ab, U+25b6, U+25c0, + U+25fb-25fe, U+2611, U+2622-2623, U+2626, U+262a, U+262e-262f, U+2638, U+2640, U+2642, + U+2648-2653, U+2660, U+2663, U+2665-2666, U+2668, U+267b, U+267e-267f, U+2695, U+269b-269c, + U+26a0, U+26a7, U+26aa-26ab, U+26ce, U+26d4, U+2705, U+2714, U+2716, U+271d, U+2721, + U+2733-2734, U+2747, U+274c, U+274e, U+2753-2755, U+2757, U+2764, U+2795-2797, U+27a1, U+27b0, + U+27bf, U+2934-2935, U+2b05-2b07, U+2b1b-2b1c, U+2b55, U+3030, U+303d, U+3297, U+3299, U+fe0f, + U+1f170-1f171, U+1f17e-1f17f, U+1f18e, U+1f191-1f19a, U+1f201-1f202, U+1f21a, U+1f22f, + U+1f232-1f23a, U+1f250-1f251, U+1f310, U+1f3a6, U+1f3b5-1f3b6, U+1f3bc, U+1f3e7, U+1f441, + U+1f499-1f49c, U+1f49f-1f4a0, U+1f4ac-1f4ad, U+1f4b1-1f4b2, U+1f4b9, U+1f4db, U+1f4f2-1f4f6, + U+1f500-1f507, U+1f515, U+1f518-1f524, U+1f52f-1f53d, U+1f549, U+1f54e, U+1f5a4, U+1f5e8, + U+1f5ef, U+1f6ab, U+1f6ad-1f6b1, U+1f6b3, U+1f6b7-1f6bc, U+1f6be, U+1f6c2-1f6c5, + U+1f6d0-1f6d1, U+1f6d7, U+1f6dc, U+1f7e0-1f7eb, U+1f7f0, U+1f90d-1f90e, U+1f9d1-1f9d2, + U+1f9e1, U+1fa75-1fa77, U+1faaf; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.3.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+200d, U+231a-231b, U+2328, U+23f0-23f3, U+2602, U+260e, U+2692, U+2694, U+2696-2697, U+2699, + U+26b0-26b1, U+26cf, U+26d1, U+26d3, U+2702, U+2709, U+270f, U+2712, U+fe0f, U+1f302, U+1f321, + U+1f392-1f393, U+1f3a9, U+1f3bd, U+1f3ee, U+1f3f7, U+1f3fa, U+1f451-1f462, U+1f484, + U+1f489-1f48a, U+1f48c-1f48e, U+1f4a1, U+1f4a3, U+1f4a5, U+1f4b0, U+1f4b3-1f4b8, + U+1f4bb-1f4da, U+1f4dc-1f4f1, U+1f4ff, U+1f508-1f514, U+1f516-1f517, U+1f526-1f529, + U+1f52c-1f52e, U+1f550-1f567, U+1f56f-1f570, U+1f576, U+1f587, U+1f58a-1f58d, U+1f5a5, + U+1f5a8, U+1f5b1-1f5b2, U+1f5c2-1f5c4, U+1f5d1-1f5d3, U+1f5dc-1f5de, U+1f5e1, U+1f5f3, + U+1f6aa, U+1f6ac, U+1f6bd, U+1f6bf, U+1f6c1, U+1f6cb, U+1f6cd-1f6cf, U+1f6d2, U+1f6e0-1f6e1, + U+1f6f0, U+1f97b-1f97f, U+1f9af, U+1f9ba, U+1f9e2-1f9e6, U+1f9ea-1f9ec, U+1f9ee-1f9f4, + U+1f9f7-1f9ff, U+1fa71-1fa74, U+1fa79-1fa7b, U+1fa86, U+1fa8e-1fa8f, U+1fa91-1fa93, U+1fa96, + U+1fa99-1faa0, U+1faa2-1faa7, U+1faaa-1faae; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.4.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+265f, U+26bd-26be, U+26f3, U+26f8, U+fe0f, U+1f004, U+1f0cf, U+1f380-1f384, U+1f386-1f38b, + U+1f38d-1f391, U+1f396-1f397, U+1f399-1f39b, U+1f39e-1f39f, U+1f3a3-1f3a5, U+1f3a7-1f3a9, + U+1f3ab-1f3b4, U+1f3b7-1f3bb, U+1f3bd-1f3c0, U+1f3c5-1f3c6, U+1f3c8-1f3c9, U+1f3cf-1f3d3, + U+1f3f8-1f3f9, U+1f47e, U+1f4e2, U+1f4f7-1f4fd, U+1f52b, U+1f579, U+1f58c-1f58d, U+1f5bc, + U+1f6f7, U+1f6f9, U+1f6fc, U+1f93f, U+1f941, U+1f945, U+1f947-1f94f, U+1f9e7-1f9e9, + U+1f9f5-1f9f6, U+1fa70-1fa71, U+1fa80-1fa81, U+1fa83-1fa85, U+1fa87-1fa8a, U+1fa94-1fa95, + U+1fa97-1fa98, U+1faa1, U+1faa9, U+1fadf; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.5.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+2693, U+26e9-26ea, U+26f1-26f2, U+26f4-26f5, U+26fa, U+26fd, U+2708, U+fe0f, U+1f301, + U+1f303, U+1f306-1f307, U+1f309, U+1f310, U+1f3a0-1f3a2, U+1f3aa, U+1f3cd-1f3ce, U+1f3d5, + U+1f3d7-1f3db, U+1f3df-1f3e6, U+1f3e8-1f3ed, U+1f3ef-1f3f0, U+1f488, U+1f492, U+1f4ba, + U+1f54b-1f54d, U+1f5fa-1f5ff, U+1f680-1f6a2, U+1f6a4-1f6a8, U+1f6b2, U+1f6d1, U+1f6d5-1f6d6, + U+1f6dd-1f6df, U+1f6e2-1f6e5, U+1f6e9, U+1f6eb-1f6ec, U+1f6f3-1f6f6, U+1f6f8, U+1f6fa-1f6fb, + U+1f9bc-1f9bd, U+1f9ed, U+1f9f3, U+1fa7c; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.6.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+200d, U+2615, U+fe0f, U+1f32d-1f330, U+1f336, U+1f33d, U+1f344-1f37f, U+1f382, U+1f52a, + U+1f7e9, U+1f7eb, U+1f942-1f944, U+1f950-1f96f, U+1f99e, U+1f9aa, U+1f9c0-1f9cb, U+1fad0-1fadc; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.7.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+200d, U+2600-2601, U+2603-2604, U+2614, U+2618, U+26a1, U+26c4-26c5, U+26c8, U+26f0, U+2728, + U+2744, U+2b1b, U+2b50, U+fe0f, U+1f300, U+1f304-1f305, U+1f308, U+1f30a-1f30f, U+1f311-1f321, + U+1f324-1f32c, U+1f331-1f335, U+1f337-1f33c, U+1f33e-1f344, U+1f3d4, U+1f3d6, U+1f3dc-1f3de, + U+1f3f5, U+1f400-1f43f, U+1f490, U+1f4a7, U+1f4ae, U+1f525, U+1f54a, U+1f573, U+1f577-1f578, + U+1f648-1f64a, U+1f6d8, U+1f940, U+1f980-1f9ae, U+1f9ba, U+1fa90, U+1faa8, U+1fab0-1fabf, + U+1facd-1facf, U+1fae7; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.8.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+200d, U+2640, U+2642, U+2695-2696, U+26f7, U+26f9, U+2708, U+2764, U+27a1, U+fe0f, U+1f33e, + U+1f373, U+1f37c, U+1f384-1f385, U+1f393, U+1f3a4, U+1f3a8, U+1f3c2-1f3c4, U+1f3c7, + U+1f3ca-1f3cc, U+1f3eb, U+1f3ed, U+1f3fb-1f3ff, U+1f430, U+1f466-1f469, U+1f46b-1f478, + U+1f47c, U+1f481-1f483, U+1f486-1f487, U+1f48b, U+1f48f, U+1f491, U+1f4bb-1f4bc, U+1f527, + U+1f52c, U+1f574-1f575, U+1f57a, U+1f645-1f647, U+1f64b, U+1f64d-1f64e, U+1f680, U+1f692, + U+1f6a3, U+1f6b4-1f6b6, U+1f6c0, U+1f6cc, U+1f91d, U+1f926, U+1f930-1f931, U+1f934-1f93a, + U+1f93c-1f93e, U+1f977, U+1f9af-1f9b3, U+1f9b8-1f9b9, U+1f9bc-1f9bd, U+1f9cc-1f9cf, + U+1f9d1-1f9df, U+1fa70, U+1fa82, U+1fac3-1fac5, U+1fac8, U+1faef; + } + + @font-face { + font-family: 'Noto Color Emoji'; + src: url('/fonts/noto-color-emoji.9.woff2') format('woff2'); + font-weight: 400; + font-display: swap; + unicode-range: + U+200d, U+2194-2195, U+2603, U+261d, U+2620, U+2639-263a, U+2665, U+26a1, U+26c4, U+270a-270d, + U+2728, U+2763-2764, U+2b50, U+fe0f, U+1f31a-1f31f, U+1f32b, U+1f389-1f38a, U+1f3fb-1f3ff, + U+1f440-1f450, U+1f463-1f465, U+1f479-1f47b, U+1f47d-1f480, U+1f485, U+1f48b-1f48c, + U+1f493-1f49f, U+1f4a2, U+1f4a4-1f4a6, U+1f4a8-1f4ab, U+1f4af, U+1f525, U+1f573, U+1f590, + U+1f595-1f596, U+1f5a4, U+1f5e3, U+1f600-1f644, U+1f648-1f64a, U+1f64c, U+1f64f, + U+1f90c-1f925, U+1f927-1f92f, U+1f932-1f933, U+1f970-1f976, U+1f978-1f97a, U+1f9a0, + U+1f9b4-1f9b7, U+1f9bb, U+1f9be-1f9bf, U+1f9d0, U+1f9e0-1f9e1, U+1fa75-1fa79, U+1fac0-1fac2, + U+1fac6, U+1fae0-1fae6, U+1fae8-1faea, U+1faef-1faf8; + } + .prose-xs ul { margin-top: 0.5rem; list-style-type: '- '; @@ -131,8 +280,9 @@ } /* Subtle scrollbar: a thin, rounded thumb that only appears on hover, on both - axes. Shared by ScrollableX (tab strips, code blocks) and the AI chat. Size - via the `--wm-scrollbar-size` var (default 6px). Higher specificity than the + axes. Shared by ScrollableX (code blocks, tab headers) and the AI chat. Size + via the `--wm-scrollbar-size` var (default 6px) — WebKit only; Firefox sizes + `thin` itself and spends ~11px of the box on it. Higher specificity than the app-wide `*::-webkit-scrollbar`, so it overrides it. */ .scrollbar-subtle { scrollbar-width: thin; diff --git a/frontend/src/lib/components/AddUser.svelte b/frontend/src/lib/components/AddUser.svelte index e3b1455d77..aef1dbd400 100644 --- a/frontend/src/lib/components/AddUser.svelte +++ b/frontend/src/lib/components/AddUser.svelte @@ -12,11 +12,61 @@ import Toggle from './Toggle.svelte' import Tooltip from './Tooltip.svelte' import { UserPlus } from 'lucide-svelte' + import Select from './select/Select.svelte' + import { resource } from 'runed' const dispatch = createEventDispatcher() let email: string | undefined = $state() let username: string | undefined = $state() + let emailFilterText: string = $state('') + let typedEmail: string = $state('') + let popoverOpen: boolean = $state(false) + let dropdownOpen: boolean = $state(false) + let addedCount: number = $state(0) + + // Only superadmins may list the instance users, everyone else just types the email in. + // The list stays capped: what is typed narrows it server-side rather than client-side. + const instanceUsers = resource( + () => ({ + search: emailFilterText, + isSuperadmin: $superadmin, + open: popoverOpen, + workspace: $workspaceStore, + addedCount + }), + async ({ search, isSuperadmin, open, workspace }, _previous, { onCleanup }) => { + if (!isSuperadmin || !open || !workspace) return [] + const request = UserService.listAddableInstanceUsers({ + workspace, + perPage: 10, + search: search || undefined + }) + // A superseded run is only aborted through the promise it registers here. Without this a + // slow earlier search can land last and overwrite the results of the newer one. + onCleanup(() => request.cancel()) + return await request + }, + { debounce: 300, initialValue: [] } + ) + + let emailItems = $derived( + instanceUsers.current.map((u) => ({ + value: u.email, + label: u.username ? `${u.username} (${u.email})` : u.email + })) + ) + + const EMAIL_REGEX = /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/ + // A whole address typed into the picker is submittable on its own, so `Add` stays reachable + // without going through the "Add new" row. A search term that is not an address is not: it + // would be submitted verbatim and rejected instead of adding whoever the list is showing. + let effectiveEmail = $derived( + email ?? + (EMAIL_REGEX.test(typedEmail.trim().toLowerCase()) + ? typedEmail.trim().toLowerCase() + : undefined) + ) function handleKeyUp(event: KeyboardEvent) { const key = event.key @@ -47,20 +97,24 @@ }) sendUserToast(`Service account '${username}' created`) } else { + // Read once: the picker's own state changes as soon as the request lands. + const email = effectiveEmail! await WorkspaceService.addUser({ workspace: $workspaceStore!, requestBody: { - email: email!, + email, username: automateUsernameCreation ? undefined : username, is_admin: selected == 'admin', operator: selected == 'operator' } }) sendUserToast(`Added ${email}`) - if (!(await UserService.existsEmail({ email: email! }))) { + // The picker excludes members server-side, so it has to refetch to drop this one. + addedCount++ + if (!(await UserService.existsEmail({ email }))) { let isSuperadmin = $superadmin if (!isCloudHosted()) { - const emailCopy = email! + const emailCopy = email sendUserToast( `User ${email} is not registered yet on the instance. ${ !isSuperadmin @@ -94,7 +148,7 @@ let isServiceAccount = $derived(selected === 'service_account') - + (typedEmail = '')}> {#snippet trigger()} + + + + {/if} +{/snippet} + +{#if isThrowawayFork} +

+ Dev workspace pairing is not available on a throwaway fork. This workspace is a fork of + {parentId}. +

+ {@render deployTarget?.()} +{:else if isDev && parentId}

- This is a {devLabelNoun(currentWs?.dev_workspace_label)} paired with root workspace + This is a {devLabelNoun(currentWs?.dev_workspace_label)} paired with + {parentWs?.is_dev_workspace ? devLabelNoun(parentWs.dev_workspace_label) : 'root workspace'} {parentId}. Promote changes from the home page banner or the Compare & Deploy page.

@@ -183,115 +560,43 @@ {currentLabel} branch.
+
+ {@render deployTarget?.()} +
+ + {#if !parentRules.readFailed} + {@render protectionsPanel({ + title: `Protections in force on ${parentId}`, + rules: parentRules, + onOpen: canEditParentRules ? openRulesetsInParent : undefined, + editLabel: `Edit in ${parentId}`, + manageLabel: `Manage in ${parentId}` + })} + {/if}
-{:else if pairedDev} -
-

- This workspace's {devLabelNoun(pairedDev.label)} is {pairedDev.name} ({pairedDev.id}). - Edits to this workspace are redirected there. -

-
- {#if pairedDev.isMember} - - {/if} - -
-
-{:else if parentId} -

- Dev workspace pairing is only available for root workspaces. This workspace is a fork of - {parentId}. -

-{:else} -
-

- Pair this workspace with a dev workspace: the same code with a different environment (resource - and variable values). Edits are made in the dev workspace and promoted here. -

+
- Attach an existing workspace as dev - - Bypass restriction - - {/if} -
- + bind:checked={overrideChecked} + options={{ right: 'Bypass restriction' }} + /> + {/if} +
+ {/snippet} +
{/if} diff --git a/frontend/src/lib/components/OnBehalfOfSelector.svelte b/frontend/src/lib/components/OnBehalfOfSelector.svelte index 651b76f94a..67648936ae 100644 --- a/frontend/src/lib/components/OnBehalfOfSelector.svelte +++ b/frontend/src/lib/components/OnBehalfOfSelector.svelte @@ -28,6 +28,7 @@ - e.detail && loadUsers()}> + e.detail && loadUsers()} +> {#snippet trigger()} @@ -228,56 +259,65 @@ {/snippet} - - -
-
- {#if isTrigger} - Choose the user this trigger will be permissioned as {isDeployment - ? 'in the target workspace' - : 'in this workspace'}. The selected user's permissions will be used when the trigger - fires. - {:else} - Choose the user this {kind} will run on behalf of {isDeployment - ? 'in the target workspace' - : 'in this workspace'}. The selected user's permissions will be used when executing. - {/if} - - Learn more - - -
- - - -
- {#each filteredUsers as user (user.email)} - + {:else} +
+ {#if !usersLoaded} + Loading users… + {:else} + No users found + {/if}
- {#if selected === 'custom' && (customValue === `u/${user.username}` || customValue === user.email)} - - {/if} - - {:else} -
- {#if !usersLoaded} - Loading users… - {:else} - No users found - {/if} -
- {/each} + {/each} +
- -
+ + diff --git a/frontend/src/lib/components/ResourceEditor.svelte b/frontend/src/lib/components/ResourceEditor.svelte index 7acc4e485b..8b10841512 100644 --- a/frontend/src/lib/components/ResourceEditor.svelte +++ b/frontend/src/lib/components/ResourceEditor.svelte @@ -130,7 +130,11 @@ const schema = rt.schema as Schema return { ...schema, - order: schema.order ?? Object.keys(schema.properties).sort() + // A resource type may declare no properties at all — `dbt_profile` is a + // `profiles.yml` block whose keys are its adapter's, not Windmill's — and + // the form renders those as one JSON editor. `Object.keys(undefined)` + // threw here, which left the editor on its loading skeleton forever. + order: schema.order ?? Object.keys(schema.properties ?? {}).sort() } }) let loadingSchema = $derived(resourceTypeResource.loading) diff --git a/frontend/src/lib/components/ResourceEditorDrawer.svelte b/frontend/src/lib/components/ResourceEditorDrawer.svelte index 6c0da69675..5067972cfa 100644 --- a/frontend/src/lib/components/ResourceEditorDrawer.svelte +++ b/frontend/src/lib/components/ResourceEditorDrawer.svelte @@ -3,17 +3,21 @@ import DrawerContent from './common/drawer/DrawerContent.svelte' - import { Loader2, Save } from 'lucide-svelte' + import { History, Loader2, Save } from 'lucide-svelte' import WsSpecificVersions from './WsSpecificVersions.svelte' - import { workspaceStore } from '$lib/stores' + import { userStore, workspaceStore } from '$lib/stores' + import { isOwner } from '$lib/utils' import LocalDraftBanner from './LocalDraftBanner.svelte' + import ResourceVersionHistory from './ResourceVersionHistory.svelte' let { workspace = undefined, - disableChatOffset = false - }: { workspace?: string; disableChatOffset?: boolean } = $props() + disableChatOffset = false, + onRestored = undefined + }: { workspace?: string; disableChatOffset?: boolean; onRestored?: () => void } = $props() let drawer: Drawer | undefined = $state() + let historyDrawer: Drawer | undefined = $state() let canSave = $state(true) let resource_type: string | undefined = $state(undefined) let defaultValues: Record | undefined = $state(undefined) @@ -33,6 +37,16 @@ let selected: string | undefined = $state(undefined) let effectiveWorkspace = $derived(workspace ?? $workspaceStore!) + // The editor renders whichever workspace-specific variant `selected` points at, so history has + // to follow it too — otherwise a restore would write over the variant the user is not looking at. + let historyWorkspace = $derived(selected ?? effectiveWorkspace) + // Clearing is irreversible and the backend gates it on ownership, not write access. $userStore + // describes the user in the workspace they are signed into, so it can only answer for that one: + // history pointed anywhere else — a ws-specific variant, or an explicit `workspace` prop — gets + // no Clear button rather than a verdict computed from the wrong membership. + let canClearSelected = $derived( + historyWorkspace === $workspaceStore && isOwner(path ?? '', $userStore, $workspaceStore) + ) export async function initEdit(p: string): Promise { resource_type = undefined @@ -89,6 +103,14 @@ {/snippet} {#snippet actions()} {#if mode == 'edit' && path && effectiveWorkspace} + + + + + {#if path && historyWorkspace} + { + historyDrawer?.closeDrawer() + // Close the editor too. It holds a baseline captured before the restore, and + // any local draft on top of it, so saving from it afterwards would write the + // pre-restore value straight back over the version just restored. + drawer?.closeDrawer() + // Its own callback rather than the `refresh` event: callers bind that to + // reopening a picker (EditorBar), which a restore should not trigger. + onRestored?.() + }} + /> + {/if} + + diff --git a/frontend/src/lib/components/ResourceForm.svelte b/frontend/src/lib/components/ResourceForm.svelte index 2c563a8e95..3820004593 100644 --- a/frontend/src/lib/components/ResourceForm.svelte +++ b/frontend/src/lib/components/ResourceForm.svelte @@ -291,7 +291,7 @@ {:else if !can_write} {:else} - {#if !viewJsonSchema} + {#if !viewJsonSchema && !resourceSchema}

Resource type '{resource_type}' not found in your workspace diff --git a/frontend/src/lib/components/ResourceVersionHistory.svelte b/frontend/src/lib/components/ResourceVersionHistory.svelte new file mode 100644 index 0000000000..90e41b74f0 --- /dev/null +++ b/frontend/src/lib/components/ResourceVersionHistory.svelte @@ -0,0 +1,314 @@ + + + + + {#if versions === undefined} +

+ {#each Array(4) as _} + + {/each} +
+ {:else if versions.length === 0} +
+ {#if neverVersioned} + Resources of this type are written by every job that uses them, so their values are + deliberately not kept in history. + {:else} + No history yet. Versions are recorded from the next edit onwards. + {/if} +
+ {:else} + {#if versions.length > 1 && canClear} +
+ {#if confirmingClear} +

+ Delete every past version, keeping only the current value? Past values can no longer + be compared or restored. +

+
+ + +
+ {:else} + + {/if} +
+ {/if} +
+ {#each versions as version, index (version.id)} + selectVersion(version.id)} + > +
+
+ {#if index === 0} + + {/if} + {index === 0 ? 'Current' : `Version ${version.id}`} +
+
+ {displayDate(version.created_at)}{version.created_by + ? ` by ${version.created_by}` + : ''} +
+
+
+ {/each} +
+ {/if} + + + {#if loaded === undefined && selectedId !== undefined} + + {:else if loaded === undefined} +
Select a version to inspect its value.
+ {:else} +
+
+ {#if isCurrent} + This is the current value + {:else if currentValue === undefined} + + The current value could not be loaded, so there is no diff to show + + {:else} + + {#snippet children({ item })} + + + {/snippet} + + {/if} + +
+ + {#if loaded.missing.length > 0} +
+ + This version references {loaded.missing.join(', ')}. Restoring it leaves the resource + pointing at something unresolvable until you recreate it. + +
+ {/if} + +
+ {#if view === 'diff' && !isCurrent && currentValue !== undefined} + + {#await import('./DiffEditor.svelte')} + + {:then Module} + + {/await} + {:else} +
+ +
+ {/if} +
+
+ {/if} +
+ diff --git a/frontend/src/lib/components/S3FilePicker.svelte b/frontend/src/lib/components/S3FilePicker.svelte index 9f702a53c3..5006e2df0b 100644 --- a/frontend/src/lib/components/S3FilePicker.svelte +++ b/frontend/src/lib/components/S3FilePicker.svelte @@ -18,6 +18,11 @@ selectedFileKey?: { s3: string; storage?: string } | undefined folderOnly?: boolean regexFilter?: RegExp | undefined + /** + * Expand one folder level at a time. Turn off to keep listing every key up + * front, which is what lets `regexFilter` prune folders with no match. + */ + lazyFolders?: boolean /** Workspace to browse S3 storage in — the acting workspace of the editor that * opened the picker, else the nav workspace. */ workspace?: string | undefined @@ -33,6 +38,7 @@ selectedFileKey = $bindable(undefined), folderOnly = false, regexFilter = undefined, + lazyFolders = true, workspace = undefined, onClose, onSelectAndClose @@ -125,6 +131,7 @@ {s3ResourcePath} {folderOnly} {regexFilter} + {lazyFolders} workspace={effectiveWorkspace} /> {#snippet actions()} diff --git a/frontend/src/lib/components/S3FilePickerInner.svelte b/frontend/src/lib/components/S3FilePickerInner.svelte index 8c6ee879fa..5737744034 100644 --- a/frontend/src/lib/components/S3FilePickerInner.svelte +++ b/frontend/src/lib/components/S3FilePickerInner.svelte @@ -3,6 +3,7 @@ File as FileIcon, FolderClosed, FolderOpen, + ChevronDown, RotateCw, Loader2, Download, @@ -18,6 +19,8 @@ type DeleteS3FileData, type DeleteS3FileResponse, type ListStoredFilesData, + type ListStoredFilesPagedData, + type ListStoredFilesPagedResponse, type ListStoredFilesResponse, type LoadFileMetadataData, type LoadFileMetadataResponse, @@ -86,7 +89,16 @@ > allowDelete?: boolean replaceUnauthorizedWarning?: Snippet + /** + * Expand one folder level at a time instead of listing every key up front. + * Callers that override `listStoredFilesRequest` with a listing that has no + * paged counterpart (e.g. git repo files) must turn this off. + */ + lazyFolders?: boolean listStoredFilesRequest?: (d: ListStoredFilesData) => CancelablePromise + listStoredFilesPagedRequest?: ( + d: ListStoredFilesPagedData + ) => CancelablePromise loadFilePreviewRequest?: (d: LoadFilePreviewData) => CancelablePromise loadFileMetadataRequest?: ( d: LoadFileMetadataData @@ -115,7 +127,9 @@ allFilesByKey = $bindable({}), allowDelete = false, replaceUnauthorizedWarning, + lazyFolders = true, listStoredFilesRequest = HelpersService.listStoredFiles, + listStoredFilesPagedRequest = HelpersService.listStoredFilesPaged, loadFilePreviewRequest = HelpersService.loadFilePreview, loadFileMetadataRequest = HelpersService.loadFileMetadata, deleteS3FileRequest = HelpersService.deleteS3File, @@ -163,13 +177,26 @@ } | undefined = $state(undefined) + /** Identifies the metadata request that currently owns the preview pane. */ + let metadataRequestId = 0 + + /** + * Flat pagination cursor: `listMarkers[n]` is where page `n + 1` resumes, so `page` + * may only advance over a page that actually loaded. Running ahead of `listMarkers` + * sends no marker and silently replays the first page, and never recovers, because + * the `listMarkers.length == page` guard stops recording from then on. + */ let listMarkers: string[] let page = $state(0) const maxKeys = 1000 + /** Entries fetched per folder level before a "Load more" row appears. */ + const pageSize = 500 let count = $state(0) let displayedCount = $state(0) + /** Flat (non-lazy) listing: whether the last page came back full. */ + let flatHasMore = $state(false) let filter = $state('') @@ -187,17 +214,356 @@ } } + /** + * Marks the synthetic "Load more" row belonging to a folder. Appended to the + * folder's own prefix so the plain lexicographic sort that orders the tree also + * places the row last among that folder's children. + */ + const LOAD_MORE_SUFFIX = '￿' + + type FolderState = { nextPageToken?: string; loading: boolean; loaded: boolean } + let folderState: Record = $state({}) + + /** Per-level listing only makes sense while browsing; searching stays flat. */ + let lazyMode = $derived(lazyFolders && filter.trim() === '') + + function nestingLevelOf(key: string): number { + const slashes = (key.match(/\//g) ?? []).length + return (key.endsWith('/') ? slashes - 1 : slashes) * 2 + } + + function parentPathOf(key: string): string | undefined { + const body = key.endsWith('/') ? key.slice(0, -1) : key + const idx = body.lastIndexOf('/') + return idx === -1 ? undefined : body.slice(0, idx + 1) + } + + /** The parentPath value carried by entries sitting at the browsing root. */ + let rootParentPath = $derived(rootPath === '' ? undefined : rootPath) + + function addEntry(key: string, type: 'folder' | 'leaf', displayName: string) { + if (allFilesByKey[key] !== undefined) return + allFilesByKey[key] = { + type, + full_key: key, + display_name: displayName, + collapsed: true, + parentPath: parentPathOf(key), + nestingLevel: nestingLevelOf(key), + count: 0 + } + } + + /** + * A level shows only when every folder above it is expanded. The browsing root is + * spelled `rootPath` by `folderState` but `undefined` by an entry's `parentPath` + * (there is no parent entry), so both spellings must resolve here — otherwise the + * root's own "Load more" row is filtered out and the top level is stuck on one page. + */ + function isLevelVisible(prefix: string | undefined): boolean { + if (prefix === rootParentPath || prefix === rootPath) return true + if (prefix === undefined) return false + const info = allFilesByKey[prefix] + if (info === undefined || info.collapsed) return false + return isLevelVisible(info.parentPath) + } + + /** + * Every key whose ancestors are all expanded, in depth-first order — full keys + * sort that way because a child always starts with its parent's + * delimiter-terminated prefix. + */ + function computeVisibleKeys(): string[] { + const visible: string[] = [] + for (const key in allFilesByKey) { + if (!key.startsWith(rootPath)) continue + if (isLevelVisible(allFilesByKey[key].parentPath)) visible.push(key) + } + for (const prefix in folderState) { + if (folderState[prefix].nextPageToken && isLevelVisible(prefix)) { + visible.push(prefix + LOAD_MORE_SUFFIX) + } + } + return visible.sort() + } + + function refreshDisplayed() { + const visible = computeVisibleKeys() + displayedFileKeys = visible + displayedCount = visible.filter((k) => !k.endsWith(LOAD_MORE_SUFFIX)).length + } + + /** + * In-flight request per level. Callers that await `loadFolderPage` need the data + * to be there when it resolves; returning early on a concurrent load would hand + * them a resolved promise and no entries. + */ + let inFlightFolderLoads: Record> = {} + + /** + * Bumped when a single level is invalidated on its own (a delete refetches it from + * page one). Requests are keyed by prefix alone, so without this a pending "Load + * more" for that level would be joined by the refetch, which would then return + * believing page one had been fetched — leaving the level showing only its later + * pages until a full reload. + */ + let folderEpoch: Record = {} + + /** + * Bumped whenever the listing is thrown away (storage switch, filter change, + * reload). A request started before the bump belongs to the previous listing, so + * its response must not repopulate the cleared state — otherwise switching + * storage mid-load leaves the previous bucket's entries on screen. + */ + let listingGeneration = 0 + + async function loadFolderPage(prefix: string, append: boolean = false): Promise { + const generation = listingGeneration + const epoch = folderEpoch[prefix] ?? 0 + const pending = inFlightFolderLoads[prefix] + if (pending) { + await pending + // Only reuse the joined result if it belongs to the same listing *and* the + // same epoch of this level. + if (generation === listingGeneration && epoch === (folderEpoch[prefix] ?? 0) && !append) + return + } + const run = loadFolderPageInner(prefix, append) + inFlightFolderLoads[prefix] = run.catch(() => {}) + try { + await run + } finally { + delete inFlightFolderLoads[prefix] + } + } + + async function loadFolderPageInner(prefix: string, append: boolean) { + const generation = listingGeneration + const epoch = folderEpoch[prefix] ?? 0 + /** Whether this response still belongs to the listing and level that asked for it. */ + const stillCurrent = () => + generation === listingGeneration && epoch === (folderEpoch[prefix] ?? 0) + const current = folderState[prefix] ?? { loading: false, loaded: false } + if (append && current.nextPageToken === undefined) return + folderState[prefix] = { ...current, loading: true } + try { + const page = await listStoredFilesPagedRequest({ + workspace: ws!, + prefix, + maxKeys: pageSize, + pageToken: append ? current.nextPageToken : undefined, + storage, + s3ResourcePath + }) + // This listing, or just this level, has been thrown away since the request + // went out. Writing the entries back would resurrect the previous storage's + // contents, or reinstate a file that was just deleted. + if (!stillCurrent()) return + // Absent counts as restricted, matching `loadFlatFiles`: the two paths must + // not disagree on which way an omitted value falls. + if ( + page.restricted_access === null || + page.restricted_access === undefined || + page.restricted_access === true + ) { + fileListUnavailable = true + folderState[prefix] = { loading: false, loaded: true } + return + } + fileListUnavailable = false + for (const folder of page.folders ?? []) { + addEntry(folder.prefix, 'folder', folder.name) + } + for (const file of page.files ?? []) { + if (regexFilter && !regexFilter.test(file.key)) continue + addEntry(file.key, 'leaf', file.name) + } + folderState[prefix] = { + loading: false, + loaded: true, + // An exhausted level serializes as JSON `null`, which is not `undefined` + // — leaving it as-is makes "no more pages" look like another page and + // sends the level round again with no token. + nextPageToken: page.next_page_token ?? undefined + } + } catch (e) { + if (stillCurrent()) { + folderState[prefix] = { ...current, loading: false } + } + throw e + } finally { + // A response that lands after the user typed a filter must not rebuild the + // list: flat mode owns `displayedFileKeys` then, and rebuilding it under + // the lazy visibility rules would prune most of the search results. + if (lazyMode && stillCurrent()) { + refreshDisplayed() + } + } + } + + /** Surface listing failures instead of leaving a folder looking empty. */ + function reportFolderError(prefix: string, e: unknown) { + console.error('Error listing folder', prefix, e) + sendUserToast(`Could not list ${prefix || 'the bucket root'}`, true) + } + + function expandFolder(prefix: string) { + loadFolderPage(prefix).catch((e) => reportFolderError(prefix, e)) + } + + function loadMore(prefix: string) { + loadFolderPage(prefix, true).catch((e) => reportFolderError(prefix, e)) + } + + /** + * Bound on the pages fetched while hunting for one entry, so a preselected key + * that no longer exists cannot walk an entire bucket. + */ + const MAX_PAGES_WHILE_REVEALING = 20 + + /** + * Page through `parent` until `child` shows up. A single page is not enough: the + * target can sort past the first page of its own folder, and giving up there + * leaves the selection looking absent. + */ + async function revealChild(parent: string, child: string, generation: number) { + for (let fetched = 0; fetched < MAX_PAGES_WHILE_REVEALING; fetched++) { + if (allFilesByKey[child] !== undefined) return + const state = folderState[parent] + if (state === undefined || !state.loaded) { + await loadFolderPage(parent) + } else if (state.nextPageToken !== undefined) { + await loadFolderPage(parent, true) + } else { + return + } + if (generation !== listingGeneration) return + } + } + + /** + * Reveal a key by loading each folder above it in turn — with per-level listing + * an ancestor's children are not known until that level is fetched. + * + * `generation` is the listing this walk belongs to. A reveal is a chain of round + * trips, so the check has to be repeated after every one of them: a filter typed + * mid-walk switches the picker to the search, and the loads that follow would date + * themselves to that listing and graft browse results onto it. + */ + async function expandToKey(key: string, generation: number) { + if (!key.startsWith(rootPath)) return + const rest = key.slice(rootPath.length).split('/') + let prefix = rootPath + for (let i = 0; i < rest.length - 1; i++) { + const child = prefix + rest[i] + '/' + await revealChild(prefix, child, generation) + if (generation !== listingGeneration) return + prefix = child + const info = allFilesByKey[prefix] + // The folder is genuinely absent; deeper levels cannot exist either. + if (info === undefined) break + info.collapsed = false + } + if (allFilesByKey[key] === undefined) { + await revealChild(prefix, key, generation) + if (generation !== listingGeneration) return + } + refreshDisplayed() + } + let lastKeyFolders: string[] = $state([]) - async function loadFiles() { + + /** Reports whether the listing completed, so a caller that moved the cursor can undo it. */ + async function loadFiles(): Promise { + if (lazyMode) { + // Typing in the filter switches the picker to the flat listing while this is in + // flight, and `loadFolderPage` resolves rather than throwing once superseded. + // What follows belongs to the listing that started it: expanding a preselected + // file would graft browse results into the search, and clearing the flags would + // retire the search's spinner. + const generation = listingGeneration + fileListLoading = true + try { + await loadFolderPage(rootPath) + if ( + generation === listingGeneration && + selectedFileKey !== undefined && + !emptyString(selectedFileKey.s3) + ) { + await expandToKey(selectedFileKey.s3, generation) + } + } catch (e) { + // `reloadContent` is called un-awaited from `open()`, so without this a + // failing root listing surfaces as an unhandled rejection and an empty + // tree indistinguishable from an empty bucket. + reportFolderError(rootPath, e) + return false + } finally { + if (generation === listingGeneration) { + fileListLoading = false + fileInfoLoading = false + } + } + return true + } + // Same contract as the lazy branch above: every caller here is un-awaited, so an + // uncaught rejection would leave the drawer on a spinner with nothing said. The + // generation guard keeps a superseded listing from clearing the spinner of the one + // that replaced it — `loadFlatFiles` returns early rather than throwing in that case. + const generation = listingGeneration + fileListLoading = true + try { + await loadFlatFiles() + } catch (e) { + console.error('Error listing files', e) + sendUserToast('Could not list the files', true) + // Nothing will load a preview now, so the right-hand pane has to stop waiting + // too. On the other exits it is owned by whoever is still fetching metadata. + if (generation === listingGeneration) { + fileInfoLoading = false + } + return false + } finally { + if (generation === listingGeneration) { + fileListLoading = false + } + } + return true + } + + /** Flat "Load more" / "Keep looking": a page that fails has to give the cursor back. */ + async function loadNextFlatPage() { + // The page number alone does not identify our advance: a filter or storage change + // resets the cursor, and the replacement listing can reach the same number before + // this request fails. Rolling that one back would strand *its* cursor instead. + const generation = listingGeneration + page += 1 + const requested = page + const ok = await loadFiles() + // Only undo our own advance: another click may have moved it on meanwhile. + if (!ok && generation === listingGeneration && page === requested) { + page = requested - 1 + } + } + + async function loadFlatFiles() { + // Debounced searches overlap: an older response must not add its keys to a newer + // search's results or overwrite its pagination state. + const generation = listingGeneration fileListLoading = true let availableFiles = await listStoredFilesRequest({ workspace: ws!, maxKeys: maxKeys, // fixed pages of 1000 files for now marker: page == 0 ? undefined : listMarkers[page - 1], - prefix: rootPath ?? (filter.trim() != '' ? filter : undefined), + // `prefix` is evaluated per path *segment* by the storage layer, so sending the + // query there matched only whole folder names. `search` matches the raw key + // prefix instead, which is what the box means. + prefix: rootPath !== '' ? rootPath : undefined, + search: filter.trim() !== '' ? filter.trim() : undefined, storage: storage, s3ResourcePath }) + if (generation !== listingGeneration) return if ( availableFiles.restricted_access === null || availableFiles.restricted_access === undefined || @@ -212,7 +578,12 @@ if (regexFilter && !regexFilter.test(file_path.s3)) { continue } - displayedCount += 1 + // Only count keys not already in the tree: a page can legitimately repeat + // entries, and counting them again makes the total climb while the list + // stays put. + if (allFilesByKey[file_path.s3] === undefined) { + displayedCount += 1 + } let split_path = file_path.s3.split('/') let parent_path: string | undefined = undefined let current_path: string | undefined = undefined @@ -249,9 +620,19 @@ } } } + // A short page means the listing is exhausted. Deriving "there is more" from + // `count % maxKeys` instead would keep offering another page whenever the total + // happens to be an exact multiple of the page size. + // Searching returns an explicit cursor because it skips over keys that did not + // match, so the last *returned* key is not where the next page resumes. + const serverMarker = availableFiles.next_marker ?? undefined + flatHasMore = + serverMarker !== undefined || + (filter.trim() === '' && availableFiles.windmill_large_files.length === maxKeys) if (listMarkers.length == page) { count += availableFiles.windmill_large_files.length const nextMarker = + serverMarker ?? availableFiles.windmill_large_files?.[availableFiles.windmill_large_files.length - 1]?.s3 if (nextMarker) listMarkers.push(nextMarker) } @@ -277,7 +658,12 @@ } } } - displayedFileKeys = [...new Set(displayedFileKeys)].sort() + // The loop above only lists entries at the browsing root, so a later page's + // keys land in `allFilesByKey` without ever being displayed — the folder they + // belong to is already expanded and nothing re-scans it. Recomputing from the + // expansion state picks them up without needing a collapse/expand round trip. + // `displayedCount` is left alone: in this mode it counts files loaded, not rows shown. + displayedFileKeys = computeVisibleKeys() fileListLoading = false fileInfoLoading = false } @@ -287,14 +673,36 @@ fileInfoLoading = false return } + // The pane belongs to the newest request, not to a key: switching storage reloads + // the same key, so comparing keys would let an older request speak for a newer one. + const requestId = ++metadataRequestId fileInfoLoading = true - let fileMetadataRaw = await loadFileMetadataRequest({ - workspace: ws!, - fileKey: fileKey, - storage: storage, - s3ResourcePath - }) + let fileMetadataRaw: LoadFileMetadataResponse + try { + fileMetadataRaw = await loadFileMetadataRequest({ + workspace: ws!, + fileKey: fileKey, + storage: storage, + s3ResourcePath + }) + } catch (e) { + // Every caller invokes this un-awaited, so a key that no longer exists would + // otherwise leave the preview pane on "Loading..." forever. + console.error('Error loading metadata for', fileKey, e) + // Unless a later request has taken the pane over: it will report its own + // outcome, including the loading flag, and blanking here would undo it. + if (requestId !== metadataRequestId) { + return + } + fileMetadata = undefined + filePreview = undefined + fileInfoLoading = false + return + } + if (requestId !== metadataRequestId) { + return + } if (fileMetadataRaw !== undefined) { fileMetadata = { fileKey: fileKey, @@ -305,10 +713,15 @@ } } // async call - loadFilePreview(fileKey, fileMetadataRaw.size_in_bytes, fileMetadataRaw.mime_type) + loadFilePreview(fileKey, requestId, fileMetadataRaw.size_in_bytes, fileMetadataRaw.mime_type) } - async function loadFilePreview(fileKey: string, fileSizeInBytes?: number, fileMimeType?: string) { + async function loadFilePreview( + fileKey: string, + requestId: number, + fileSizeInBytes?: number, + fileMimeType?: string + ) { let filePreviewRaw = await loadFilePreviewRequest({ workspace: ws!, fileKey: fileKey, @@ -333,6 +746,9 @@ '\n\n ... FILE CONTENT TRUNCATED ...\n\n' } + if (requestId !== metadataRequestId) { + return + } if (filePreviewRaw !== undefined) { filePreview = { fileKey: fileKey, @@ -371,11 +787,45 @@ } sendUserToast(`${fileKey} deleted from S3 bucket`) selectedFileKey = { s3: '', storage } + // The preview pane and its toolbar render from these rather than from the + // selection, so the deleted file stays previewed — and offers to download, move + // and delete itself — until they are cleared too. + fileMetadata = undefined + filePreview = undefined + // A metadata load still in flight belongs to the file just deleted; retire it, or + // its response repopulates the pane it was cleared from. Retiring it also means + // nobody is left to report its outcome, so the pane's loading flag is ours. + metadataRequestId += 1 + fileInfoLoading = false + if (lazyMode) { + // Only the level the file lived in changed; refetch it from its first page + // and keep the rest of the expanded tree. Its already-fetched entries have + // to go with the cursor, or the reset cursor would hand out a "Load more" + // for pages that are still displayed. + const parent = parentPathOf(fileKey) ?? rootPath + for (const key of Object.keys(allFilesByKey)) { + if (allFilesByKey[key].parentPath === (parent === '' ? undefined : parent)) { + delete allFilesByKey[key] + } + } + delete folderState[parent] + // Invalidate this level so a pending "Load more" for it is not joined below. + folderEpoch[parent] = (folderEpoch[parent] ?? 0) + 1 + delete inFlightFolderLoads[parent] + await loadFolderPage(parent).catch((e) => reportFolderError(parent, e)) + return + } const currentPage = page - await clearAndLoadFiles() - for (let i = 0; i < currentPage; i++) { - page = i + 1 - await loadFiles() + // Every page here has to land, starting with the fresh first one. + if (await clearAndLoadFiles()) { + for (let i = 0; i < currentPage; i++) { + page = i + 1 + if (!(await loadFiles())) { + // Stop at the last page that actually loaded. + page = i + break + } + } } const fileKeyFolders = fileKey.split('/').slice(0, -1) let current_path: string | undefined = undefined @@ -399,11 +849,20 @@ displayedFileKeys = [...new Set(displayedFileKeys)].sort() } - async function clearAndLoadFiles({ keepFilter }: { keepFilter?: boolean } = {}) { + /** Reports whether the fresh listing loaded, so a caller replaying pages can stop. */ + async function clearAndLoadFiles({ + keepFilter + }: { keepFilter?: boolean } = {}): Promise { + // Anything already in flight belongs to the listing being discarded. + listingGeneration += 1 + inFlightFolderLoads = {} + folderEpoch = {} displayedFileKeys = [] allFilesByKey = {} + folderState = {} count = 0 displayedCount = 0 + flatHasMore = false page = 0 listMarkers = [] fileMetadata = undefined @@ -411,7 +870,7 @@ if (!keepFilter) { filter = '' } - await loadFiles() + return await loadFiles() } async function moveS3File(srcFileKey: string | undefined, destFileKey: string | undefined) { @@ -483,9 +942,18 @@ } await clearAndLoadFiles() if (selectedFileKey !== undefined) { - if (allFilesByKey[selectedFileKey.s3] === undefined) { + const entry = allFilesByKey[selectedFileKey.s3] + if (entry !== undefined) { + if (entry.type !== 'folder') { + loadFileMetadataPlusPreviewAsync(selectedFileKey.s3) + } + } else if (!lazyMode) { + // Flat mode has listed everything it is going to, so a missing key really + // is missing. Per-level listing has not: only the levels on the way to + // the key were fetched, and `selectedFileKey` is bound out to the caller + // — blanking it there would silently clear the configured object. selectedFileKey = { s3: '', storage } - } else if (allFilesByKey[selectedFileKey.s3].type !== 'folder') { + } else { loadFileMetadataPlusPreviewAsync(selectedFileKey.s3) } } @@ -506,6 +974,27 @@ function selectItem(index: number, toggleCollapsed: boolean = true) { let item_key = displayedFileKeys[index] let item = allFilesByKey[item_key] + if (item === undefined) return + if (lazyMode) { + if (item.type === 'folder') { + if (folderOnly) { + selectedFileKey = { s3: item_key, storage } + } + if (toggleCollapsed) { + item.collapsed = !item.collapsed + } + if (!item.collapsed && !folderState[item_key]?.loaded) { + // Children are unknown until this level is fetched. + expandFolder(item_key) + } else { + refreshDisplayed() + } + } else { + selectedFileKey = { s3: item_key, storage } + loadFileMetadataPlusPreviewAsync(selectedFileKey.s3) + } + return + } if (item.type === 'folder') { if (folderOnly) { selectedFileKey = { @@ -612,7 +1101,12 @@
{#if !rootPath}
- +
{/if} {#if displayedFileKeys.length === 0} @@ -624,8 +1118,22 @@
{:else}
- No files in the workspace S3 bucket at that prefix + {#if filter.trim() !== ''} + No files starting with "{filter.trim()}" + {:else} + No files in the workspace S3 bucket + {/if}
+ {#if flatHasMore} + +
+ +
+ {/if} {/if} {:else}
@@ -638,7 +1146,12 @@ {#snippet header()}{/snippet} {#snippet footer()}{/snippet} {#snippet item({ index, style })} - {@const file_info = allFilesByKey[displayedFileKeys[index]]} + + {@const item_key = displayedFileKeys[index] ?? ''} + {@const is_load_more = item_key.endsWith(LOAD_MORE_SUFFIX)} + {@const load_more_prefix = is_load_more ? item_key.slice(0, -1) : ''} + {@const file_info = allFilesByKey[item_key]}
- {#if file_info} + {#if is_load_more} + + {@const loadMoreNesting = + nestingLevelOf(load_more_prefix + 'x') - 2 * rootPathNestingLevel} + {@const loadingMore = folderState[load_more_prefix]?.loading === true} + + +
!loadingMore && loadMore(load_more_prefix)} + class={twMerge( + 'flex flex-row h-full text-xs items-center justify-start text-secondary', + loadingMore ? 'cursor-default' : 'cursor-pointer' + )} + > +
+ +
+ {#if loadingMore} + + {:else} + + {/if} +
+
Load more
+
+
+ {:else if file_info} {@const nestingLevel = file_info.nestingLevel - 2 * rootPathNestingLevel} @@ -665,14 +1209,23 @@ style={`margin-left: ${(2 + nestingLevel) * 0.25}rem;`} > {#if file_info.type === 'folder'} - {#if file_info.collapsed}{:else} + {:else if file_info.collapsed}{:else}{/if}
- {file_info.display_name} ({file_info.count}{count % 1000 === 0 && - lastKeyFolders[file_info.nestingLevel / 2] === file_info.display_name - ? '+' - : ''} item{file_info.count === 1 ? '' : 's'}) + + {#if file_info.display_name === ''} + (empty name) + {:else}{file_info.display_name}{/if} + {#if !lazyMode} + ({file_info.count}{count % 1000 === 0 && + lastKeyFolders[file_info.nestingLevel / 2] === file_info.display_name + ? '+' + : ''} item{file_info.count === 1 ? '' : 's'}) + {/if}
{:else} @@ -688,27 +1241,24 @@
{#if fileListLoading === true}
Loading content
+ {:else if lazyMode} + +
{displayedCount} item{displayedCount === 1 ? '' : 's'} shown
{:else}
- {displayedCount}{count % maxKeys === 0 ? '+' : ''} + {displayedCount}{flatHasMore ? '+' : ''} {displayedCount !== count ? 'filtered ' : ''}items (including inside folders)
- {#if count % maxKeys === 0} - {/if} diff --git a/frontend/src/lib/components/SchemaFormWithArgPicker.svelte b/frontend/src/lib/components/SchemaFormWithArgPicker.svelte index c4275f48b0..4879579de7 100644 --- a/frontend/src/lib/components/SchemaFormWithArgPicker.svelte +++ b/frontend/src/lib/components/SchemaFormWithArgPicker.svelte @@ -196,7 +196,10 @@
diff --git a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte index f9d5dbf777..2502cd075b 100644 --- a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte +++ b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte @@ -889,7 +889,7 @@ white-space: pre-wrap; word-break: break-words; width: 100%; - min-height: 2.25rem; + min-height: 1.9rem; } /* Hide the textarea's own glyphs (the highlight overlay renders the text) diff --git a/frontend/src/lib/components/copilot/chat/ContextUsageIndicator.svelte b/frontend/src/lib/components/copilot/chat/ContextUsageIndicator.svelte index 80b317ba12..a307fad255 100644 --- a/frontend/src/lib/components/copilot/chat/ContextUsageIndicator.svelte +++ b/frontend/src/lib/components/copilot/chat/ContextUsageIndicator.svelte @@ -1,6 +1,6 @@ + + + {#snippet trigger()} + + v{shown} + + + {/snippet} + + {#snippet content()} +
+ +
+ {#if versions.length > 0 && versions.length < latest} + {versions.length} most recent of {latest} versions + {:else} + Versions + {/if} +
+
+ {#each versions as v (v.version)} + + {/each} +
+
+ {/snippet} +
diff --git a/frontend/src/lib/components/copilot/chat/artifacts/ArtifactViewer.svelte b/frontend/src/lib/components/copilot/chat/artifacts/ArtifactViewer.svelte index 2aadcca63d..1ce7fe1d35 100644 --- a/frontend/src/lib/components/copilot/chat/artifacts/ArtifactViewer.svelte +++ b/frontend/src/lib/components/copilot/chat/artifacts/ArtifactViewer.svelte @@ -9,14 +9,68 @@ import { copyToClipboard, download } from '$lib/utils' import CodeDisplay from '../script/CodeDisplay.svelte' import LinkRenderer from '../LinkRenderer.svelte' - import { artifactFilename, artifactMimeType, type PersistedArtifact } from './artifactsDB' + import { + artifactFilename, + artifactMimeType, + currentVersion, + type ArtifactVersion, + type PersistedArtifact + } from './artifactsDB' import { markdownProse } from '$lib/components/markdownProse' + import ArtifactVersionPicker from './ArtifactVersionPicker.svelte' + import type { SessionArtifactsStore } from './artifactsState.svelte' + import { History } from 'lucide-svelte' + import TimeAgo from '$lib/components/TimeAgo.svelte' interface Props { artifact: PersistedArtifact + store: SessionArtifactsStore } - let { artifact }: Props = $props() + let { artifact, store }: Props = $props() + + const latest = $derived(currentVersion(artifact)) + // Nothing to pick between until a second version exists. + const hasHistory = $derived(latest > 1) + + // undefined = following the current version. An explicit pick survives later edits, so + // the AI writing v8 does not yank the reader out of the v3 they chose to read. + let pinned = $state(undefined) + let pinnedContent = $state(undefined) + + // The store hands us a fresh object on every edit, so this effect reruns constantly. + // Compare the id against the last one seen: clearing on every rerun would drop the + // reader's pin the moment the assistant writes a new version. + let pinnedFor: string | undefined + $effect(() => { + if (artifact.id === pinnedFor) return + pinnedFor = artifact.id + pinned = undefined + }) + + $effect(() => { + const version = pinned + if (version === undefined) { + pinnedContent = undefined + return + } + const id = artifact.id + void store.getVersion(id, version).then((snapshot) => { + if (pinned !== version || artifact.id !== id) return + // Pruned out from under the pin (history is capped): fall back to current rather + // than showing an empty document. + if (!snapshot) { + pinned = undefined + return + } + pinnedContent = snapshot + }) + }) + + const shown = $derived(pinnedContent ?? artifact) + // Label from the snapshot that is rendered, not from the one just requested: the read is + // async, so `pinned` names a version the body has not swapped to yet. + const shownVersion = $derived(pinnedContent?.version) // Markdown is the only rendered kind in v1; anything else shows source only. const canPreview = $derived(artifact.kind === 'md') @@ -25,12 +79,16 @@ let copied = $state(false) async function copyRaw() { - if (!(await copyToClipboard(artifact.content))) return + if (!(await copyToClipboard(shown.content))) return copied = true setTimeout(() => (copied = false), 1500) } function downloadFile() { - download(artifactFilename(artifact), artifact.content, artifactMimeType(artifact.kind)) + download( + artifactFilename({ name: shown.name, kind: artifact.kind }), + shown.content, + artifactMimeType(artifact.kind) + ) } const plugins = [gfmPlugin(), { renderer: { pre: CodeDisplay, a: LinkRenderer } }] @@ -40,9 +98,17 @@
- - {artifact.name} + + {shown.name} + {#if hasHistory} + (pinned = v)} + /> + {/if}
@@ -78,11 +144,31 @@
+ {#if pinnedContent} + +
+ + + + Viewing v{shownVersion} of {latest} · saved + ago + +
+ +
+
+ {/if} +
{#if source} - {#key `${artifact.id}:${artifact.updatedAt}`} - + {#key `${artifact.id}:${pinnedContent ? `v${pinnedContent.version}` : artifact.updatedAt}`} + {/key} {:else} + + {descriptorError ? `descriptor: ${descriptorError}` : 'descriptor is not valid'} + + {/if} +
+ {#if testIsLoading} + + {:else} +
+ + + + + {#snippet trigger()} + + {#if argsOverridden} + + {/if} + {/snippet} + {#snippet content()} + {#if schema} + + {:else} +

This descriptor takes no arguments.

+ {/if} + {/snippet} +
+
+ {/if} +
+
+ + + + +
+ + {#snippet addFile()} + + {#snippet trigger()} +
+ +
+ {/snippet} + {#snippet content({ close })} +
+ + (newFileError = dbtPathError(newFile, modules ?? undefined))} + onkeydown={(e) => { + if (e.key === 'Enter') createFile() + if (e.key === 'Escape') close() + }} + /> + {#if newFileError} +

{newFileError}

+ {/if} +

+ Anywhere in the project, e.g. macros/cents.sql. + {DBT_MODULE_EXTENSIONS.join(', ')} +

+
+ + +
+
+ {/snippet} +
+ {/snippet} +
+
+ {#key fileLang} + { + if (openFile === null) { + code = editorCode + lastSyncedCode = code + inferSchema(editorCode) + } else { + flushOpenFile() + } + }} + on:saveDraft + cmdEnterAction={async () => { + if (openFile === null) await inferSchema(editorCode) + runTest() + }} + formatAction={async () => { + if (openFile === null) await inferSchema(editorCode) + dispatch('format') + }} + class="flex flex-1 h-full !overflow-visible" + scriptLang={fileLang} + automaticLayout={true} + fixedOverflowWidgets={true} + {args} + customTag={tag} + /> + {/key} +
+
+
+ + + + + { + graphSelection = sel + selectedDbt = dbt + selectedBuffer = buffer + }} + /> + + + + {#if selectedAsset && selectedDbt} + (graphSelection = undefined)} + /> + {/if} +
+ +
+
+
+
+
+
diff --git a/frontend/src/lib/components/dbt/DbtModelDetails.svelte b/frontend/src/lib/components/dbt/DbtModelDetails.svelte new file mode 100644 index 0000000000..2d25dbdc00 --- /dev/null +++ b/frontend/src/lib/components/dbt/DbtModelDetails.svelte @@ -0,0 +1,323 @@ + + +
+
+ {assetPath.split('/').pop()} + {#if dbt.materialized} + {dbt.materialized} + {/if} + {#if dbt.resource_type !== 'model'} + {dbt.resource_type} + {/if} + {#each dbt.tags ?? [] as t (t)} + {t} + {/each} +
+ {#if previewable} + {#if showRows && preview && !('error' in preview)} + + {:else} + + {/if} + {/if} + {#if fileInBundle && dbt.original_file_path && onOpenFile} + + {/if} + +
+
+ + {#if stalePlaceholders} +
+ The run arguments have changed since this graph was parsed, so these rows need not + describe the models on screen — arguments reach schemas, aliases and which models exist + at all. Refresh the models to draw and preview them under the current ones. +
+ {:else if staleVars} +
+ The run form's vars have changed since this graph was parsed. Rows are previewed under + the vars it was parsed with, so they still describe the models on screen — refresh the + models to draw and preview them under the current ones. +
+ {/if} + +
+
+
+ relation + {assetPath} +
+ {#if dbt.original_file_path} +
+ file + {dbt.original_file_path} +
+ {/if} + {#if dbt.description} +
+ description + {dbt.description} +
+ {/if} +
+ + {#if columns.length > 0 || (dbt.data_tests?.length ?? 0) > 0} +
+ {#if columns.length > 0} +
+
columns declared
+
+ {#each columns as [name, desc] (name)} +
+ {name} + {desc} +
+ {/each} +
+ +
+ Declared metadata — dbt reports no column-level lineage. +
+
+ {/if} + {#if (dbt.data_tests?.length ?? 0) > 0} +
+
tests
+
+ {#each dbt.data_tests ?? [] as t, i (i)} + + {t.kind}{t.column ? ` · ${t.column}` : ''} + + {/each} +
+
+ {/if} +
+ {/if} + + {#if showRows && preview} + {#if 'error' in preview} +
{preview.error}
+ {#if dbt.raw_code} + + {/if} + {:else if 'pending' in preview} +
+ + Running `dbt show` — this is a job, so it waits on a worker and the engine. +
+ {:else} + {@const cols = Object.keys(preview.rows[0] ?? {})} + {#if cols.length === 0} +
The model returned no rows.
+ {:else} + + + + {#each cols as c (c)} + + {/each} + + + + {#each preview.rows as row, i (i)} + + {#each cols as c (c)} + + {/each} + + {/each} + +
{c}
{cellText(row[c])}
+
+ {preview.rows.length} rows in {(preview.tookMs / 1000).toFixed(1)}s{preview.node + ? ` · ${preview.node}` + : ''} +
+ {/if} + {/if} + {:else if dbt.raw_code} + + {:else} +
+ {dbt.resource_type === 'source' + ? 'A source is declared rather than built, so it has no transform of its own.' + : 'No SQL stored for this node.'} +
+ {/if} +
+
diff --git a/frontend/src/lib/components/dbt/DbtModelGraph.svelte b/frontend/src/lib/components/dbt/DbtModelGraph.svelte new file mode 100644 index 0000000000..e6d948d926 --- /dev/null +++ b/frontend/src/lib/components/dbt/DbtModelGraph.svelte @@ -0,0 +1,504 @@ + + +
+
+ + {provenance} + + {#if modelCount > 0} + · {modelCount} models + {/if} +
+ +
+
+ + {#if refreshPending} +
+ Still parsing. A cold worker provisions the dbt engine before it starts; a project + pinned to a worker tag nothing serves waits here indefinitely. + Open the parse job +
+ {/if} + + {#if refreshError} +
+
The parse failed
+
{refreshError.message}
+ {#if refreshError.job} + Open the parse job + {/if} +
+ {/if} + + {#if loading} +
+ Loading the model graph +
+ {:else if failed} +
Could not load the model graph.
+ {:else if !graph} +
+ {#if refreshJob || deployedHash != undefined} + + This project has no models in the asset graph. A project that brings its own + profiles.yml without naming a + profile.warehouse has no warehouse identity to key + dbt:// assets on, so its relations cannot be drawn. + + {:else} + + Nothing has parsed this project yet. Refresh models runs + dbt parse over the files as they are here and draws what dbt + reports — deploying does the same. + + + It needs a warehouse it can reach: the profile is rendered before dbt runs, so a project + whose profile.warehouse is not configured under Settings → dbt + fails the parse the way it would fail a run. + + {/if} +
+ {:else} +
+ +
+ {/if} +
diff --git a/frontend/src/lib/components/dbt/DbtProjectPanel.svelte b/frontend/src/lib/components/dbt/DbtProjectPanel.svelte new file mode 100644 index 0000000000..89b275aab3 --- /dev/null +++ b/frontend/src/lib/components/dbt/DbtProjectPanel.svelte @@ -0,0 +1,162 @@ + + +{#snippet branch(nodes: Node[], depth: number)} + {#each nodes as node (node.name + node.path + depth)} + {@const key = `${depth}:${node.name}:${node.path}`} + {#if node.children.length > 0} + + {#if !collapsed[key]} + {@render branch(node.children, depth + 1)} + {/if} + {:else} +
+ + + {#if onDelete && node.path !== projectFileKey} + + {/if} +
+ {/if} + {/each} +{/snippet} + +
+
+ {scriptPath}__dbt/ +
+ {fileCount + 1} + {@render addFile?.()} +
+
+
+ + + {@render branch(tree, 0)} +
+ {#if fileCount === 0} +
+ No project yet. Copy one in and push it: +
cp -r my-dbt-project/. {scriptPath}__dbt/
+wmill sync push
+
+ {/if} +
diff --git a/frontend/src/lib/components/dbt/DbtRunGraph.svelte b/frontend/src/lib/components/dbt/DbtRunGraph.svelte new file mode 100644 index 0000000000..cffb663b44 --- /dev/null +++ b/frontend/src/lib/components/dbt/DbtRunGraph.svelte @@ -0,0 +1,774 @@ + + +{#snippet sqlPane()} + {#if selectedIsForeign} +
+ Another dbt project in this workspace also materializes this relation, and the graph keeps one + project's model per relation — so the SQL shown here would not be this run's. Open that + project's own run to see it. +
+ {:else if selectedDbt?.raw_code} +
+
+ {selectedDbt.original_file_path ?? selectedDbt.unique_id} + {#if selectedDbt.materialized} + {selectedDbt.materialized} + {/if} + + {#if selectedDbt.resource_type === 'model'} + {#if showRows && preview && !('error' in preview)} + + {:else} + + {/if} + {/if} + {#if selectedRelation} + + {/if} + + + read-only · edit in the script + +
+
+ {#if showRows && preview} + {#if 'error' in preview} +
{preview.error}
+ + {:else if 'pending' in preview} +
+ + Running `dbt show` — this is a job, so it waits on a worker and the engine. +
+ {:else} + {@const cols = Object.keys(preview.rows[0] ?? {})} + {#if cols.length === 0} +
The model returned no rows.
+ {:else} + + + + {#each cols as c (c)} + + {/each} + + + + {#each preview.rows as row, i (i)} + + {#each cols as c (c)} + + {/each} + + {/each} + +
{c}
{cellText(row[c])}
+
+ {preview.rows.length} rows in {(preview.tookMs / 1000).toFixed(1)}s{preview.node + ? ` · ${preview.node}` + : ''} +
+ {/if} + {/if} + {:else} + + {/if} +
+
+ {/if} +{/snippet} + +{#if resumable} +
+ {(run?.totals?.error ?? 0) > 0 ? `${run?.totals?.error} failed` : ''}{(run?.totals?.error ?? + 0) > 0 && (run?.totals?.skipped ?? 0) > 0 + ? ', ' + : ''}{(run?.totals?.skipped ?? 0) > 0 ? `${run?.totals?.skipped} skipped` : ''}. Rebuild only + those with dbt retry, instead of the whole project. + + + +
+{/if} + +{#if loading} +
+ Loading the model graph +
+{:else if failed} +
Could not load the model graph.
+{:else if !graph} +
+ {#if ranTestsOnly} + This run selected tests alone, so it built no models. A dbt test is an assertion rather than a + relation, so it has no node here — the models it asserts against belong to the runs that build + them. Its results are in the table below. + {:else} + + No models stored for this version of the project. + {#if scriptHash} + A version's graph is written by its deploy, so this is what a deploy still in + flight looks like — it fills in when that job lands. + {/if} + A project that brings its own profiles.yml without naming + a profile.warehouse also has no warehouse identity to key + models on, and stores none. + {/if} +
+{:else} +
+ {#if relationDrift > 0} +
+ {relationDrift} + {relationDrift === 1 ? 'model has' : 'models have'} been renamed or moved since this run — + {relationDrift === 1 ? 'its node shows' : 'their nodes show'} today's relation, not the one this + run wrote. +
+ {/if} + {#if goneSinceRun > 0} +
+ {goneSinceRun} + {goneSinceRun === 1 ? 'model' : 'models'} this run built {goneSinceRun === 1 ? 'is' : 'are'} + no longer in the project — renamed or removed since, so + {goneSinceRun === 1 ? 'it is' : 'they are'} not drawn. +
+ {/if} +
+ (selection = s)} + showMinimap={false} + scrollZoom={false} + /> +
+ {@render sqlPane()} +
+{/if} diff --git a/frontend/src/lib/components/dbt/DbtRunResult.svelte b/frontend/src/lib/components/dbt/DbtRunResult.svelte new file mode 100644 index 0000000000..34d7886b30 --- /dev/null +++ b/frontend/src/lib/components/dbt/DbtRunResult.svelte @@ -0,0 +1,146 @@ + + +
+
+ {#each [{ k: 'success', label: 'passed', cls: 'text-green-600 dark:text-green-400' }, { k: 'warn', label: 'warned', cls: 'text-yellow-600 dark:text-yellow-400' }, { k: 'error', label: 'failed', cls: 'text-red-600 dark:text-red-400' }, { k: 'skipped', label: 'skipped', cls: 'text-secondary' }] as t (t.k)} + {@const n = (totals as Record)[t.k] ?? 0} + {#if n > 0} + {n} {t.label} + {/if} + {/each} + of {totals.total ?? nodes.length} nodes + + {run.command ?? 'build'} · {run.engine ?? ''} + {run.engine_version ?? ''} + +
+ + {#if nodes.length > 0} +
+ + + + + + + + + + + + {#each nodes as node (node.unique_id)} + {@const s = split(node.unique_id)} + {@const r = rank(node.status, node.outcome)} + + + + + + + + {/each} + +
NodeKindRelationRowsTime
+
+ + {#if r === 0} + + {:else if r === 1} + + {:else if r === 2} + + {:else} + + {/if} + + {s.name} + {#if node.message && r < 2} + {node.message} + {/if} +
+ {#if node.message && r < 2} +
+ {node.message} +
+ {/if} +
+ + {#if s.kind === 'test'} + + {/if} + {s.kind} + + + {fmtRelation(node.relation_name) ?? ''} + + {node.rows_affected ?? ''} + + {fmtTime(node.execution_time)} +
+
+ {#if hasTests} +
+ A test's severity decides the outcome: dbt's own warn surfaces + without failing the job. +
+ {/if} + {/if} +
diff --git a/frontend/src/lib/components/dbt/parseDbtRun.test.ts b/frontend/src/lib/components/dbt/parseDbtRun.test.ts new file mode 100644 index 0000000000..47d6073ef7 --- /dev/null +++ b/frontend/src/lib/components/dbt/parseDbtRun.test.ts @@ -0,0 +1,193 @@ +import { describe, it, expect } from 'vitest' +import { + parseDbtRun, + relationOutcome, + splitRelation, + statusRank, + splitUniqueId, + nodeSelector +} from './parseDbtRun' + +const run = { + engine: 'dbt-core-1x', + engine_version: '1.12.0', + command: 'build', + totals: { total: 1, success: 1, error: 0, warn: 0, skipped: 0 }, + nodes: [{ unique_id: 'model.p.customers', status: 'success' }] +} + +describe('parseDbtRun', () => { + it('takes a successful run as-is', () => { + expect(parseDbtRun(run)?.engine).toBe('dbt-core-1x') + }) + + // The worker puts the same JSON in the error message after the exit-status + // line, and this is the case worth rendering: the failing node is what the + // user came for. + it('recovers the run from a failed job’s error message', () => { + const failed = { + error: { + name: 'ExecutionErr', + message: `execution error:\nNon-zero exit status for dbt build: 1\n\n${JSON.stringify(run)}` + } + } + expect(parseDbtRun(failed)?.totals?.total).toBe(1) + }) + + // The failures worth reading are the ones whose message carries braces of its + // own — a Jinja template, the compiled SQL, an adapter's own JSON — and the + // payload is appended after all of it. + it('finds the run past braces in the error text', () => { + const failed = { + error: { + name: 'ExecutionErr', + message: + 'execution error:\nCompilation Error in model x\n {{ ref("missing") }} depends on {"a": 1}\n' + + `Non-zero exit status for dbt build: 1\n\n${JSON.stringify(run)}` + } + } + expect(parseDbtRun(failed)?.totals?.total).toBe(1) + }) + + // `{nodes, totals}` alone is a shape an ordinary script can return, and it + // would then be rendered as somebody's dbt run. + it('does not claim an ordinary result that happens to have nodes and totals', () => { + expect(parseDbtRun({ nodes: [], totals: {} })).toBeUndefined() + expect(parseDbtRun({ engine: 'v8', nodes: [], totals: {} })).toBeUndefined() + }) + + it('accepts every engine the worker stamps', () => { + for (const engine of ['dbt-core-1x', 'dbt-core-2x', 'fusion']) { + expect(parseDbtRun({ ...run, engine })?.engine).toBe(engine) + } + }) + + // The payload carries one object per node, so a scan bounded by brace COUNT + // gives up on an ordinary project — a few hundred nodes, tests included — and + // silently loses the per-model table on exactly the runs it exists for. + it('finds the run in a payload with hundreds of nodes', () => { + const big = { + ...run, + totals: { total: 400, success: 399, error: 1, warn: 0, skipped: 0 }, + nodes: Array.from({ length: 400 }, (_, i) => ({ + unique_id: `model.p.m${i}`, + status: i === 0 ? 'error' : 'success' + })) + } + const failed = { + error: { + name: 'ExecutionErr', + message: + 'execution error:\nCompilation Error {{ ref("x") }} {"a": 1}\n\n' + + JSON.stringify(big, null, 2) + } + } + expect(parseDbtRun(failed)?.nodes?.length).toBe(400) + }) + + it('is undefined for anything unparseable', () => { + expect(parseDbtRun(undefined)).toBeUndefined() + expect(parseDbtRun('a string')).toBeUndefined() + expect(parseDbtRun({ error: { message: 'failed with {not json' } })).toBeUndefined() + }) +}) + +describe('statusRank', () => { + // dbt counts `partial success` in totals.error and a retry redoes it, so + // ranking it as a pass would contradict the job's own outcome. + it('ranks partial success with the failures', () => { + expect(statusRank('partial success')).toBe(statusRank('error')) + expect(statusRank('PARTIAL SUCCESS')).toBe(0) + }) + + // The worker publishes `unknown` for a status it does not recognise and counts + // it in `totals.error`; ranking it as a pass drew a green check on a node the + // same result called an error. + // The worker counts `no_op` in totals.skipped — dbt built nothing for that + // node — so a green check would claim a run that never happened. + it('ranks a no-op with the skips, not with the passes', () => { + expect(statusRank('no-op', 'no_op')).toBe(statusRank('skipped', 'skipped')) + expect(statusRank('success', 'no_op')).toBe(2) + }) + + it('ranks an unknown outcome with the failures, not with the passes', () => { + expect(statusRank('some-future-dbt-status', 'unknown')).toBe(0) + expect(statusRank('success', 'unknown')).toBe(0) + }) + + it('orders failed before warned before skipped before passed', () => { + expect( + ['success', 'skipped', 'warn', 'error'].sort((a, b) => statusRank(a) - statusRank(b)) + ).toEqual(['error', 'warn', 'skipped', 'success']) + }) +}) + +describe('splitUniqueId', () => { + it('splits kind from name and drops a generic test’s uniqueness hash', () => { + expect(splitUniqueId('model.jaffle.stg_orders')).toEqual({ + kind: 'model', + name: 'stg_orders' + }) + expect(splitUniqueId('test.jaffle.not_null_orders_id.4e687af8d0')).toEqual({ + kind: 'test', + name: 'not_null_orders_id' + }) + // A model whose name contains a dot keeps it: only tests carry the hash. + expect(splitUniqueId('model.jaffle.a.b').name).toBe('a.b') + }) +}) + +describe('relationOutcome', () => { + it('agrees with the worker classifier on every status it names', () => { + expect(relationOutcome('started')).toBe('running') + for (const s of ['success', 'pass', 'PASS', ' Success ']) { + expect(relationOutcome(s)).toBe('materialized') + } + // `partial success` built the relation and then failed its tests; the + // worker records it failed, so the colour must agree. + for (const s of ['error', 'fail', 'runtime error', 'partial success', 'PARTIAL SUCCESS']) { + expect(relationOutcome(s)).toBe('failed') + } + // Nothing was built, so nothing is coloured. + for (const s of ['warn', 'skipped', 'no-op', 'something new']) { + expect(relationOutcome(s)).toBeUndefined() + } + }) +}) + +describe('splitRelation', () => { + it('keeps a period that lives inside a quoted identifier', () => { + // The backend supports it, so rendering it as `v2.orders` names a + // relation that does not exist. + expect(splitRelation('"wh"."analytics.v2"."orders"')).toEqual(['wh', 'analytics.v2', 'orders']) + expect(splitRelation('"db"."schema"."name"')).toEqual(['db', 'schema', 'name']) + expect(splitRelation('db.schema.name')).toEqual(['db', 'schema', 'name']) + // BigQuery backticks and T-SQL brackets quote too. + expect(splitRelation('`proj`.`data.set`.`t`')).toEqual(['proj', 'data.set', 't']) + expect(splitRelation('[db].[my.schema].[t]')).toEqual(['db', 'my.schema', 't']) + }) + + // Every one of these dialects escapes its delimiter by doubling it. Dropping + // the pair renames the relation, and the manifest keeps the real spelling — + // so the run's status would be recorded against a key no graph node has. + it('keeps a delimiter the identifier escaped by doubling', () => { + expect(splitRelation('"wh"."schema"."a""b"')).toEqual(['wh', 'schema', 'a"b']) + expect(splitRelation('`proj`.`da``ta`.`t`')).toEqual(['proj', 'da`ta', 't']) + expect(splitRelation('[db].[my]]schema].[t]')).toEqual(['db', 'my]schema', 't']) + }) +}) + +describe('nodeSelector', () => { + // Verified against dbt-core 1.12, dbt-core 2.0.0-alpha.5 and fusion + // 2.0.0-preview.202: the intersection resolves to the one node whatever the + // project's `model-paths` is, while a path-derived FQN resolves to nothing + // as soon as that root is more than one segment deep. + it('intersects the name with its package, wherever the model sits', () => { + expect(nodeSelector('model.jaffle_shop.fct_orders')).toBe('fct_orders,package:jaffle_shop') + }) + + // Ambiguous across packages, but a selector dbt resolves rather than rejects. + it('falls back to the bare name without a package', () => { + expect(nodeSelector('fct_orders')).toBe('fct_orders') + }) +}) diff --git a/frontend/src/lib/components/dbt/parseDbtRun.ts b/frontend/src/lib/components/dbt/parseDbtRun.ts new file mode 100644 index 0000000000..aecd3dcf03 --- /dev/null +++ b/frontend/src/lib/components/dbt/parseDbtRun.ts @@ -0,0 +1,247 @@ +export type DbtNode = { + unique_id: string + status: string + /** Windmill's stable word for the same result, published beside dbt's own. + * Preferred wherever a decision is made: `status` is dbt's vocabulary and + * dbt may rename it. */ + outcome?: DbtOutcome + execution_time?: number + rows_affected?: number + relation_name?: string + message?: string +} + +export type DbtRun = { + engine?: string + engine_version?: string + command?: string + totals?: { total?: number; success?: number; error?: number; warn?: number; skipped?: number } + nodes?: DbtNode[] + /** The arguments the run actually used, as submitted. A `dbt retry` restores + * the failed run's arguments inside the worker, so the retry job's own args + * name only the run it resumed — this is the sole way to recover what it + * really ran with. */ + invocation_args?: Record +} + +/** The engines the worker stamps on a result. This is the discriminator: a + * `{nodes, totals}` shape alone is one an ordinary script can return, and it + * would then be rendered as somebody's dbt run. */ +const ENGINES = ['dbt-core-1x', 'dbt-core-2x', 'fusion'] + +function asDbtRun(v: unknown): DbtRun | undefined { + if (!v || typeof v !== 'object') return undefined + const o = v as Record + return ENGINES.includes(o.engine as string) && + Array.isArray(o.nodes) && + o.totals != undefined && + typeof o.totals === 'object' + ? (o as DbtRun) + : undefined +} + +/** + * The dbt invocation a job result describes, if it describes one. + * + * On success the result IS the run. On failure the worker puts the same JSON in + * the error message after the exit-status line, and that is the case worth + * rendering: the failing node is what the user came for. + */ +export function parseDbtRun(result: any): DbtRun | undefined { + const direct = asDbtRun(result) + if (direct) return direct + const msg = result?.error?.message + if (typeof msg !== 'string') return undefined + // The payload is appended pretty-printed, so its `{` is the only one at COLUMN + // ZERO — everything nested is indented, and dbt's own error text carries its + // braces mid-line. Counting braces instead needs a cap that a real project + // blows: forwards on an error full of them, backwards on one `{` per node. + for (const line of lineStarts(msg)) { + if (msg[line] !== '{') continue + try { + const run = asDbtRun(JSON.parse(msg.slice(line))) + if (run) return run + } catch { + // A `{` alone on a line inside the error text; the payload is later. + } + } + return undefined +} + +/** Index of the first character of each line, the payload's own `{` among them. */ +function* lineStarts(s: string): Generator { + let at = 0 + while (at !== -1 && at < s.length) { + yield at + const next = s.indexOf('\n', at) + at = next === -1 ? -1 : next + 1 + } +} + +/** Ordering rank of a node's status: 0 failed, 1 warned, 2 skipped, 3 passed. + * + * Both `unknown` and `no_op` rank where the RESULT counts them, not where the + * default would put them: the worker counts `unknown` in `totals.error` and + * `no_op` in `totals.skipped`, so falling through to 3 drew a green check on a + * node the same result called an error, and on one it never built. */ +export function statusRank(status: string, outcome?: DbtOutcome): number { + switch (outcome ?? classifyStatus(status)) { + case 'failed': + case 'unknown': + return 0 + case 'warned': + return 1 + case 'skipped': + case 'no_op': + return 2 + default: + return 3 + } +} + +/** The worker's stable vocabulary for a node result, published as `outcome`. */ +export type DbtOutcome = + | 'started' + | 'passed' + | 'failed' + | 'warned' + | 'skipped' + | 'no_op' + | 'unknown' + +/** + * dbt's node status, reduced to the outcomes the UI distinguishes. + * + * Only for results that predate `outcome`, or for the live event stream, which + * carries dbt's word alone. Anything holding a node from a job result should + * read `outcome` instead — that is the field the worker publishes precisely so + * this mapping is not the contract. + */ +function classifyStatus( + status: string +): 'started' | 'passed' | 'failed' | 'warned' | 'skipped' | 'other' { + // `partial success` is dbt's word for a node that built but whose tests + // failed. The worker counts it in `totals.error` and a retry redoes it, so + // showing it green would contradict the job's own outcome. + switch (status.trim().toLowerCase()) { + case 'started': + return 'started' + case 'success': + case 'pass': + return 'passed' + case 'error': + case 'fail': + case 'runtime error': + case 'partial success': + return 'failed' + case 'warn': + return 'warned' + case 'skipped': + return 'skipped' + default: + return 'other' + } +} + +/** + * The kind and name behind a dbt `unique_id`, which dbt builds as + * `..`. A generic test's name carries a trailing + * hash dbt adds for uniqueness; it is noise in a run summary. + */ +export function splitUniqueId(uniqueId: string): { kind: string; name: string } { + const parts = uniqueId.split('.') + const kind = parts[0] ?? '' + let name = parts.slice(2).join('.') + if (kind === 'test') name = name.replace(/\.[0-9a-f]{6,}$/, '') + return { kind, name: name || uniqueId } +} + +/** + * What a node's status says happened to the relation it builds, or `undefined` + * when it says nothing. + * + * Mirrors the worker's `classify_status`, and must keep mirroring it: the two + * decide the same thing about the same string, one for the record it writes and + * one for the colour drawn over it. `warn`, `skipped` and `no-op` leave the + * relation untouched, so they get no colour rather than a misleading one. + */ +export function relationOutcome( + status: string, + outcome?: DbtOutcome +): 'running' | 'materialized' | 'failed' | undefined { + switch (outcome ?? classifyStatus(status)) { + case 'started': + return 'running' + case 'passed': + return 'materialized' + case 'failed': + return 'failed' + // `warn`, `skipped` and `no-op` say nothing about the relation: nothing + // was written, so its state is whatever the last run left. + default: + return undefined + } +} + +/** + * dbt's `relation_name` split into its parts, honouring quoting. + * + * Mirrors the worker's `split_relation`: `"`, `` ` `` and `[` open a quoted + * identifier, and a `.` inside one is part of the name. Splitting on every `.` + * turns `"wh"."analytics.v2"."orders"` into a relation called `orders` in a + * schema called `v2` — a table that does not exist. + */ +export function splitRelation(relation: string): string[] { + const parts: string[] = [] + let current = '' + let quote: string | undefined + for (let i = 0; i < relation.length; i++) { + const c = relation[i] + if (quote !== undefined) { + const close = quote === '[' ? ']' : quote + if (c === close) { + // Doubled, which is how each of these dialects escapes its own + // delimiter: one literal character, not the end of the identifier. + if (relation[i + 1] === close) { + current += close + i++ + } else { + quote = undefined + } + } else current += c + } else if (c === '"' || c === '`' || c === '[') { + quote = c + } else if (c === '.') { + parts.push(current) + current = '' + } else { + current += c + } + } + parts.push(current) + return parts.map((p) => p.trim()) +} + +/** + * A selector naming exactly one node: `,package:`. + * + * The comma is dbt's intersection operator, so this reads "the node whose name + * is `` and whose package is ``" — one node, since dbt refuses + * two models of one name inside a package. A bare name would match the leaf of + * every package's FQN, and a package can ship a model whose name the project + * also uses. + * + * Not the FQN (`..`), which cannot be rebuilt from + * `original_file_path`: how many leading segments are the resource root is + * `model-paths`, and dropping exactly one turns `src/models/marts/orders.sql` + * into `pkg.models.marts.orders`, which dbt's matcher — equal lengths, from the + * front — resolves to nothing at all. + * + * Without a package, the bare name — ambiguous across packages, but a selector + * dbt resolves rather than one it rejects. + */ +export function nodeSelector(uniqueId: string): string { + const { name } = splitUniqueId(uniqueId) + const pkg = uniqueId.split('.')[1] + return pkg ? `${name},package:${pkg}` : name +} diff --git a/frontend/src/lib/components/dbt/previewRows.ts b/frontend/src/lib/components/dbt/previewRows.ts new file mode 100644 index 0000000000..9364196d4a --- /dev/null +++ b/frontend/src/lib/components/dbt/previewRows.ts @@ -0,0 +1,108 @@ +import { JobService, type ScriptModule } from '$lib/gen' + +/** An unsaved project, submitted as its own preview job. Held as it was sent, + * never re-read from the editor: a graph describes the project it was parsed + * from, and so must anything run against that graph's nodes. */ +export type DbtPreviewBuffer = { + content: string + modules: Record | undefined + tag?: string + timeout?: number + /** The arguments the project was parsed under, in run-form shape. Part of the + * snapshot because vars decide `enabled`, schemas and aliases: run a preview + * under later ones and it can address a relation this graph never had. */ + args: Record | undefined +} + +/** A model's rows, as `dbt show` returns them. */ +export type DbtPreview = + | { pending: true } + | { rows: Record[]; node?: string; tookMs: number } + | { error: string } + +/** + * Preview one model's rows by running its own project's `dbt show`. + * + * A job, not a query: the rows come from the warehouse through the project's + * profile, with its vars and its adapter, which is the only place that knows how + * to resolve `ref()` and where the relation actually lives. `show` is therefore + * not a run-form command — it is what a table's preview is made of, here and on + * the run page's graph. + * + * Which project runs is decided by the caller, and must match the graph the SQL + * on screen came from: a deployed version by hash, a buffer parse by shipping + * that same buffer. + * + * `stillWanted` is asked before each poll and before the result is used, so a + * preview outlives neither the page that asked for it nor a navigation. + */ +export async function previewDbtRows(opts: { + workspace: string + scriptPath: string + /** Pins the preview to a deployed version, for a graph showing that version. */ + scriptHash?: string | number + /** The project the graph was parsed from, when that was a buffer. Required + * then, because there may be no deployed version at all — and when there + * is, it can lack the model or build it from other SQL. */ + buffer?: DbtPreviewBuffer + /** One node: a model name, or `package.model` where two packages share one. */ + model: string + /** The run's own vars, so a descriptor with a required `{{ }}` var resolves. */ + vars?: Record + limit?: number + /** Extra top-level arguments — a run's `{{ placeholder }}` values. */ + args?: Record + stillWanted?: () => boolean +}): Promise { + const { workspace, scriptPath, scriptHash, buffer, model, vars, limit, args, stillWanted } = opts + const startedAt = Date.now() + const requestBody = { + ...(args ?? {}), + command: { label: 'show', vars: vars ?? {}, model, limit: limit ?? 25 } + } + try { + // The BUFFER when the caller has one, because then the graph and the SQL + // above these rows are the buffer's: a model added since the deploy exists + // in no other project, and one whose SQL changed builds different rows + // there. It ships whole — dbt resolves `ref()` project-wide. + // + // Otherwise by HASH whenever the caller pins one: the SQL on screen is that + // version's, and running the deployed one would show today's rows under + // it — or fail outright for a model since removed. + const id = buffer + ? await JobService.runScriptPreview({ + workspace, + timeout: buffer.timeout, + requestBody: { + path: scriptPath, + content: buffer.content, + language: 'dbt', + tag: buffer.tag, + modules: buffer.modules, + args: requestBody + } + }) + : scriptHash + ? await JobService.runScriptByHash({ + workspace, + hash: String(scriptHash), + requestBody + }) + : await JobService.runScriptByPath({ workspace, path: scriptPath, requestBody }) + // Polled rather than awaited: a preview is a job, and its engine may need + // provisioning on a cold worker. + for (let i = 0; i < 90; i++) { + await new Promise((r) => setTimeout(r, 1000)) + if (stillWanted && !stillWanted()) return undefined + const done = await JobService.getCompletedJobResultMaybe({ workspace, id }) + if (!done.completed) continue + const res = done.result as { node?: string; show?: Record[] } | undefined + return done.success && res?.show + ? { rows: res.show, node: res.node, tookMs: Date.now() - startedAt } + : { error: 'The preview job failed — open it from Runs for the detail.' } + } + return { error: 'The preview is still running; open it from Runs.' } + } catch (e) { + return { error: e instanceof Error ? e.message : String(e) } + } +} diff --git a/frontend/src/lib/components/dbt/projectFiles.test.ts b/frontend/src/lib/components/dbt/projectFiles.test.ts new file mode 100644 index 0000000000..d9ed58b41a --- /dev/null +++ b/frontend/src/lib/components/dbt/projectFiles.test.ts @@ -0,0 +1,72 @@ +import { describe, expect, it } from 'vitest' +import type { ScriptModule } from '$lib/gen' +import { dbtModelSelector, dbtModulePath, dbtProjectFileKey } from './projectFiles' + +const mod = (content: string): ScriptModule => ({ content, language: 'dbt' as any }) + +const PROJECT = `name: jaffle +model-paths: ["models"] +` + +describe('dbtModulePath', () => { + // The worker resolves `.` and `//` away when it materialises the bundle, so a + // redundant spelling is a second key for one file on disk and walks straight + // past a check that compares the typed string. + it('canonicalises before the reserved-name and duplicate checks', () => { + expect(dbtModulePath('./models//x.sql', {})).toEqual({ path: 'models/x.sql' }) + expect(dbtModulePath('./wm_dbt.yaml', {})).toEqual({ + error: expect.stringContaining('is the descriptor') + }) + const bundle = { './models/x.sql': mod('select 1') } + expect(dbtModulePath('models/x.sql', bundle)).toEqual({ + error: expect.stringContaining('./models/x.sql already exists') + }) + }) + + // A path outside the bundle has no canonical form inside it, and the worker + // drops it — which would read as a file that was added and never written. + it('refuses a path that escapes the bundle', () => { + expect(dbtModulePath('../secrets.sql', {})).toEqual({ error: expect.stringContaining('..') }) + expect(dbtModulePath('/etc/x.sql', {})).toEqual({ error: expect.stringContaining('relative') }) + }) + + it('refuses an extension dbt does not read', () => { + expect(dbtModulePath('models/x.txt', {})).toEqual({ + error: expect.stringContaining('must end with') + }) + }) +}) + +// Every write path canonicalises, so a read that compares the constant exactly +// would neither find a project imported under a redundant spelling nor protect +// it from deletion. +it('resolves the key a bundle actually holds the project file under', () => { + expect(dbtProjectFileKey({ './dbt_project.yml': mod(PROJECT) })).toBe('./dbt_project.yml') + expect(dbtProjectFileKey({})).toBeUndefined() +}) + +describe('dbtModelSelector', () => { + // Package-qualified, because a bare leaf name also matches a dependency + // package's model of the same name. + it('selects a model under the project’s own model-paths', () => { + const bundle = { 'dbt_project.yml': mod(PROJECT), 'models/orders.sql': mod('select 1') } + expect(dbtModelSelector(bundle, 'models/orders.sql')).toBe('orders,package:jaffle') + }) + + // A project may put its models anywhere; a macro or singular test is `.sql` + // too and is not selectable by name, so those fall back to the whole project. + it('honours a custom model-paths and skips what is not a model', () => { + const bundle = { + 'dbt_project.yml': mod('name: jaffle\nmodel-paths: ["transform"]\n'), + 'transform/orders.sql': mod('select 1'), + 'macros/cents.sql': mod('{% macro cents() %}{% endmacro %}') + } + expect(dbtModelSelector(bundle, 'transform/orders.sql')).toBe('orders,package:jaffle') + expect(dbtModelSelector(bundle, 'macros/cents.sql')).toBeUndefined() + }) + + it('finds the project file under a redundant spelling', () => { + const bundle = { './dbt_project.yml': mod(PROJECT), 'models/orders.sql': mod('select 1') } + expect(dbtModelSelector(bundle, 'models/orders.sql')).toBe('orders,package:jaffle') + }) +}) diff --git a/frontend/src/lib/components/dbt/projectFiles.ts b/frontend/src/lib/components/dbt/projectFiles.ts new file mode 100644 index 0000000000..36902ac74a --- /dev/null +++ b/frontend/src/lib/components/dbt/projectFiles.ts @@ -0,0 +1,162 @@ +/** + * The rules for the files of a dbt project, which is what a dbt script's module + * bundle is. + * + * A `__mod` bundle is homogeneous — one language, one extension — and a dbt + * project is not: models are SQL or Python, schemas and the project file YAML, + * seeds CSV, doc blocks Markdown. Every module is stored with `language: dbt` + * because they are dbt's to read, so everything below is decided by the file + * name. + */ +import type { Preview, ScriptModule } from '$lib/gen' +import { canonicalModulePath, findModulePathClash } from '../scriptModulePath' +import YAML from 'yaml' + +/** The descriptor. It is the script's CONTENT, not a module: a module at the + * same path would be a second, independent value for one file, since the export + * writes the content there and the bundle would emit over it. */ +export const DBT_DESCRIPTOR = 'wm_dbt.yaml' + +/** What makes the bundle a project. The worker refuses a version without it, so + * it is the one file the tree will not let you delete. */ +export const DBT_PROJECT_FILE = 'dbt_project.yml' + +/** The key a bundle actually holds `dbt_project.yml` under. + * + * Not the constant: nothing on the push path rewrites existing keys, so a + * project imported with `./dbt_project.yml` in it holds that spelling, and an + * exact lookup would neither find the project nor protect it from deletion — + * while every write path canonicalises. Resolved the same way the duplicate + * check resolves a clash. */ +export function dbtProjectFileKey( + modules: Record | null | undefined +): string | undefined { + return findModulePathClash(modules, DBT_PROJECT_FILE) +} + +/** + * Extensions a dbt project's own files take. `.py` because dbt Python models are + * first-class on Snowflake, BigQuery and Databricks. + */ +export const DBT_MODULE_EXTENSIONS = ['.sql', '.py', '.yml', '.yaml', '.csv', '.md'] + +/** + * The editor language for one project file, by extension. `postgresql` and + * `ansible` are how Windmill spells "SQL" and "YAML" to the editor. + */ +export function dbtFileLang(path: string): Preview['language'] { + if (path.endsWith('.sql')) return 'postgresql' + if (path.endsWith('.yml') || path.endsWith('.yaml')) return 'ansible' + if (path.endsWith('.py')) return 'python3' + // `.md` (a doc block) and `.csv` (a seed) have no grammar of their own here; + // `bash` leaves prose alone, where the default would colour it as TypeScript. + return 'bash' +} + +/** Whether this file may be added to the bundle at all. */ +export function dbtModuleLang(filePath: string): ScriptModule['language'] | undefined { + return DBT_MODULE_EXTENSIONS.some((e) => filePath.endsWith(e)) + ? ('dbt' as ScriptModule['language']) + : undefined +} + +/** The canonical key a typed path becomes, or the reason it cannot be one. + * + * Canonicalised before the reserved-name and duplicate checks, not after: the + * worker resolves `.` and `//` away when it materialises the bundle, so + * `./dbt_project.yml` and `dbt_project.yml` are two keys for one file on disk + * and either check is trivially walked past by the redundant spelling. */ +export function dbtModulePath( + path: string, + modules: Record | undefined +): { path: string } | { error: string } { + const canonical = canonicalModulePath(path) + if ('error' in canonical) return canonical + if (canonical.path === DBT_DESCRIPTOR) { + return { + error: `${DBT_DESCRIPTOR} is the descriptor, edited from the tree — it cannot also be a file` + } + } + if (!dbtModuleLang(canonical.path)) { + return { error: `File must end with one of: ${DBT_MODULE_EXTENSIONS.join(', ')}` } + } + const clash = findModulePathClash(modules, canonical.path) + if (clash) return { error: `${clash} already exists` } + return canonical +} + +/** Why this path cannot be a file, or `undefined` when it can. */ +export function dbtPathError( + path: string, + modules: Record | undefined +): string | undefined { + if (!path.trim()) return undefined + const resolved = dbtModulePath(path, modules) + return 'error' in resolved ? resolved.error : undefined +} + +/** A new file's starting content. A model compiles on its own, so it is runnable + * before it is edited; anything else starts empty rather than with a guess at + * which dbt schema it is. */ +export function dbtDefaultContent(filePath: string): string { + return filePath.endsWith('.sql') ? 'select 1 as id\n' : '' +} + +/** + * What `dbt build --select` should be given for an open file, or `undefined` + * when the file is not a model. + * + * Only files under the project's `model-paths` are models; a project also holds + * macros, analyses and singular tests, all `.sql`, none of them selectable by + * name. The selector is package-qualified because a bare leaf name also matches a + * dependency package's model of the same name. + */ +export function dbtModelSelector( + modules: Record, + filePath: string +): string | undefined { + // `.py` as well as `.sql`: a dbt Python model is a model, and leaving it out + // would run the whole project to check one file — the larger warehouse bill + // this narrowing exists to avoid. + const ext = ['.sql', '.py'].find((e) => filePath.endsWith(e)) + if (!ext) return undefined + const projectKey = dbtProjectFileKey(modules) + let project: any + try { + project = YAML.parse((projectKey ? modules[projectKey] : undefined)?.content ?? '') + } catch { + return undefined + } + if (!project?.name) return undefined + const modelPaths: string[] = Array.isArray(project['model-paths']) + ? project['model-paths'] + : ['models'] + if (!modelPaths.some((d) => filePath === d || filePath.startsWith(d + '/'))) return undefined + const name = filePath.split('/').pop()!.slice(0, -ext.length) + return `${name},package:${project.name}` +} + +/** The descriptor's `engine` and `profile.warehouse`, for the editor's header. + * Best effort: a descriptor mid-edit is often not valid YAML, and a header that + * blanked on every keystroke would be worse than one that lags. */ +export function dbtDescriptorSummary(content: string): { + engine: string + warehouse: string +} { + try { + const d = YAML.parse(content ?? '') + return { + engine: typeof d?.engine === 'string' ? d.engine : 'dbt-core-1x', + // A descriptor naming its own `profiles_yml` and no warehouse has no + // warehouse identity at all, which is what leaves it with no graph. + warehouse: + typeof d?.profile?.warehouse === 'string' + ? d.profile.warehouse + : d?.profile?.profiles_yml + ? 'own profiles.yml' + : 'main' + } + } catch { + return { engine: 'dbt-core-1x', warehouse: 'main' } + } +} diff --git a/frontend/src/lib/components/dbt/runStatus.svelte.ts b/frontend/src/lib/components/dbt/runStatus.svelte.ts new file mode 100644 index 0000000000..aaca172717 --- /dev/null +++ b/frontend/src/lib/components/dbt/runStatus.svelte.ts @@ -0,0 +1,100 @@ +/** + * What a dbt run is doing to each relation on a model graph, live and then + * settled. + * + * Two sources, in that order of authority: + * + * * the run's own `run_results.json`, once it has one — joined on dbt's + * `unique_id`, which is what both sides carry. The per-relation state table + * holds ONE row per relation stamped with its last writer, so reading that + * for a finished run would show it a later run's outcomes. + * * `dbt_run_progress` while it is in flight, polled. Only `dbt-core-1x` emits + * the node events behind it, and only a database-connected worker records + * them, so on the other engines and on agent workers this stays empty and the + * graph colours at the end instead. + * + * One definition, used by the run page and by the editor: a second one would be + * a second answer to "what colour is this model right now". + */ +import { JobService } from '$lib/gen' +import type { AssetGraphResponse, AssetRunState } from '$lib/components/assets/AssetGraph/types' +import { parseDbtRun, relationOutcome } from './parseDbtRun' + +export function useDbtRunStatus(opts: { + workspace: () => string | undefined + /** The run whose progress to show, or `undefined` for a graph with no run. */ + jobId: () => string | undefined + running: () => boolean + /** The finished job's result, which carries a status per dbt node. */ + result: () => unknown + /** The graph on screen, for the `unique_id` → relation mapping. */ + graph: () => AssetGraphResponse | undefined + /** Guards every response against a navigation: a poll that outlives the run + * it was issued for would colour the next one's models. */ + generation: () => number + destroyed: () => boolean +}) { + let polled = $state>(new Map()) + + async function load() { + const ws = opts.workspace() + const id = opts.jobId() + if (!ws || !id) return + const gen = opts.generation() + try { + const rows = await JobService.getRunProgress({ workspace: ws, id }) + if (gen !== opts.generation() || opts.destroyed()) return + const next = new Map() + for (const r of rows) { + next.set(`asset:${r.asset_kind}:${r.asset_path}`, { + status: r.status, + rowCount: r.row_count + }) + } + polled = next + } catch { + // A progress hiccup must not blank the graph. + } + } + + let run = $derived(parseDbtRun(opts.result())) + + let settled = $derived.by(() => { + if (opts.running()) return undefined + const g = opts.graph() + if (!run?.nodes?.length || !g) return undefined + const assetByNode = new Map() + for (const a of g.assets) { + if (a.dbt?.unique_id) assetByNode.set(a.dbt.unique_id, `asset:${a.kind}:${a.path}`) + } + const out = new Map() + for (const n of run.nodes) { + const id = assetByNode.get(n.unique_id) + const outcome = id && relationOutcome(n.status, n.outcome) + // A test or an analysis matches no relation, and a skipped node says + // nothing about one; both are left uncoloured rather than guessed at. + if (id && outcome) out.set(id, { status: outcome, rowCount: n.rows_affected }) + } + return out.size > 0 ? out : undefined + }) + + return { + /** Poll once. The caller owns the cadence — a run page ticks while its job + * runs, the editor while a build does. */ + load, + /** Drop what a previous run left, so its colours do not survive into the + * next one. */ + reset: () => (polled = new Map()), + get status() { + return settled ?? polled + }, + /** Whether the run settled itself, which is what tells a caller it no + * longer needs to poll. */ + get isSettled() { + return settled != undefined + }, + get run() { + return run + } + } +} diff --git a/frontend/src/lib/components/debug/dapClient.ts b/frontend/src/lib/components/debug/dapClient.ts index 71783b4ec0..d124637b9b 100644 --- a/frontend/src/lib/components/debug/dapClient.ts +++ b/frontend/src/lib/components/debug/dapClient.ts @@ -82,6 +82,14 @@ const initialState: DebugState = { export const debugState = writable({ ...initialState }) +const DEFAULT_REQUEST_TIMEOUT_MS = 10_000 + +// `launch` waits on dependency installation in the debug server, which can take minutes on a +// cold cache; anything shorter here reports a timeout while the install is still running. +const REQUEST_TIMEOUT_MS_BY_COMMAND: Record = { + launch: 180_000 +} + export class DAPClient { private ws: WebSocket | null = null private seq = 1 @@ -120,7 +128,13 @@ export class DAPClient { logs: s.logs, output: s.output })) + // Reject rather than drop: a dropped `launch` leaves its caller awaiting + // until the timeout below fires, which is minutes rather than seconds. + const aborted = Array.from(this.pendingRequests.values()) this.pendingRequests.clear() + for (const pending of aborted) { + pending.reject(new Error('DAP connection closed')) + } } this.ws.onerror = (error) => { @@ -164,11 +178,13 @@ export class DAPClient { arguments: args } + const timeoutMs = REQUEST_TIMEOUT_MS_BY_COMMAND[command] ?? DEFAULT_REQUEST_TIMEOUT_MS + return new Promise((resolve, reject) => { const timeout = setTimeout(() => { this.pendingRequests.delete(seq) - reject(new Error(`Request timeout: ${command}`)) - }, 10000) + reject(new Error(`Request timeout: ${command} (after ${timeoutMs}ms)`)) + }, timeoutMs) this.pendingRequests.set(seq, { resolve: (value) => { diff --git a/frontend/src/lib/components/details/OnBehalfOfBadge.svelte b/frontend/src/lib/components/details/OnBehalfOfBadge.svelte new file mode 100644 index 0000000000..13bd94b122 --- /dev/null +++ b/frontend/src/lib/components/details/OnBehalfOfBadge.svelte @@ -0,0 +1,34 @@ + + +{#if onBehalfOf} + + {#snippet text()} + Every run of this {kind} is permissioned as {detailed}, whoever starts it. + {/snippet} + + On behalf of {onBehalfOf} + + +{/if} diff --git a/frontend/src/lib/components/flows/DebounceLimit.svelte b/frontend/src/lib/components/flows/DebounceLimit.svelte index 1889d964ba..3ab55caeed 100644 --- a/frontend/src/lib/components/flows/DebounceLimit.svelte +++ b/frontend/src/lib/components/flows/DebounceLimit.svelte @@ -5,6 +5,7 @@ import { enterpriseLicense } from '$lib/stores' import Label from '../Label.svelte' import type { Schema } from '$lib/common' + import { slideDynamic } from '$lib/transitions' let { debounce_delay_s = $bindable(), @@ -16,7 +17,8 @@ placeholder, size = 'xs', color = undefined, - fontClass = 'font-normal' + fontClass = 'font-normal', + indentContent = false }: { debounce_delay_s: number | undefined debounce_key: string | undefined @@ -28,6 +30,8 @@ size: 'xs' | 'sm' color?: 'nord' | undefined fontClass?: string + /** Align the fields under the toggle's label, as the flow step settings do. */ + indentContent?: boolean } = $props() // Check if an originalType like "string | string[]" is a top-level @@ -85,6 +89,10 @@ max_total_debounces_amount = undefined } }) + + // Presence, not truthiness: a single backspace takes the seeded 1s to 0 while the + // user is still typing, and reading that as off would disable the field mid-edit. + let off = $derived(!$enterpriseLicense || debounce_delay_s === undefined)
@@ -94,9 +102,9 @@ {color} {size} disabled={!$enterpriseLicense} - checked={Boolean(debounce_delay_s)} + checked={debounce_delay_s !== undefined} on:change={() => { - if (debounce_delay_s) { + if (debounce_delay_s !== undefined) { debounce_delay_s = undefined debounce_key = undefined } else { @@ -113,10 +121,10 @@ />
- {#if debounce_delay_s} -
+ {#if !off} +
{/if} diff --git a/frontend/src/lib/components/flows/FlowEditor.svelte b/frontend/src/lib/components/flows/FlowEditor.svelte index a484675f83..a8c18f70ba 100644 --- a/frontend/src/lib/components/flows/FlowEditor.svelte +++ b/frontend/src/lib/components/flows/FlowEditor.svelte @@ -1,12 +1,18 @@ +{#snippet panelBody()} + +{/snippet} +
panelController.measure(w)} id="flow-editor" - class={'h-full overflow-hidden transition-colors duration-[400ms] ease-linear border-t'} + class={'relative h-full overflow-hidden transition-colors duration-[400ms] ease-linear border-t'} use:triggerableByAI={{ id: 'flow-editor', description: 'Component to edit a flow' }} > - + +
{#if graphOverlay} @@ -226,36 +396,96 @@ {/if}
- - {#if loading} -
-
- + {#if panelMode === 'docked'} + + + {#if loading} +
+
+ +
-
- {:else} - - {/if} - + {:else if modalPanel} +
+ + {#if detachClaims === 0} +
+ +
+ {/if} +
+ {@render panelBody()} +
+
+ {:else} + {@render panelBody()} + {/if} + + {/if} {#if !disableAi} {/if} + + {#if showStepHint} +
+ + {stepHintText} +
+ {/if}
+ + + + + {#snippet children({ zIndex })} + {#if panelMode === 'modal' && panelModalOpen} + + + + + + {/if} + {/snippet} + diff --git a/frontend/src/lib/components/flows/branchOps.test.ts b/frontend/src/lib/components/flows/branchOps.test.ts new file mode 100644 index 0000000000..844477c968 --- /dev/null +++ b/frontend/src/lib/components/flows/branchOps.test.ts @@ -0,0 +1,55 @@ +import { describe, it, expect } from 'vitest' +import { writable, get } from 'svelte/store' +import type { FlowModule } from '$lib/gen' +import type { StateStore } from '$lib/utils' +import type { ExtendedOpenFlow } from './types' +import { reorderBranches } from './branchOps' + +function branch(expr: string) { + return { summary: expr, expr, modules: [] as FlowModule[] } +} + +function ctx(branches: ReturnType[]) { + const flow = { + summary: '', + value: { + modules: [{ id: 'a', value: { type: 'branchone', branches, default: [] } } as FlowModule] + } + } as ExtendedOpenFlow + const flowStore: StateStore = { val: flow } + const history = writable({ history: [] as ExtendedOpenFlow[], index: -1 }) + const read = () => + (flowStore.val.value.modules[0].value as { branches: ReturnType[] }).branches + return { flowStore, history, read } +} + +describe('reorderBranches', () => { + it('commits the new order', () => { + const [x, y, z] = [branch('x'), branch('y'), branch('z')] + const { flowStore, history, read } = ctx([x, y, z]) + + reorderBranches('a', [z, x, y], { flowStore, history }) + + expect(read().map((b) => b.expr)).toEqual(['z', 'x', 'y']) + }) + + it('records an undo entry, so a drag can be undone like an add or a delete', () => { + const [x, y] = [branch('x'), branch('y')] + const { flowStore, history } = ctx([x, y]) + + reorderBranches('a', [y, x], { flowStore, history }) + + expect(get(history).history).toHaveLength(1) + }) + + it('spends nothing when the drag lands where it started', () => { + // `finalize` fires on every drop, so a no-op drag must not consume an undo step. + const [x, y] = [branch('x'), branch('y')] + const { flowStore, history, read } = ctx([x, y]) + + reorderBranches('a', [x, y], { flowStore, history }) + + expect(get(history).history).toHaveLength(0) + expect(read().map((b) => b.expr)).toEqual(['x', 'y']) + }) +}) diff --git a/frontend/src/lib/components/flows/branchOps.ts b/frontend/src/lib/components/flows/branchOps.ts new file mode 100644 index 0000000000..f42026bccb --- /dev/null +++ b/frontend/src/lib/components/flows/branchOps.ts @@ -0,0 +1,82 @@ +import type { FlowModule } from '$lib/gen' +import type { ExtendedOpenFlow } from './types' +import type { FlowState } from './flowState' +import type { StateStore } from '$lib/utils' +import type { History } from '$lib/history.svelte' +import { push } from '$lib/history.svelte' +import { dfs } from './dfs' +import { findModuleInFlow } from './flowTree' + +type BranchList = Array<{ summary?: string; expr?: string; modules: FlowModule[] }> + +type Ctx = { + flowStore: StateStore + flowStateStore: StateStore + history: History +} + +/** Append an empty branch to a branchone/branchall step. */ +export function addBranch(moduleId: string, { flowStore, history }: Omit) { + push(history, flowStore.val) + const module = findModuleInFlow(flowStore.val.value, moduleId) + if (!module) throw new Error(`Node ${moduleId} not found`) + + if (module.value.type === 'branchone' || module.value.type === 'branchall') { + module.value.branches.push({ summary: '', expr: 'false', modules: [] }) + } +} + +/** + * Drop a branch and the flow state of every step inside it. + * + * `index` counts the way the graph lays the branches out, where a branchone's default + * occupies slot 0 — one ahead of the same branch's position in `value.branches`. Callers + * working from the array (the settings panel) must add that offset back. + */ +export function removeBranch( + moduleId: string, + index: number, + { flowStore, flowStateStore, history }: Ctx +) { + push(history, flowStore.val) + const module = findModuleInFlow(flowStore.val.value, moduleId) + if (!module) throw new Error(`Node ${moduleId} not found`) + + if (module.value.type === 'branchone' || module.value.type === 'branchall') { + const offset = module.value.type === 'branchone' ? 1 : 0 + const at = index - offset + + if (module.value.branches[at]?.modules) { + const leaves = dfs(module.value.branches[at].modules, (mod) => mod.id) + leaves.forEach((leafId: string) => delete flowStateStore.val[leafId]) + } + + module.value.branches.splice(at, 1) + } +} + +/** + * Commit a reordered branch list. Undoable like add/remove — a drag is a structural edit, + * and for a branchone it changes which predicate is evaluated first. + */ +export function reorderBranches( + moduleId: string, + ordered: BranchList, + { flowStore, history }: Omit +) { + const module = findModuleInFlow(flowStore.val.value, moduleId) + if (!module) throw new Error(`Node ${moduleId} not found`) + if (module.value.type !== 'branchone' && module.value.type !== 'branchall') return + + const current = module.value.branches + // A drag that lands where it started must not spend an undo entry. + if (ordered.length === current.length && ordered.every((b, i) => b === current[i])) return + + push(history, flowStore.val) + module.value.branches = ordered as typeof current +} + +/** Slot a branch occupies in the graph's numbering, from its index in `value.branches`. */ +export function graphBranchIndex(type: 'branchone' | 'branchall', arrayIndex: number): number { + return type === 'branchone' ? arrayIndex + 1 : arrayIndex +} diff --git a/frontend/src/lib/components/flows/common/FlowCard.svelte b/frontend/src/lib/components/flows/common/FlowCard.svelte index aadb9d459d..091d3a9feb 100644 --- a/frontend/src/lib/components/flows/common/FlowCard.svelte +++ b/frontend/src/lib/components/flows/common/FlowCard.svelte @@ -6,6 +6,8 @@ title?: string | undefined summary?: string | undefined description?: string | undefined + subtitle?: string | undefined + subtitleDocLink?: string | undefined noEditor: boolean noHeader?: boolean flowModuleValue?: FlowModuleValue | undefined @@ -20,6 +22,8 @@ title = undefined, summary = $bindable(undefined), description = $bindable(undefined), + subtitle = undefined, + subtitleDocLink = undefined, noEditor, noHeader = false, flowModuleValue = undefined, @@ -37,9 +41,12 @@ - let cachedValues: Record< - string, - { - latestHash: string | undefined - } - > = {} - - @@ -101,8 +186,8 @@ class="overflow-x-auto scrollbar-hidden flex items-center justify-between flex-nowrap w-full" > {#if flowModuleValue} - -
+ +
{#if flowModuleValue.type === 'identity'} Identity (input copied to output) {:else if flowModuleValue.type === 'rawscript'} @@ -121,75 +206,16 @@ {siblingToolNames} /> {:else if flowModuleValue.type === 'script' && 'path' in flowModuleValue && flowModuleValue.path} - - - {#if hubVersionId} - + + {#if scriptItems.length > 0} + {/if} - {#if flowModuleValue.hash} - {#if latestHash != flowModuleValue.hash} - - {/if} - - {:else if latestHash} -
- -
- {/if} -
+
{/if} {#if title} -
{title}
+ +
{title}
{/if} {@render children?.()} {@render action?.()} +
+ {#if subtitle} +

+ {subtitle} + {#if subtitleDocLink} + Docs + {/if} +

+ {/if} {#if isAgentTool} {#if toolNameError}

{toolNameError}

diff --git a/frontend/src/lib/components/flows/common/FlowPanelChrome.svelte b/frontend/src/lib/components/flows/common/FlowPanelChrome.svelte new file mode 100644 index 0000000000..0a3d822fd1 --- /dev/null +++ b/frontend/src/lib/components/flows/common/FlowPanelChrome.svelte @@ -0,0 +1,29 @@ + + +
+ +
+{#if panelDetach?.modalOpen()} + -
-
-{/if} + {/snippet} + + diff --git a/frontend/src/lib/components/flows/content/DynamicInputHelpBox.svelte b/frontend/src/lib/components/flows/content/DynamicInputHelpBox.svelte index 93bfd2d533..8ed18d73bf 100644 --- a/frontend/src/lib/components/flows/content/DynamicInputHelpBox.svelte +++ b/frontend/src/lib/components/flows/content/DynamicInputHelpBox.svelte @@ -3,11 +3,12 @@ import Button from '$lib/components/common/button/Button.svelte' import { ChevronDown } from 'lucide-svelte' - let opened = $state(false); - + let opened = $state(false) -
+ +
diff --git a/frontend/src/lib/components/flows/content/ExpandedSubflowStep.svelte b/frontend/src/lib/components/flows/content/ExpandedSubflowStep.svelte new file mode 100644 index 0000000000..4aede5e3d1 --- /dev/null +++ b/frontend/src/lib/components/flows/content/ExpandedSubflowStep.svelte @@ -0,0 +1,149 @@ + + +
+
+
+ {leafId} + {#if resolved} + in + {#each resolved.pathChain as path, i (i)} + {#if i > 0} + + {/if} + {path} + {/each} + {/if} +
+
+ {#if resolved} + {@const { containingFlowPath, module } = resolved} + {#if $flowEditorDrawer} + + {/if} +
+
+
+ {#if loaded == undefined} +
+ +
+ {:else if loaded.error} +
+ Could not load the subflow this step belongs to: {loaded.error.message} +
+ {:else if resolved?.module} + + {:else} +
+ This step is part of an expanded subflow and is not editable from this flow. +
+ {/if} +
+
diff --git a/frontend/src/lib/components/flows/content/FlowBranchAllWrapper.svelte b/frontend/src/lib/components/flows/content/FlowBranchAllWrapper.svelte index 17f5767c46..da06034a8d 100644 --- a/frontend/src/lib/components/flows/content/FlowBranchAllWrapper.svelte +++ b/frontend/src/lib/components/flows/content/FlowBranchAllWrapper.svelte @@ -12,7 +12,7 @@ } } - let { noEditor, branch = $bindable() }: Props = $props() + let { noEditor, branch }: Props = $props()
@@ -23,13 +23,8 @@
{/snippet}
-
Skip failures
- +
Skip failures
+
diff --git a/frontend/src/lib/components/flows/content/FlowBranchOneWrapper.svelte b/frontend/src/lib/components/flows/content/FlowBranchOneWrapper.svelte index fd7bbc82e8..f118e0859d 100644 --- a/frontend/src/lib/components/flows/content/FlowBranchOneWrapper.svelte +++ b/frontend/src/lib/components/flows/content/FlowBranchOneWrapper.svelte @@ -16,13 +16,7 @@ enableAi?: boolean } - let { - branch = $bindable(), - parentModule, - previousModule, - noEditor, - enableAi = false - }: Props = $props() + let { branch, parentModule, previousModule, noEditor, enableAi = false }: Props = $props()
@@ -32,19 +26,8 @@
{/snippet} -
-

Predicate expression

- { - if (!branch.summary) { - branch.summary = e.detail - } - }} - /> +
+
diff --git a/frontend/src/lib/components/flows/content/FlowBranchesAllWrapper.svelte b/frontend/src/lib/components/flows/content/FlowBranchesAllWrapper.svelte index 8fa98b33d1..8374962083 100644 --- a/frontend/src/lib/components/flows/content/FlowBranchesAllWrapper.svelte +++ b/frontend/src/lib/components/flows/content/FlowBranchesAllWrapper.svelte @@ -1,20 +1,26 @@
- - - - - {#if !noEditor} - - The result of this step is the list of the result of each branch. - - {/if} -
-

{value.branches.length} branch{value.branches.length > 1 ? 'es' : ''}

-
- {#each value.branches as branch, i} -
-
- Branch {i + 1} - -
-
- +
+ + + + {#snippet extra()} + + {/snippet} + + + +
+ {#if selectedTab === 'branches'} +
+
+
+ {#each items as item, i (item.id)} + +
+ +
+ +
+
+ Branch {i + 1} + item.branch.summary ?? '', (v) => (item.branch.summary = String(v)) + } + inputProps={{ placeholder: 'Summary' }} + /> +
+
-
- {/each} + {/each} +
+
-

Add branches and steps directly on the graph.

-
Run in parallel
- -
- - {#if flowModule} - - - - - - - - - {#snippet content()} -
- -
- -
-
- -
- -
-
- -
- -
-
- -
- -
-
- -
- -
-
- -
- -
-
-
- {/snippet} -
-
+
+ + +
+ + {:else} + {/if} - - +
+
diff --git a/frontend/src/lib/components/flows/content/FlowBranchesOneWrapper.svelte b/frontend/src/lib/components/flows/content/FlowBranchesOneWrapper.svelte index aa6d1af97c..25b843268a 100644 --- a/frontend/src/lib/components/flows/content/FlowBranchesOneWrapper.svelte +++ b/frontend/src/lib/components/flows/content/FlowBranchesOneWrapper.svelte @@ -1,24 +1,29 @@
- - - - - {#if !noEditor} - - The result of this step is the result of the branch. - - {/if} -
-

- {value.branches.length + 1} branch{value.branches.length + 1 > 1 ? 'es' : ''} -

-
-
- Default branch -

If none of the predicates' expressions evaluated in-order match, this branch is - chosen

-
- {#each value.branches as branch, i} -
-
- Branch {i + 1} - -
-
- { - if (!branch.summary) { - branch.summary = e.detail + +
+ + + + {#snippet extra()} + + {/snippet} + + + +
+ {#if selectedTab === 'branches'} +
+
+
+ {#each items as item, i (item.id)} + +
+ +
+ +
+
+ Branch {i + 1} + item.branch.summary ?? '', (v) => (item.branch.summary = String(v)) } - }} - parentModule={flowModule} - {previousModule} - {enableAi} + inputProps={{ placeholder: 'Summary' }} + /> +
+
-
- {/each} + {/each} +
+
+ Default +

Runs if none of the above match

+
-

Add branches and steps directly on the graph.

-
- - {#if flowModule} - - - - - - - - - {#snippet content()} -
- -
- -
-
- -
- -
-
- -
- -
-
- -
- -
-
- -
- -
-
- -
- -
-
-
- {/snippet} -
-
+ + + {:else} + {/if} - - +
+
diff --git a/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte b/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte index 00643588ae..3103f310b4 100644 --- a/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte +++ b/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte @@ -12,6 +12,7 @@ import { initFlow } from '$lib/components/flows/flowStore.svelte' import type { FlowState } from '$lib/components/flows/flowState' import type { FlowEditorContext } from '../types' + import { base } from '$app/paths' let flowEditorDrawer: Drawer | undefined = $state() @@ -20,11 +21,12 @@ const flowEditorContext = getContext('FlowEditorContext') let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) - export async function openDrawer(path: string, cb: () => void): Promise { + export async function openDrawer(path: string, cb: () => void, stepId?: string): Promise { flowPath = path flow = undefined loading = true callback = cb + selectedStepId = stepId flowEditorDrawer?.openDrawer?.() try { @@ -47,6 +49,7 @@ let callback: (() => void) | undefined = undefined let flowPath: string = $state('') + let selectedStepId: string | undefined = $state(undefined) let flow: Flow | undefined = $state(undefined) let savedFlow: Flow | undefined = $state(undefined) let loading = $state(true) @@ -66,6 +69,7 @@ const flowStateStore: StateStore = $state({ val: {} }) let diffDrawer: DiffDrawer | undefined = $state() + let flowBuilder: FlowBuilder | undefined = $state(undefined) @@ -87,12 +91,13 @@
{:else if flow} { - window.open(`/flows/edit/${flowPath}`, '_blank', 'noopener,noreferrer') + // Follow the drawer's own selection, which the user may have moved since it opened. + const stepId = flowBuilder?.getSelectedId() ?? selectedStepId + const selected = stepId ? `?selected=${encodeURIComponent(stepId)}` : '' + window.open(`${base}/flows/edit/${flowPath}${selected}`, '_blank', 'noopener,noreferrer') flowEditorDrawer?.closeDrawer() }} startIcon={{ icon: ExternalLink }} diff --git a/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte b/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte index 471790d539..81600e2dcc 100644 --- a/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte +++ b/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte @@ -2,6 +2,7 @@ import { getContext } from 'svelte' import type { FlowEditorContext } from '../types' import FlowModuleWrapper from './FlowModuleWrapper.svelte' + import { moduleSlot, savedModuleById } from '../moduleSlot' import FlowSettings from './FlowSettings.svelte' import FlowInput from './FlowInput.svelte' import FlowFailureModule from './FlowFailureModule.svelte' @@ -14,6 +15,7 @@ import { handleSelectTriggerFromKind, type Trigger } from '$lib/components/triggers/utils' import { computeMissingInputWarnings } from '../missingInputWarnings' import FlowResult from './FlowResult.svelte' + import ExpandedSubflowStep from './ExpandedSubflowStep.svelte' import type { StateStore } from '$lib/utils' import FlowSelectionPanel from './FlowSelectionPanel.svelte' import { @@ -123,7 +125,7 @@ {noEditor} disabled={disabledFlowInputs} on:openTriggers={(ev) => { - selectionManager.selectId('Trigger') + selectionManager.selectId('Trigger', { openPanel: true }) handleSelectTriggerFromKind(triggersState, triggersCount, savedFlow?.path, ev.detail.kind) showCaptureHint.set(true) }} @@ -176,9 +178,10 @@ {onDeployTrigger} /> {:else if selectedId?.startsWith('subflow:')} -
Selected step is witin an expanded subflow and is not directly editable in the flow editor
+ flowModuleSchemaMap?.reloadExpandedSubflows()} + /> {:else} {@const dup = checkDup(flowStore.val.value.modules)} {#if dup} @@ -186,12 +189,13 @@ {:else} {#key selectedId} {#each flowStore.val.value.modules as flowModule, index (flowModule.id ?? index)} + {@const slot = moduleSlot(() => flowStore.val.value.modules, flowModule.id, flowModule)} diff --git a/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte b/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte index e0f9a7d319..4af530a08a 100644 --- a/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte +++ b/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte @@ -234,6 +234,10 @@ connectProp: () => {}, propPickerConfig: writable(undefined), clearConnect: () => {}, + openPicker: () => {}, + pickerMode: () => 'pane' as const, + setPickTarget: () => {}, + onPick: () => {}, exprBeingEdited: writable([]) }) diff --git a/frontend/src/lib/components/flows/content/FlowInputs.svelte b/frontend/src/lib/components/flows/content/FlowInputs.svelte index 335a37b0c7..e275df3976 100644 --- a/frontend/src/lib/components/flows/content/FlowInputs.svelte +++ b/frontend/src/lib/components/flows/content/FlowInputs.svelte @@ -12,7 +12,7 @@ import { Check, Code, Zap } from 'lucide-svelte' import SuspendDrawer from './SuspendDrawer.svelte' import { defaultScripts } from '$lib/stores' - import { defaultScriptLanguages, processLangs } from '$lib/scripts' + import { defaultScriptLanguages, processInlineLangs } from '$lib/scripts' import type { SupportedLanguage } from '$lib/common' import DefaultScripts from '$lib/components/DefaultScripts.svelte' import type { FlowBuilderWhitelabelCustomUi } from '$lib/components/custom_ui' @@ -48,7 +48,7 @@ let filter = $state('') let langs = $derived( - processLangs(undefined, $defaultScripts?.order ?? Object.keys(defaultScriptLanguages)) + processInlineLangs(undefined, $defaultScripts?.order ?? Object.keys(defaultScriptLanguages)) .map((l) => [defaultScriptLanguages[l], l]) .filter( (x) => $defaultScripts?.hidden == undefined || !$defaultScripts.hidden.includes(x[1]) diff --git a/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte b/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte index dc2cba93d4..d7503b67a8 100644 --- a/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte +++ b/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte @@ -6,7 +6,7 @@ @@ -281,8 +322,9 @@ icon: owner.startsWith('f/') ? Folder : User, props: { width: 14, height: 14 } }} + title={owner.slice(2)} > - {owner.slice(2)} + {owner.slice(2)}
{/each} @@ -355,6 +397,7 @@ }} {label} selected={selectedByKeyboard === i} + onHover={() => hover(i)} /> {/each} {/if} @@ -395,7 +438,8 @@ {#each inlineScripts as [label, lang], i (lang)} hover(i + inlineOffset)} {enterpriseLangs} {label} lang={lang == 'docker' ? 'bash' : lang} @@ -419,13 +463,14 @@ {/each} {/if} - {#if !disableAi && funcDesc?.length > 0 && kind != 'failure' && kind != 'preprocessor' && (selectedKind == 'script' || selectedKind == 'trigger') && preFilter == 'all'} + {#if showAiRows}
  • hover(aiOffset)} on:click={() => { lang = 'bun' onGenerate() @@ -436,7 +481,8 @@ hover(aiOffset + 1)} on:click={() => { lang = 'python3' onGenerate() @@ -463,22 +509,24 @@ bind:filteredWithOwner={filteredWorkspaceItems} {filter} kind={selectedKind} - selected={selectedByKeyboard - inlineScripts?.length - aiLength - topLevelNodes.length} + selected={selectedByKeyboard - workspaceOffset} on:pickScript on:pickFlow {displayPath} {refreshCount} + onHover={(i) => hover(workspaceOffset + i)} /> {/await}
    {/if} - {#if selectedKind === 'script' && preFilter === 'all' && !selected && customUi?.aiSandbox != false} -
    AI Sandbox
    + {#if showAiSandbox} +
    {aiSandboxHeading}
    hover(aiSandboxOffset)} enterpriseLangs={[]} - label="Claude Code" + label={aiSandboxLabel} lang="claudesandbox" on:click={() => { dispatch('new', { @@ -511,15 +559,12 @@ } appFilter={selected?.name} kind={selectedKind} - selected={selectedByKeyboard - - inlineScripts?.length - - aiLength - - filteredWorkspaceItems?.length - - topLevelNodes.length} + selected={selectedByKeyboard - hubOffset} on:pickScript bind:loading {displayPath} {refreshCount} + onHover={(i) => hover(hubOffset + i)} /> {/await} {/if} diff --git a/frontend/src/lib/components/flows/content/FlowLoop.svelte b/frontend/src/lib/components/flows/content/FlowLoop.svelte index a7bbe35251..bfb80addd6 100644 --- a/frontend/src/lib/components/flows/content/FlowLoop.svelte +++ b/frontend/src/lib/components/flows/content/FlowLoop.svelte @@ -1,37 +1,28 @@ +{#snippet parallelismToggle()} + { + ;(mod.value as ForloopFlow).parallelism = detail + ? { type: 'static', value: DEFAULT_PARALLELISM } + : undefined + }} + options={{ + right: PARALLELISM_LABEL, + rightTooltip: PARALLELISM_TOOLTIP, + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/flow_loops' + }} + /> +{/snippet} + {#snippet header()}
    -
    +
    Add steps inside the loop and specify an iterator expression that defines the sequence @@ -157,7 +200,7 @@
    @@ -165,332 +208,169 @@
    {/snippet} - - - {#if mod.value.type === 'forloopflow'} -
    -
    -
    Skip failures If disabled, the flow will fail as soon as one of the iteration fail. Otherwise, - the error will be collected as the result of the iteration. Regardless of this - setting, if a flow level error handler is defined, it will process the error. - (Workspace error handlers will NOT be used to process errors if enabled.)
    + {#if mod.value.type === 'forloopflow'} + + + + {#snippet extra()} + + {/snippet} + + + +
    + {#if selectedTab === 'loop'} +
    + { + editor?.insertAtCursor(detail) + editor?.focus() + }} > + (mod.value as ForloopFlow).iterator, + (v) => { + ;(mod.value as ForloopFlow).iterator = v + } + } + argName="iterator" + label={ITERATOR_LABEL} + headerTooltip={ITERATOR_TOOLTIP} + error={iteratorMissing ? ITERATOR_MISSING : undefined} + schema={iteratorSchema} + noDynamicToggle + extraLib={stepPropPicker.extraLib} + pickableProperties={stepPropPicker.pickableProperties} + previousModuleId={previousModule?.id} + bind:suggestion + bind:focused={iteratorFieldFocused} + aiOnKeyUp={iteratorGen?.onKeyUp} + bind:editor + > + {#snippet aiGen()} + {#if enableAi} + (suggestion = e.detail || undefined)} + on:setExpr={(e) => setExpr(e.detail)} + pickableProperties={stepPropPicker.pickableProperties} + /> + {/if} + {/snippet} + + +
    +
    -
    -
    -
    Squash - - Squashing a for loop runs all iterations on the same worker, using a single runner - per step for the entire loop. This eliminates cold starts between iterations for - supported languages (Bun, Deno, and Python). - -
    { ;(mod.value as ForloopFlow).squash = detail }} - options={{ - right: 'Squash' - }} - class="whitespace-nowrap" disabled={mod.value.parallel} - /> -
    -
    -
    Run in parallel
    - { - if (detail === false) { - ;(mod.value as ForloopFlow).parallelism = undefined - } - }} options={{ - right: 'All iterations run in parallel' + title: mod.value.parallel ? SQUASH_PARALLEL_CONFLICT : undefined, + right: 'Squash', + rightTooltip: + 'Squashing a for loop runs all iterations on the same worker, using a single runner per step for the entire loop. This eliminates cold starts between iterations for supported languages (Bun, Deno, and Python).', + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/flow_loops' }} - class="whitespace-nowrap" - disabled={mod.value.squash} /> -
    -
    -
    Parallelism Assign a maximum number of branches run in parallel to control huge for-loops. -
    -
    - { - const parallelismExpr = (mod.value as ForloopFlow).parallelism - - return parallelismExpr && parallelismExpr.type === 'static' - ? parallelismExpr.value - : '' - }, - (value) => { - if (value === '' || value === null || value === undefined) { - ;(mod.value as ForloopFlow).parallelism = undefined - } else { - ;(mod.value as ForloopFlow).parallelism = { - type: 'static', - value - } - } - } - } - /> - { - const forLoopFlow = mod.value as ForloopFlow - if (e.detail == parallelismType) return - if (e.detail === 'javascript') { - if (!forLoopFlow.parallelism || forLoopFlow.parallelism.type !== 'javascript') { - ;(mod.value as ForloopFlow).parallelism = { - type: 'javascript', - expr: '' - } - } - } else { - if (!forLoopFlow.parallelism || forLoopFlow.parallelism.type !== 'static') { - ;(mod.value as ForloopFlow).parallelism = { - type: 'static', - value: 0 - } - } - } + +
    + { + // An absent `parallelism` means "no cap" to the worker, so switching + // parallelism on must not seed one — the cap below is opted into. + if (!detail) (mod.value as ForloopFlow).parallelism = undefined + }} + disabled={mod.value.squash} + options={{ + title: mod.value.squash ? SQUASH_PARALLEL_CONFLICT : undefined, + right: 'Run in parallel', + rightTooltip: 'Run the iterations concurrently instead of one after the other.', + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/flow_loops' }} - > - {#snippet children({ item })} - - - - {/snippet} - -
    -
    -
    - - {#if mod.value.type === 'forloopflow' && mod.value.parallel && mod.value.parallelism?.type == 'javascript'} -
    -
    - Parallelism expression - - JavaScript expression that defines the maximum number of parallel executions. - Example: flow_input.max_parallel || 3 - -
    -
    - -
    - { - parallelismEditor?.insertAtCursor(detail) - parallelismEditor?.focus() - }} - > - { - const parallelismExpr = (mod.value as ForloopFlow).parallelism - - return parallelismExpr && parallelismExpr.type === 'javascript' - ? parallelismExpr.expr - : '' - }, - (expr) => { - ;(mod.value as ForloopFlow).parallelism = { - type: 'javascript', - expr - } - } - } - class="h-full" - shouldBindKey={false} - extraLib={stepPropPicker.extraLib} /> - -
    - {/if} -
    -
    - Iterator expression - - The JavaScript expression that will be evaluated to get the list of items to iterate - over. Example : ["banana", "apple", flow_input.my_fruit]. - -
    - { - const config = { - onSelect: (code) => { - setExpr(code) - return true - }, - clearFocus: () => { - flowPropPickerConfig.set(undefined) - } - } - flowPropPickerConfig.set({ - ...config, - clearFocus: () => { - flowPropPickerConfig.set(undefined) - } - }) - }} - /> - {#if enableAi} - (suggestion = e.detail || undefined)} - on:setExpr={(e) => { - setExpr(e.detail) - }} - pickableProperties={stepPropPicker.pickableProperties} - /> - {/if} -
    + {#if mod.value.parallel} +
    + { + parallelismEditor?.insertAtCursor(detail) + parallelismEditor?.focus() + }} + > + + {#key parallelismCapped} + (mod.value as ForloopFlow).parallelism, + (v) => { + ;(mod.value as ForloopFlow).parallelism = v + } + } + argName="parallelism" + collapsed={!parallelismCapped} + header={parallelismToggle} + schema={parallelismSchema} + argExtra={{ min: 1, step: 1 }} + animateAppear + previousModuleId={previousModule?.id} + bind:editor={parallelismEditor} + /> + {/key} + +
    + {/if} +
    - {#if mod.value.iterator.type == 'javascript'} - -
    - { - if ($flowPropPickerConfig) { - setExpr(detail) - flowPropPickerConfig.set(undefined) - return - } - editor?.insertAtCursor(detail) - editor?.focus() - }} - noPadding - > -
    - { - iteratorFieldFocused = true - }} - on:blur={() => { - iteratorFieldFocused = false - }} - lang="javascript" - bind:code={mod.value.iterator.expr} - class="h-full" - shouldBindKey={false} - extraLib={stepPropPicker.extraLib} - {suggestion} - /> -
    -
    -
    + + {:else} -
    + {/if} +
    diff --git a/frontend/src/lib/components/flows/content/FlowModuleCache.svelte b/frontend/src/lib/components/flows/content/FlowModuleCache.svelte index 99fc4c7b3a..981e8a5aa9 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleCache.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleCache.svelte @@ -1,12 +1,12 @@ -
    - {#snippet header()} - - If defined, the result of the step will be cached for the number of seconds defined such that - if this step were to be re-triggered with the same input it would retrieve and return its - cached value instead of recomputing it. - - {/snippet} - +
    {#if flowModule.value.type == 'script'} {:else} { - if (isCacheEnabled && flowModule.cache_ttl != undefined) { + if (isCacheEnabled) { flowModule.cache_ttl = undefined } else { - flowModule.cache_ttl = 600 + flowModule.cache_ttl = stepSettingDefaults('cache') } }} options={{ - right: 'Cache the results for each possible inputs' + right: 'Cache results', + rightTooltip: + 'The result of the step is cached for the configured number of seconds; a re-trigger with the same input returns the cached value instead of recomputing it.', + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/cache' }} /> - {#if flowModule.cache_ttl} - - flowModule.cache_ignore_s3_path, - (v) => (flowModule.cache_ignore_s3_path = v || undefined) - } - options={{ - right: 'Ignore S3 Object paths for caching purposes', - rightTooltip: - 'If two S3 objects passed as input have the same content, they will hit the same cache entry, regardless of their path.' - }} - /> + {#if isCacheEnabled} +
    + + flowModule.cache_ignore_s3_path, + (v) => (flowModule.cache_ignore_s3_path = v || undefined) + } + options={{ + right: 'Ignore S3 object paths', + rightTooltip: + 'If two S3 objects passed as input have the same content, they will hit the same cache entry, regardless of their path.' + }} + /> +
    {/if} {/if} -
    +
    diff --git a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte index 75c0d03d65..d1fdf8f412 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte @@ -1,10 +1,8 @@ -
    - {#snippet header()} - - The logs, arguments and results of this flow step will be completely deleted from Windmill - after the specified delay once the flow is complete. They might be temporarily visible in UI - while the flow is running. -
    - This also applies to a flow step that has failed: the error will not be accessible. -
    -
    - The deletion is irreversible. Set to 0 for immediate deletion. - {#if disabled} -
    -
    - This option is only available on Windmill Enterprise Edition. - {/if} -
    - {/snippet} - +
    { if (enabled) { flowModule.delete_after_secs = undefined } else { - flowModule.delete_after_secs = 0 + flowModule.delete_after_secs = stepSettingDefaults('lifetime') } }} options={{ - right: 'Delete logs, arguments and results after the flow is complete' + right: 'Delete after use', + rightTooltip: tip }} /> {#if enabled} -
    +
    {/if} -
    +
diff --git a/frontend/src/lib/components/flows/content/FlowModuleEarlyStop.svelte b/frontend/src/lib/components/flows/content/FlowModuleEarlyStop.svelte index fbe747cff3..e9a824c294 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleEarlyStop.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleEarlyStop.svelte @@ -1,26 +1,43 @@ - -
- {#if !isBranchAll && !isParallelLoop} -
- {#snippet header()} - - If defined, at the end of the step, the predicate expression will be evaluated to decide - if the flow should stop early, skip rest of steps in iteration/branch if inside a parallel - for loop or branch all, or break if inside a for/while loop or branch all. - - {/snippet} + ? { + label: 'Skip rest of iteration if', + tooltip: + 'Evaluated after this step. When it returns true the remaining steps of this iteration are skipped; the other iterations are unaffected.' + } + : { + label: 'Skip rest of branch if', + tooltip: + 'Evaluated after this step. When it returns true the remaining steps of this branch are skipped; the other branches are unaffected.' + } + : { + label: 'Break parent loop if', + tooltip: `Evaluated after this step. When it returns true the enclosing loop ${breakableParent.stepId} stops iterating and the flow carries on.` + } + : { + label: 'Stop flow if', + tooltip: + "Evaluated after this step. When it returns true the flow stops here and returns this step's result." + } + ) - { - if (isStopAfterIfEnabled && flowModule.stop_after_if) { - flowModule.stop_after_if = undefined - } else { - flowModule.stop_after_if = { - expr: 'result == undefined', - skip_if_stopped: false, - error_message: undefined, - error_include_result: false + // The all-iterations predicate runs once the loop or branch-all is done, over what + // every iteration returned, so it can decide on the whole rather than on one result. + let allItersPrefix = $derived( + `Evaluated once ${isBranchAll ? 'every branch' : 'every iteration'} has completed, over their collected results.` + ) + let stopAfterAllItersCopy = $derived( + breakableParent + ? breakableParent.isParallel + ? breakableParent.type === 'loop' + ? { + label: 'Skip rest of iteration if', + tooltip: `${allItersPrefix} When it returns true the remaining steps of the enclosing iteration are skipped.` + } + : { + label: 'Skip rest of branch if', + tooltip: `${allItersPrefix} When it returns true the remaining steps of the enclosing branch are skipped.` + } + : { + label: `Break parent loop ${breakableParent.stepId} if`, + tooltip: `${allItersPrefix} When it returns true the enclosing loop stops iterating and the flow carries on.` + } + : { + label: 'Stop flow if', + tooltip: `${allItersPrefix} When it returns true the flow stops here and returns them.` + } + ) + + let earlyStopResult = $derived( + isLoop + ? Array.isArray(result) && result.length > 0 + ? result[result.length - 1] + : result === NEVER_TESTED_THIS_FAR + ? result + : undefined + : result + ) + + +{#snippet stopStatusPicker(stop: StopAfterIf)} +
+ + {#if stop.error_message != undefined} +
+ stop.error_message ?? '', (v) => (stop.error_message = String(v))} + inputProps={{ placeholder: 'Enter custom error message (optional)' }} + /> + stop.error_include_result ?? false, (v) => (stop.error_include_result = v) + } + options={{ + right: 'Include result in error', + rightTooltip: + "When enabled, this step's output is embedded inside the raised error object (as error.result) instead of being discarded. The flow result stays { error }." + }} + /> +
+ {/if} +
+{/snippet} + +{#snippet stopAfterToggle()} + { + if (isStopAfterIfEnabled && flowModule.stop_after_if) { + flowModule.stop_after_if = undefined + } else { + flowModule.stop_after_if = stepSettingDefaults('early-stop') + } + }} + options={{ + title: isParallelLoop ? stopAfterCopy.tooltip : undefined, + right: stopAfterCopy.label, + rightTooltip: stopAfterCopy.tooltip, + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/early_stop' + }} + /> +{/snippet} + +{#snippet stopAfterAllItersToggle()} + { + if (isStopAfterAllIterationsEnabled && flowModule.stop_after_all_iters_if) { + flowModule.stop_after_all_iters_if = undefined + } else { + flowModule.stop_after_all_iters_if = stepSettingDefaults('early-stop') + } + }} + options={{ + right: stopAfterAllItersCopy.label, + rightTooltip: stopAfterAllItersCopy.tooltip, + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/early_stop' + }} + /> +{/snippet} + +
+ {#if blocks !== 'all-iters' && !isBranchAll} +
+ { + stopAfterEditor?.insertAtCursor(detail) + stopAfterEditor?.focus() + }} + > + flowModule.stop_after_if, + (v) => { + flowModule.stop_after_if = v } } - }} - options={{ - right: isLoop - ? 'Break loop' - : breakableParent - ? breakableParent.isParallel - ? breakableParent.type === 'loop' - ? 'Skip rest of steps in iteration' - : 'Skip rest of steps in branch' - : 'Break parent loop module' - : 'Stop flow if condition met' - }} - /> - -
- {#if flowModule.stop_after_if} - {@const earlyStopResult = isLoop - ? Array.isArray(result) && result.length > 0 - ? result[result.length - 1] - : result === NEVER_TESTED_THIS_FAR - ? result - : undefined - : result} - {#if !breakableParent && !isLoop} -
- { - if (flowModule.stop_after_if && event.detail) { - flowModule.stop_after_if.error_message = undefined - flowModule.stop_after_if.error_include_result = false - raise_error_message_stop_after_if = false - } - }} - options={{ - right: 'Label flow as "skipped" if stopped' - }} - /> - { - if (flowModule.stop_after_if) { - if (event.detail) { - flowModule.stop_after_if.error_message = '' - flowModule.stop_after_if.skip_if_stopped = false - } else { - flowModule.stop_after_if.error_message = undefined - flowModule.stop_after_if.error_include_result = false - } - } - }} - options={{ - right: 'Raise an error message if stopped', - rightTooltip: - 'If enabled and the stop condition is met, an error message will be raised. A custom message can be provided; otherwise, a default message will be used. Mutually exclusive with "Label flow as skipped".' - }} - /> -
- {/if} - {#if raise_error_message_stop_after_if} - - - {/if} - Stop condition expression -
- { - editor?.insertAtCursor(detail) - editor?.focus() - }} - > - - -
- {:else} - {#if !breakableParent && !isLoop} -
- - -
- {/if} - Stop condition expression - - {/if} -
-
+ argName="stop_after_if" + argType="javascript" + collapsed={!isStopAfterIfEnabled || isParallelLoop} + animateAppear + header={stopAfterToggle} + noDynamicToggle + schema={predicateSchema} + previousModuleId={undefined} + pickableProperties={stepPropPicker.pickableProperties} + extraLib={`declare const result = ${JSON.stringify(earlyStopResult)};\n` + + stepPropPicker.extraLib + + (isLoop ? `\ndeclare const all_iters = ${JSON.stringify(result)};` : '')} + bind:editor={stopAfterEditor} + /> + + {#if isStopAfterIfEnabled && !breakableParent && !isLoop && flowModule.stop_after_if} + {@render stopStatusPicker(flowModule.stop_after_if)} + {/if} +
{/if} - {#if isLoop || isBranchAll} -
- {#snippet header()} - - If defined, at the end of the step, the predicate expression will be evaluated to decide - if the flow should stop early, skip rest of steps in iteration/branch if inside a parallel - for loop or branch all, or break if inside a for/while loop or branch all. - - {/snippet} - - { - if (isStopAfterAllIterationsEnabled && flowModule.stop_after_all_iters_if) { - flowModule.stop_after_all_iters_if = undefined - } else { - flowModule.stop_after_all_iters_if = { - expr: 'result == undefined', - skip_if_stopped: false, - error_message: undefined, - error_include_result: false + {#if blocks !== 'stop-after' && (isLoop || isBranchAll)} +
+ { + stopAfterAllItersEditor?.insertAtCursor(detail) + stopAfterAllItersEditor?.focus() + }} + > + flowModule.stop_after_all_iters_if, + (v) => { + flowModule.stop_after_all_iters_if = v } } - }} - options={{ - right: - (breakableParent - ? breakableParent.isParallel - ? breakableParent.type === 'loop' - ? 'Skip rest of steps in iteration' - : 'Skip rest of steps in branch' - : 'Break parent loop module ' + breakableParent.stepId - : 'Stop flow') + ' if condition met' - }} - /> - -
- {#if flowModule.stop_after_all_iters_if} - {#if !breakableParent} -
- { - if (flowModule.stop_after_all_iters_if && event.detail) { - flowModule.stop_after_all_iters_if.error_message = undefined - flowModule.stop_after_all_iters_if.error_include_result = false - raise_error_message_stop_after_all_if = false - } - }} - options={{ - right: 'Label flow as "skipped" if stopped' - }} - /> - { - if (flowModule.stop_after_all_iters_if) { - if (event.detail) { - flowModule.stop_after_all_iters_if.error_message = '' - flowModule.stop_after_all_iters_if.skip_if_stopped = false - } else { - flowModule.stop_after_all_iters_if.error_message = undefined - flowModule.stop_after_all_iters_if.error_include_result = false - } - } - }} - options={{ - right: 'Raise an error message if stopped', - rightTooltip: - 'If enabled and the stop condition is met, an error message will be raised. A custom message can be provided; otherwise, a default message will be used. Mutually exclusive with "Label flow as skipped".' - }} - /> -
- {/if} - {#if raise_error_message_stop_after_all_if} - - - {/if} - Stop condition expression -
- { - editor?.insertAtCursor(detail) - editor?.focus() - }} - > - - -
- {:else} - {#if !breakableParent} -
- - -
- {/if} - Stop condition expression - - {/if} -
-
+ argName="stop_after_all_iters_if" + argType="javascript" + collapsed={!isStopAfterAllIterationsEnabled} + animateAppear + header={stopAfterAllItersToggle} + noDynamicToggle + schema={predicateSchema} + previousModuleId={undefined} + pickableProperties={stepPropPicker.pickableProperties} + extraLib={`declare const result = ${JSON.stringify(result)};\n` + stepPropPicker.extraLib} + bind:editor={stopAfterAllItersEditor} + /> + + {#if isStopAfterAllIterationsEnabled && !breakableParent && flowModule.stop_after_all_iters_if} + {@render stopStatusPicker(flowModule.stop_after_all_iters_if)} + {/if} +
{/if}
diff --git a/frontend/src/lib/components/flows/content/FlowModuleHeader.svelte b/frontend/src/lib/components/flows/content/FlowModuleHeader.svelte index b34a78e1db..63d053fe14 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleHeader.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleHeader.svelte @@ -2,24 +2,10 @@ import Button from '$lib/components/common/button/Button.svelte' import { type FlowModule } from '$lib/gen' import { createEventDispatcher, getContext } from 'svelte' - import { - Bed, - Database, - Gauge, - GitFork, - Pen, - PhoneIncoming, - RefreshCcw, - Repeat, - Square, - Pin, - Save, - Settings - } from 'lucide-svelte' - import Popover from '../../Popover.svelte' + import { Pen, RefreshCcw, Save } from 'lucide-svelte' + import DropdownV2 from '../../DropdownV2.svelte' import type { FlowEditorContext } from '../types' import { sendUserToast } from '$lib/utils' - import { getLatestHashForScript } from '$lib/scripts' import type { FlowBuilderWhitelabelCustomUi } from '$lib/components/custom_ui' import FlowModuleWorkerTagSelect from './FlowModuleWorkerTagSelect.svelte' @@ -29,134 +15,14 @@ } let { module, tag }: Props = $props() - const { scriptEditorDrawer, workspaceScriptSettingsDrawer, flowEditorDrawer, opWorkspace } = - getContext('FlowEditorContext') + const { flowEditorDrawer } = getContext('FlowEditorContext') const dispatch = createEventDispatcher() let customUi: undefined | FlowBuilderWhitelabelCustomUi = getContext('customUi') - - let popoverClasses = - 'center-center rounded p-2 bg-blue-100 text-blue-800 border border-blue-300 hover:bg-blue-200 dark:bg-frost-700 dark:text-frost-100 dark:border-frost-600' -
- {#if module.value.type === 'script' || module.value.type === 'rawscript' || module.value.type == 'flow'} - {#if module.retry?.constant || module.retry?.exponential} - dispatch('toggleRetry')}> - - {#snippet text()} - Retries - {/snippet} - - {/if} - {#if module?.value?.['concurrent_limit'] != undefined} - dispatch('toggleConcurrency')} - > - - {#snippet text()} - Concurrency Limits - {/snippet} - - {/if} - {#if module.cache_ttl != undefined} - dispatch('toggleCache')}> - - {#snippet text()} - Cache - {/snippet} - - {/if} - {#if module.stop_after_if || module.stop_after_all_iters_if} - dispatch('toggleStopAfterIf')} - > - - {#snippet text()} - Early stop/break - {/snippet} - - {/if} - {#if module.suspend} - dispatch('toggleSuspend')}> - - {#snippet text()} - Suspend - {/snippet} - - {/if} - {#if module.sleep} - dispatch('toggleSleep')}> - - {#snippet text()} - Sleep - {/snippet} - - {/if} - {#if module.mock?.enabled} - dispatch('togglePin')}> - - {#snippet text()} - This step is pinned - {/snippet} - - {/if} - {/if} +
{#if module.value.type === 'script'} - {#if !module.value.path.startsWith('hub/') && customUi?.scriptEdit != false} - - + dispatch('createScriptFromInlineScript') + } + ]} + /> {/if}
diff --git a/frontend/src/lib/components/flows/content/FlowModuleMock.svelte b/frontend/src/lib/components/flows/content/FlowModuleMock.svelte index e3d4cb3bea..e56fee962c 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleMock.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleMock.svelte @@ -2,11 +2,11 @@ import { run } from 'svelte/legacy' import Toggle from '$lib/components/Toggle.svelte' - import Tooltip from '$lib/components/Tooltip.svelte' import type { FlowModule } from '$lib/gen' - import { Section } from '$lib/components/common' + import Label from '$lib/components/Label.svelte' import JsonEditor from '$lib/components/JsonEditor.svelte' import { untrack } from 'svelte' + import { slideDynamic } from '$lib/transitions' interface Props { flowModule: FlowModule @@ -56,47 +56,38 @@ } -
- {#snippet header()} -
- - If defined and enabled, the step will immediately return the mock value instead of being - executed. - - { - if (isMockEnabled) { - flowModule.mock = { - enabled: false, - return_value: flowModule.mock?.return_value - } - } else { - flowModule.mock = { - enabled: true, - return_value: flowModule.mock?.return_value ?? { example: 'value' } - } - code = JSON.stringify(flowModule.mock?.return_value, null, 2) - } - }} - size="xs" - /> +
+ { + if (isMockEnabled) { + flowModule.mock = { + enabled: false, + return_value: flowModule.mock?.return_value + } + } else { + flowModule.mock = { + enabled: true, + return_value: flowModule.mock?.return_value ?? { example: 'value' } + } + code = JSON.stringify(flowModule.mock?.return_value, null, 2) + } + }} + options={{ + right: 'Pin output', + rightTooltip: + 'While pinned, the step returns this value immediately instead of executing. The same control lives on the step in the graph.' + }} + /> + {#if isMockEnabled} +
+
- {/snippet} - -
- Mocked Return value - - {#if isMockEnabled} - {#key renderCount} - - {/key} - {:else} -
{flowModule.mock?.return_value
-					? JSON.stringify(flowModule.mock?.return_value, null, 2)
-					: ''}
- {/if} -
-
+ {/if} +
diff --git a/frontend/src/lib/components/flows/content/FlowModuleMockTransitionMessage.svelte b/frontend/src/lib/components/flows/content/FlowModuleMockTransitionMessage.svelte deleted file mode 100644 index 34384cc506..0000000000 --- a/frontend/src/lib/components/flows/content/FlowModuleMockTransitionMessage.svelte +++ /dev/null @@ -1,145 +0,0 @@ - - - - -
- -
-
-
NEW
- Mock has evolved into - - - PIN - -
-
- Find it in: - "Test this step" tab -
-
- - -
- - How to use the PIN feature: - - -
- -
-
1
-
-
Pick a result from history
-
- - - History picker -
-
-
- - -
-
2
-
-
Pin it as a fixed output
-
- - - Pin action -
-
-
- - -
-
-
-
The last pin can be recovered from history
-
- - - Recover pins -
-
-
- - -
-
- -
-
-
All of this can be done from the flow view
-
- - - Flow view pinning -
-
-
-
-
-
diff --git a/frontend/src/lib/components/flows/content/FlowModuleSkip.svelte b/frontend/src/lib/components/flows/content/FlowModuleSkip.svelte index ffb0a8d502..40fb32b064 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleSkip.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleSkip.svelte @@ -1,12 +1,13 @@ -
-
- {#snippet header()} - - If the condition is met, the step will behave as an identity step, passing the previous - step's result through unchanged. - - {/snippet} +{#snippet skipToggle()} + { + if (isSkipEnabled && flowModule.skip_if) { + flowModule.skip_if = undefined + } else { + flowModule.skip_if = stepSettingDefaults('skip') + } + }} + options={{ + right: 'Skip step if', + rightTooltip: + "If the condition is met, the step behaves as an identity step, passing the previous step's result through unchanged." + }} + /> +{/snippet} - { - if (isSkipEnabled && flowModule.skip_if) { - flowModule.skip_if = undefined - } else { - flowModule.skip_if = { - expr: 'false' - } +
+ { + editor?.insertAtCursor(detail) + editor?.focus() + }} + > + flowModule.skip_if, + (v) => { + flowModule.skip_if = v } - }} - options={{ - right: 'Skip step if condition is met' - }} + } + argName="skip_if" + argType="javascript" + collapsed={!isSkipEnabled} + animateAppear + header={skipToggle} + noDynamicToggle + {schema} + previousModuleId={previousModule?.id} + pickableProperties={stepPropPicker.pickableProperties} + extraLib={`declare const result = ${JSON.stringify(result)};\n` + stepPropPicker.extraLib} + bind:editor /> - -
- {#if flowModule.skip_if} - Skip condition expression -
- { - editor?.insertAtCursor(detail) - editor?.focus() - }} - > - - -
- {:else} - Skip condition expression - - {/if} -
-
+
diff --git a/frontend/src/lib/components/flows/content/FlowModuleSleep.svelte b/frontend/src/lib/components/flows/content/FlowModuleSleep.svelte index 92086ccd37..f0645c4854 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleSleep.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleSleep.svelte @@ -2,16 +2,14 @@ import InputTransformForm from '$lib/components/InputTransformForm.svelte' import type SimpleEditor from '$lib/components/SimpleEditor.svelte' import Toggle from '$lib/components/Toggle.svelte' - import Tooltip from '$lib/components/Tooltip.svelte' import type { FlowModule } from '$lib/gen' + import { stepSettingDefaults } from '../flowStepSettings' import { emptySchema } from '$lib/utils' import { getContext } from 'svelte' import PropPickerWrapper from '../propPicker/PropPickerWrapper.svelte' import type { FlowEditorContext } from '../types' - import { SecondsInput } from '../../common' - import Section from '$lib/components/Section.svelte' - import Label from '$lib/components/Label.svelte' import { getStepPropPicker } from '../previousResults' + import { slideDynamic } from '$lib/transitions' import { SAME_WORKER_INCOMPATIBLE_MSG } from '../utils.svelte' import { Alert } from '$lib/components/common' @@ -51,66 +49,53 @@ let sameWorker = $derived(Boolean(!isAgentTool && flowStore.val.value.same_worker)) -
- {#snippet header()} - - If defined, at the end of the step, the flow will sleep for a number of seconds before - scheduling the next job (if any, no effect if the step is the last one). - - {/snippet} - +
{#if sameWorker} - + {SAME_WORKER_INCOMPATIBLE_MSG} Disable `Same Worker` in the flow settings to use a sleep. {/if} - { if (isSleepEnabled && flowModule.sleep != undefined) { flowModule.sleep = undefined } else { - flowModule.sleep = { - type: 'static', - value: 0 - } + flowModule.sleep = stepSettingDefaults('sleep') } }} options={{ - right: 'Sleep after module successful execution' + right: 'Sleep after step', + rightTooltip: + 'At the end of the step, the flow sleeps for a number of seconds before scheduling the next job (no effect if the step is the last one).', + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/sleep' }} /> - -
+ {#if flowModule.sleep && schema.properties['sleep'] && !sameWorker} +
+ { + editor?.insertAtCursor(detail) + editor?.focus() + }} + > + + +
+ {/if} +
diff --git a/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte b/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte index 3a9e791d2d..43c65bc482 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte @@ -1,22 +1,23 @@ -
- {#snippet action()} - - {/snippet} - {#snippet header()} -
- - If defined, at the end of the step, the flow will be suspended until it receives external - requests to be resumed or canceled. This is most useful to implement approval steps but can - be used flexibly for other purposes. - - { - if (isSuspendEnabled && flowModule.suspend != undefined) { - flowModule.suspend = undefined - } else { - flowModule.suspend = { - required_events: 1, - timeout: 1800 - } - } - }} - options={{ - right: 'Suspend flow execution until events/approvals received' - }} - /> -
- {/snippet} +
+ { + if (isSuspendEnabled && flowModule.suspend != undefined) { + flowModule.suspend = undefined + } else { + flowModule.suspend = stepSettingDefaults('suspend') + } + }} + options={{ + right: 'Suspend until approval/resume', + rightTooltip: + 'At the end of the step, the flow is suspended until it receives external requests to resume or cancel it. Most useful for approval steps, but can be used flexibly for other purposes.', + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/flow_approval' + }} + /> -
- - - - - -
+ {#if isSuspendEnabled} +
+
+ + + + + +
- {#if suspendTabSelected === 'core'} -
- - + {#if suspendTabSelected === 'core'} +
+ + - { - if (flowModule.suspend) { - flowModule.suspend.continue_on_disapprove_timeout = e.detail - } - }} - /> - {#if Boolean(flowModule.suspend?.continue_on_disapprove_timeout)} - - We recommend using the expr resume?.error to handle null payload values. -
- To filter timeout, use resume?.error?.name === "SuspendedTimedOut".
- To filter disapproval, use resume?.error?.name === "SuspendedDisapproved" -
- {/if} -
- {:else if suspendTabSelected === 'permissions'} -
- {#if emptyString($enterpriseLicense)} - - {/if} - {#if flowModule.suspend} -
- { if (flowModule.suspend) { - flowModule.suspend.user_auth_required = e.detail - if (e.detail && flowModule.suspend?.user_groups_required === undefined) { - flowModule.suspend.user_groups_required = { - type: 'static', - value: [] + flowModule.suspend.continue_on_disapprove_timeout = e.detail + } + }} + /> + {#if Boolean(flowModule.suspend?.continue_on_disapprove_timeout)} + + We recommend using the expr resume?.error to handle null payload values. +
+ To filter timeout, use resume?.error?.name === "SuspendedTimedOut". +
+ To filter disapproval, use resume?.error?.name === "SuspendedDisapproved" +
+ {/if} +
+ {:else if suspendTabSelected === 'permissions'} +
+
+ { + if (flowModule.suspend) { + flowModule.suspend.user_auth_required = e.detail + if (e.detail && flowModule.suspend?.user_groups_required === undefined) { + flowModule.suspend.user_groups_required = { + type: 'static', + value: [] + } + } else if (!e.detail) { + flowModule.suspend.user_groups_required = undefined + flowModule.suspend.self_approval_disabled = false } - } else if (!e.detail) { - flowModule.suspend.user_groups_required = undefined - flowModule.suspend.self_approval_disabled = false } - } - }} - /> + }} + /> - { - if (flowModule.suspend) { - flowModule.suspend.self_approval_disabled = e.detail - } - }} - /> + { + if (flowModule.suspend) { + flowModule.suspend.self_approval_disabled = e.detail + } + }} + /> -
+
- {#if Boolean(flowModule.suspend.user_auth_required) && allUserGroups.length !== 0 && flowModule.suspend && schema.properties['groups']} - Require approvers to be members of one of the following user groups (leave empty for - any) - -
+ {#if Boolean(flowModule.suspend?.user_auth_required) && allUserGroups.length !== 0 && flowModule.suspend && schema.properties['groups']} + Require approvers to be members of one of the following user groups (leave empty + for any) + { editor?.insertAtCursor(detail) @@ -231,75 +237,79 @@ bind:editor /> + {/if} +
+
+ {:else} +
+ {#if flowModule?.suspend?.resume_form} +
+
+ +
+ +
+ {:else if emptyString($enterpriseLicense)} + + {:else} + + {/if} + + flowModule.suspend?.resume_form?.schema ?? draftFormSchema, + (v) => { + if (flowModule.suspend) { + flowModule.suspend.resume_form = { schema: v } + } + } + } + drawerOnly + /> + + {#if flowModule.suspend?.resume_form} + { + if (flowModule.suspend) { + flowModule.suspend.hide_cancel = e.detail + } + }} + options={{ + right: 'Hide cancel button on approval page' + }} + /> {/if}
{/if} -
- {:else} -
-
- {#if flowModule?.suspend?.resume_form} - - {:else if emptyString($enterpriseLicense)} - - {:else} -
- { - if (flowModule.suspend) { - flowModule.suspend.resume_form = { - schema: emptySchema() - } - } - jsonView = true - }} - /> -
- { - jsonView = false - if (flowModule.suspend) { - flowModule.suspend.resume_form = { - schema: e.detail - } - } - }} - schema={{}} - /> - {/if} -
-
- {#if flowModule.suspend} - {#if emptyString($enterpriseLicense)} - - {/if} -
-
- -
-
- {/if} - {#if flowModule.suspend} - - {/if} +
+
{/if} -
+
diff --git a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte index 62a3432995..ae4c614ee2 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte @@ -1,19 +1,17 @@ -
- {#snippet header()} - - If defined, the custom timeout will be used instead of the instance timeout for the step. The - step's timeout cannot be greater than the instance timeout. - - {/snippet} - +
{ if (istimeoutEnabled && flowModule.timeout != undefined) { @@ -77,39 +70,37 @@ } }} options={{ - right: 'Add a custom timeout for this step' + right: 'Custom timeout', + rightTooltip: + "The custom timeout is used instead of the instance timeout for the step. The step's timeout cannot be greater than the instance timeout." }} /> - + {#if flowModule.timeout && schema.properties['timeout']} +
+ { + editor?.insertAtCursor(detail) + editor?.focus() + }} + > + + +
+ {/if} {#if flowModule.timeout && flowModule.timeout.type !== 'static'} -
+

A dynamic timeout expression is evaluated when running the full flow. It is ignored when @@ -118,4 +109,4 @@

{/if} -
+
diff --git a/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte b/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte index 0cae21b630..6f832902d0 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte @@ -1,8 +1,11 @@ @@ -44,7 +38,6 @@ {#if isOwner !== undefined && suspendStatus} { + const r = flowModuleRetry + if (!r) return + untrack(() => { + if (delayType === 'constant' && !r.constant && r.exponential) { + delayType = 'exponential' + } else if (delayType === 'exponential' && !r.exponential && r.constant) { + delayType = 'constant' + } else if (delayType === 'disabled' && (r.constant || r.exponential)) { + // Retries added from outside: without this the row keeps reading "off" while + // rendering the incoming values greyed out, and toggling on would overwrite them. + delayType = r.constant ? 'constant' : 'exponential' + } + }) + }) + let displayRetry = $derived( + flowModuleRetry ?? { + constant: { attempts: 1, seconds: 5 }, + exponential: { attempts: 1, multiplier: 1, seconds: 5, random_factor: 0 } + } + ) + // Always-defined sub-configs so the read-only off-state can bind without + // undefined checks; when the real config exists these are the same refs. + let displayConstant = $derived(displayRetry.constant ?? { attempts: 1, seconds: 5 }) + let displayExponential = $derived( + displayRetry.exponential ?? { attempts: 1, multiplier: 1, seconds: 5, random_factor: 0 } + ) + // Only feed the preview the branch that's actually shown, so the off-state + // defaults don't render a bogus combined schedule. + let previewRetry = $derived( + retriesOff + ? displayDelayType === 'constant' + ? { constant: displayRetry.constant } + : { exponential: displayRetry.exponential } + : flowModuleRetry + ) let result = $derived( flowModule && flowStateStore?.val ? (flowStateStore.val[flowModule.id]?.previewResult ?? NEVER_TESTED_THIS_FAR) @@ -99,10 +157,12 @@ } function initialLoad() { + // Presence, not attempts > 0: a retry block with zero attempts is still + // configured, and flowStepSettings describes it that way. delayType = - (flowModuleRetry?.constant?.attempts ?? 0) > 0 + flowModuleRetry?.constant != undefined ? 'constant' - : (flowModuleRetry?.exponential?.attempts ?? 0) > 0 + : flowModuleRetry?.exponential != undefined ? 'exponential' : 'disabled' loaded = true @@ -122,258 +182,288 @@ const u32Max = 4294967295 -
+
{#if sameWorker} {SAME_WORKER_INCOMPATIBLE_MSG} Disable `Same Worker` in the flow settings to use retries. {/if} - - { - flowModuleRetry = undefined - if (e.detail === 'constant') { + checked={delayType === 'constant' || delayType === 'exponential'} + on:change={() => { + if (delayType === 'constant' || delayType === 'exponential') { flowModuleRetry = undefined + delayType = 'disabled' + } else { setConstantRetries() - } else if (e.detail === 'exponential') { - flowModuleRetry = undefined - setExponentialRetries() + delayType = 'constant' } }} - > - {#snippet children({ item })} - - - - {/snippet} - + options={{ + right: 'Retry on failure', + rightTooltip: + 'Upon error this step is retried with a delay and a maximum number of attempts as defined below.', + rightDocumentationLink: 'https://www.windmill.dev/docs/flows/retries' + }} + /> - {#if (delayType === 'constant' || delayType === 'exponential') && !sameWorker} -
- {#snippet header()} - - Optional condition to determine when to retry. If not specified, will retry on any failure - within the configured attempt limits. - - {/snippet} - - { - if (!flowModuleRetry) { - return + {#if !retriesOff && !sameWorker} +
+ { + flowModuleRetry = undefined + if (e.detail === 'constant') { + setConstantRetries() + delayType = 'constant' + } else if (e.detail === 'exponential') { + setExponentialRetries() + delayType = 'exponential' } - if (isRetryConditionEnabled && flowModuleRetry.retry_if) { - const { retry_if, ...rest } = flowModuleRetry - flowModuleRetry = rest - } else { - flowModuleRetry = { - ...flowModuleRetry, - retry_if: { - expr: 'error && error.name !== "PERMANENT_FAILURE"' + }} + > + {#snippet children({ item })} + + + {/snippet} + + + {#snippet retryConditionToggle()} + { + if (!flowModuleRetry) { + return + } + if (isRetryConditionEnabled && flowModuleRetry.retry_if) { + const { retry_if, ...rest } = flowModuleRetry + flowModuleRetry = rest + } else { + flowModuleRetry = { + ...flowModuleRetry, + retry_if: { + expr: 'error && error.name !== "PERMANENT_FAILURE"' + } } } - } - }} - options={{ - right: 'Only retry if condition is met' - }} - /> + }} + options={{ + right: 'Conditional retry', + rightTooltip: + 'Optional condition to determine when to retry. Expression should return true to retry, false to skip retry. If not specified, retries on any failure within the configured attempt limits.' + }} + /> + {/snippet} -
- {#if flowModuleRetry?.retry_if} - Retry condition expression - Expression should return true to retry, false to skip retry -
- {#if stepPropPicker} - { - editor?.insertAtCursor(detail) - editor?.focus() - }} - > - - - {:else} + {#if stepPropPicker} + { + retryIfEditor?.insertAtCursor(detail) + retryIfEditor?.focus() + }} + > + flowModuleRetry?.retry_if, + (v) => { + if (flowModuleRetry) flowModuleRetry.retry_if = v + } + } + argName="retry_if" + argType="javascript" + collapsed={!isRetryConditionEnabled} + animateAppear + header={retryConditionToggle} + noDynamicToggle + schema={predicateSchema} + previousModuleId={undefined} + pickableProperties={stepPropPicker.pickableProperties} + extraLib={`declare const result = ${JSON.stringify(result)};` + + `\ndeclare const flow_input = ${JSON.stringify(stepPropPicker.pickableProperties.flow_input || {})};`} + bind:editor={retryIfEditor} + /> + + {:else} +
+ {@render retryConditionToggle()} + {#if flowModuleRetry?.retry_if} +
- {/if} -
- {:else} - Retry condition expression - Expression should return true to retry, false to skip retry - - {/if} -
-
- {/if} +
+ {/if} +
+ {/if} - {#if (delayType === 'constant' || delayType === 'exponential') && !sameWorker} -
-
- {#if delayType === 'constant'} - {#if flowModuleRetry?.constant} -
Attempts
-
+
+
+ {#if displayDelayType === 'constant'} +
Attempts
+
+ + +
+
Delay
+ + {:else if displayDelayType === 'exponential'} +
Attempts
+
+ + +
+
Multiplier
+ delay = multiplier * base ^ (number of attempt) - -
-
Delay
- - {/if} - {:else if delayType === 'exponential'} - {#if flowModuleRetry?.exponential} -
Attempts
-
- - -
-
Multiplier
- delay = multiplier * base ^ (number of attempt) - -
Base (in seconds)
- - {#if validationError} - {validationError} - {:else} - Must be ≥ 1. A base of 0 would cause immediate retries. - {/if} -
Randomization factor (percentage)
-
- {#if !$enterpriseLicense} - - {/if} +
Base (in seconds)
-
+ {#if validationError} + {validationError} + {:else} + Must be ≥ 1. A base of 0 would cause immediate retries. + {/if} +
Randomization factor (percentage)
+
+ {#if !$enterpriseLicense} + + {/if} +
+ +
-
- {/if} - {/if} -
-
- {#if true} - {@const { attempts: cAttempts, seconds: cSeconds } = flowModuleRetry?.constant || {}} - {@const { - attempts: eAttempts, - seconds: eSeconds, - multiplier, - random_factor - } = flowModuleRetry?.exponential || {}} - {@const cArray = Array.from({ length: Math.min(cAttempts || 0, 100) }, () => cSeconds)} - {@const eArray = Array.from( - { length: Math.min(eAttempts || 0, 100) }, - (_, i) => (multiplier || 0) * (eSeconds || 0) ** (i + cArray.length + 1) - )} - {@const array = [...cArray, ...eArray]} -
-
Retry attempts
- {#if array.length > 0} - - - - - - - - - {#each array.slice(1, 100) as delay, i} - {@const index = i + 2} - - - -
1:After {array[0]} second{array[0] === 1 ? '' : 's'} - {#if (random_factor ?? 0) > 0}(+/- {((array[0] ?? 0) * (random_factor ?? 0)) / - 100} - seconds){/if}
{index}: - {delay} second{delay === 1 ? '' : 's'} - {#if (random_factor ?? 0) > 0}(+/- {((delay ?? 0) * (random_factor ?? 0)) / - 100} - seconds){/if} - after attempt #{index - 1} - {#if i > cArray.length - 2} - - ({multiplier} * {eSeconds}{index}) - + {/if} + +
+ {#if true} + {@const { attempts: cAttempts, seconds: cSeconds } = previewRetry?.constant || {}} + {@const { + attempts: eAttempts, + seconds: eSeconds, + multiplier, + random_factor + } = previewRetry?.exponential || {}} + {@const cArray = Array.from({ length: Math.min(cAttempts || 0, 100) }, () => cSeconds)} + {@const eArray = Array.from( + { length: Math.min(eAttempts || 0, 100) }, + (_, i) => (multiplier || 0) * (eSeconds || 0) ** (i + cArray.length + 1) + )} + {@const array = [...cArray, ...eArray]} +
+
Retry attempts
+ {#if array.length > 0} + + + + + + + + + {#each array.slice(1, 100) as delay, i} + {@const index = i + 2} + + + + + {/each} + {#if (cAttempts ?? 0) > 100 || (eAttempts ?? 0) > 100} + + + + {/if} - - - {/each} - {#if (cAttempts ?? 0) > 100 || (eAttempts ?? 0) > 100} - - - - + +
1:After {array[0]} second{array[0] === 1 ? '' : 's'} + {#if (random_factor ?? 0) > 0}(+/- {((array[0] ?? 0) * + (random_factor ?? 0)) / + 100} + seconds){/if}
{index}: + {delay} second{delay === 1 ? '' : 's'} + {#if (random_factor ?? 0) > 0}(+/- {((delay ?? 0) * + (random_factor ?? 0)) / + 100} + seconds){/if} + after attempt #{index - 1} + {#if i > cArray.length - 2} + + ({multiplier} * {eSeconds}{index}) + + {/if} +
......
......
{/if} -
- {/if} +
+ {/if} +
- {/if}
-
{/if}
diff --git a/frontend/src/lib/components/flows/content/FlowRunSettings.svelte b/frontend/src/lib/components/flows/content/FlowRunSettings.svelte new file mode 100644 index 0000000000..a1dae0f3a4 --- /dev/null +++ b/frontend/src/lib/components/flows/content/FlowRunSettings.svelte @@ -0,0 +1,385 @@ + + +{#snippet sectionHeader(title: string)} +
+ {title} +
+{/snippet} + +
+ {#if !isFailure} +
+ {@render sectionHeader('Flow control')} + +
+
+ +
+ +
+ +
+ +
+ +
+ +
+ +
+
+ {/if} + +
+ {@render sectionHeader('Execution policy')} + +
+ {#if !loopSubset} +
+ +
+ +
+ + {#if suspendNeedsItsOwnContinueToggle} + + This only applies when the step's own code fails. A disapproval or an approval timeout + is not a failure of this step, so it still stops the flow. To continue past those, + turn on "Continue on disapproval/timeout" in the approval settings. + + {/if} +
+ + + {#if !isFailure} +
+ +
+ + {#if isRawScript || isWorkspaceScript} +
+ {#if flowModule.value.type === 'script'} + + {:else if flowModule.value.type === 'rawscript'} +
+ { + if (flowModule.value.type !== 'rawscript') return + flowModule.value.concurrent_limit = concurrencyOn ? undefined : 1 + }} + options={{ + right: 'Concurrency limit', + rightTooltip: 'Allowed concurrency within a given timeframe.', + rightDocumentationLink: + 'https://www.windmill.dev/docs/flows/concurrency_limit' + }} + /> + {#if concurrencyOn} +
+ + + +
+ {/if} +
+ {/if} +
+ {/if} + +
+ { + if (flowModule.priority !== undefined) { + flowModule.priority = undefined + } else { + flowModule.priority = stepSettingDefaults('priority') + } + }} + options={{ + right: 'High priority', + rightTooltip: + 'Jobs scheduled from this step take precedence over other jobs in the queue when the flow runs.' + }} + /> + {#if flowModule.priority !== undefined} +
+ +
+ {/if} + + {#if isCloudHosted()} + + Setting priority is not available on the cloud. + + {/if} +
+ +
+ +
+ +
+ +
+ {/if} + {/if} + + {#if !isFailure} +
+ +
+ {/if} + + {#if loopSubset} +
+ +
+ {/if} +
+ + {#if s3Language && onApplyS3Snippet && !isFailure} +
+ {@render sectionHeader('S3 snippets')} +

+ Read and write S3 objects, and use Polars or DuckDB to run efficient ETL processes. +

+
+ + {#snippet children({ item })} + {#if s3Language === 'deno'} + + {:else} + + + + {/if} + {/snippet} + + +
+ {#if s3Snippet} +
+ +
+ {/if} +
+ {/if} +
diff --git a/frontend/src/lib/components/flows/content/FlowSettings.svelte b/frontend/src/lib/components/flows/content/FlowSettings.svelte index c63921d989..8e78a04269 100644 --- a/frontend/src/lib/components/flows/content/FlowSettings.svelte +++ b/frontend/src/lib/components/flows/content/FlowSettings.svelte @@ -47,6 +47,7 @@ customUi, preserveOnBehalfOf, savedOnBehalfOfEmail, + savedOnBehalfOfPermissionedAs, opWorkspace } = getContext('FlowEditorContext') @@ -55,6 +56,7 @@ let canPreserve = $derived(!!$userStore?.is_admin || !!$userStore?.is_super_admin || isDeployer) let onBehalfOfChoice: OnBehalfOfChoice = $state(undefined) let customOnBehalfOfEmail: string = $state('') + let myPermissionedAs = $derived($userStore?.username ? `u/${$userStore.username}` : undefined) function asSchema(x: any) { return x as Schema @@ -114,7 +116,7 @@
-
+
diff --git a/frontend/src/lib/components/flows/content/GenAiQuick.svelte b/frontend/src/lib/components/flows/content/GenAiQuick.svelte index e0e681d568..5d3814fb63 100644 --- a/frontend/src/lib/components/flows/content/GenAiQuick.svelte +++ b/frontend/src/lib/components/flows/content/GenAiQuick.svelte @@ -1,17 +1,18 @@ -
- - {#snippet children()} -

- MCP clients allow AI agents to access and execute a list of tools made available by an MCP - server. -
- Choose an MCP resource to make its tools available to the agent. -
-
- Note: Only HTTP streamable MCP servers are supported. -

- {/snippet} -
+ +
+ + {#snippet children()} +

+ MCP clients allow AI agents to access and execute a list of tools made available by an MCP + server. +
+ Choose an MCP resource to make its tools available to the agent. +
+
+ Note: Only HTTP streamable MCP servers are supported. +

+ {/snippet} +
-
- -
- - {#if !resourcePath} - {#if !showOAuthForm} - - {:else} - (showOAuthForm = false)} - /> - {/if} - {/if} - - {#if resourcePath?.length > 0}
-
-
- {#snippet action()} - - {/snippet} -
- {#if error} -
{`Failed to load tools from MCP server: ${error}`}
- {:else if tools.status === 'loading'} -
-
Loading tools...
-
- {:else if (tools.value ?? []).length === 0 && !error} -
-
- No tools loaded yet. Click "Refresh Tools" to fetch tools from the MCP server. -
-
- {:else if (tools.value ?? []).length > 0} -
-
- {#each tools.value ?? [] as mcpTool} -
- {mcpTool.name} - {#if mcpTool.description} - — {mcpTool.description} - {/if} -
- {/each} -
-
- {/if} -
-
+ {:else} + (showOAuthForm = false)} + /> + {/if} + {/if} - {#if tool.value.include_tools && tool.value.exclude_tools} -
-
-
- -
-
- -
+ {#if resourcePath?.length > 0} +
+ +
+ +
+ {#snippet action()} + + {/snippet} +
+ {#if error} +
{`Failed to load tools from MCP server: ${error}`}
+ {:else if tools.status === 'loading'} +
+
Loading tools...
+
+ {:else if (tools.value ?? []).length === 0 && !error} +
+
+ No tools loaded yet. Click "Refresh Tools" to fetch tools from the MCP server. +
+
+ {:else if (tools.value ?? []).length > 0} +
+
+ {#each tools.value ?? [] as mcpTool} +
+ {mcpTool.name} + {#if mcpTool.description} + — {mcpTool.description} + {/if} +
+ {/each} +
+
+ {/if}
+ + {#if tool.value.include_tools && tool.value.exclude_tools} +
+
+
+ +
+
+ +
+
+
+ {/if} {/if} - {/if} -
+
+
diff --git a/frontend/src/lib/components/flows/content/StepSettingsBadges.svelte b/frontend/src/lib/components/flows/content/StepSettingsBadges.svelte new file mode 100644 index 0000000000..b92a8a05a5 --- /dev/null +++ b/frontend/src/lib/components/flows/content/StepSettingsBadges.svelte @@ -0,0 +1,30 @@ + + +{#if configured.length > 0} +
+ {#each configured as s (s.key)} + {@const Icon = s.icon} + + + {#snippet text()} + {s.tooltip} + · {s.summary.text} + {/snippet} + + {/each} +
+{/if} diff --git a/frontend/src/lib/components/flows/content/SuspendDrawer.svelte b/frontend/src/lib/components/flows/content/SuspendDrawer.svelte index 3ac2afe001..e351c2812c 100644 --- a/frontend/src/lib/components/flows/content/SuspendDrawer.svelte +++ b/frontend/src/lib/components/flows/content/SuspendDrawer.svelte @@ -16,21 +16,26 @@
- To add a form, go to the Form tab, inside the Advanced {'->'} Suspend tab, and add a form. - You can then get back the payloads using `resume` (single approver), or `resumes` (multiple approvers) - in the next step. Forms are an EE feature only. The approver list itself is fetchable using `approvers` + To add a form, open the Form tab of this Suspend until approval/resume setting + and click + Add a form. You can then get back the payloads using `resume` (single approver), or + `resumes` (multiple approvers) in the next step. Forms are an EE feature only. The approver + list itself is fetchable using `approvers`
A prompt is simply an approval step that can be self-approved. To do this, include the diff --git a/frontend/src/lib/components/flows/content/WebsearchToolDisplay.svelte b/frontend/src/lib/components/flows/content/WebsearchToolDisplay.svelte index 77e4d07c61..2d679ff665 100644 --- a/frontend/src/lib/components/flows/content/WebsearchToolDisplay.svelte +++ b/frontend/src/lib/components/flows/content/WebsearchToolDisplay.svelte @@ -1,10 +1,15 @@ -
- - Gives the AI Agent the ability to search the web. Only works for openai, anthropic and google - models for now. - -
+ +
+ + Gives the AI Agent the ability to search the web. Only works for openai, anthropic and google + models for now. + +
+
diff --git a/frontend/src/lib/components/flows/expandedSubflowStep.test.ts b/frontend/src/lib/components/flows/expandedSubflowStep.test.ts new file mode 100644 index 0000000000..33b5515850 --- /dev/null +++ b/frontend/src/lib/components/flows/expandedSubflowStep.test.ts @@ -0,0 +1,52 @@ +import { describe, it, expect } from 'vitest' +import type { FlowModule } from '$lib/gen' +import { expandedSubflowParentId, resolveExpandedSubflowStep } from './expandedSubflowStep' + +const script = (id: string): FlowModule => + ({ id, value: { type: 'rawscript', language: 'bun', content: '', input_transforms: {} } }) as any +const subflow = (id: string, path: string): FlowModule => + ({ id, value: { type: 'flow', path } }) as any +const forloop = (id: string, modules: FlowModule[]): FlowModule => + ({ id, value: { type: 'forloopflow', modules, iterator: { type: 'static', value: [] } } }) as any + +// `outer` inlines `f/inner`, which inlines `f/innermost`; each subflow step also sits +// inside a for loop, which the graph node id does not record. +const flows: Record = { + 'f/inner': [forloop('l', [subflow('b', 'f/innermost')])], + 'f/innermost': [forloop('m', [script('leaf')])] +} +const rootModules = [forloop('k', [subflow('a', 'f/inner')])] +const loadFlowModules = async (path: string) => flows[path] + +describe('expandedSubflowParentId', () => { + // The last segment of `subflowSteps` is the parent's own step, so it belongs to the + // parent key: `subflow:a:b` sits in expansion `a`, not in itself. + it('is the enclosing expansion, not the expansion itself', () => { + expect(expandedSubflowParentId('subflow:a:b')).toBe('a') + expect(expandedSubflowParentId('subflow:a:b:c')).toBe('subflow:a:b') + expect(expandedSubflowParentId('a')).toBeUndefined() + }) +}) + +describe('resolveExpandedSubflowStep', () => { + it('follows every subflow boundary down to the selected step', async () => { + const resolved = await resolveExpandedSubflowStep( + 'subflow:a:b:leaf', + rootModules, + loadFlowModules + ) + expect(resolved?.pathChain).toEqual(['f/inner', 'f/innermost']) + expect(resolved?.containingFlowPath).toBe('f/innermost') + expect(resolved?.module?.id).toBe('leaf') + }) + + it('still reports the containing flow when the step is not one of its modules', async () => { + const resolved = await resolveExpandedSubflowStep( + 'subflow:a:b:leaf-tool-0', + rootModules, + loadFlowModules + ) + expect(resolved?.containingFlowPath).toBe('f/innermost') + expect(resolved?.module).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/flows/expandedSubflowStep.ts b/frontend/src/lib/components/flows/expandedSubflowStep.ts new file mode 100644 index 0000000000..0075897b09 --- /dev/null +++ b/frontend/src/lib/components/flows/expandedSubflowStep.ts @@ -0,0 +1,62 @@ +import type { FlowModule } from '$lib/gen' +import { findStepPath, parseExpandedSubflowId } from '$lib/components/restartFromStepPath' + +export type ResolvedExpandedSubflowStep = { + /** The flow the selected step lives in, i.e. the innermost subflow crossed. */ + containingFlowPath: string + /** Path of each subflow crossed from the edited flow down to the containing one. */ + pathChain: string[] + /** The selected step, or `undefined` when the id points at something that isn't a + * module of the containing flow (an AI agent tool node, a stale selection...). */ + module: FlowModule | undefined +} + +/** + * Graph node id of the expansion an inline-expanded subflow node sits in, i.e. the key + * whose modules hold the step it stands for: `subflow:a:b` → `a`, `subflow:a:b:c` → + * `subflow:a:b`. `undefined` for a top-level expansion, whose step belongs to the edited + * flow itself. Note that the last segment of `subflowSteps` is the parent's own step, so + * every segment stays in the parent key. + */ +export function expandedSubflowParentId(nodeId: string): string | undefined { + const steps = parseExpandedSubflowId(nodeId)?.subflowSteps + if (!steps) { + return undefined + } + return steps.length > 1 ? 'subflow:' + steps.join(':') : steps[0] +} + +/** + * Resolves a `subflow:A:B:leaf` graph node id (an inline-expanded subflow step) to the + * step it stands for, following each `Flow{path}` boundary through the flows it points + * at. `loadFlowModules` fetches a flow's modules by path. + * + * Returns `undefined` when `id` is not an expanded-subflow node or a boundary can't be + * followed; a resolution whose `module` is undefined still carries the containing flow, + * which is enough to open it in the editor. + */ +export async function resolveExpandedSubflowStep( + id: string, + rootModules: FlowModule[], + loadFlowModules: (path: string) => Promise +): Promise { + const parsed = parseExpandedSubflowId(id) + if (!parsed) { + return undefined + } + let modules = rootModules + const pathChain: string[] = [] + for (const stepId of parsed.subflowSteps) { + const value = findStepPath(modules, stepId)?.target.value + if (value?.type !== 'flow') { + return undefined + } + pathChain.push(value.path) + modules = await loadFlowModules(value.path) + } + return { + containingFlowPath: pathChain[pathChain.length - 1], + pathChain, + module: findStepPath(modules, parsed.leaf)?.target + } +} diff --git a/frontend/src/lib/components/flows/flowDeleteController.test.ts b/frontend/src/lib/components/flows/flowDeleteController.test.ts index b029ce702f..0a37e1ec7c 100644 --- a/frontend/src/lib/components/flows/flowDeleteController.test.ts +++ b/frontend/src/lib/components/flows/flowDeleteController.test.ts @@ -134,7 +134,11 @@ describe('flowDeleteController', () => { expect(flowStore.val.value.modules.map((module) => module.id)).toEqual(['dependent_step']) expect(flowStore.val.value.groups ?? []).toEqual([]) expect(Object.keys(flowStateStore.val)).toEqual(['dependent_step']) - expect(selectionManager.selectId).toHaveBeenCalledWith('dependent_step') + // The surviving step is selected as a side effect of the delete, so it must be + // marked as such — otherwise the modal step panel pops open on its own. + expect(selectionManager.selectId).toHaveBeenCalledWith('dependent_step', { + openPanel: false + }) expect(onDelete).toHaveBeenCalledWith('agent_step') }) }) diff --git a/frontend/src/lib/components/flows/flowDeleteController.ts b/frontend/src/lib/components/flows/flowDeleteController.ts index 5fea6a4cc3..a1933ba080 100644 --- a/frontend/src/lib/components/flows/flowDeleteController.ts +++ b/frontend/src/lib/components/flows/flowDeleteController.ts @@ -56,7 +56,9 @@ export function executeDeletePlan( if (plan.selection.kind === 'clear') { args.selectionManager.clearSelection() } else { - args.selectionManager.selectId(plan.selection.id) + // Whatever remains selected after a delete was not asked for, so it must not + // pop the modal panel open. + args.selectionManager.selectId(plan.selection.id, { openPanel: false }) } if (plan.targets.some((target) => target.kind === 'preprocessor')) { diff --git a/frontend/src/lib/components/flows/flowEditorTelemetry.test.ts b/frontend/src/lib/components/flows/flowEditorTelemetry.test.ts new file mode 100644 index 0000000000..22174c7e14 --- /dev/null +++ b/frontend/src/lib/components/flows/flowEditorTelemetry.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from 'vitest' +import { + createBreakpointTracker, + forcedPlacementEvent, + type FlowPanelPlacementEvent +} from './flowEditorTelemetry' + +describe('createBreakpointTracker', () => { + it('counts a crossing into modal once, however many times the width settles', () => { + const events: FlowPanelPlacementEvent[] = [] + const tracker = createBreakpointTracker((event) => events.push(event)) + + tracker.observe('auto', 'docked', true) + tracker.observe('auto', 'modal', true) + // A window drag re-resolves the mode continuously; the breakpoint activated once. + tracker.observe('auto', 'modal', true) + tracker.observe('auto', 'docked', true) + tracker.observe('auto', 'modal', true) + + expect(events).toEqual(['breakpoint_modal', 'breakpoint_modal']) + }) + + it('does not re-count when a pin is released back to auto without moving the panel', () => { + const events: FlowPanelPlacementEvent[] = [] + const tracker = createBreakpointTracker((event) => events.push(event)) + + tracker.observe('auto', 'modal', true) + // Pinning Detached on a narrow editor, then releasing it, leaves the panel modal + // throughout — re-arming on the preference alone would read that as a second crossing. + tracker.observe('modal', 'modal', true) + tracker.observe('auto', 'modal', true) + + expect(events).toEqual(['breakpoint_modal']) + }) + + it('does not re-count when a remount reports an unmeasured editor', () => { + const events: FlowPanelPlacementEvent[] = [] + const tracker = createBreakpointTracker((event) => events.push(event)) + + tracker.observe('auto', 'modal', true) + // A reload rebuilds the controller, which resolves to docked until its width lands. + // Reading that as the panel having docked would arm the tracker for a second crossing + // the user never made. + tracker.observe('auto', 'docked', false) + tracker.observe('auto', 'modal', true) + + expect(events).toEqual(['breakpoint_modal']) + }) + + it('ignores a modal the user pinned', () => { + const events: FlowPanelPlacementEvent[] = [] + const tracker = createBreakpointTracker((event) => events.push(event)) + + tracker.observe('modal', 'modal', true) + tracker.observe('docked', 'docked', true) + + expect(events).toEqual([]) + }) +}) + +describe('forcedPlacementEvent', () => { + it('reports only the pins that move the panel', () => { + expect(forcedPlacementEvent('docked', 'modal')).toBe('force_attach') + expect(forcedPlacementEvent('modal', 'docked')).toBe('force_detach') + // Already where it is pinned to: a stated preference, not an override. + expect(forcedPlacementEvent('docked', 'docked')).toBeUndefined() + expect(forcedPlacementEvent('modal', 'modal')).toBeUndefined() + expect(forcedPlacementEvent('auto', 'modal')).toBeUndefined() + expect(forcedPlacementEvent('auto', 'docked')).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/flows/flowEditorTelemetry.ts b/frontend/src/lib/components/flows/flowEditorTelemetry.ts new file mode 100644 index 0000000000..3d826dc8d9 --- /dev/null +++ b/frontend/src/lib/components/flows/flowEditorTelemetry.ts @@ -0,0 +1,94 @@ +import { getContext, setContext } from 'svelte' +import { logFeatureUsage } from '$lib/utils/featureUsage' +import type { FlowPanelMode, FlowPanelPreference } from './panelPlacement' + +// Anonymous counters for where the flow editor's step panel ends up. Same rules as every +// other `logFeatureUsage` caller: aggregated counts only, and the three keys below are the +// whole vocabulary — no path, expression or step id ever reaches here. + +const FEATURE = 'flow_editor' +const KIND = 'panel_placement' + +export type FlowPanelPlacementEvent = + /** The width moved the panel into the modal, under `auto`. */ + | 'breakpoint_modal' + /** The user pinned the panel into the pane while it was in the modal. */ + | 'force_attach' + /** The user pinned the panel out into the modal while it was in the pane. */ + | 'force_detach' + +type Log = (event: FlowPanelPlacementEvent) => void + +const log: Log = (event) => logFeatureUsage(FEATURE, KIND, { key: event }) + +/** + * The event a placement pin should produce, or nothing. `auto` is not a placement being + * forced, and a pin that matches where the panel already is moves nothing — the counters + * carry no width, so counting that would be indistinguishable from the override that did + * move the panel. + */ +export function forcedPlacementEvent( + preference: FlowPanelPreference, + mode: FlowPanelMode +): FlowPanelPlacementEvent | undefined { + if (preference === mode) return undefined + if (preference === 'docked') return 'force_attach' + if (preference === 'modal') return 'force_detach' + return undefined +} + +/** + * Counts the width moving the panel into the modal, once per crossing — `mode` re-resolves + * continuously as a drag settles, so counting per evaluation would read one drag as hundreds. + * + * Both guards below are there because a modal panel is not by itself a crossing: `wasModal` + * follows where the panel was rather than whether the breakpoint put it there, and an + * unmeasured editor is no placement at all. Rune-free so each edge is testable directly. + */ +export function createBreakpointTracker(emit: Log) { + let wasModal = false + + return { + observe(preference: FlowPanelPreference, mode: FlowPanelMode, measured: boolean) { + if (!measured) return + if (mode === 'modal' && !wasModal && preference === 'auto') emit('breakpoint_modal') + wasModal = mode === 'modal' + } + } +} + +export interface FlowPanelPlacementTelemetry { + /** The panel's current placement; emits `breakpoint_modal` on a crossing into the modal. */ + observe(preference: FlowPanelPreference, mode: FlowPanelMode, measured: boolean): void + /** A placement the user pinned, against where the panel was when they pinned it. */ + forced(preference: FlowPanelPreference, mode: FlowPanelMode): void +} + +const CONTEXT_KEY = 'flowPanelPlacementTelemetry' + +const NOOP: FlowPanelPlacementTelemetry = { observe: () => {}, forced: () => {} } + +/** + * Published by `FlowBuilder`, above the `{#key}` that rebuilds the editor on a reload: a + * tracker recreated mid-edit would re-arm and count a still-narrow editor again. + * + * `enabled` is false in session preview tabs. Those stay mounted and laid out at panel width + * even while hidden, and that panel is narrower than the breakpoint by construction: their + * crossings would bury the ones this measures, and their overrides would then be read + * against a denominator that no longer contains them. + */ +export function setFlowPanelPlacementTelemetry(enabled: boolean): void { + const emit: Log = enabled ? log : () => {} + const tracker = createBreakpointTracker(emit) + setContext(CONTEXT_KEY, { + observe: tracker.observe, + forced: (preference, mode) => { + const event = forcedPlacementEvent(preference, mode) + if (event) emit(event) + } + }) +} + +export function useFlowPanelPlacementTelemetry(): FlowPanelPlacementTelemetry { + return getContext(CONTEXT_KEY) ?? NOOP +} diff --git a/frontend/src/lib/components/flows/flowPanelMode.svelte.ts b/frontend/src/lib/components/flows/flowPanelMode.svelte.ts new file mode 100644 index 0000000000..1b595c53ea --- /dev/null +++ b/frontend/src/lib/components/flows/flowPanelMode.svelte.ts @@ -0,0 +1,35 @@ +import { resolvePanelMode, type FlowPanelMode, type FlowPanelPreference } from './panelPlacement' + +/** + * Holds the step panel's placement preference and the editor's measured width, and reads + * the resolution off `resolvePanelMode`. The preference is not persisted: it lasts as long + * as the editor is open, so every flow opens on `auto` and a pin is a deliberate act each + * time. + */ +export function useFlowPanelMode(opts: { enabled: () => boolean }) { + let preference = $state('auto') + let width = $state(0) + + return { + get preference(): FlowPanelPreference { + return preference + }, + set preference(next: FlowPanelPreference) { + preference = next + }, + get mode(): FlowPanelMode { + return resolvePanelMode({ enabled: opts.enabled(), preference, width }) + }, + /** + * Whether `mode` reflects a real layout. Until the first measurement lands, `mode` is + * `docked` because that is the safe thing to render, not because the editor is wide. + */ + get measured(): boolean { + return width > 0 + }, + /** Fed by the editor root's measured width; drives `auto` in both directions. */ + measure(measured: number | null | undefined) { + width = measured ?? 0 + } + } +} diff --git a/frontend/src/lib/components/flows/flowStepSettings.test.ts b/frontend/src/lib/components/flows/flowStepSettings.test.ts new file mode 100644 index 0000000000..7ecb138214 --- /dev/null +++ b/frontend/src/lib/components/flows/flowStepSettings.test.ts @@ -0,0 +1,140 @@ +import { describe, it, expect } from 'vitest' +import type { FlowModule } from '$lib/gen' +import { describeStepSettings, hasInlineConcurrency } from './flowStepSettings' + +const stepSettingsByKey = (...args: Parameters) => + Object.fromEntries(describeStepSettings(...args).map((v) => [v.key, v])) + +function step(overrides: Partial = {}): FlowModule { + return { + id: 'a', + value: { type: 'rawscript', language: 'bun', content: '', input_transforms: {} }, + ...overrides + } as FlowModule +} + +describe('describeStepSettings', () => { + it('reports an untouched step as configured nowhere', () => { + expect(describeStepSettings(step()).filter((s) => s.configured)).toEqual([]) + }) + + it('treats a sleep of 0 as configured and says so, rather than claiming it is off', () => { + // Switching Sleep on seeds `{ value: 0 }`; the row must not contradict the toggle. + const s = stepSettingsByKey(step({ sleep: { type: 'static', value: 0 } }))['sleep'] + expect(s?.configured).toBe(true) + expect(s?.summary.text).toBe('0s after') + expect(s?.summary.state).toBe('configured') + }) + + it('does not let an empty stop_after_if mask a configured stop_after_all_iters_if', () => { + // Both can be set on a sequential loop. + const s = stepSettingsByKey( + step({ + stop_after_if: { expr: '', skip_if_stopped: false }, + stop_after_all_iters_if: { expr: 'result.done', skip_if_stopped: false } + }) + )['early-stop'] + expect(s?.configured).toBe(true) + expect(s?.summary.text).toBe('result.done') + }) + + it('describes retries with zero attempts instead of reporting None', () => { + const s = stepSettingsByKey(step({ retry: { constant: { attempts: 0, seconds: 5 } } }))[ + 'retries' + ] + expect(s?.configured).toBe(true) + expect(s?.summary.text).toBe('0 attempts, constant') + }) + + it('reads concurrency and cache from the referenced script for workspace-script steps', () => { + const mod = step({ value: { type: 'script', path: 'u/me/s', input_transforms: {} } as any }) + const off = stepSettingsByKey(mod) + expect(off['concurrency']?.configured).toBe(false) + + const on = stepSettingsByKey(mod, { concurrent_limit: 3, cache_ttl: 60 }) + expect(on['concurrency']?.configured).toBe(true) + expect(on['concurrency']?.summary.text).toBe('Max 3') + expect(on['cache']?.configured).toBe(true) + }) + + it("prefers the module's own cache_ttl over the referenced script's, like the worker", () => { + const mod = step({ + cache_ttl: 3600, + value: { type: 'script', path: 'u/me/s', input_transforms: {} } as any + }) + // No referenced settings loaded yet (the graph badges never load them), so a + // module-level TTL has to stand on its own here. + expect(stepSettingsByKey(mod)['cache']?.configured).toBe(true) + expect(stepSettingsByKey(mod, { cache_ttl: 60 })['cache']?.summary.text).toBe('1 h') + }) + + it('reports a non-positive inline concurrency as invalid, not as unset', () => { + const s = stepSettingsByKey( + step({ + cache_ttl: -1, + value: { + type: 'rawscript', + language: 'bun', + content: '', + input_transforms: {}, + concurrent_limit: -1 + } + } as Partial) + ) + // `configured` means the step carries the config, so a value the user set counts + // even when the runtime ignores it — the summary is what says it is a no-op. + expect(s['concurrency']?.configured).toBe(true) + expect(s['concurrency']?.summary).toMatchObject({ text: 'Invalid limit', state: 'invalid' }) + expect(s['cache']?.configured).toBe(true) + expect(s['cache']?.summary).toMatchObject({ text: 'No TTL set', state: 'invalid' }) + }) + + it('treats a cleared concurrency input as present, not as unset', () => { + // Emptying a number input binds `null`, not `undefined`. Reading that as unset is + // what disabled the field the user was editing, so presence must be strict. + const mod = step({ + value: { + type: 'rawscript', + language: 'bun', + content: '', + input_transforms: {}, + concurrent_limit: null + } + } as unknown as Partial) + // Presence keeps the setting editor's controls live while the field is empty. + expect(hasInlineConcurrency(mod)).toBe(true) + expect(stepSettingsByKey(mod)['concurrency']?.summary).toMatchObject({ + text: 'Invalid limit', + state: 'invalid' + }) + }) + + it('omits settings that do not apply to the step type', () => { + const subflow = step({ value: { type: 'flow', path: 'u/me/f' } as any }) + expect(describeStepSettings(subflow).some((s) => s.key === 'concurrency')).toBe(false) + expect(describeStepSettings(step()).some((s) => s.key === 'concurrency')).toBe(true) + }) + + it('marks an invalid retry config as invalid rather than configured', () => { + const s = stepSettingsByKey( + step({ retry: { exponential: { attempts: 2, multiplier: 1, seconds: -1 } } }) + )['retries'] + expect(s?.summary.state).toBe('invalid') + }) + + it('labels early stop for trigger steps by what it means there', () => { + const trigger = step({ + value: { + type: 'rawscript', + language: 'bun', + content: '', + input_transforms: {}, + is_trigger: true + } as any + }) + expect(stepSettingsByKey(trigger)['early-stop']?.tooltip).toBe( + 'Stop early if there are no new events' + ) + expect(stepSettingsByKey(step())['early-stop']?.tooltip).toBe('Early stop / break') + }) +}) diff --git a/frontend/src/lib/components/flows/flowStepSettings.ts b/frontend/src/lib/components/flows/flowStepSettings.ts new file mode 100644 index 0000000000..9c995a93a9 --- /dev/null +++ b/frontend/src/lib/components/flows/flowStepSettings.ts @@ -0,0 +1,362 @@ +import { + ChevronsUp, + CircleStop, + Combine, + Database, + Gauge, + Hand, + Moon, + RefreshCw, + ShieldAlert, + SkipForward, + Timer, + Trash2 +} from 'lucide-svelte' +import type { FlowModule } from '$lib/gen' +import type { ScriptAdvancedSettingsFields } from '$lib/components/scriptSettings' +import { validateRetryConfig } from '$lib/utils' + +// Single source for the per-step runtime settings: which apply, which are configured, +// and how each reads back. Graph badges, the run-settings accordion and the setting +// editors all read it here so they cannot drift. Script-level twin: `scriptSettings.ts`. + +export type StepSettingKey = + | 'skip' + | 'early-stop' + | 'suspend' + | 'sleep' + | 'retries' + | 'error-handling' + | 'timeout' + | 'concurrency' + | 'priority' + | 'cache' + | 'debounce' + | 'lifetime' + +export type StepSettingSummary = { + text: string + state: 'configured' | 'default' | 'invalid' + /** Render the text as code (it is a user-written expression). */ + mono?: boolean +} + +export type StepSettingView = { + key: StepSettingKey + label: string + /** Longer wording for hover surfaces; falls back to `label`. */ + tooltip: string + icon: any + /** The setting's config is present on the step. Deliberately not "the runtime + * would behave differently" — a configured setting can still be a no-op + * (a sleep of 0), and `summary` says so rather than claiming it is off. */ + configured: boolean + summary: StepSettingSummary +} + +/** A step whose script polls an external system and returns the new items. */ +export function isTriggerStep(module: FlowModule | undefined): boolean { + return ( + module?.value != undefined && + (module.value.type === 'script' || module.value.type === 'rawscript') && + module.value.is_trigger === true + ) +} + +const def = (text: string): StepSettingSummary => ({ text, state: 'default' }) +const inv = (text: string): StepSettingSummary => ({ text, state: 'invalid' }) +const cfg = (text: string, mono = false): StepSettingSummary => ({ + text, + state: 'configured', + mono +}) + +function formatDur(s: number | undefined): string { + if (s == null) return '' + if (s < 60) return `${s}s` + if (s < 3600) return `${Math.round(s / 60)} min` + return `${Math.round(s / 3600)} h` +} + +/** Describe a user-written predicate. An empty expression is still configured — + * the setting is on, it just has nothing to evaluate yet. */ +function exprSummary(expr: string | undefined): StepSettingSummary { + const e = expr?.trim() + if (!e) return cfg('No expression') + return e.length <= 24 ? cfg(e, true) : cfg('Expression set') +} + +type Ctx = { referenced?: ScriptAdvancedSettingsFields } + +type SettingSpec = { + label: string + tooltip?: (mod: FlowModule) => string + icon: any + applies?: (mod: FlowModule) => boolean + configured: (mod: FlowModule, ctx: Ctx) => boolean + summarize: (mod: FlowModule, ctx: Ctx) => StepSettingSummary +} + +// A value the step itself carries counts as configured even when the runtime would +// ignore it — `summary` is what reports the no-op, and the setting editors keep their +// controls live on presence so clearing a field mid-edit can't disable it. Only values +// read off a referenced workspace script use the runtime's `> 0` test: nobody is typing +// into those, and 0 there simply means the script has no limit. +const isWorkspaceScript = (mod: FlowModule) => mod.value.type === 'script' +const inlineConcurrentLimit = (mod: FlowModule) => + mod.value.type === 'rawscript' ? mod.value.concurrent_limit : undefined +const effectiveCacheTtl = (mod: FlowModule, ctx: Ctx) => + mod.cache_ttl ?? (isWorkspaceScript(mod) ? ctx.referenced?.cache_ttl : undefined) + +/** Whether an inline step carries a concurrency limit at all. The setting editor keeps its + * controls live on presence, so clearing the field mid-edit can't disable the input. A + * present-but-non-positive limit is surfaced as invalid rather than silently as "None". + * Strict: an emptied number input binds to `null`, which is still a value being typed. */ +export function hasInlineConcurrency(mod: FlowModule): boolean { + return inlineConcurrentLimit(mod) !== undefined +} + +/** Canonical order — every surface lists settings in this sequence. */ +const SPECS: { key: StepSettingKey; spec: SettingSpec }[] = [ + { + key: 'skip', + spec: { + label: 'Skip if', + icon: SkipForward, + configured: (m) => Boolean(m.skip_if), + summarize: (m) => (m.skip_if ? exprSummary(m.skip_if.expr) : def('Off')) + } + }, + { + key: 'early-stop', + spec: { + label: 'Early stop / break', + tooltip: (m) => + isTriggerStep(m) ? 'Stop early if there are no new events' : 'Early stop / break', + icon: CircleStop, + configured: (m) => m.stop_after_if != undefined || m.stop_after_all_iters_if != undefined, + summarize: (m) => { + // Both can be set on a sequential loop, so pick the first that carries an + // expression instead of letting an empty one mask the other. + const exprs = [m.stop_after_if?.expr, m.stop_after_all_iters_if?.expr].filter( + (e) => e != undefined + ) + if (exprs.length === 0) return def('Off') + return exprSummary(exprs.find((e) => e?.trim()) ?? exprs[0]) + } + } + }, + { + key: 'suspend', + spec: { + label: 'Suspend until approval/resume', + icon: Hand, + configured: (m) => Boolean(m.suspend), + summarize: (m) => { + if (!m.suspend) return def('Off') + const n = m.suspend.required_events ?? 1 + return cfg(`${n} approval${n > 1 ? 's' : ''}`) + } + } + }, + { + key: 'sleep', + spec: { + label: 'Sleep', + icon: Moon, + configured: (m) => Boolean(m.sleep), + summarize: (m) => { + const s = m.sleep + if (!s) return def('Off') + if (s.type === 'static') { + const v = Number(s.value) + return Number.isFinite(v) ? cfg(`${formatDur(v)} after`) : cfg('Dynamic') + } + return cfg('Dynamic') + } + } + }, + { + key: 'retries', + spec: { + label: 'Retries', + icon: RefreshCw, + configured: (m) => m.retry?.constant != undefined || m.retry?.exponential != undefined, + summarize: (m) => { + const r = m.retry + if (r?.constant == undefined && r?.exponential == undefined) return def('None') + if (validateRetryConfig(r)) return { text: 'Invalid', state: 'invalid' } + const isConstant = r?.constant != undefined + const n = (isConstant ? r?.constant?.attempts : r?.exponential?.attempts) ?? 0 + const kind = isConstant ? 'constant' : 'exponential' + return cfg(`${n} attempt${n === 1 ? '' : 's'}, ${kind}`) + } + } + }, + { + key: 'error-handling', + spec: { + label: 'Error handling', + icon: ShieldAlert, + configured: (m) => Boolean(m.continue_on_error), + summarize: (m) => (m.continue_on_error ? cfg('Continue on error') : def('Off')) + } + }, + { + key: 'timeout', + spec: { + label: 'Timeout', + icon: Timer, + configured: (m) => m.timeout != null, + summarize: (m) => { + const t = m.timeout + if (t == null) return def('None') + if (typeof t === 'number') return cfg(formatDur(t)) + if (t.type === 'static') { + const v = Number(t.value) + return Number.isFinite(v) ? cfg(formatDur(v)) : cfg('Dynamic') + } + return cfg('Dynamic') + } + } + }, + { + key: 'concurrency', + spec: { + label: 'Concurrency limit', + icon: Gauge, + applies: (m) => m.value.type === 'rawscript' || m.value.type === 'script', + // Presence for the step's own limit (the user may be mid-edit, and `summary` says + // when it is a no-op), effectiveness for the referenced script's — a remote + // value nobody is typing into, where 0 just means the script has no limit. + configured: (m, ctx) => + isWorkspaceScript(m) + ? ctx.referenced?.concurrent_limit != undefined && ctx.referenced.concurrent_limit > 0 + : hasInlineConcurrency(m), + summarize: (m, ctx) => { + if (isWorkspaceScript(m)) { + const l = ctx.referenced?.concurrent_limit + return l != undefined && l > 0 ? cfg(`Max ${l}`) : def('None') + } + const l = inlineConcurrentLimit(m) + if (l === undefined) return def('None') + if (!(l > 0)) return inv('Invalid limit') + const key = m.value.type === 'rawscript' ? m.value.custom_concurrency_key : undefined + return cfg(`Max ${l}${key ? ' per key' : ''}`) + } + } + }, + { + key: 'priority', + spec: { + label: 'Priority', + icon: ChevronsUp, + configured: (m) => m.priority !== undefined, + summarize: (m) => { + if (m.priority === undefined) return def('Off') + // 0 is how the runtime spells "no priority". + return m.priority > 0 ? cfg('High priority') : inv('No priority set') + } + } + }, + { + key: 'cache', + spec: { + label: 'Cache results', + icon: Database, + // The worker takes the module's cache_ttl over the referenced script's, so a + // module-level TTL must show here even for a workspace-script step. + configured: (m, ctx) => + m.cache_ttl !== undefined || (isWorkspaceScript(m) && (ctx.referenced?.cache_ttl ?? 0) > 0), + summarize: (m, ctx) => { + const ttl = effectiveCacheTtl(m, ctx) + if (ttl === undefined) return def('Off') + return ttl > 0 ? cfg(formatDur(ttl)) : inv('No TTL set') + } + } + }, + { + key: 'debounce', + spec: { + label: 'Debounce', + icon: Combine, + configured: (m) => m.debouncing?.debounce_delay_s !== undefined, + summarize: (m) => { + const d = m.debouncing?.debounce_delay_s + if (d === undefined) return def('Off') + return d > 0 ? cfg(`${formatDur(d)} debounce`) : inv('No delay set') + } + } + }, + { + key: 'lifetime', + spec: { + label: 'Lifetime', + icon: Trash2, + configured: (m) => m.delete_after_secs != null, + summarize: (m) => { + const s = m.delete_after_secs + if (s == null) return def('Off') + return s === 0 ? cfg('Delete now') : cfg(`Delete after ${formatDur(s)}`) + } + } + } +] + +/** The settings that apply to this step, in canonical order. + * `referenced` supplies the workspace script's own settings for `script` steps, + * whose concurrency and cache live on the script rather than on the step. */ +export function describeStepSettings( + mod: FlowModule, + referenced?: ScriptAdvancedSettingsFields +): StepSettingView[] { + const ctx: Ctx = { referenced } + return SPECS.filter(({ spec }) => spec.applies?.(mod) ?? true).map(({ key, spec }) => ({ + key, + label: spec.label, + tooltip: spec.tooltip?.(mod) ?? spec.label, + icon: spec.icon, + configured: spec.configured(mod, ctx), + summary: spec.summarize(mod, ctx) + })) +} + +/** How a trigger step decides it has nothing to process. Stored on the step at + * creation, so changing it only affects newly created steps. */ +// Falsy-or-empty, not `result == undefined`: a trigger returning nothing new may say so +// with any empty value, and stopping is always the right response. +const TRIGGER_STOP_EXPR = '!result || (Array.isArray(result) && result.length == 0)' + +/** The config a setting is seeded with when it is switched on. Read by the setting + * editors and by every path that creates a step, so both agree. Settings absent from + * this map have no seeded config (the editor writes the value directly). */ +const DEFAULTS = { + skip: () => ({ expr: 'false' }), + 'early-stop': (kind?: 'trigger' | 'end') => + kind === 'trigger' + ? { expr: TRIGGER_STOP_EXPR, skip_if_stopped: true } + : kind === 'end' + ? { expr: 'true', skip_if_stopped: false } + : { + expr: 'result == undefined', + skip_if_stopped: false, + error_message: undefined, + error_include_result: false + }, + suspend: () => ({ required_events: 1, timeout: 1800 }), + sleep: () => ({ type: 'static' as const, value: 0 }), + cache: () => 600, + lifetime: () => 0, + priority: () => 100 +} satisfies Partial unknown>> + +export type SeededSettingKey = keyof typeof DEFAULTS + +/** Seeded config for a setting. Typed per key, so an unhandled key is a compile + * error rather than a silent `undefined`. */ +export function stepSettingDefaults( + key: K, + kind?: 'trigger' | 'end' +): ReturnType<(typeof DEFAULTS)[K]> { + return DEFAULTS[key](kind) as ReturnType<(typeof DEFAULTS)[K]> +} diff --git a/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte b/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte index dbebc82f70..0e92ee8a5e 100644 --- a/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte +++ b/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte @@ -169,7 +169,12 @@ {#if !loading} - + ).value + if (typeof value !== 'object' || value === null || Array.isArray(value)) { + throw new Error("missing 'value' - paste a whole OpenFlow document, not just its value") + } + if (!Array.isArray((value as Record).modules)) { + throw new Error("'value.modules' must be a list") + } + } + function validateGroups(groups: { start_id: string; end_id: string }[] | undefined) { if (!groups) return const seen = new Set() @@ -46,6 +63,7 @@ function apply() { try { const parsed = YAML.parse(code) + validateShape(parsed) validateGroups(parsed.value?.groups) if (parsed.summary && typeof parsed.summary === 'string') { flowStore.val.summary = parsed.summary @@ -65,6 +83,9 @@ if (parsed['on_behalf_of_email'] !== undefined) { flowStore.val.on_behalf_of_email = parsed['on_behalf_of_email'] } + if (parsed['on_behalf_of'] !== undefined) { + flowStore.val.on_behalf_of = parsed['on_behalf_of'] + } flowStore.val.value = parsed.value flowStore.val.schema = parsed.schema flowStore.val.tag = parsed.tag @@ -101,6 +122,7 @@ minHeight={editorHeight} bind:code lang="yaml" + leadingChangeSync />
{/await} diff --git a/frontend/src/lib/components/flows/map/FlowErrorHandlerItem.svelte b/frontend/src/lib/components/flows/map/FlowErrorHandlerItem.svelte index fbf909bba9..9e6a18d82c 100644 --- a/frontend/src/lib/components/flows/map/FlowErrorHandlerItem.svelte +++ b/frontend/src/lib/components/flows/map/FlowErrorHandlerItem.svelte @@ -61,6 +61,13 @@ refreshFlowStateStore(flowStore) } + function deleteFailureModule() { + flowStore.val.value.failure_module = undefined + // The panel has to land somewhere once the error handler is gone, but that is a + // consequence of the delete — not a request to see the flow's settings. + selectionManager.selectId('settings-metadata', { openPanel: false }) + } + const smallFailureModule = $derived(!(failureModuleId && diffManager && moduleAction) && compact) @@ -103,10 +110,7 @@ title="Delete failure script" type="button" class="ml-1" - onclick={() => { - flowStore.val.value.failure_module = undefined - selectionManager.selectId('settings-metadata') - }} + onclick={deleteFailureModule} > @@ -117,10 +121,7 @@ title="Delete failure script" type="button" class="absolute -top-1.5 -right-1.5 rounded-full bg-surface border border-border p-0.5 hover:bg-surface-hover" - onclick={() => { - flowStore.val.value.failure_module = undefined - selectionManager.selectId('settings-metadata') - }} + onclick={deleteFailureModule} > diff --git a/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte b/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte index de949a6dd9..6aeaa14f35 100644 --- a/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte +++ b/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte @@ -4,23 +4,8 @@ import Popover from '$lib/components/Popover.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { classNames, type Item, type StateStore } from '$lib/utils' - import { - Bed, - Database, - Gauge, - EllipsisVertical, - PhoneIncoming, - Repeat, - Square, - SkipForward, - Pin, - X, - Play, - Loader2, - TriangleAlert, - Timer, - Maximize2 - } from 'lucide-svelte' + import { EllipsisVertical, Pin, X, Play, Loader2, TriangleAlert, Maximize2 } from 'lucide-svelte' + import type { StepSettingView } from '../flowStepSettings' import { createEventDispatcher, getContext, untrack } from 'svelte' import { fade } from 'svelte/transition' import type { FlowEditorContext } from '../types' @@ -55,12 +40,8 @@ selected?: boolean deletable?: boolean moduleAction: ModuleActionInfo | undefined - retry?: boolean - cache?: boolean - earlyStop?: boolean - skip?: boolean - suspend?: boolean - sleep?: boolean + /** Configured settings to badge, from flowStepSettings. */ + settings?: StepSettingView[] mock?: | { enabled?: boolean @@ -72,12 +53,8 @@ label: string path?: string nodeState?: FlowNodeState - concurrency?: boolean - // TODO: Implement for this one. See how concurrency is implemented. - debouncing?: boolean retries?: number | undefined warningMessage?: string | undefined - isTrigger?: boolean editMode?: boolean alwaysShowOutputPicker?: boolean loopStatus?: { type: 'inside' | 'self'; flow: 'forloopflow' | 'whileloopflow' } | undefined @@ -97,23 +74,15 @@ selected = false, deletable = false, moduleAction = undefined, - retry = false, - cache = false, - earlyStop = false, - skip = false, - suspend = false, - sleep = false, + settings = [], mock = { enabled: false }, bold = false, id = undefined, label, path = '', nodeState, - concurrency = false, - debouncing = false, retries = undefined, warningMessage = undefined, - isTrigger = false, editMode = false, alwaysShowOutputPicker = false, loopStatus = undefined, @@ -304,112 +273,23 @@ class="absolute text-sm right-2 flex flex-row gap-1 z-10 transition-all duration-100" style={`bottom: ${outputPickerBarOpen ? '-38px' : '-12px'}`} > - {#if retry} + {#each settings as s (s.key)} + {@const Icon = s.icon}
- {#if retries}{retries}{/if} - + {#if s.key === 'retries' && retries}{retries}{/if} +
{#snippet text()} - Retries + {s.tooltip} + · {s.summary.text} {/snippet}
- {/if} - - {#if concurrency} - -
- -
- {#snippet text()} - Concurrency Limits - {/snippet} -
- {/if} - {#if debouncing} - -
- -
- {#snippet text()} - Debouncing - {/snippet} -
- {/if} - {#if cache} - -
- -
- {#snippet text()} - Cached - {/snippet} -
- {/if} - {#if earlyStop} - -
- -
- {#snippet text()} - {isTrigger ? 'Stop early if there are no new events' : 'Early stop/break'} - {/snippet} -
- {/if} - {#if skip} - -
- -
- {#snippet text()} - Skip - {/snippet} -
- {/if} - {#if suspend} - -
- -
- {#snippet text()} - Suspend - {/snippet} -
- {/if} - {#if sleep} - -
- -
- {#snippet text()} - Sleep - {/snippet} -
- {/if} + {/each} {#if mock?.enabled} -
+ +
- - -{/if} + + + {/if} + diff --git a/frontend/src/lib/components/graph/flowRunStatus.svelte.ts b/frontend/src/lib/components/graph/flowRunStatus.svelte.ts new file mode 100644 index 0000000000..8f5ee97641 --- /dev/null +++ b/frontend/src/lib/components/graph/flowRunStatus.svelte.ts @@ -0,0 +1,56 @@ +import { getContext, hasContext, setContext } from 'svelte' +import { SvelteMap } from 'svelte/reactivity' +import type { Job } from '$lib/gen' +import type { GraphModuleState } from './model' + +const FLOW_RUN_STATUS_KEY = 'FlowRunStatus' + +export type SuspendStatus = Record + +/** + * Run status is read straight from here by the node and edge renderers rather than + * being carried in their `data`. Baking it into `data` means the only way to show a + * status change is to rebuild every node and edge, which re-runs the sugiyama layout + * and makes xyflow re-measure and re-create the whole graph on every poll. + */ +export class FlowRunStatus { + #moduleStates = new SvelteMap() + flowJob = $state.raw(undefined) + suspendStatus = $state.raw({}) + + getModuleState(id: string | undefined): GraphModuleState | undefined { + return id == undefined ? undefined : this.#moduleStates.get(id) + } + + setModuleStates(next: Record | undefined) { + const incoming = next ?? {} + for (const id of [...this.#moduleStates.keys()]) { + if (!(id in incoming)) { + this.#moduleStates.delete(id) + } + } + // Writing a key invalidates only that key's readers, so one step finishing + // never re-renders the other steps. + for (const [id, state] of Object.entries(incoming)) { + if (this.#moduleStates.get(id) !== state) { + this.#moduleStates.set(id, state) + } + } + } +} + +export function setFlowRunStatusContext(): FlowRunStatus { + const status = new FlowRunStatus() + setContext(FLOW_RUN_STATUS_KEY, status) + return status +} + +/** + * Graphs that never show run status (mini graph, diff viewer) provide no context, so + * every reader has to tolerate its absence. + */ +export function getFlowRunStatusContext(): FlowRunStatus | undefined { + return hasContext(FLOW_RUN_STATUS_KEY) + ? getContext(FLOW_RUN_STATUS_KEY) + : undefined +} diff --git a/frontend/src/lib/components/graph/graphBuilder.svelte.ts b/frontend/src/lib/components/graph/graphBuilder.svelte.ts index 3e56a5cd5b..450debbef4 100644 --- a/frontend/src/lib/components/graph/graphBuilder.svelte.ts +++ b/frontend/src/lib/components/graph/graphBuilder.svelte.ts @@ -1,9 +1,10 @@ -import type { FlowModule, Job, PathScript, RawScript, Script } from '$lib/gen' +import type { FlowModule, PathScript, RawScript, Script } from '$lib/gen' import { type Edge } from '@xyflow/svelte' import { getAllModules, getDependeeAndDependentComponents } from '../flows/flowExplorer' import { dfsByModule } from '../flows/previousResults' import { defaultIfEmptyString } from '$lib/utils' import type { GraphModuleState } from './model' +import type { SelectIntentOptions } from './selectionUtils.svelte' import { getFlowModuleAssets, type AssetWithAltAccessType } from '../assets/lib' import { assetDisplaysAsOutputInFlowGraph } from './renderers/nodes/AssetNode.svelte' import type { ModulesTestStates, ModuleTestState } from '../modulesTest.svelte' @@ -60,7 +61,7 @@ export type GraphEventHandlers = { isPreprocessor?: boolean }) => void deleteBranch: (detail: { id: string; index: number }, label: string) => void - select: (mod: string | FlowModule) => void + select: (mod: string | FlowModule, opts?: SelectIntentOptions) => void delete: (detail: { id: string }, label: string) => void /** Drop a node that only mirrors run state (the error handler marker). Never edits the flow. */ dismissRunNode: (id: string) => void @@ -84,14 +85,6 @@ export type GraphEventHandlers = { export type SimplifiableFlow = { simplifiedFlow: boolean } -export function isTriggerStep(module: FlowModule | undefined): boolean { - return ( - module?.value != undefined && - (module.value.type === 'script' || module.value.type === 'rawscript') && - module.value.is_trigger === true - ) -} - export function buildPrefix(prefix: string | undefined, id: string): string { return (prefix ?? '') + id + ':' } @@ -139,7 +132,6 @@ export type InputN = { editMode: boolean isRunning: boolean individualStepTests: boolean - flowJob: Job | undefined showJobStatus: boolean flowHasChanged: boolean chatInputEnabled: boolean @@ -155,11 +147,9 @@ export type ModuleN = { id: string parentIds: string[] eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined testModuleState: ModuleTestState | undefined insertable: boolean editMode: boolean - flowJob: Job | undefined isOwner: boolean assets: AssetWithAltAccessType[] | undefined moduleAction: ModuleActionInfo | undefined @@ -174,7 +164,6 @@ export type FailureModuleN = { id: string module: FlowModule eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined } } @@ -185,7 +174,6 @@ export type BranchAllStartN = { id: string branchIndex: number eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined insertable: boolean branchOne: boolean } @@ -196,7 +184,6 @@ export type BranchAllEndN = { data: { id: string eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined } } @@ -206,7 +193,6 @@ export type ForLoopEndN = { id: string eventHandlers: GraphEventHandlers simplifiedTriggerView: boolean - flowModuleState: GraphModuleState | undefined } } @@ -215,7 +201,6 @@ export type ForLoopStartN = { data: { id: string eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined selectedId: string | undefined editMode: boolean simplifiedTriggerView: boolean @@ -229,7 +214,6 @@ export type ResultN = { success: boolean | undefined eventHandlers: GraphEventHandlers editMode: boolean - job: Job | undefined showJobStatus: boolean } } @@ -251,7 +235,6 @@ export type BranchOneStartN = { data: { id: string eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined selected: boolean insertable: boolean label: string @@ -266,7 +249,6 @@ export type BranchOneEndN = { data: { id: string eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined } } @@ -287,7 +269,6 @@ export type NoBranchN = { data: { id: string eventHandlers: GraphEventHandlers - flowModuleState: GraphModuleState | undefined branchOne: boolean label: string branchIndex: number @@ -330,6 +311,11 @@ export type AiToolN = { nameError?: string eventHandlers: GraphEventHandlers moduleId: string + /** An MCP tool's server, which its calls are keyed to. */ + resourcePath?: string + /** The agent step this tool hangs off. The editor draws the declared tools, whose ids the + * run's per-call state is not keyed by, so the node needs its agent to find its calls. */ + agentModuleId: string // Set on a linked agent's display-only tools: clicking selects this module (the agent step) // instead of the tool, whose resource-owned id is not flow-unique. selectTarget?: string @@ -337,7 +323,6 @@ export type AiToolN = { // Tool of a linked agent: its inputs are editable but its structure comes from the resource, // so it can't be deleted here. readOnly?: boolean - flowModuleStates: Record | undefined } } @@ -359,10 +344,7 @@ export type CollapsedGroupN = { autocollapse: boolean | undefined stepCount: number modules: FlowModule[] - flowModuleStates: Record | undefined - flowJob: Job | undefined isOwner: boolean - suspendStatus: Record showNotes: boolean editMode: boolean eventHandlers: GraphEventHandlers @@ -423,6 +405,9 @@ export function graphBuilder( extra: { disableAi: boolean insertable: boolean + // Only for the parts of the graph's shape a run decides: which loop iteration is + // expanded and where the error-handler marker attaches. Everything a step merely + // displays comes from FlowRunStatus, never from here. flowModuleStates: Record | undefined testModuleStates: ModulesTestStates | undefined moduleActions?: Record @@ -435,9 +420,7 @@ export function graphBuilder( isOwner: boolean isRunning: boolean individualStepTests: boolean - flowJob: Job | undefined showJobStatus: boolean - suspendStatus: Record flowHasChanged: boolean chatInputEnabled: boolean additionalAssetsMap?: Record @@ -488,12 +471,10 @@ export function graphBuilder( id: module.id, parentIds: [], eventHandlers: eventHandlers, - flowModuleState: extra.flowModuleStates?.[module.id], testModuleState: extra.testModuleStates?.states?.[module.id], insertable: extra.insertable && !module.id.startsWith('subflow:'), editMode: extra.editMode, isOwner: extra.isOwner, - flowJob: extra.flowJob, assets: getFlowModuleAssets(module, extra.additionalAssetsMap), moduleAction: extra.moduleActions?.[module.id], ...extraData @@ -518,8 +499,7 @@ export function graphBuilder( data: { id: module.id, module, - eventHandlers: eventHandlers, - flowModuleState: extra.flowModuleStates?.[module.id] + eventHandlers: eventHandlers }, type: 'failureModule', selectable: false @@ -616,7 +596,11 @@ export function graphBuilder( disableMoveIds: options?.disableMoveIds, enableTrigger: sourceId === 'Input', index, - ...extra, + // Only what the edge renderer reads. Anything listed here lands on every edge, + // so a value that changes each poll invalidates the whole edge set and makes + // Svelte re-create each one. + disableAi: extra.disableAi, + isOwner: extra.isOwner, insertable: extra.insertable && !options?.disableInsert && prefix == undefined, shouldOffsetInsertBtnDueToAssetNode: nodeIdsWithOutputAssets.has(sourceId) }, @@ -638,7 +622,6 @@ export function graphBuilder( editMode: extra.editMode, isRunning: extra.isRunning, individualStepTests: extra.individualStepTests, - flowJob: extra.flowJob, showJobStatus: extra.showJobStatus, flowHasChanged: extra.flowHasChanged, chatInputEnabled: extra.chatInputEnabled, @@ -679,7 +662,6 @@ export function graphBuilder( eventHandlers: eventHandlers, success: success, editMode: extra.editMode, - job: extra.flowJob, showJobStatus: extra.showJobStatus }, type: 'result' @@ -748,10 +730,7 @@ export function graphBuilder( modules: leafIds .map((id) => moduleMap.get(id)) .filter((m): m is FlowModule => !!m), - flowModuleStates: extra.flowModuleStates, - flowJob: extra.flowJob, isOwner: extra.isOwner, - suspendStatus: extra.suspendStatus, showNotes, editMode: prefix == undefined && extra.editMode, eventHandlers @@ -873,8 +852,7 @@ export function graphBuilder( id: `${module.id}-end`, data: { id: module.id, - eventHandlers: eventHandlers, - flowModuleState: extra.flowModuleStates?.[module.id] + eventHandlers: eventHandlers }, type: 'branchAllEnd' } @@ -889,7 +867,6 @@ export function graphBuilder( id: module.id, branchIndex: -1, eventHandlers: eventHandlers, - flowModuleState: extra.flowModuleStates?.[module.id], branchOne: false, label: 'No branches' }, @@ -915,7 +892,6 @@ export function graphBuilder( id: module.id, branchIndex: branchIndex, eventHandlers: eventHandlers, - flowModuleState: extra.flowModuleStates?.[module.id], insertable: extra.insertable, branchOne: false }, @@ -961,7 +937,6 @@ export function graphBuilder( simplifiedTriggerView, eventHandlers: eventHandlers, editMode: extra.editMode, - flowModuleState: extra.flowModuleStates?.[module.id], selectedId: extra.selectedId }, type: 'forLoopStart' @@ -982,8 +957,7 @@ export function graphBuilder( data: { id: module.id, eventHandlers: eventHandlers, - simplifiedTriggerView, - flowModuleState: extra.flowModuleStates?.[module.id] + simplifiedTriggerView }, type: 'forLoopEnd' } @@ -1053,7 +1027,6 @@ export function graphBuilder( id: `${module.id}-end`, data: { eventHandlers: eventHandlers, - flowModuleState: extra.flowModuleStates?.[module.id], id: module.id }, type: 'branchOneEnd' @@ -1069,7 +1042,6 @@ export function graphBuilder( eventHandlers: eventHandlers, insertable: extra.insertable, preLabel: undefined, - flowModuleState: extra.flowModuleStates?.[module.id], selected: false, modules: module.value.default }, @@ -1105,7 +1077,6 @@ export function graphBuilder( branchIndex: branchIndex, eventHandlers: eventHandlers, insertable: extra.insertable, - flowModuleState: extra.flowModuleStates?.[module.id], selected: false, modules: branch.modules }, diff --git a/frontend/src/lib/components/graph/model.ts b/frontend/src/lib/components/graph/model.ts index 3618406371..07c7fb0e19 100644 --- a/frontend/src/lib/components/graph/model.ts +++ b/frontend/src/lib/components/graph/model.ts @@ -66,6 +66,9 @@ export type GraphModuleState = { isListJob?: boolean skipped?: boolean agent_actions?: FlowStatusModule['agent_actions'] + /** Positionally aligned with `agent_actions`: every push of an action appends one entry, so a + * missing entry means that action has not finished yet. */ + agent_actions_success?: FlowStatusModule['agent_actions_success'] script_hash?: string workflow_as_code_status?: WorkflowStatus } diff --git a/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte b/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte index 5fbf084374..89e7c88c94 100644 --- a/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte +++ b/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte @@ -6,15 +6,15 @@ import { getStraightLinePath } from '../utils' import { twMerge } from 'tailwind-merge' import { NODE_WITH_WRITE_ASSET_Y_OFFSET } from '../nodes/AssetNode.svelte' - import { workspaceStore } from '$lib/stores' import FlowStatusWaitingForEvents from '$lib/components/FlowStatusWaitingForEvents.svelte' import type { Job } from '$lib/gen' - import type { GraphModuleState } from '../../model' import InsertModuleButton from '$lib/components/flows/map/InsertModuleButton.svelte' import { getGraphContext } from '../../graphContext' + import { getFlowRunStatusContext } from '../../flowRunStatus.svelte' import { GROUP_TOP_PADDING } from '$lib/components/graph/compoundLayout' const { useDataflow, showAssets, moveManager } = getGraphContext() + const flowRunStatus = getFlowRunStatusContext() let { id, @@ -40,10 +40,7 @@ enableTrigger: boolean disableAi: boolean disableMoveIds: string[] - flowModuleStates: Record | undefined isOwner: boolean - flowJob: Job | undefined - suspendStatus?: Record shouldOffsetInsertBtnDueToAssetNode?: boolean } } = $props() @@ -76,12 +73,13 @@ // TODO: this is a hack to show the waiting for events indicator on the edge a proper way would be to have a edge state // and handle the edge state in the graph builder let waitingForEvents = $derived( - data?.flowModuleStates?.[data.targetId]?.type === 'WaitingForEvents' || - data?.flowModuleStates?.[`${data.sourceId}-v`]?.type === 'WaitingForEvents' + flowRunStatus?.getModuleState(data.targetId)?.type === 'WaitingForEvents' || + flowRunStatus?.getModuleState(`${data.sourceId}-v`)?.type === 'WaitingForEvents' ) + let flowJob: Job | undefined = $derived(flowRunStatus?.flowJob) let suspendStatus: Record | undefined = $derived( - data?.suspendStatus + flowRunStatus?.suspendStatus ) let centerY = $derived( @@ -133,7 +131,7 @@ - {#if waitingForEvents && data.flowJob && data.flowJob.type === 'QueuedJob'} + {#if waitingForEvents && flowJob && flowJob.type === 'QueuedJob'}
@@ -147,10 +145,10 @@
- {#if data?.flowJob && data.flowJob.flow_status?.modules?.[data.flowJob.flow_status?.step]?.type === 'WaitingForEvents'} + {#if flowJob && flowJob.flow_status?.modules?.[flowJob.flow_status?.step]?.type === 'WaitingForEvents'} @@ -159,7 +157,7 @@ {#each Object.values(suspendStatus) as suspendCount (suspendCount.job.id)} diff --git a/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte index 9660ee3fa7..9465f5b49f 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte @@ -43,7 +43,7 @@ let computeAIToolNodesCache: | { nodes: (Node & NodeLayout)[] - hasFlowModuleStates: boolean + agentActions: Record linkedAgentTools: Record | undefined ret: ReturnType } @@ -55,6 +55,58 @@ : AI_TOOL_MESSAGE_PREFIX + '-' + agentModuleId + '-' + idx } + export type AgentAction = NonNullable[number] + + function bareResourcePath(path: string | undefined): string | undefined { + return path?.startsWith('$res:') ? path.slice('$res:'.length) : path + } + + /** Whether a run's action was a call of this declared tool. The editor keeps one node per + * declared tool, so each kind of action has to find its way back to the right one: a flow + * module by id, web search by being the only one of its kind, an MCP server by its resource + * path. A miss leaves the node undecorated rather than decorating the wrong tool. */ + export function agentActionMatchesTool( + action: AgentAction, + tool: { moduleId: string; type?: string; resourcePath?: string } + ): boolean { + switch (action.type) { + case 'tool_call': + return action.module_id === tool.moduleId + case 'web_search': + return tool.type === 'websearch' + case 'mcp_tool_call': + // One MCP node stands for a whole server and many function names, so the server path + // is the only join that holds. The worker strips `$res:` before building the action, + // while a flow authored outside the resource picker can still carry it. + return ( + tool.type === 'mcp' && + bareResourcePath(action.resource_path) === bareResourcePath(tool.resourcePath) + ) + case 'message': + return false + } + } + + /** The one id an agent action's state is written and read under. Every writer and reader has + * to rebuild the same key, so they all come here; a switch with no default makes a new action + * kind a compile error rather than a status that silently never resolves. */ + export function getAgentActionStateId( + idx: number, + agentModuleId: string, + action: AgentAction + ): string { + switch (action.type) { + case 'tool_call': + return getToolCallId(idx, agentModuleId, action.module_id) + case 'mcp_tool_call': + return AI_MCP_TOOL_CALL_PREFIX + '-' + agentModuleId + '-' + idx + case 'web_search': + return AI_WEBSEARCH_PREFIX + '-' + agentModuleId + '-' + idx + case 'message': + return getToolCallId(idx, agentModuleId) + } + } + function getComparableNode(node: Node & NodeLayout): Node & NodeLayout { if (node.type === 'module' && node.data.module.value.type === 'aiagent') { return { @@ -69,6 +121,22 @@ } } + function agentActionsOf( + nodes: (Node & NodeLayout)[], + flowModuleStates: Record | undefined, + insertable: boolean + ): Record { + const actions: Record = {} + // The editor renders the static tool set and ignores the run's actions, so snapshotting + // them there would deep-clone a value that changes every poll and never matches. + if (insertable) return actions + for (const node of nodes) { + if (node.type !== 'module' || node.data.module.value.type !== 'aiagent') continue + actions[node.id] = $state.snapshot(flowModuleStates?.[node.id]?.agent_actions) + } + return actions + } + export function computeAIToolNodes( nodes: (Node & NodeLayout)[], eventHandlers: GraphEventHandlers, @@ -83,7 +151,10 @@ } { if ( computeAIToolNodesCache && - !!flowModuleStates === computeAIToolNodesCache.hasFlowModuleStates && + deepEqual( + agentActionsOf(nodes, flowModuleStates, insertable), + computeAIToolNodesCache.agentActions + ) && deepEqual(nodes.map(getComparableNode), computeAIToolNodesCache.nodes) && deepEqual(linkedAgentTools, computeAIToolNodesCache.linkedAgentTools) ) { @@ -109,6 +180,7 @@ name: string type?: string stateType?: GraphModuleState['type'] + resourcePath?: string }[] = sourceTools.map((t, idx) => { // Handle FlowModule, MCP, and Websearch tools const toolType = @@ -122,7 +194,8 @@ return { id: t.id, name: t.summary ?? '', - type: toolType + type: toolType, + resourcePath: t.value.tool_type === 'mcp' ? t.value.resource_path : undefined } }) @@ -132,26 +205,13 @@ baseOffset = BELOW_ADDITIONAL_OFFSET + AI_TOOL_BASE_OFFSET rowOffset = AI_TOOL_ROW_OFFSET tools = agentActions.map((a, idx) => { + const id = getAgentActionStateId(idx, node.id, a) if (a.type === 'tool_call' || a.type === 'mcp_tool_call') { - const id = - a.type === 'tool_call' - ? getToolCallId(idx, node.id, a.module_id) - : AI_MCP_TOOL_CALL_PREFIX + '-' + node.id + '-' + idx - return { - id, - name: a.function_name - } + return { id, name: a.function_name } } else if (a.type === 'web_search') { - return { - id: AI_WEBSEARCH_PREFIX + '-' + node.id + '-' + idx, - name: 'Web Search', - type: 'websearch' - } + return { id, name: 'Web Search', type: 'websearch' } } else { - return { - id: getToolCallId(idx, node.id), - name: 'Message' - } + return { id, name: 'Message' } } }) } @@ -192,7 +252,8 @@ selectTarget: isLinkedAgent && !agentActions ? node.id : undefined, insertable, readOnly: isLinkedAgent, - flowModuleStates + agentModuleId: node.id, + resourcePath: tool.resourcePath }, id: `${node.id}-tool-${tool.id}`, width: inputToolWidth, @@ -253,7 +314,7 @@ computeAIToolNodesCache = { nodes: nodes.map(getComparableNode), - hasFlowModuleStates: !!flowModuleStates, + agentActions: agentActionsOf(nodes, flowModuleStates, insertable), linkedAgentTools: $state.snapshot(linkedAgentTools), ret } @@ -277,6 +338,7 @@ import { getNodeColorClasses } from '../../util' import { deepEqual } from 'fast-equals' import { getGraphContext } from '../../graphContext' + import { getFlowRunStatusContext } from '../../flowRunStatus.svelte' let hover = $state(false) @@ -286,13 +348,48 @@ } let { data, id }: Props = $props() + const flowRunStatus = getFlowRunStatusContext() const { selectionManager } = getGraphContext() - const flowModuleState = $derived(data.flowModuleStates?.[data.moduleId]) + const flowModuleState = $derived(flowRunStatus?.getModuleState(data.moduleId)) + + /** + * The editor draws the agent's declared tools, one node per tool, while a run keys its state + * per call. Roll every call of this tool into the one node it already has, so a run shows up + * here without adding nodes and shifting the graph. A run graph looks its own state up + * directly, so it never gets here. + */ + const toolCalls = $derived.by(() => { + if (flowModuleState) return undefined + const agentState = flowRunStatus?.getModuleState(data.agentModuleId) + const actions = agentState?.agent_actions + if (!actions) return undefined + let count = 0 + let failed = 0 + let pending = 0 + actions.forEach((action, index) => { + if ( + !agentActionMatchesTool(action, { + moduleId: data.moduleId, + type: data.type, + resourcePath: data.resourcePath + }) + ) + return + count++ + const success = agentState?.agent_actions_success?.[index] + if (success === undefined) pending++ + else if (!success) failed++ + }) + if (count === 0) return undefined + const type = pending > 0 ? 'InProgress' : failed > 0 ? 'Failure' : 'Success' + return { type, count } as const + }) + let colorClasses = $derived( getNodeColorClasses( - data.nameError ? 'Failure' : flowModuleState?.type, + data.nameError ? 'Failure' : (flowModuleState?.type ?? toolCalls?.type), selectionManager?.getSelectedId() === (data.selectTarget ?? data.moduleId) ) ) @@ -303,9 +400,9 @@ // agent's module id through the manual path would race the agent node's own click selection. function onSelect() { if (data.selectTarget) { - selectionManager?.selectId(data.selectTarget) + selectionManager?.selectId(data.selectTarget, { openPanel: true }) } - data.eventHandlers.select(data.selectTarget ?? data.moduleId) + data.eventHandlers.select(data.selectTarget ?? data.moduleId, { openPanel: true }) } @@ -343,6 +440,17 @@ {data.tool || 'Missing name'} + + + {#if toolCalls && toolCalls.count > 1} + + {toolCalls.count} + + {/if} {#if data.insertable && !data.readOnly}
- {#if waitingForEvents && data.flowJob && data.flowJob.type === 'QueuedJob'} + {#if waitingForEvents && flowRunStatus?.flowJob && flowRunStatus.flowJob.type === 'QueuedJob'}
@@ -80,19 +78,19 @@
- {#if data.flowJob.flow_status?.modules?.[data.flowJob.flow_status?.step]?.type === 'WaitingForEvents'} + {#if flowRunStatus?.flowJob?.flow_status?.modules?.[flowRunStatus.flowJob.flow_status?.step]?.type === 'WaitingForEvents'} - {:else if data.suspendStatus && Object.keys(data.suspendStatus).length > 0} + {:else if flowRunStatus?.suspendStatus && Object.keys(flowRunStatus.suspendStatus).length > 0}
- {#each Object.values(data.suspendStatus) as suspendCount (suspendCount.job.id)} + {#each Object.values(flowRunStatus.suspendStatus) as suspendCount (suspendCount.job.id)} diff --git a/frontend/src/lib/components/graph/renderers/nodes/FailureModuleNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/FailureModuleNode.svelte index 49ccbe46d6..63288fe7bb 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/FailureModuleNode.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/FailureModuleNode.svelte @@ -6,14 +6,16 @@ import type { FailureModuleN } from '../../graphBuilder.svelte' import { getNodeColorClasses, NODE } from '$lib/components/graph' import { msToSec } from '$lib/utils' + import { getFlowRunStatusContext } from '../../flowRunStatus.svelte' interface Props { data: FailureModuleN['data'] } let { data }: Props = $props() + const flowRunStatus = getFlowRunStatusContext() - let state = $derived(data.flowModuleState) + let state = $derived(flowRunStatus?.getModuleState(data.id)) let colorClasses = $derived(getNodeColorClasses(state?.skipped ? '_Skipped' : state?.type, false)) diff --git a/frontend/src/lib/components/graph/renderers/nodes/InputNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/InputNode.svelte index 195f920d03..c59a775bd2 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/InputNode.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/InputNode.svelte @@ -12,6 +12,7 @@ import type { FlowEditorContext } from '$lib/components/flows/types' import { MessageSquare } from 'lucide-svelte' import { getGraphContext } from '../../graphContext' + import { getFlowRunStatusContext } from '../../flowRunStatus.svelte' import FunnelCog from '$lib/components/icons/FunnelCog.svelte' interface Props { @@ -21,6 +22,7 @@ let { data }: Props = $props() const { selectionManager, diffManager } = getGraphContext() + const flowRunStatus = getFlowRunStatusContext() const flowEditorContext = getContext('FlowEditorContext') const { previewArgs, flowStore } = flowEditorContext || {} @@ -110,7 +112,7 @@ data.eventHandlers.hideJobStatus() }} individualStepTests={data.individualStepTests} - job={data.flowJob} + job={flowRunStatus?.flowJob} showJobStatus={data.showJobStatus} flowHasChanged={data.flowHasChanged} > diff --git a/frontend/src/lib/components/graph/renderers/nodes/ModuleNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/ModuleNode.svelte index c15c23a13e..4ba08a57a8 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/ModuleNode.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/ModuleNode.svelte @@ -1,11 +1,14 @@ {#if !isMoving} - - - {#snippet trigger()} - - {/snippet} - {#snippet content({ close })} - { - close() - }} - on:new={(e) => { - data.eventHandlers.insert({ - index: -1, // ignored when agentId is set - agentId: data.agentModuleId, - ...e.detail - }) - close() - }} - on:insert={(e) => { - data.eventHandlers.insert({ - index: -1, // ignored when agentId is set - agentId: data.agentModuleId, - ...e.detail - }) - close() - }} - on:pickScript={(e) => { - data.eventHandlers.insert({ - index: -1, // ignored when agentId is set - agentId: data.agentModuleId, - kind: e.detail.kind, - script: { - ...e.detail, - summary: e.detail.summary - ? e.detail.summary.replace(/\s/, '_').replace(/[^a-zA-Z0-9_]/g, '') - : e.detail.path.split('/').pop() - } - }) - close() - }} - on:pickMcpTool={(e) => { - data.eventHandlers.insert({ - index: -1, - agentId: data.agentModuleId, - kind: 'mcpTool' - }) - close() - }} - on:pickWebsearchTool={(e) => { - data.eventHandlers.insert({ - index: -1, - agentId: data.agentModuleId, - kind: 'websearchTool' - }) - close() - }} - on:pickAiAgentTool={(e) => { - data.eventHandlers.insert({ - index: -1, - agentId: data.agentModuleId, - kind: 'aiAgentTool' - }) - close() - }} - /> - {/snippet} - + + + {#snippet trigger()} + + {/snippet} + {#snippet content({ close })} + { + close() + }} + on:new={(e) => { + data.eventHandlers.insert({ + index: -1, // ignored when agentId is set + agentId: data.agentModuleId, + ...e.detail + }) + close() + }} + on:insert={(e) => { + data.eventHandlers.insert({ + index: -1, // ignored when agentId is set + agentId: data.agentModuleId, + ...e.detail + }) + close() + }} + on:pickScript={(e) => { + data.eventHandlers.insert({ + index: -1, // ignored when agentId is set + agentId: data.agentModuleId, + kind: e.detail.kind, + script: { + ...e.detail, + summary: e.detail.summary + ? e.detail.summary.replace(/\s/, '_').replace(/[^a-zA-Z0-9_]/g, '') + : e.detail.path.split('/').pop() + } + }) + close() + }} + on:pickMcpTool={(e) => { + data.eventHandlers.insert({ + index: -1, + agentId: data.agentModuleId, + kind: 'mcpTool' + }) + close() + }} + on:pickWebsearchTool={(e) => { + data.eventHandlers.insert({ + index: -1, + agentId: data.agentModuleId, + kind: 'websearchTool' + }) + close() + }} + on:pickAiAgentTool={(e) => { + data.eventHandlers.insert({ + index: -1, + agentId: data.agentModuleId, + kind: 'aiAgentTool' + }) + close() + }} + /> + {/snippet} + {/if} diff --git a/frontend/src/lib/components/graph/renderers/nodes/ResultNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/ResultNode.svelte index 13ede87991..88febcd6ca 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/ResultNode.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/ResultNode.svelte @@ -3,6 +3,7 @@ import NodeWrapper from './NodeWrapper.svelte' import type { ResultN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { getFlowRunStatusContext } from '../../flowRunStatus.svelte' interface Props { data: ResultN['data'] @@ -12,6 +13,7 @@ let { data, id }: Props = $props() const { selectionManager } = getGraphContext() + const flowRunStatus = getFlowRunStatusContext() @@ -27,7 +29,7 @@ }} nodeKind="result" editMode={data.editMode} - job={data.job} + job={flowRunStatus?.flowJob} showJobStatus={data.showJobStatus} /> {/snippet} diff --git a/frontend/src/lib/components/graph/renderers/nodes/TriggersNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/TriggersNode.svelte index 58f95d4b33..43af1fef48 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/TriggersNode.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/TriggersNode.svelte @@ -84,13 +84,15 @@ }} on:select={() => data?.eventHandlers?.select('Trigger')} onSelect={async (triggerIndex: number) => { - data?.eventHandlers?.select('Trigger') + data?.eventHandlers?.select('Trigger', { openPanel: true }) await tick() triggersState.selectedTriggerIndex = triggerIndex }} onAddDraftTrigger={async (type: TriggerType) => { const newTrigger = triggersState.addDraftTrigger(triggersCount, type) - data?.eventHandlers?.select('Trigger') + // A scheduled poll continues in the trigger-script picker that opens + // alongside this, so revealing the panel would cover it. + data?.eventHandlers?.select('Trigger', { openPanel: type !== 'poll' }) await tick() triggersState.selectedTriggerIndex = newTrigger }} diff --git a/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte b/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte index ccc5643981..3755b5cf77 100644 --- a/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte +++ b/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte @@ -1,6 +1,7 @@
@@ -98,25 +113,35 @@
-{#if showTriggerScriptPicker} - -
- { - showTriggerScriptPicker = false - dispatch('new', e.detail) - }} - on:pickScript={(e) => { - showTriggerScriptPicker = false - dispatch('pickScript', e.detail) - }} - kind="trigger" - /> -
-
-{/if} + + + {#snippet children()} + {#if showTriggerScriptPicker} + + +
(showTriggerScriptPicker = false) }} + > + { + showTriggerScriptPicker = false + dispatch('new', e.detail) + }} + on:pickScript={(e) => { + showTriggerScriptPicker = false + dispatch('pickScript', e.detail) + }} + kind="trigger" + /> +
+
+ {/if} + {/snippet} +
diff --git a/frontend/src/lib/components/graph/selectionUtils.svelte.ts b/frontend/src/lib/components/graph/selectionUtils.svelte.ts index daa9de3ecf..eddbae4fca 100644 --- a/frontend/src/lib/components/graph/selectionUtils.svelte.ts +++ b/frontend/src/lib/components/graph/selectionUtils.svelte.ts @@ -1,9 +1,34 @@ import type { Node } from '@xyflow/svelte' +/** Intent attached to a `selectId` call. `true` opens the panel even for ids that + * would not normally trigger it; `false` marks an incidental selection (what remains + * after a delete) and keeps it shut; omitted uses the default rules. */ +export type SelectIntentOptions = { + openPanel?: boolean +} + +/** Panels reached from toolbar buttons or dedicated graph nodes rather than step + * modules. They open on a single selection; step modules deliberately do not. */ +const FLOW_LEVEL_PANEL_IDS = new Set([ + 'constants', + 'failure', + 'preprocessor', + 'Input', + 'Result', + 'Trigger' +]) + +export function isFlowLevelPanelTarget(id: string): boolean { + // 'settings-' prefixed, not 'settings' prefixed: step ids are user-editable, so a + // step renamed settings_v2 must not be mistaken for the flow's settings panel. + return id === 'settings' || id.startsWith('settings-') || FLOW_LEVEL_PANEL_IDS.has(id) +} + export class SelectionManager { #selectedNodes = $state([]) #selectionMode = $state<'normal' | 'rect-select'>('normal') #clearGraphSelection: () => void = () => {} + #onSelectIntent: ((id: string, opts?: SelectIntentOptions) => void) | undefined = undefined constructor() {} @@ -11,7 +36,15 @@ export class SelectionManager { this.#clearGraphSelection = clearGraphSelection } - selectId(id: string) { + /** Fires on every `selectId` call, BEFORE the same-id dedup early-return — so a + * consumer can react even when the id is re-selected (e.g. clicking the already + * selected "Settings" toolbar button to re-open a modal panel). */ + setOnSelectIntent(cb: ((id: string, opts?: SelectIntentOptions) => void) | undefined) { + this.#onSelectIntent = cb + } + + selectId(id: string, opts?: SelectIntentOptions) { + this.#onSelectIntent?.(id, opts) if (this.#selectedNodes.length === 1 && this.#selectedNodes[0].id === id) { return } @@ -77,6 +110,12 @@ export class SelectionManager { return } + // Before the same-id early return, like `selectId`: re-selecting an already + // selected node must still be able to reopen its panel. + if (nodes.length === 1) { + this.#onSelectIntent?.(nodes[0].id) + } + // If the new selection is the same as the current selection, do nothing const newIds = nodes.map((n) => n.id).join(',') const currentIds = this.#selectedNodes.map((n) => n.id).join(',') diff --git a/frontend/src/lib/components/home/BulkActionsBar.svelte b/frontend/src/lib/components/home/BulkActionsBar.svelte new file mode 100644 index 0000000000..f36ed4848a --- /dev/null +++ b/frontend/src/lib/components/home/BulkActionsBar.svelte @@ -0,0 +1,340 @@ + + +
+ + {selection.size} selected + +
+ + + {#if targets('archive').length > 0 || targets('unarchive').length === 0} + + {/if} + {#if targets('unarchive').length > 0} + + {/if} + open('discard'), + disabled: targets('discard').length === 0, + // Only while blocked: a disabled entry can't open its modal, so the reason + // has to live here — and an enabled one would render a pointless ⓘ. + tooltip: targets('discard').length === 0 ? actionTitle('discard') : undefined + }, + { + displayName: `Delete${countSuffix('delete')}`, + icon: Trash, + type: 'delete' as const, + action: () => open('delete'), + disabled: targets('delete').length === 0, + tooltip: targets('delete').length === 0 ? actionTitle('delete') : undefined + } + ]} + > + {#snippet buttonReplacement()} + + {/snippet} + +
+
+ + 0 ? 'Retry' : pending ? ACTION_LABEL[pending] : ''} + type={pending === 'delete' || pending === 'discard' ? 'danger' : 'info'} + loading={running} + onCanceled={close} + onConfirmed={confirm} +> +
+ {#if pending === 'move'} +