diff --git a/.agents/skills/ai-chat/SKILL.md b/.agents/skills/ai-chat/SKILL.md new file mode 100644 index 0000000000..f670b2ffb7 --- /dev/null +++ b/.agents/skills/ai-chat/SKILL.md @@ -0,0 +1,40 @@ +--- +name: ai-chat +description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window. +--- + +## Always benchmark before and after + +No context or behavior change ships without an `ai_evals` A/B on the affected mode. +Add or adjust cases for exactly what you changed — see the `ai-evals` skill for +authoring and the full run reference. + +Run the affected mode **before** your change and **after**, same model(s), same cases. + +## Measure the window first, and cumulative second + +Optimize **`finalContextTokens`** (window occupancy — what drives overflow and +compaction), then cumulative prompt tokens. + +## Context discipline + +The dominant fixed cost is per-iteration overhead: the system prompt **plus every +tool schema** is re-sent on every loop iteration. So: + +- **Every tool and every parameter is a permanent tax.** Justify each one and measure + it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead + params rather than leaving them in the schema. +- **Tool results return the minimum.** Never echo content the model already has. The + canonical mistake: a write tool that returns the whole edited artifact right after + the model authored it — return `{ success, message }` instead. When you touch a + *shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check + this invariant for **all** the write tools routing through it — the echo has + regressed before via a shared refactor. + +## Prompts and tool descriptions are part of the surface + +The system prompt and tool descriptions steer behavior as much as the tools +themselves, and are benchmarkable the same way. A description that advertises +truncation makes the model self-limit; the path-conventions block changes where +drafts land. Treat prompt/description edits as real changes and A/B them — a +pure-prompt change is a legitimate, measurable improvement. \ No newline at end of file diff --git a/.agents/skills/ai-evals/SKILL.md b/.agents/skills/ai-evals/SKILL.md new file mode 100644 index 0000000000..ad334e2c6b --- /dev/null +++ b/.agents/skills/ai-evals/SKILL.md @@ -0,0 +1,87 @@ +--- +name: ai-evals +description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes. +--- + +# AI evals — authoring and running benchmark cases + +`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes: +`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production +prompts, tools, and guidance in this checkout. Each attempt runs the real production +path, deterministic validation, then LLM judging. + +The goal is to test current production guidance with realistic user requests — **not** +to pin one exact implementation shape. + +## Running benchmarks + +```bash +cd ai_evals +bun install # first time; frontend modes also need `cd frontend && bun install` +bun run cli -- models # list model aliases +bun run cli -- cases global # list cases for a mode +bun run cli -- run global global-test1-script-create --model sonnet +``` + +Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill +backend's `/api/w//ai/proxy`, so you need **any** reachable backend: + +```bash +WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1: WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \ + bun run cli -- run global --models sonnet,gpt-5.5,gemini-3.1-pro-preview +``` + +- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace + creation 400s ("reached workspace limit"). Always set + `WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to + reuse one. The only side effect of a run is upserting an `f/evals/ai/` + resource there. +- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a + separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under + test. + +## Authoring core rules + +1. Write prompts like a real user request. +2. Prefer behavior, inputs, constraints, and outcomes over internal implementation. +3. Keep deterministic validation narrow and hard. +4. Put semantic expectations in `judgeChecklist`. +5. Use `expected` fixtures only when exact structure really matters. + +### Prompt writing + +Prompts should sound like something a user would naturally ask. Do not write prompts +as if the user knows Windmill internals unless the case explicitly tests a power-user +workflow. + +Good: +- "Create a flow that routes support requests based on customer tier." +- "Add a reset button that sets the counter back to 0." +- "Create a flow that reuses the existing greeting script instead of duplicating the logic." + +Bad: +- "Use `branchone` with 3 branches and a default branch." +- "Create a `rawscript` step with this exact topology." +- "This is a benchmark harness." + +### Deterministic validation + +Use deterministic checks only for hard failures: missing required files; unexpected +extra files when the prompt says not to create them; syntax errors; unresolved flow +refs; missing required special modules or suspend config; obvious corruption. + +Do **not** encode one preferred implementation. Bad hard checks: exact step topology +for a creation flow; exact branch structure when the prompt only asked for routing; +exact input shape when multiple reasonable shapes are acceptable. + +### Judge checklist + +Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior +that must be present, important constraints, and key completion criteria — not +low-level implementation details unless truly required. + +Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing +workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a +`rawscript` node". + +See `ai_evals/README.md` for the full case format, fields, and fixture details. \ No newline at end of file diff --git a/.claude/skills/ai-chat/SKILL.md b/.claude/skills/ai-chat/SKILL.md new file mode 120000 index 0000000000..f8c527f3c9 --- /dev/null +++ b/.claude/skills/ai-chat/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-chat/SKILL.md \ No newline at end of file diff --git a/.claude/skills/ai-evals/SKILL.md b/.claude/skills/ai-evals/SKILL.md new file mode 120000 index 0000000000..8b714cb50d --- /dev/null +++ b/.claude/skills/ai-evals/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-evals/SKILL.md \ No newline at end of file diff --git a/.github/DockerfileBackendTests b/.github/DockerfileBackendTests index 88275f204b..9b29f72a64 100644 --- a/.github/DockerfileBackendTests +++ b/.github/DockerfileBackendTests @@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv ENV TZ=Etc/UTC diff --git a/.github/scripts/check-docs-links.mjs b/.github/scripts/check-docs-links.mjs new file mode 100644 index 0000000000..122d098197 --- /dev/null +++ b/.github/scripts/check-docs-links.mjs @@ -0,0 +1,126 @@ +// Extracts every windmill.dev/docs link referenced in the frontend source and +// verifies none of them 404. Run: `node .github/scripts/check-docs-links.mjs`. +// Used by the check-docs-links GitHub workflow (release / manual trigger only). + +import { readdir, readFile } from 'node:fs/promises' +import { join, extname } from 'node:path' + +const ROOT = 'frontend/src' +const EXTS = new Set(['.ts', '.js', '.svelte', '.mjs', '.cjs']) +const DOCS_RE = /https?:\/\/(?:www\.)?windmill\.dev\/docs\/[^\s"'`)>\]}]*/g +// `const someBaseUrl = 'https://www.windmill.dev/docs/...'` used later as `${someBaseUrl}/foo` +const BASE_RE = /(?:const|let|var)\s+([A-Za-z_$][\w$]*)\s*=\s*['"`](https?:\/\/(?:www\.)?windmill\.dev\/docs\/[^'"`]+)['"`]/g + +const CONCURRENCY = 24 +const TIMEOUT_MS = 20000 +const RETRIES = 2 + +async function walk(dir) { + const out = [] + for (const entry of await readdir(dir, { withFileTypes: true })) { + const p = join(dir, entry.name) + if (entry.isDirectory()) { + if (entry.name === 'node_modules' || entry.name === '.svelte-kit') continue + out.push(...(await walk(p))) + } else if (EXTS.has(extname(entry.name))) { + out.push(p) + } + } + return out +} + +// url (no fragment) -> Set of source files it appears in +const urls = new Map() +const unresolved = [] + +function record(url, file) { + const clean = url + .replace(/\\.*$/, '') // cut at an escape sequence embedded in a string literal (e.g. \n) + .replace(/#.*$/, '') // drop anchor fragment — irrelevant to a 404 check + .replace(/[.,;:'")\]]+$/, '') + if (!clean) return + // A `{`/`${` means the URL is built from an unresolved template/interpolation var. + if (clean.includes('{')) { + unresolved.push(`${clean} (${file})`) + return + } + if (!urls.has(clean)) urls.set(clean, new Set()) + urls.get(clean).add(file) +} + +for (const file of await walk(ROOT)) { + let content = await readFile(file, 'utf8') + // Inline file-local base-url constants so `${base}/page` template literals resolve. + const bases = [] + for (const m of content.matchAll(BASE_RE)) bases.push({ name: m[1], value: m[2], decl: m[0] }) + for (const { name, value } of bases) { + content = content.replaceAll('${' + name + '}', value) + } + // Blank each base declaration so a prefix-only base (no index page of its own, + // e.g. .../app_configuration_settings) isn't checked as a standalone link. + // A genuinely bare `${base}` usage was already inlined above, so it's still covered. + for (const { decl } of bases) content = content.replace(decl, '') + for (const m of content.matchAll(DOCS_RE)) record(m[0], file) +} + +const allUrls = [...urls.keys()].sort() +console.log(`Found ${allUrls.length} distinct docs links across ${ROOT}`) +if (unresolved.length) { + console.log(`\n⚠️ ${unresolved.length} link(s) built from an unrecognized base URL — skipped (register the base const so they can be checked):`) + for (const u of [...new Set(unresolved)].sort()) console.log(` ${u}`) +} + +async function check(url) { + for (let attempt = 0; attempt <= RETRIES; attempt++) { + const ctrl = new AbortController() + const timer = setTimeout(() => ctrl.abort(), TIMEOUT_MS) + try { + let res = await fetch(url, { + method: 'HEAD', + redirect: 'follow', + signal: ctrl.signal, + headers: { 'user-agent': 'windmill-docs-link-check' } + }) + // Some hosts reject HEAD — fall back to GET. + if (res.status === 405 || res.status === 501) { + res = await fetch(url, { + method: 'GET', + redirect: 'follow', + signal: ctrl.signal, + headers: { 'user-agent': 'windmill-docs-link-check' } + }) + } + clearTimeout(timer) + return { url, status: res.status, ok: res.status < 400 } + } catch (err) { + clearTimeout(timer) + if (attempt === RETRIES) return { url, status: 0, ok: false, error: String(err?.message || err) } + await new Promise((r) => setTimeout(r, 500 * (attempt + 1))) + } + } +} + +// Simple concurrency pool. +const results = [] +let idx = 0 +async function worker() { + while (idx < allUrls.length) { + const url = allUrls[idx++] + results.push(await check(url)) + } +} +await Promise.all(Array.from({ length: CONCURRENCY }, worker)) + +const failures = results.filter((r) => !r.ok) +if (failures.length === 0) { + console.log(`\n✅ All ${allUrls.length} docs links are reachable.`) + process.exit(0) +} + +console.log(`\n❌ ${failures.length} broken docs link(s):`) +for (const f of failures.sort((a, b) => a.url.localeCompare(b.url))) { + console.log(`\n ${f.url}`) + console.log(` status: ${f.error ? `error (${f.error})` : f.status}`) + for (const file of urls.get(f.url)) console.log(` ↳ ${file}`) +} +process.exit(1) diff --git a/.github/workflows/ai-agent-tests.yml b/.github/workflows/ai-agent-tests.yml new file mode 100644 index 0000000000..3a5c51f153 --- /dev/null +++ b/.github/workflows/ai-agent-tests.yml @@ -0,0 +1,132 @@ +name: AI Agent Integration Tests + +# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against +# real LLM providers. Runs only when AI-agent backend code or the tests change, +# because each run makes real (paid) LLM calls. To avoid spending on every commit, +# the PR side triggers only when a PR is marked ready for review (out of draft) — +# not on `synchronize` — plus push to main and manual dispatch. +on: + workflow_dispatch: + push: + branches: [main] + paths: + - "integration_tests/ai_agent_tests/**" + - "backend/windmill-ai/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-worker/src/ai_executor.rs" + - "backend/windmill-worker/src/ai/**" + - "backend/windmill-worker/src/memory_common.rs" + - "backend/windmill-common/src/flow_conversations.rs" + - ".github/workflows/ai-agent-tests.yml" + pull_request: + types: [opened, reopened, ready_for_review] + paths: + - "integration_tests/ai_agent_tests/**" + - "backend/windmill-ai/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-worker/src/ai_executor.rs" + - "backend/windmill-worker/src/ai/**" + - "backend/windmill-worker/src/memory_common.rs" + - "backend/windmill-common/src/flow_conversations.rs" + - ".github/workflows/ai-agent-tests.yml" + +concurrency: + group: ai-agent-tests-${{ github.ref }} + cancel-in-progress: true + +jobs: + ai_agent_e2e: + # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while + # still a draft. `ready_for_review` always arrives non-draft. + if: github.event_name != 'pull_request' || github.event.pull_request.draft == false + runs-on: ubicloud-standard-16 + services: + postgres: + image: postgres:16 + ports: + - 5432:5432 + env: + POSTGRES_DB: windmill + POSTGRES_PASSWORD: changeme + options: >- + --health-cmd pg_isready --health-interval 10s --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache-workspaces: backend + toolchain: 1.93.0 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.3.10 + + - uses: actions/setup-node@v4 + with: + node-version: "20" + + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + + # CE build (no enterprise/license needed for AI agents). `quickjs` powers + # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool + # test. Bun tool scripts run via the always-on worker (BUN_PATH). + - name: Build Windmill + working-directory: ./backend + env: + SQLX_OFFLINE: true + CARGO_BUILD_JOBS: 12 + RUSTFLAGS: "" + run: cargo build --features quickjs,mcp + + - name: Start Windmill + working-directory: ./backend + env: + DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill + BUN_PATH: bun + NODE_BIN_PATH: node + RUST_LOG: info + run: | + mkdir -p ../integration_tests/logs + ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 & + echo "Waiting for Windmill to be ready..." + for i in $(seq 1 60); do + if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then + echo "Windmill is ready" + break + fi + sleep 2 + done + curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; } + + - name: Run AI agent integration tests + timeout-minutes: 20 + working-directory: ./integration_tests/ai_agent_tests + env: + WINDMILL_URL: http://localhost:8000 + # Only the providers we have org secrets for. Other providers + # (Azure, Bedrock, OpenRouter) are skipped by conftest when their + # keys are absent — see skip_provider_without_credentials. + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }} + run: | + python -m venv .venv + .venv/bin/pip install -r requirements.txt + # The S3/vision-attachment tests need MinIO large-file storage and + # image-capable provider setup; out of scope for this cost-controlled + # smoke. Add MinIO secrets + a storage service to enable them. + .venv/bin/python -m pytest -v \ + --ignore=test_user_attachments.py \ + --ignore=test_user_images.py \ + --ignore=test_image_output.py + + - name: Archive Windmill logs + uses: actions/upload-artifact@v4 + if: always() + with: + name: ai-agent-tests-windmill-logs + path: integration_tests/logs diff --git a/.github/workflows/ai-evals-test.yml b/.github/workflows/ai-evals-test.yml new file mode 100644 index 0000000000..a255a4df94 --- /dev/null +++ b/.github/workflows/ai-evals-test.yml @@ -0,0 +1,166 @@ +name: AI Evals (global mode) + +# Smoke-tests the production global AI chat proxy/frontend execution path via +# the ai_evals harness, one case across one cheap model per provider. Runs only +# when the eval harness or the global chat code change, since each run makes real +# (paid) LLM calls. The backend is built from source purely as the AI proxy the +# harness routes model calls through; the global tools/drafts run in-process in +# the Vitest bridge against production frontend code. To avoid spending on every +# commit, the PR side triggers only when a PR is marked ready for review (out of +# draft) — not on `synchronize` — plus push to main and manual dispatch. +on: + workflow_dispatch: + push: + branches: [main] + paths: + - "ai_evals/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-ai/**" + - "frontend/src/lib/components/copilot/**" + # The eval harness runs production frontend code in-process; these are the + # AI/draft-specific deps outside copilot/ that the global smoke exercises. + - "frontend/src/lib/userDraft.svelte.ts" + - "frontend/src/lib/userDraftDbSyncer.svelte.ts" + - "frontend/src/lib/infer.ts" + - ".github/workflows/ai-evals-test.yml" + pull_request: + types: [opened, reopened, ready_for_review] + paths: + - "ai_evals/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-ai/**" + - "frontend/src/lib/components/copilot/**" + # The eval harness runs production frontend code in-process; these are the + # AI/draft-specific deps outside copilot/ that the global smoke exercises. + - "frontend/src/lib/userDraft.svelte.ts" + - "frontend/src/lib/userDraftDbSyncer.svelte.ts" + - "frontend/src/lib/infer.ts" + - ".github/workflows/ai-evals-test.yml" + +concurrency: + group: ai-evals-test-${{ github.ref }} + cancel-in-progress: true + +jobs: + ai_evals_global: + # Provider secrets are unavailable to forked and Dependabot PRs. + if: >- + github.event_name != 'pull_request' || + ( + github.event.pull_request.draft == false && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.user.login != 'dependabot[bot]' + ) + runs-on: ubicloud-standard-16 + services: + postgres: + image: postgres:16 + ports: + - 5432:5432 + env: + POSTGRES_DB: windmill + POSTGRES_PASSWORD: changeme + options: >- + --health-cmd pg_isready --health-interval 10s --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache-workspaces: backend + toolchain: 1.93.0 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.3.10 + + - uses: actions/setup-node@v4 + with: + # Node 22.19+ is required by the frontend's undici 8.x, which the + # Vitest bridge loads; Node 20 fails with markAsUncloneable. + node-version: "22" + + # CE build used only as the AI proxy (login, workspace, provider resource, + # /ai/proxy). No worker execution or MCP needed — global tools/drafts run + # in the Vitest bridge. quickjs matches the standard CE feature set. + - name: Build Windmill (AI proxy) + working-directory: ./backend + env: + SQLX_OFFLINE: true + CARGO_BUILD_JOBS: 12 + RUSTFLAGS: "" + run: cargo build --features quickjs + + - name: Start Windmill + working-directory: ./backend + env: + DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill + RUST_LOG: info + run: | + mkdir -p ../ai_evals/logs + ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 & + echo "Waiting for Windmill to be ready..." + for i in $(seq 1 60); do + if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then + echo "Windmill is ready" + break + fi + sleep 2 + done + curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; } + + - name: Install frontend deps + generate client + working-directory: ./frontend + run: | + npm ci + npm run generate-backend-client + + - name: Run global AI evals + timeout-minutes: 20 + working-directory: ./ai_evals + env: + WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000 + WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests + # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY. + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }} + DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }} + run: | + bun install + mkdir -p results + # One cheap model per provider (anthropic/openai/googleai/deepseek). + fail=0 + for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do + echo "::group::global-test1-script-create ($m)" + if ! bun run cli -- run global global-test1-script-create \ + --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then + echo "$m: harness/proxy errored" + fail=1 + echo "::endgroup::" + continue + fi + # The CLI exits 0 when the harness records failed attempts, so gate + # on execution-only pass counts while ignoring model output quality. + if jq -e \ + '.attemptCount > 0 and .passedAttempts == .attemptCount' \ + "results/ci-$m.json" > /dev/null; then + echo "$m: OK — proxy/frontend execution completed" + else + echo "$m: FAILED proxy/frontend execution" + jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true + fail=1 + fi + echo "::endgroup::" + done + [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; } + + - name: Archive logs and results + uses: actions/upload-artifact@v4 + if: always() + with: + name: ai-evals-global-logs + path: | + ai_evals/logs + ai_evals/results diff --git a/.github/workflows/backend-test-windows.yml b/.github/workflows/backend-test-windows.yml index 1c73e5d429..7065547b28 100644 --- a/.github/workflows/backend-test-windows.yml +++ b/.github/workflows/backend-test-windows.yml @@ -58,7 +58,9 @@ jobs: - uses: denoland/setup-deno@v2 with: - deno-version: v2.x + # Pin to the Deno version shipped in the runtime image (Dockerfile) so CI + # tests what production runs, instead of floating on the latest v2.x. + deno-version: 2.2.1 - uses: actions/setup-go@v2 with: @@ -74,7 +76,7 @@ jobs: - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: diff --git a/.github/workflows/backend-test.yml b/.github/workflows/backend-test.yml index 8f1f15447c..0be727d4bd 100644 --- a/.github/workflows/backend-test.yml +++ b/.github/workflows/backend-test.yml @@ -50,7 +50,9 @@ jobs: dotnet-version: "9.0.x" - uses: denoland/setup-deno@v2 with: - deno-version: v2.x + # Pin to the Deno version shipped in the runtime image (Dockerfile) so CI + # tests what production runs, instead of floating on the latest v2.x. + deno-version: 2.2.1 - uses: actions/setup-go@v2 with: go-version: 1.21.5 @@ -62,7 +64,7 @@ jobs: node-version: "20" - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: php-version: "8.3" diff --git a/.github/workflows/check-docs-links.yml b/.github/workflows/check-docs-links.yml new file mode 100644 index 0000000000..ec17c06771 --- /dev/null +++ b/.github/workflows/check-docs-links.yml @@ -0,0 +1,23 @@ +name: Check frontend docs links + +on: + push: + tags: + - "v*" + workflow_dispatch: + +jobs: + check-docs-links: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + with: + sparse-checkout: | + frontend/src + .github/scripts + - uses: actions/setup-node@v4 + with: + node-version: "22.x" + - name: Verify docs links are not 404 + run: node .github/scripts/check-docs-links.mjs diff --git a/.github/workflows/check-org-membership.yml b/.github/workflows/check-org-membership.yml deleted file mode 100644 index eb338d3188..0000000000 --- a/.github/workflows/check-org-membership.yml +++ /dev/null @@ -1,83 +0,0 @@ -name: Check Organization Membership - -on: - workflow_call: - inputs: - commenter: - required: false - type: string - default: '' - description: 'The username to check. Auto-detected from the event context if not provided.' - organization: - required: false - type: string - default: 'windmill-labs' - description: 'The organization to check membership for' - trusted_bot: - required: false - type: string - default: 'windmill-internal-app[bot]' - description: 'The trusted bot username to allow' - secrets: - access_token: - required: true - description: 'The access token to use for org membership check' - outputs: - is_member: - description: 'Whether the user is an organization member or trusted bot' - value: ${{ jobs.check-membership.outputs.is_member }} - -jobs: - check-membership: - runs-on: ubicloud-standard-2 - outputs: - is_member: ${{ steps.check-membership.outputs.is_member }} - steps: - - name: Determine commenter - id: determine-commenter - run: | - COMMENTER="${{ inputs.commenter }}" - if [[ -z "$COMMENTER" ]]; then - if [[ "${{ github.event_name }}" == "issue_comment" || \ - "${{ github.event_name }}" == "pull_request_review_comment" ]]; then - COMMENTER="${{ github.event.comment.user.login }}" - elif [[ "${{ github.event_name }}" == "pull_request_review" ]]; then - COMMENTER="${{ github.event.review.user.login }}" - else - COMMENTER="${{ github.event.issue.user.login }}" - fi - fi - echo "commenter=$COMMENTER" >> $GITHUB_OUTPUT - - - name: Check organization membership - id: check-membership - env: - ORG_ACCESS_TOKEN: ${{ secrets.access_token }} - COMMENTER: ${{ steps.determine-commenter.outputs.commenter }} - ORG: ${{ inputs.organization }} - TRUSTED_BOT: ${{ inputs.trusted_bot }} - run: | - # 1. Allow the trusted bot straight away - if [[ "$COMMENTER" == "$TRUSTED_BOT" ]]; then - echo "is_member=true" >> $GITHUB_OUTPUT - exit 0 - fi - - # 2. Disallow other bots - if [[ "${COMMENTER}" =~ \[bot\]$ ]]; then - echo "is_member=false" >> $GITHUB_OUTPUT - exit 0 - fi - - # 3. Otherwise check if the user is a member of the organization - STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ - -H "Authorization: token $ORG_ACCESS_TOKEN" \ - -H "Accept: application/vnd.github+json" \ - -H "X-GitHub-Api-Version: 2022-11-28" \ - "https://api.github.com/orgs/$ORG/members/$COMMENTER") - - if [ "$STATUS" -eq 204 ]; then - echo "is_member=true" >> $GITHUB_OUTPUT - else - echo "is_member=false" >> $GITHUB_OUTPUT - fi \ No newline at end of file diff --git a/.github/workflows/check-write-access.yml b/.github/workflows/check-write-access.yml new file mode 100644 index 0000000000..5e9494e74a --- /dev/null +++ b/.github/workflows/check-write-access.yml @@ -0,0 +1,66 @@ +name: Check Write Access + +# Authorizes a user to trigger privileged command workflows (/review, /ai, /plan, +# /updatesqlx, ...). The webhook author_association reports PRIVATE org members as +# CONTRIBUTOR/NONE (only public members show as MEMBER), so command jobs can't gate on +# it alone. This mints the internal GitHub App token — which can see private members — +# and confirms the user is a member or has write access to the repo. The app token is +# minted fresh per run, so unlike the old ORG_ACCESS_TOKEN PAT it never expires. + +on: + workflow_call: + inputs: + username: + required: true + type: string + description: 'The user whose access to verify' + trusted_bot: + required: false + type: string + default: 'windmill-internal-app[bot]' + description: 'A bot login that is always authorized' + outputs: + authorized: + description: 'true if the user is the trusted bot, an org member, or has repo write access' + value: ${{ jobs.check.outputs.authorized }} + +jobs: + check: + runs-on: ubuntu-latest + outputs: + authorized: ${{ steps.check.outputs.authorized }} + steps: + - name: Mint internal app token + id: app + uses: actions/create-github-app-token@v2 + with: + app-id: ${{ vars.INTERNAL_APP_ID }} + private-key: ${{ secrets.INTERNAL_APP_KEY }} + owner: ${{ github.repository_owner }} + + - name: Resolve authorization + id: check + env: + GH_TOKEN: ${{ steps.app.outputs.token }} + USERNAME: ${{ inputs.username }} + TRUSTED_BOT: ${{ inputs.trusted_bot }} + REPO: ${{ github.repository }} + run: | + if [ "$USERNAME" = "$TRUSTED_BOT" ]; then + echo "authorized=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + ORG="${REPO%%/*}" + # Org membership resolves private members too (204 = member, 404 = not). + if gh api "orgs/$ORG/members/$USERNAME" --silent 2>/dev/null; then + echo "authorized=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + # Fallback: effective repo permission (also covers outside collaborators). + PERM=$(gh api "repos/$REPO/collaborators/$USERNAME/permission" --jq '.permission' 2>/dev/null || echo none) + if [ "$PERM" = "admin" ] || [ "$PERM" = "write" ]; then + echo "authorized=true" >> "$GITHUB_OUTPUT" + else + echo "authorized=false" >> "$GITHUB_OUTPUT" + echo "$USERNAME is neither the trusted bot, an org member, nor a repo writer." + fi diff --git a/.github/workflows/claude-plan.yml b/.github/workflows/claude-plan.yml index c63e3fe1aa..ef25554f83 100644 --- a/.github/workflows/claude-plan.yml +++ b/.github/workflows/claude-plan.yml @@ -11,20 +11,24 @@ on: types: [submitted] jobs: - check-membership: + # author_association misses private org members; check-access resolves them via the + # internal app token. Both are OR'd below so public members still pass instantly. + check-access: if: | (github.event_name == 'issue_comment' && contains(github.event.comment.body, '/plan')) || (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/plan')) || (github.event_name == 'pull_request_review' && contains(github.event.review.body, '/plan')) || (github.event_name == 'issues' && contains(github.event.issue.body, '/plan')) - uses: ./.github/workflows/check-org-membership.yml - secrets: - access_token: ${{ secrets.ORG_ACCESS_TOKEN }} + uses: ./.github/workflows/check-write-access.yml + with: + username: ${{ github.event.comment.user.login || github.event.review.user.login || github.event.issue.user.login }} + secrets: inherit claude-plan-action: - needs: check-membership + needs: [check-access] if: | - needs.check-membership.outputs.is_member == 'true' + needs.check-access.outputs.authorized == 'true' || + contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association || github.event.review.author_association || github.event.issue.author_association) runs-on: ubicloud-standard-4 timeout-minutes: 20 permissions: diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index 0d9df9f0ac..115115dac6 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -11,20 +11,24 @@ on: types: [submitted] jobs: - check-membership: + # author_association misses private org members; check-access resolves them via the + # internal app token. Both are OR'd below so public members still pass instantly. + check-access: if: | (github.event_name == 'issue_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) || (github.event_name == 'pull_request_review_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) || (github.event_name == 'pull_request_review' && startsWith(github.event.review.body, '/ai') && !startsWith(github.event.review.body, '/ai-fast')) || (github.event_name == 'issues' && startsWith(github.event.issue.body, '/ai') && !startsWith(github.event.issue.body, '/ai-fast')) - uses: ./.github/workflows/check-org-membership.yml - secrets: - access_token: ${{ secrets.ORG_ACCESS_TOKEN }} + uses: ./.github/workflows/check-write-access.yml + with: + username: ${{ github.event.comment.user.login || github.event.review.user.login || github.event.issue.user.login }} + secrets: inherit claude-code-action: - needs: check-membership + needs: [check-access] if: | - needs.check-membership.outputs.is_member == 'true' + needs.check-access.outputs.authorized == 'true' || + contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association || github.event.review.author_association || github.event.issue.author_association) runs-on: ubicloud-standard-8 permissions: contents: write @@ -37,6 +41,44 @@ jobs: with: fetch-depth: 1 + # Make the EE source (the *_ee.rs files in the companion repo) available so the + # reviewer can see EE-only code (e.g. windmill-queue/src/jobs_ee.rs), not just the + # CE surface. The EE ref is read from the PR head's backend/ee-repo-ref.txt (via the + # API, so it reflects the PR's EE pin regardless of which ref is checked out here). + - name: Check EE access + id: ee + env: + EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number }} + run: | + if [ -z "$EE_TOKEN" ] || [ -z "$PR_NUMBER" ]; then + echo "available=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + HEAD_SHA=$(gh api "repos/${{ github.repository }}/pulls/$PR_NUMBER" --jq .head.sha) + REF=$(gh api "repos/${{ github.repository }}/contents/backend/ee-repo-ref.txt?ref=$HEAD_SHA" --jq .content | base64 -d | tr -d '[:space:]') + if [ -z "$REF" ]; then + echo "available=false" >> "$GITHUB_OUTPUT" + else + echo "available=true" >> "$GITHUB_OUTPUT" + echo "ee_repo_ref=$REF" >> "$GITHUB_OUTPUT" + fi + + - name: Checkout EE repository + if: steps.ee.outputs.available == 'true' + uses: actions/checkout@v4 + with: + repository: windmill-labs/windmill-ee-private + path: ./windmill-ee-private + ref: ${{ steps.ee.outputs.ee_repo_ref }} + token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} + fetch-depth: 1 + + - name: Substitute EE code + if: steps.ee.outputs.available == 'true' + run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private + - name: Run Claude PR Action uses: anthropics/claude-code-action@v1 with: diff --git a/.github/workflows/codex-pr-review.yml b/.github/workflows/codex-pr-review.yml index 66cc4b5f10..ea622914bc 100644 --- a/.github/workflows/codex-pr-review.yml +++ b/.github/workflows/codex-pr-review.yml @@ -32,27 +32,19 @@ concurrency: cancel-in-progress: true jobs: - check-membership: - if: github.event_name == 'pull_request' - uses: ./.github/workflows/check-org-membership.yml - with: - commenter: ${{ github.event.pull_request.user.login }} - secrets: - access_token: ${{ secrets.ORG_ACCESS_TOKEN }} - codex-review: - needs: check-membership runs-on: ubicloud-standard-2 timeout-minutes: 30 + # A non-fork PR (head.repo.fork == false) can only be opened by someone with push + # access to this repo, so fork==false already enforces write access. Do NOT re-add + # an author_association gate: the pull_request webhook payload reports private org + # members as CONTRIBUTOR/NONE (only public members show as MEMBER), which silently + # skips auto-review for every private member. if: | - always() && + github.event_name == 'workflow_call' || ( - needs.check-membership.result == 'skipped' || - (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true') - ) && - ( - github.event_name == 'workflow_call' || - (github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false) + github.event.pull_request.draft == false && + github.event.pull_request.head.repo.fork == false ) permissions: contents: read diff --git a/.github/workflows/git-commands.yaml b/.github/workflows/git-commands.yaml index f1cc380563..3443b7e649 100644 --- a/.github/workflows/git-commands.yaml +++ b/.github/workflows/git-commands.yaml @@ -5,21 +5,22 @@ on: types: [created] jobs: - check-membership: - if: >- - github.event.issue.pull_request && ( - startsWith(github.event.comment.body, '/updatesqlx') || - startsWith(github.event.comment.body, '/demo') || - startsWith(github.event.comment.body, '/eeref') || - startsWith(github.event.comment.body, '/docs') - ) - uses: ./.github/workflows/check-org-membership.yml - secrets: - access_token: ${{ secrets.ORG_ACCESS_TOKEN }} + # /command comments can come from anyone; author_association misses private org + # members, so check-access resolves them via the internal app token. Runs once and is + # OR'd into each job's guard (public members still pass on author_association alone). + check-access: + if: github.event.issue.pull_request != null && startsWith(github.event.comment.body, '/') + uses: ./.github/workflows/check-write-access.yml + with: + username: ${{ github.event.comment.user.login }} + secrets: inherit update-sqlx: - needs: check-membership - if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/updatesqlx') + needs: [check-access] + if: >- + github.event.issue.pull_request && + (contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || needs.check-access.outputs.authorized == 'true') && + startsWith(github.event.comment.body, '/updatesqlx') runs-on: ubicloud-standard-8 permissions: contents: write @@ -147,8 +148,11 @@ jobs: }) demo: - needs: check-membership - if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/demo') + needs: [check-access] + if: >- + github.event.issue.pull_request && + (contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || needs.check-access.outputs.authorized == 'true') && + startsWith(github.event.comment.body, '/demo') runs-on: ubicloud-standard-2 permissions: contents: read @@ -227,8 +231,11 @@ jobs: fi update-ee-ref: - needs: check-membership - if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/eeref') + needs: [check-access] + if: >- + github.event.issue.pull_request && + (contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || needs.check-access.outputs.authorized == 'true') && + startsWith(github.event.comment.body, '/eeref') runs-on: ubicloud-standard-2 permissions: contents: write @@ -313,8 +320,11 @@ jobs: }) update-docs: - needs: check-membership - if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/docs') + needs: [check-access] + if: >- + github.event.issue.pull_request && + (contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || needs.check-access.outputs.authorized == 'true') && + startsWith(github.event.comment.body, '/docs') runs-on: ubicloud-standard-2 permissions: contents: read diff --git a/.github/workflows/pi-pr-review.yml b/.github/workflows/pi-pr-review.yml index 9fbe43e9f0..72553b0d83 100644 --- a/.github/workflows/pi-pr-review.yml +++ b/.github/workflows/pi-pr-review.yml @@ -30,27 +30,19 @@ concurrency: cancel-in-progress: true jobs: - check-membership: - if: github.event_name == 'pull_request' - uses: ./.github/workflows/check-org-membership.yml - with: - commenter: ${{ github.event.pull_request.user.login }} - secrets: - access_token: ${{ secrets.ORG_ACCESS_TOKEN }} - pi-review: - needs: check-membership runs-on: ubicloud-standard-2 timeout-minutes: 30 + # A non-fork PR (head.repo.fork == false) can only be opened by someone with push + # access to this repo, so fork==false already enforces write access. Do NOT re-add + # an author_association gate: the pull_request webhook payload reports private org + # members as CONTRIBUTOR/NONE (only public members show as MEMBER), which silently + # skips auto-review for every private member. if: | - always() && + github.event_name == 'workflow_call' || ( - needs.check-membership.result == 'skipped' || - (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true') - ) && - ( - github.event_name == 'workflow_call' || - (github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false) + github.event.pull_request.draft == false && + github.event.pull_request.head.repo.fork == false ) permissions: contents: read diff --git a/.github/workflows/pr-ready-review.yml b/.github/workflows/pr-ready-review.yml index 4d722df4af..eb634977d1 100644 --- a/.github/workflows/pr-ready-review.yml +++ b/.github/workflows/pr-ready-review.yml @@ -30,26 +30,18 @@ concurrency: cancel-in-progress: true jobs: - check-membership: - if: github.event_name == 'pull_request' - uses: ./.github/workflows/check-org-membership.yml - with: - commenter: ${{ github.event.pull_request.user.login }} - secrets: - access_token: ${{ secrets.ORG_ACCESS_TOKEN }} - auto-review: - needs: check-membership runs-on: ubuntu-latest + # A non-fork PR (head.repo.fork == false) can only be opened by someone with push + # access to this repo, so fork==false already enforces write access. Do NOT re-add + # an author_association gate: the pull_request webhook payload reports private org + # members as CONTRIBUTOR/NONE (only public members show as MEMBER), which silently + # skips auto-review for every private member. if: | - always() && + github.event_name == 'workflow_call' || ( - needs.check-membership.result == 'skipped' || - (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true') - ) && - ( - github.event_name == 'workflow_call' || - (github.event.pull_request.draft == false || github.event.pull_request.ready_for_review == true) + (github.event.pull_request.draft == false || github.event.pull_request.ready_for_review == true) && + github.event.pull_request.head.repo.fork == false ) permissions: contents: read diff --git a/.github/workflows/pr-review-commands.yml b/.github/workflows/pr-review-commands.yml index ba55bfea2f..ef93274d7e 100644 --- a/.github/workflows/pr-review-commands.yml +++ b/.github/workflows/pr-review-commands.yml @@ -42,16 +42,24 @@ jobs: ;; esac - check-membership: - needs: parse + # author_association misses private org members; check-access resolves them via the + # internal app token. Both are OR'd so public members still pass instantly. + check-access: + needs: [parse] if: needs.parse.outputs.command != '' - uses: ./.github/workflows/check-org-membership.yml - secrets: - access_token: ${{ secrets.ORG_ACCESS_TOKEN }} + uses: ./.github/workflows/check-write-access.yml + with: + username: ${{ github.event.comment.user.login }} + secrets: inherit acknowledge: - needs: [parse, check-membership] - if: needs.parse.outputs.command != '' && needs.check-membership.outputs.is_member == 'true' + needs: [parse, check-access] + if: | + needs.parse.outputs.command != '' && + ( + contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || + needs.check-access.outputs.authorized == 'true' + ) runs-on: ubuntu-latest permissions: issues: write @@ -68,9 +76,12 @@ jobs: -f content=eyes >/dev/null claude: - needs: [parse, check-membership] + needs: [parse, check-access] if: | - needs.check-membership.outputs.is_member == 'true' && + ( + contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || + needs.check-access.outputs.authorized == 'true' + ) && (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'claude') permissions: contents: read @@ -86,9 +97,12 @@ jobs: WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} codex: - needs: [parse, check-membership] + needs: [parse, check-access] if: | - needs.check-membership.outputs.is_member == 'true' && + ( + contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || + needs.check-access.outputs.authorized == 'true' + ) && (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'codex') permissions: contents: read @@ -105,9 +119,12 @@ jobs: WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} pi: - needs: [parse, check-membership] + needs: [parse, check-access] if: | - needs.check-membership.outputs.is_member == 'true' && + ( + contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) || + needs.check-access.outputs.authorized == 'true' + ) && (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'pi') permissions: contents: read diff --git a/.github/workflows/refresh-docs-snapshot.yml b/.github/workflows/refresh-docs-snapshot.yml new file mode 100644 index 0000000000..a000f44df8 --- /dev/null +++ b/.github/workflows/refresh-docs-snapshot.yml @@ -0,0 +1,52 @@ +name: Refresh docs snapshot + +# The backend embeds a vendored docs snapshot (backend/windmill-api/docs_snapshot/*.gz) +# so in-product docs search works with no runtime egress. This job re-fetches it from +# windmill.dev on a schedule and opens a PR when it changed, keeping the embedded docs +# fresh independently of the release cadence (the binary embeds whatever is on the +# source tree at build time, so a merged refresh rides into the next release build). +on: + schedule: + - cron: "0 6 * * 1" # Mondays 06:00 UTC + workflow_dispatch: + +jobs: + refresh: + runs-on: ubicloud + permissions: + contents: write + pull-requests: write + steps: + - uses: actions/create-github-app-token@v2 + id: app + with: + app-id: ${{ vars.INTERNAL_APP_ID }} + private-key: ${{ secrets.INTERNAL_APP_KEY }} + - uses: actions/checkout@v4 + with: + token: ${{ steps.app.outputs.token }} + - name: Fetch + re-gzip docs snapshot + run: cd backend/windmill-api/docs_snapshot && ./fetch.sh + - name: Sanity-check the fetched corpus + # curl -f in fetch.sh rejects HTTP errors, but not a valid-but-garbage 200 + # (truncated file, error page). Guard against embedding a broken snapshot. + run: | + cd backend/windmill-api/docs_snapshot + test "$(wc -c < llms-full.txt.gz)" -gt 100000 + test "$(wc -c < llms.txt.gz)" -gt 1000 + pages=$(gzip -dc llms-full.txt.gz | grep -c '^Source:' || true) + echo "pages in snapshot: $pages" + test "${pages:-0}" -ge 200 + - uses: peter-evans/create-pull-request@v6 + with: + token: ${{ steps.app.outputs.token }} + branch: chore/refresh-docs-snapshot + add-paths: backend/windmill-api/docs_snapshot/*.gz + commit-message: "chore: refresh vendored docs snapshot" + title: "chore: refresh vendored docs snapshot" + body: | + Automated refresh of the embedded docs snapshot + (`backend/windmill-api/docs_snapshot/*.gz`) from windmill.dev. + + Review the diff for unexpected churn (a bad upstream docs deploy would + show up as a large drop in pages or content) before merging. diff --git a/AGENTS.md b/AGENTS.md index aead237e59..83b68f7d36 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,8 @@ Open-source platform for internal tools, workflows, API integrations, background ## Dev Environment - **Backend**: `cargo run` from `backend/` (API at http://localhost:8000) +- **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing. The bundled DuckDB compile (~2min) is cached in a per-user dir shared across worktrees, so a fresh worktree reuses it and the build is near-instant. +- **Data pipelines (DuckLake) from source**: a plain `cargo run` (even `--features quickjs`) advertises a `duckdb` worker tag but **cannot** execute DuckDB scripts and has **no** working S3 proxy (DuckLake writes 404). Build CE DuckLake with `cargo run --features quickjs,duckdb,parquet,private` (add `,python` for Python scripts, `,enterprise,license` for EE) **and** build the FFI (bullet above). See `backend/CLAUDE.md` → "Running data pipelines (DuckLake) from source" for the exact feature sets and the two feature-gate gotchas. - **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+) - **DB**: `psql postgres://postgres:changeme@localhost:5432/windmill` - **Login**: `admin@windmill.dev` / `changeme` diff --git a/CHANGELOG.md b/CHANGELOG.md index b0bc5876af..5c7cea1e37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,436 @@ # Changelog +## [1.753.0](https://github.com/windmill-labs/windmill/compare/v1.752.0...v1.753.0) (2026-07-08) + + +### Features + +* AI chat background jobs tray with detach, approval and preview ([#9982](https://github.com/windmill-labs/windmill/issues/9982)) ([286da00](https://github.com/windmill-labs/windmill/commit/286da005ef2faad1d193640f74f8e96999358707)) +* condensed top bar for session preview editors ([#10011](https://github.com/windmill-labs/windmill/issues/10011)) ([b847ca2](https://github.com/windmill-labs/windmill/commit/b847ca2bc7f06f494aa802d4350d6f032ba2bb58)) +* **db-health:** add connection sizing guidance ([#10014](https://github.com/windmill-labs/windmill/issues/10014)) ([f28ea9c](https://github.com/windmill-labs/windmill/commit/f28ea9cb991bbda32ed1b7a37a5f1b3552a589a8)) +* **sessions:** scoped preview refresh + multi-target live editors + pipeline preview ([#10006](https://github.com/windmill-labs/windmill/issues/10006)) ([32c398f](https://github.com/windmill-labs/windmill/commit/32c398f27de8cd5b1478ef60d247c13705b6b50f)) +* shared tab system, universal markdown code blocks, subtle scrollbars ([#10003](https://github.com/windmill-labs/windmill/issues/10003)) ([a00ee51](https://github.com/windmill-labs/windmill/commit/a00ee5196b2c013e9672ab029f5477079ac5da21)) + + +### Bug Fixes + +* bump bundled Go CLIs to patched versions to clear image CVEs ([#9996](https://github.com/windmill-labs/windmill/issues/9996)) ([d467161](https://github.com/windmill-labs/windmill/commit/d467161117444d7d9b18def627e90d9622512e02)) +* name the offending item when a fork fails on a NUL escape ([#10013](https://github.com/windmill-labs/windmill/issues/10013)) ([99d0047](https://github.com/windmill-labs/windmill/commit/99d00475156def6faad255c4e728923253f9169f)) +* preserve worker group tag override on 'Run again' ([#10004](https://github.com/windmill-labs/windmill/issues/10004)) ([c4cb2f3](https://github.com/windmill-labs/windmill/commit/c4cb2f373b6361f0f3ce6b1c8e32a4c010207760)) +* replicate external secret backend secrets when forking a workspace ([#10007](https://github.com/windmill-labs/windmill/issues/10007)) ([f65fe7b](https://github.com/windmill-labs/windmill/commit/f65fe7bf585d353f7d88746e947d68e2f351e516)) +* session preview editors and picker dropdown overflow ([#10010](https://github.com/windmill-labs/windmill/issues/10010)) ([fb12b23](https://github.com/windmill-labs/windmill/commit/fb12b23e0169ba2cdcf454a27dcf814a2caf26b3)) + +## [1.752.0](https://github.com/windmill-labs/windmill/compare/v1.751.0...v1.752.0) (2026-07-07) + + +### Features + +* add fork_parent_workspace claim to OIDC tokens for fork workspaces ([#9987](https://github.com/windmill-labs/windmill/issues/9987)) ([7efeae2](https://github.com/windmill-labs/windmill/commit/7efeae26d821b10667b6e3edd220468f6ae48936)) +* add SQL migrations for data tables ([#9693](https://github.com/windmill-labs/windmill/issues/9693)) ([e47aeda](https://github.com/windmill-labs/windmill/commit/e47aedac0a4af40dd697d5fc4d54dd3c8efe9ab8)) +* **cli:** clarify fork-branch workspace auto-targeting in output ([#9988](https://github.com/windmill-labs/windmill/issues/9988)) ([88c2d0e](https://github.com/windmill-labs/windmill/commit/88c2d0e8e32c218787c01daed80ef41efc39dd11)) +* open runs/schedules pages from AI chat in session preview tabs ([#9976](https://github.com/windmill-labs/windmill/issues/9976)) ([4bb82ad](https://github.com/windmill-labs/windmill/commit/4bb82ad6cdb62eae7b69b1054714333e54558632)) +* **raw-apps:** runtime-error overlay + AI import-React instruction ([#9966](https://github.com/windmill-labs/windmill/issues/9966)) ([8df613b](https://github.com/windmill-labs/windmill/commit/8df613b4d2f88765f49cc988a894ca323c4ec4f7)) +* **sessions:** v2 unified sidebar with family/fork scoping and preview router ([#9816](https://github.com/windmill-labs/windmill/issues/9816)) ([9503190](https://github.com/windmill-labs/windmill/commit/95031903ebe223dc03b49a6bcd3e4ee67cefc4bb)) +* smooth bursty AI chat streaming with a typewriter reveal ([#9991](https://github.com/windmill-labs/windmill/issues/9991)) ([a6276b5](https://github.com/windmill-labs/windmill/commit/a6276b590082d06480434a8ea002c335ea1cfb59)) +* update base image to debian 13 (trixie) ([#9973](https://github.com/windmill-labs/windmill/issues/9973)) ([c5c1ead](https://github.com/windmill-labs/windmill/commit/c5c1eadeb18e509a98d1e787206c0438417683fc)) + + +### Bug Fixes + +* **ai-agent:** align agent_actions_success with agent_actions for mcp and websearch ([#9983](https://github.com/windmill-labs/windmill/issues/9983)) ([87f8d46](https://github.com/windmill-labs/windmill/commit/87f8d46aafffd5e88a336192c51e0c95ff2e6f18)) +* **ai:** flow writer builds approval steps as scripts, not identity ([#9985](https://github.com/windmill-labs/windmill/issues/9985)) ([6b01caa](https://github.com/windmill-labs/windmill/commit/6b01caaf26a4f0a08f643db4e22a70e27d0dc554)) +* clear old path asset usage when renaming a script ([#9979](https://github.com/windmill-labs/windmill/issues/9979)) ([927b8d0](https://github.com/windmill-labs/windmill/commit/927b8d064f693384978184992b8f8a1cd708e711)) +* **cli:** auto-derive cascade triggers in --local pipeline graph ([#9978](https://github.com/windmill-labs/windmill/issues/9978)) ([edfe7b4](https://github.com/windmill-labs/windmill/commit/edfe7b415af6670c5855b7a0b52db4c1f7781964)) +* **pipelines:** live materialize/dataset editing — stale graph, phantom drafts, stale Save-all deploys ([#9990](https://github.com/windmill-labs/windmill/issues/9990)) ([f7efb64](https://github.com/windmill-labs/windmill/commit/f7efb646bf1f2e132d1e3ff031b142383ae01c5e)) +* **sessions:** auto-rename regression + preview-panel and fork nits ([#9993](https://github.com/windmill-labs/windmill/issues/9993)) ([804178f](https://github.com/windmill-labs/windmill/commit/804178f5e1c904c3f8e35e2b660f33c78964c6eb)) +* **sessions:** scope fork session Edits to session-edited items only ([#9989](https://github.com/windmill-labs/windmill/issues/9989)) ([7046dc6](https://github.com/windmill-labs/windmill/commit/7046dc6dfb474ef49313377855bb2bd60294e25a)) + +## [1.751.0](https://github.com/windmill-labs/windmill/compare/v1.750.0...v1.751.0) (2026-07-06) + + +### Features + +* add cosmetic dev/staging label for dev workspaces ([#9959](https://github.com/windmill-labs/windmill/issues/9959)) ([fd8e64d](https://github.com/windmill-labs/windmill/commit/fd8e64d11fea3ffdb7858100c67cb2e9ca841ed6)) +* **auth:** add runtime NO_AUTH mode for authentication bypass ([#9962](https://github.com/windmill-labs/windmill/issues/9962)) ([91e1b08](https://github.com/windmill-labs/windmill/commit/91e1b087a206efb7189824b4184e1f3f4cda7211)) +* **frontend:** custom skills — detail modal, batch manage, shared validation ([#9847](https://github.com/windmill-labs/windmill/issues/9847)) ([2e14302](https://github.com/windmill-labs/windmill/commit/2e14302e4abbad595584806bff12548d520fcb58)) +* **pipelines:** auto-derive cascade edges from ducklake/s3 reads (+ muted-read badge) ([#9963](https://github.com/windmill-labs/windmill/issues/9963)) ([3dcd394](https://github.com/windmill-labs/windmill/commit/3dcd3949a14199b106506994ea31ca3de7e636b3)) + + +### Bug Fixes + +* **ai:** centralize Anthropic Messages API routing across completion paths ([#9960](https://github.com/windmill-labs/windmill/issues/9960)) ([cc2f638](https://github.com/windmill-labs/windmill/commit/cc2f638de6cebeffb9fee1d4835a0cfd565af86c)) +* **assets:** responsive layout for small screens ([#9961](https://github.com/windmill-labs/windmill/issues/9961)) ([45946d1](https://github.com/windmill-labs/windmill/commit/45946d1185c0bd07948d4d8454880c2801571f9d)) +* **cli:** quote non-identifier property names in resource-type namespace ([#9964](https://github.com/windmill-labs/windmill/issues/9964)) ([dc6b997](https://github.com/windmill-labs/windmill/commit/dc6b99775b550e7433fee8a159c30eaf296500c5)) +* critical alerts modal mute toggles no longer close popover or fail to save ([#9969](https://github.com/windmill-labs/windmill/issues/9969)) ([6587019](https://github.com/windmill-labs/windmill/commit/6587019d263374ee5707d258f5d8eec7e73c690d)) +* **frontend:** theme-aware code block background in prose markdown ([#9968](https://github.com/windmill-labs/windmill/issues/9968)) ([9821596](https://github.com/windmill-labs/windmill/commit/9821596251cff698958ffbfbd11fffa6a7988c6c)) + +## [1.750.0](https://github.com/windmill-labs/windmill/compare/v1.749.0...v1.750.0) (2026-07-06) + + +### Features + +* chat-scoped session changes bar + unified diff drawer ([#9762](https://github.com/windmill-labs/windmill/issues/9762)) ([a6c0b37](https://github.com/windmill-labs/windmill/commit/a6c0b3756be78ca3fadc7bad6bae98c0887fd538)) +* **pipelines:** require data uploads before running a pipeline ([#9953](https://github.com/windmill-labs/windmill/issues/9953)) ([a1c5b7a](https://github.com/windmill-labs/windmill/commit/a1c5b7aa3ed2841f09f4f148ded5c5b5ef10fd3d)) +* **pipelines:** wm_partition macro for grain-agnostic partition filters ([#9950](https://github.com/windmill-labs/windmill/issues/9950)) ([43044c2](https://github.com/windmill-labs/windmill/commit/43044c2e28139b1dbde6844c781a821f8de68f58)) + + +### Bug Fixes + +* **ai:** test key routes Azure Foundry Claude models via Anthropic Messages API ([#9956](https://github.com/windmill-labs/windmill/issues/9956)) ([ea19cc9](https://github.com/windmill-labs/windmill/commit/ea19cc9dc459bd259e27f7fcc29601a010c5f8f0)) +* **cli:** HD-1 test_edges + HD-2 scd2 _current write in --local pipeline graph ([#9947](https://github.com/windmill-labs/windmill/issues/9947)) ([ad6f23d](https://github.com/windmill-labs/windmill/commit/ad6f23d6bfcf1056bcb6d8c6b552114e88177328)) +* **pipelines:** make node & pipeline-level run affordances always visible ([#9948](https://github.com/windmill-labs/windmill/issues/9948)) ([6eabb96](https://github.com/windmill-labs/windmill/commit/6eabb96ae78fb966f9916f907bb693d569b04c0b)) +* read chat drafts via own-draft route so drawer-kind drafts deploy ([#9913](https://github.com/windmill-labs/windmill/issues/9913)) ([056ebdb](https://github.com/windmill-labs/windmill/commit/056ebdb03543a93094c80ca354c117236cd8d6c8)) +* resolve extensionless bun relative imports on windows loader ([#9949](https://github.com/windmill-labs/windmill/issues/9949)) ([bf96621](https://github.com/windmill-labs/windmill/commit/bf9662172ad7e0ff53d39adc338fd7886672c8f9)) + +## [1.749.0](https://github.com/windmill-labs/windmill/compare/v1.748.0...v1.749.0) (2026-07-05) + + +### Features + +* **pipelines:** mid-DAG selective execution (dbt `model+`) for pipeline runs ([#9945](https://github.com/windmill-labs/windmill/issues/9945)) ([2d3a773](https://github.com/windmill-labs/windmill/commit/2d3a77344104a587548f23f1b614ceffd52a5778)) +* **pipelines:** partition run-arg picker + first-run setup signpost ([#9943](https://github.com/windmill-labs/windmill/issues/9943)) ([475b072](https://github.com/windmill-labs/windmill/commit/475b072987b33d50111f5251a5f69f4245f930ae)) +* **pipelines:** self-teaching custom data_test errors + scaffold ([#9937](https://github.com/windmill-labs/windmill/issues/9937)) ([0ad174f](https://github.com/windmill-labs/windmill/commit/0ad174fa490e17eeb26280b5bdfd62956dfed9ff)) + + +### Bug Fixes + +* **cli:** macro-library parity in --local pipeline graph + read-only run --dry-run ([#9942](https://github.com/windmill-labs/windmill/issues/9942)) ([e3f4303](https://github.com/windmill-labs/windmill/commit/e3f43033cafcdb5df253aeb55ce93e599b2584d2)) +* **datatable:** self-teaching error for unresolved datatable:// references ([#9941](https://github.com/windmill-labs/windmill/issues/9941)) ([55451db](https://github.com/windmill-labs/windmill/commit/55451db009e3060c21948ece2c97e102a3c9b171)) +* **object-storage:** remove 20-file bucket-browser listing cap in CE ([#9935](https://github.com/windmill-labs/windmill/issues/9935)) ([22452ce](https://github.com/windmill-labs/windmill/commit/22452ce54034a9bea8f7d48946818fd148b938c0)) +* **pipelines:** dedup guard for keyed merge + deploy-time SCD2 validation ([#9936](https://github.com/windmill-labs/windmill/issues/9936)) ([52ce805](https://github.com/windmill-labs/windmill/commit/52ce805f619747af4f998cde7819a164c754205a)) +* **pipelines:** link SCD2 <dim>_current view to its producer across all graph surfaces ([#9933](https://github.com/windmill-labs/windmill/issues/9933)) ([574d3ac](https://github.com/windmill-labs/windmill/commit/574d3ac9ff5015b5d3f53040c9d4dfbfd161a076)) +* **pipelines:** order data_test relationships refs before the tested script in a cascade ([#9934](https://github.com/windmill-labs/windmill/issues/9934)) ([46be39d](https://github.com/windmill-labs/windmill/commit/46be39dfb7fbfb2b70e61819d6065b45810c41c9)) +* **pipelines:** pipeline-level run control, tables label, data-test rollback + fork badges ([#9944](https://github.com/windmill-labs/windmill/issues/9944)) ([6ae8dd3](https://github.com/windmill-labs/windmill/commit/6ae8dd37b1de930ab17344cebf7c28385c6cfdba)) +* rebuild windows bun loader main.ts filter from forward-slash cdir ([#9946](https://github.com/windmill-labs/windmill/issues/9946)) ([a582e04](https://github.com/windmill-labs/windmill/commit/a582e04bf40cf685f88bceaf88e3d24bde3d420a)) + +## [1.748.0](https://github.com/windmill-labs/windmill/compare/v1.747.0...v1.748.0) (2026-07-05) + + +### Features + +* **ai-agent:** support reasoning effort in AI agent workflow steps ([#9886](https://github.com/windmill-labs/windmill/issues/9886)) ([a368d49](https://github.com/windmill-labs/windmill/commit/a368d49bd8786a2dca6771f2051f1d44d1b2363d)) +* **ducklake:** scheduled lake maintenance (expiry, compaction, orphan cleanup) ([#9916](https://github.com/windmill-labs/windmill/issues/9916)) ([3352150](https://github.com/windmill-labs/windmill/commit/33521505dbc34f22b575d21fda1cc76d698a8840)) +* **pipelines:** asset freshness — fresh/stale badge (CE) + watchdog (EE) ([#9909](https://github.com/windmill-labs/windmill/issues/9909)) ([5d7fb6d](https://github.com/windmill-labs/windmill/commit/5d7fb6deca3e02e89d77e5d3856483beb8b8bfeb)) +* **pipelines:** capture violating-row samples for data tests ([#9919](https://github.com/windmill-labs/windmill/issues/9919)) ([d4b4374](https://github.com/windmill-labs/windmill/commit/d4b4374de8f8a7875b050c16d1236fcd0355812b)) +* **pipelines:** fork data environments for ducklake materialization (dev data) ([#9915](https://github.com/windmill-labs/windmill/issues/9915)) ([39eb9de](https://github.com/windmill-labs/windmill/commit/39eb9de1bce400109c130a081807e40e995ae068)) +* **pipelines:** on_schema_change write guardrails + data_test deploy validation ([#9930](https://github.com/windmill-labs/windmill/issues/9930)) ([377c02e](https://github.com/windmill-labs/windmill/commit/377c02ec47389e64b7ef5cbbae0de05df648266d)) +* **pipelines:** record upstream snapshot ids on cascade-dispatched jobs ([#9910](https://github.com/windmill-labs/windmill/issues/9910)) ([af36498](https://github.com/windmill-labs/windmill/commit/af36498432e643108308e1c03b5d986d0f0f8888)) +* **pipelines:** schema contracts — save-time consumer checks vs captured schemas ([#9917](https://github.com/windmill-labs/windmill/issues/9917)) ([42e11c6](https://github.com/windmill-labs/windmill/commit/42e11c6570b62ffaa86598438fa8ddf462c4035f)) +* **pipeline:** write-audit-publish for materialization data tests ([#9911](https://github.com/windmill-labs/windmill/issues/9911)) ([dce247c](https://github.com/windmill-labs/windmill/commit/dce247c6d2678a2c95bd728027e17ae3965638e2)) +* **sdk:** enforce s3:// URIs for string S3 params + ingestion (EL) docs ([#9912](https://github.com/windmill-labs/windmill/issues/9912)) ([5ad2de9](https://github.com/windmill-labs/windmill/commit/5ad2de91a26b312bf27124ceca16ef331621bde8)) + + +### Bug Fixes + +* **cli:** pipeline + workspace UX batch (init/bind stub, run errors, macro libs, lock-job report, upgrade errors) ([#9929](https://github.com/windmill-labs/windmill/issues/9929)) ([28a6b08](https://github.com/windmill-labs/windmill/commit/28a6b086c842105298f236baa0a61868f71a5eb1)) +* **cli:** publish all windmill-parser-wasm-* deps so local pipeline graph keeps write edges ([#9926](https://github.com/windmill-labs/windmill/issues/9926)) ([744a759](https://github.com/windmill-labs/windmill/commit/744a7597edaf3ca9a7fd2b21a34fb33457913a64)) +* **pipelines:** activity-axis label clarity + select failed node on cascade failure ([#9931](https://github.com/windmill-labs/windmill/issues/9931)) ([5769b60](https://github.com/windmill-labs/windmill/commit/5769b6036cf14b0cb424c5b3d9d878c600a5652e)) + +## [1.747.0](https://github.com/windmill-labs/windmill/compare/v1.746.0...v1.747.0) (2026-07-03) + + +### Features + +* **frontend:** add federatedTokenFile field to instance object storage Azure config ([#9904](https://github.com/windmill-labs/windmill/issues/9904)) ([ae85d27](https://github.com/windmill-labs/windmill/commit/ae85d274371a24c5badb6081f00deeb409123252)) + + +### Bug Fixes + +* **ai:** route Azure Foundry Claude models via Anthropic Messages API ([#9908](https://github.com/windmill-labs/windmill/issues/9908)) ([d600c7e](https://github.com/windmill-labs/windmill/commit/d600c7ecfe305533798e82e8d05e5f2f297f9b54)) +* **forks:** clone only the current raw-app bundle, via server-side copy ([#9899](https://github.com/windmill-labs/windmill/issues/9899)) ([5c521d8](https://github.com/windmill-labs/windmill/commit/5c521d808a2b5d6d6bb7cf3da17fb2addc53fdf4)) +* **kafka:** set https.ca.location=probe for OAUTHBEARER OIDC token endpoint ([#9897](https://github.com/windmill-labs/windmill/issues/9897)) ([1b6065f](https://github.com/windmill-labs/windmill/commit/1b6065fa9201fd548c4b2ef199f1009200645929)) +* prevent truncated tool call args from bricking AI chat sessions ([#9902](https://github.com/windmill-labs/windmill/issues/9902)) ([4ba17d0](https://github.com/windmill-labs/windmill/commit/4ba17d0f9cd70489f89c84f982a0c8f0062fed1a)) +* strip NUL characters from app values at save time ([#9903](https://github.com/windmill-labs/windmill/issues/9903)) ([3ec1f16](https://github.com/windmill-labs/windmill/commit/3ec1f164be9c8c6c40e003188ce593a963c65a43)) + +## [1.746.0](https://github.com/windmill-labs/windmill/compare/v1.745.0...v1.746.0) (2026-07-02) + + +### Features + +* **ai:** add Azure AI Foundry as a native AI provider ([#9879](https://github.com/windmill-labs/windmill/issues/9879)) ([d9b080f](https://github.com/windmill-labs/windmill/commit/d9b080f57fa0be144cefa773d39742c45b40f043)) +* **frontend:** group compare & deploy items by folder ([#9880](https://github.com/windmill-labs/windmill/issues/9880)) ([7b04820](https://github.com/windmill-labs/windmill/commit/7b04820f8ef8c7f02f79dd4239a877f667d23e6a)) +* **frontend:** pipelines index page and sql editor hint ([#9881](https://github.com/windmill-labs/windmill/issues/9881)) ([20351a6](https://github.com/windmill-labs/windmill/commit/20351a6b4c262184c5f815eeb5de007ab1eaf4a0)) +* **pipeline:** backfill a range of partitions from the asset drawer ([#9885](https://github.com/windmill-labs/windmill/issues/9885)) ([53bbb92](https://github.com/windmill-labs/windmill/commit/53bbb92953178eb6d0017818ef870f3cb2399dfd)) +* **pipelines:** workspace duckdb macro libraries (// macros / // use) ([#9890](https://github.com/windmill-labs/windmill/issues/9890)) ([84141ad](https://github.com/windmill-labs/windmill/commit/84141add1ddf35c2573e3c213366ce7c5f1f2258)) +* **s3:** replace CE 50MB upload cap with 10GiB workspace storage quota ([#9874](https://github.com/windmill-labs/windmill/issues/9874)) ([af01e90](https://github.com/windmill-labs/windmill/commit/af01e90b5c65d1b1cfacf4433f8cff7effe73768)) +* support workspace forks on cloud using parent workspace limits ([#9864](https://github.com/windmill-labs/windmill/issues/9864)) ([7c7d747](https://github.com/windmill-labs/windmill/commit/7c7d7474cc86a4052272032f281cc4d7a85db37b)) + + +### Bug Fixes + +* **duckdb:** auto-declare partition arg for `// partitioned` scripts ([#9878](https://github.com/windmill-labs/windmill/issues/9878)) ([b883adb](https://github.com/windmill-labs/windmill/commit/b883adbc0011073da592dc5b39e1b79db492c83c)) +* **frontend:** home New submenus fall back below, hugging the right edge ([#9894](https://github.com/windmill-labs/windmill/issues/9894)) ([186ac49](https://github.com/windmill-labs/windmill/commit/186ac4933b79aed57fce23ebcf3b525fcfd1c474)) +* **frontend:** show inline workspace name editor on general settings (Fixes GIT-911) ([#9892](https://github.com/windmill-labs/windmill/issues/9892)) ([a49c087](https://github.com/windmill-labs/windmill/commit/a49c0871d7ab2aaf78a7713b8a786ead937434da)) +* **frontend:** stack cron field and cron builder button on narrow screens ([#9871](https://github.com/windmill-labs/windmill/issues/9871)) ([7989795](https://github.com/windmill-labs/windmill/commit/79897950e7646b00d92a28a009174d91c705b251)) +* invalidate bun bundle cache on transitive relative-import changes ([#9891](https://github.com/windmill-labs/windmill/issues/9891)) ([d15033c](https://github.com/windmill-labs/windmill/commit/d15033cde6a474b548ebbaf18ff02223fc21f701)) +* make SMTP username and password optional in frontend validation ([#9895](https://github.com/windmill-labs/windmill/issues/9895)) ([37bb574](https://github.com/windmill-labs/windmill/commit/37bb57474e8336823bb31527f2a708ef41cd39c4)) +* **parsers:** infer py s3 assets from S3Object constructor and dict forms ([#9877](https://github.com/windmill-labs/windmill/issues/9877)) ([659642e](https://github.com/windmill-labs/windmill/commit/659642e4889361f86e8addb038cda62fc3471006)) +* pipeline dogfooding fixes — SCD2 data-test scope, --partition, s3object upload binding ([#9875](https://github.com/windmill-labs/windmill/issues/9875)) ([d65f58c](https://github.com/windmill-labs/windmill/commit/d65f58c388d88fff71cda22dfa21aecdae70c450)) +* polish pipeline graph view (layout, viewport, minimap, lineage, timestamps) ([#9883](https://github.com/windmill-labs/windmill/issues/9883)) ([b92a86b](https://github.com/windmill-labs/windmill/commit/b92a86b8b3a60b877540c3a7f0ffefe36ccbb053)) +* stale AI chat context picker after workspace item changes ([#9893](https://github.com/windmill-labs/windmill/issues/9893)) ([5af91a6](https://github.com/windmill-labs/windmill/commit/5af91a677cad88faccba702e3556fc4fb7b6e640)) +* **triggers:** retry transient websocket connect failures before disabling ([#9887](https://github.com/windmill-labs/windmill/issues/9887)) ([7894507](https://github.com/windmill-labs/windmill/commit/789450731b0a3c8dffa336f7bfc3f3de528c09fb)) + +## [1.745.0](https://github.com/windmill-labs/windmill/compare/v1.744.0...v1.745.0) (2026-07-01) + + +### Features + +* **forks:** partial-visibility deploy + surface hidden items ([#9868](https://github.com/windmill-labs/windmill/issues/9868)) ([20cd1a0](https://github.com/windmill-labs/windmill/commit/20cd1a02d582c0715bedacce52cc5c1e1e8d70ca)) +* **frontend:** add zoom and download to Mermaid graphs ([#9859](https://github.com/windmill-labs/windmill/issues/9859)) ([289017b](https://github.com/windmill-labs/windmill/commit/289017bcb28c049c8258b2ffd7da0ec3e6ef120b)) +* use derived username instead of email for non-member superadmins ([#9857](https://github.com/windmill-labs/windmill/issues/9857)) ([76a9523](https://github.com/windmill-labs/windmill/commit/76a95230095ca3f43c9dc9eecde0e9de6520242f)) + + +### Bug Fixes + +* **cli:** correct misleading delete-fork command description ([#9870](https://github.com/windmill-labs/windmill/issues/9870)) ([a73b14d](https://github.com/windmill-labs/windmill/commit/a73b14d902d759226d0af2f2faf9bdd6588e358c)) +* **folders:** allow dots and at-signs in folder owner validation ([#9856](https://github.com/windmill-labs/windmill/issues/9856)) ([383c705](https://github.com/windmill-labs/windmill/commit/383c70523bf81c5c07784a4379ef6b1c93ff86e5)) +* **forks:** require admin of both sides for the compare visibility guard ([#9869](https://github.com/windmill-labs/windmill/issues/9869)) ([7363d2c](https://github.com/windmill-labs/windmill/commit/7363d2c217cb04391f03b2f9958be70a9d0b5325)) +* **forks:** reset diff tally on trigger delete + guard compare visibility for admins ([#9866](https://github.com/windmill-labs/windmill/issues/9866)) ([6a6f129](https://github.com/windmill-labs/windmill/commit/6a6f12960e29c314d11ad541519c71412f42567b)) +* **jobs:** give flow dynselect a path and its worker tag, like scripts ([#9867](https://github.com/windmill-labs/windmill/issues/9867)) ([1a9debb](https://github.com/windmill-labs/windmill/commit/1a9debb689f756f38db085d1360c8fe7691ada48)) +* **offboarding:** make global reassignment per-workspace and optional ([#9863](https://github.com/windmill-labs/windmill/issues/9863)) ([3586164](https://github.com/windmill-labs/windmill/commit/35861641f807a02b5c205608fb592e20ee7cad7f)) + +## [1.744.0](https://github.com/windmill-labs/windmill/compare/v1.743.0...v1.744.0) (2026-07-01) + + +### Features + +* add copy-to-clipboard button to rendered Mermaid diagrams in AI chat ([#9838](https://github.com/windmill-labs/windmill/issues/9838)) ([a27e814](https://github.com/windmill-labs/windmill/commit/a27e814a03c615259381eaf684aa90d56569b0af)) +* add dev workspaces paired with a lockable prod workspace ([#9793](https://github.com/windmill-labs/windmill/issues/9793)) ([b4b0c6a](https://github.com/windmill-labs/windmill/commit/b4b0c6a93e52152251fadefe319773faf42549b2)) +* **ansible:** support repo-provided ansible.cfg in delegate_to_git_repo ([#9851](https://github.com/windmill-labs/windmill/issues/9851)) ([68bf0da](https://github.com/windmill-labs/windmill/commit/68bf0daf5815307cda6ce23214dd5159b6aa33b4)) +* **licensing:** enforce offline license seat cap ([#9845](https://github.com/windmill-labs/windmill/issues/9845)) ([83f3d7f](https://github.com/windmill-labs/windmill/commit/83f3d7f910b331c09f60cc9ff556728afa3dec07)) +* **object-store:** make GCS service account key optional for Workload Identity ([#9842](https://github.com/windmill-labs/windmill/issues/9842)) ([83ed011](https://github.com/windmill-labs/windmill/commit/83ed011e264f20ffa66a7bf933f2fe3615cf6b67)) +* **pipeline:** local development for data pipelines (CLI --local + pipeline dev preview) ([#9840](https://github.com/windmill-labs/windmill/issues/9840)) ([74f579e](https://github.com/windmill-labs/windmill/commit/74f579e6d9ef08e74460f904a4c22ed9d6a3b5b0)) +* **pipelines:** add managed SCD2 history materialize strategy ([#9850](https://github.com/windmill-labs/windmill/issues/9850)) ([5a66127](https://github.com/windmill-labs/windmill/commit/5a661279a3690e2393b9b16996f5d1a5a509259c)) + + +### Bug Fixes + +* **ai-chat:** replay anthropic turns verbatim to keep thinking valid ([#9843](https://github.com/windmill-labs/windmill/issues/9843)) ([a37a144](https://github.com/windmill-labs/windmill/commit/a37a144e81cf6b3de935688a617e9d0e1756004a)) +* grant dispatch_event table to windmill roles ([#9852](https://github.com/windmill-labs/windmill/issues/9852)) ([f05b50d](https://github.com/windmill-labs/windmill/commit/f05b50d29ac2fdbb808a97057fb92c8e425b4a2f)) +* grant workspace_diff, materialized_partition, debounce_stale_data to windmill roles ([#9853](https://github.com/windmill-labs/windmill/issues/9853)) ([293647d](https://github.com/windmill-labs/windmill/commit/293647de4c13cb8468cbd81ff1924cba90e164b4)) +* honor verify-ca/verify-full sslmode for postgres connections ([#9835](https://github.com/windmill-labs/windmill/issues/9835)) ([bf6be96](https://github.com/windmill-labs/windmill/commit/bf6be967fa8c74e1299cf63f813c1cfa34b97f3e)) +* **mcp:** stop double-escaping string query params in build_query_string ([#9855](https://github.com/windmill-labs/windmill/issues/9855)) ([1c46f89](https://github.com/windmill-labs/windmill/commit/1c46f899ca03edf62053f4f14d65b4eabff4255d)) +* **s3_proxy:** preserve URL-encoding on Hive-partition proxy writes ([#9848](https://github.com/windmill-labs/windmill/issues/9848)) ([6b79bdd](https://github.com/windmill-labs/windmill/commit/6b79bddd42fe55f891c17cb71a7e36ee31337bac)) +* validate workspace name length (max 50 chars) on create and fork ([#9854](https://github.com/windmill-labs/windmill/issues/9854)) ([b52972d](https://github.com/windmill-labs/windmill/commit/b52972d0de89004e98d18241d238ca028e4eecba)) + +## [1.743.0](https://github.com/windmill-labs/windmill/compare/v1.742.0...v1.743.0) (2026-06-29) + + +### Features + +* **home:** redesign create-new popover and home header ([#9827](https://github.com/windmill-labs/windmill/issues/9827)) ([2493eaf](https://github.com/windmill-labs/windmill/commit/2493eaf031f30072637a297398674e761f039005)) +* **pipeline:** AI-chat data-pipeline editor (route + in-session) + home surfacing ([#9805](https://github.com/windmill-labs/windmill/issues/9805)) ([c910278](https://github.com/windmill-labs/windmill/commit/c91027824be1f1f49cdd14148baf6aad092a1dd0)) + + +### Bug Fixes + +* **gcp:** require token verification for authenticated push delivery ([#9834](https://github.com/windmill-labs/windmill/issues/9834)) ([9b65161](https://github.com/windmill-labs/windmill/commit/9b65161c643bf3f120d2ebd82f786c17233a971b)) + +## [1.742.0](https://github.com/windmill-labs/windmill/compare/v1.741.0...v1.742.0) (2026-06-28) + + +### Features + +* **apps:** add labels input to app editor deploy drawer ([#9828](https://github.com/windmill-labs/windmill/issues/9828)) ([da45e69](https://github.com/windmill-labs/windmill/commit/da45e699c8aefeede172c90769ef4f4b182fec0c)) +* column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) ([#9814](https://github.com/windmill-labs/windmill/issues/9814)) ([003a262](https://github.com/windmill-labs/windmill/commit/003a262a4e9d6c2a63ada01aa8429aea1fbb6031)) + + +### Bug Fixes + +* **audit:** don't read pg_authid from an elevated context in S3 export migration ([#9832](https://github.com/windmill-labs/windmill/issues/9832)) ([75ba81b](https://github.com/windmill-labs/windmill/commit/75ba81b2d27fb0722095780312064cb93d20287e)) +* close unauthenticated DAP debugger program-mode launch bypass ([#9829](https://github.com/windmill-labs/windmill/issues/9829)) ([c0768de](https://github.com/windmill-labs/windmill/commit/c0768de0acdf63eaba5fb97d04bfc64f2f03b93d)) +* redeploy older app version from deployment history ([#9826](https://github.com/windmill-labs/windmill/issues/9826)) ([c479afa](https://github.com/windmill-labs/windmill/commit/c479afab8ebceccbee050e923dc5c27a6712ea62)) + +## [1.741.0](https://github.com/windmill-labs/windmill/compare/v1.740.0...v1.741.0) (2026-06-26) + + +### Features + +* **ai-chat:** add create_folder tool to global chat ([#9819](https://github.com/windmill-labs/windmill/issues/9819)) ([44c25de](https://github.com/windmill-labs/windmill/commit/44c25de418612ab98341adb15d5671222b54367e)) +* **ai-chat:** hint /compact in context usage tooltip ([#9777](https://github.com/windmill-labs/windmill/issues/9777)) ([aadfb62](https://github.com/windmill-labs/windmill/commit/aadfb620c0b7dcd7e94367b761875e14ef9abe69)) +* **ai-chat:** let global chat edit the user's personal instructions ([#9771](https://github.com/windmill-labs/windmill/issues/9771)) ([3be2752](https://github.com/windmill-labs/windmill/commit/3be27521b05de33e48582e80c6651071f889f048)) +* **ai-chat:** surface raw apps in the @-mention context picker ([#9800](https://github.com/windmill-labs/windmill/issues/9800)) ([1602244](https://github.com/windmill-labs/windmill/commit/16022447c7b445be753b9545b10b4c67da0893d5)) +* capture managed-materialize output schema as asset metadata ([#2](https://github.com/windmill-labs/windmill/issues/2)a) ([#9812](https://github.com/windmill-labs/windmill/issues/9812)) ([ade74b2](https://github.com/windmill-labs/windmill/commit/ade74b297f6a03441e700a20ffc2d7291c8a85fd)) +* **sdk:** allow overriding worker tag when running jobs (WIN-2105) ([#9807](https://github.com/windmill-labs/windmill/issues/9807)) ([52fc7bf](https://github.com/windmill-labs/windmill/commit/52fc7bf94cf3f87f68d9dba9884944d87e7d5d57)) + + +### Bug Fixes + +* apply step timeout to 'Test this step' preview ([#9810](https://github.com/windmill-labs/windmill/issues/9810)) ([d04062b](https://github.com/windmill-labs/windmill/commit/d04062bff58c9c4c79ce542a4321e71bcbcf0e98)) +* **flows:** reject corrupt step paths at deploy + atomic cache writes ([#9751](https://github.com/windmill-labs/windmill/issues/9751)) ([#9813](https://github.com/windmill-labs/windmill/issues/9813)) ([3cda447](https://github.com/windmill-labs/windmill/commit/3cda44762148bcd2ee5c0ea821db884950376ead)) +* **frontend:** clarify instance data table unavailable on cloud ([#9806](https://github.com/windmill-labs/windmill/issues/9806)) ([c3e8c78](https://github.com/windmill-labs/windmill/commit/c3e8c789ac05c9c28991d9ab6f2358f61fa87971)) +* hide GCS service account key behind a reveal in object storage settings ([#9815](https://github.com/windmill-labs/windmill/issues/9815)) ([0ec5061](https://github.com/windmill-labs/windmill/commit/0ec5061270749ed078e01f5a4bc7397a1755ca32)) +* ping job during volume setup to prevent false zombie restarts ([#9803](https://github.com/windmill-labs/windmill/issues/9803)) ([43bb676](https://github.com/windmill-labs/windmill/commit/43bb676dc5652cb06fe1414b8d3aacf295bae36b)) +* skipped suspend step no longer parks the flow forever ([#9821](https://github.com/windmill-labs/windmill/issues/9821)) ([40110bc](https://github.com/windmill-labs/windmill/commit/40110bc7158bc42c3d84bd4637a12b82fcd72a9a)) + + +### Performance Improvements + +* **audit:** re-anchor S3 audit export on enable + opt-in backfill ([#9818](https://github.com/windmill-labs/windmill/issues/9818)) ([577ceee](https://github.com/windmill-labs/windmill/commit/577ceeee8679f054c6898d1a7889df30ab830f8f)) + +## [1.740.0](https://github.com/windmill-labs/windmill/compare/v1.739.0...v1.740.0) (2026-06-25) + + +### Features + +* **api:** add structured endpoint for flow logs ([#9797](https://github.com/windmill-labs/windmill/issues/9797)) ([ba768fe](https://github.com/windmill-labs/windmill/commit/ba768fee888682cb50142d6e76c0422c40307f46)) +* bounded-cascade selective execution for pipelines (UI + CLI) ([#9695](https://github.com/windmill-labs/windmill/issues/9695)) ([248540a](https://github.com/windmill-labs/windmill/commit/248540ac4d6e4ee9ee7c3e6f2cc822c63cc6426e)) +* data tests for ducklake pipeline materialization ([#9708](https://github.com/windmill-labs/windmill/issues/9708)) ([f6998ec](https://github.com/windmill-labs/windmill/commit/f6998ec54cba2507703790bf33427e7567d42c4b)) +* detect and guard against deploying stale drafts ([#9768](https://github.com/windmill-labs/windmill/issues/9768)) ([d865518](https://github.com/windmill-labs/windmill/commit/d8655189347f58df9d17e83dc55798baf7964279)) +* ducklake time-travel UX (snapshot history + AT VERSION reads) ([#9709](https://github.com/windmill-labs/windmill/issues/9709)) ([d131d75](https://github.com/windmill-labs/windmill/commit/d131d754e1fc9674abf5de383d2bc93596df9bd1)) +* self-host docs search for chat, mcp, cli; drop inkeep ([#9772](https://github.com/windmill-labs/windmill/issues/9772)) ([9d61e4e](https://github.com/windmill-labs/windmill/commit/9d61e4e59e4101de84217f7c7846f1aa94e84d89)) + + +### Bug Fixes + +* allow hyphens in postgresql database name validation ([#9782](https://github.com/windmill-labs/windmill/issues/9782)) ([170cd79](https://github.com/windmill-labs/windmill/commit/170cd79aaf92152fc3c0f675f155853c7f0e5b25)) +* **debounce:** never supersede a running debounce survivor ([#9780](https://github.com/windmill-labs/windmill/issues/9780)) ([5549bdc](https://github.com/windmill-labs/windmill/commit/5549bdc67a5559a764616c44b1018543bc0568fe)) +* decrypt secret variables via external backend in common resolvers ([#9784](https://github.com/windmill-labs/windmill/issues/9784)) ([cd42c6c](https://github.com/windmill-labs/windmill/commit/cd42c6ca18261328055554788932c3fe876a4a5b)) +* enforce containment of python module dir for preview jobs ([#9704](https://github.com/windmill-labs/windmill/issues/9704)) ([88fca6a](https://github.com/windmill-labs/windmill/commit/88fca6a8c130b9e3b0f0cd410e422d4e074fc11f)) +* **frontend:** apply script editor timeout to preview/Test runs ([#9794](https://github.com/windmill-labs/windmill/issues/9794)) ([6664ce6](https://github.com/windmill-labs/windmill/commit/6664ce6dc0c5fbc283303148de06d6bb85e4acf7)) +* **frontend:** nested-loop "Test this step" resolves iter to innermost loop ([#9778](https://github.com/windmill-labs/windmill/issues/9778)) ([74ebfc6](https://github.com/windmill-labs/windmill/commit/74ebfc67f069047875db738926865bd4bd6fe9e9)) +* opt out of Deno minimum-dependency-age for private npm registries ([#9802](https://github.com/windmill-labs/windmill/issues/9802)) ([b28f974](https://github.com/windmill-labs/windmill/commit/b28f974e5069f635419d9ea56fad6a0e417894e8)) +* pass SSL cert env vars to `uv python install` ([#9790](https://github.com/windmill-labs/windmill/issues/9790)) ([962758c](https://github.com/windmill-labs/windmill/commit/962758c02de5f6d962c681fe9c39769b99429e8d)) +* **python:** re-verify wheel RECORD on local cache reuse (once per worker) ([#9775](https://github.com/windmill-labs/windmill/issues/9775)) ([6c71c33](https://github.com/windmill-labs/windmill/commit/6c71c33470e3ea547f3b994db829eb4d04882443)) +* **python:** serialize concurrent installs into shared wheel cache dir ([#9787](https://github.com/windmill-labs/windmill/issues/9787)) ([11d83ab](https://github.com/windmill-labs/windmill/commit/11d83ab1ec559be5d3263010228e6db65358e04b)) +* re-pin stale-draft fork base when restoring an app deployment ([#9792](https://github.com/windmill-labs/windmill/issues/9792)) ([b9711e5](https://github.com/windmill-labs/windmill/commit/b9711e5ace8585315a1c2b85bb25ac8dd7832d6f)) +* restore libargon2-1 for PHP runtime in server image ([#9795](https://github.com/windmill-labs/windmill/issues/9795)) ([e9cb806](https://github.com/windmill-labs/windmill/commit/e9cb80639b2dec63ede69fc3a4e3720bb1a3c319)) +* use transaction for parallel_monitor_lock DELETE in last-iteration path ([#9789](https://github.com/windmill-labs/windmill/issues/9789)) ([754cae9](https://github.com/windmill-labs/windmill/commit/754cae956ac8d431ddeb055453835e249cdd07b7)) + + +### Performance Improvements + +* drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes ([#9786](https://github.com/windmill-labs/windmill/issues/9786)) ([aa098c7](https://github.com/windmill-labs/windmill/commit/aa098c70c0271b2b1917749d1f607c0559cf04de)) +* eliminate dual-connection DB pool contention across worker, queue, and api ([#9798](https://github.com/windmill-labs/windmill/issues/9798)) ([0dbd9c1](https://github.com/windmill-labs/windmill/commit/0dbd9c1231b00d4693af68835fe1d9e7c8869b43)) + +## [1.739.0](https://github.com/windmill-labs/windmill/compare/v1.738.0...v1.739.0) (2026-06-24) + + +### Features + +* add /compact session chat command ([#9764](https://github.com/windmill-labs/windmill/issues/9764)) ([83cc553](https://github.com/windmill-labs/windmill/commit/83cc5533ee92e59356a117eefeaf42dad23287f6)) +* add session chat slash commands ([#9748](https://github.com/windmill-labs/windmill/issues/9748)) ([24b95e9](https://github.com/windmill-labs/windmill/commit/24b95e9fe12ba4abdfe1ff6e9f9fe42cb2ded011)) +* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209)) +* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381)) +* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d)) +* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548)) +* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38)) +* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2)) +* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c)) + + +### Bug Fixes + +* **ai-chat:** strip unclosed <summary> tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d)) +* **apps:** realign legacy raw-app drafts to raw_app draft kind ([#9761](https://github.com/windmill-labs/windmill/issues/9761)) ([288318a](https://github.com/windmill-labs/windmill/commit/288318ac269714fc03b15622dbb86b1c28268a36)) +* **backend:** resolve folder_labels search_path on non-public (PG_SCHEMA) schemas ([#9758](https://github.com/windmill-labs/windmill/issues/9758)) ([f582878](https://github.com/windmill-labs/windmill/commit/f5828780fd6a8be070b2933ebd41ee6dff98a9e1)) +* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d)) +* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43)) +* **frontend:** keep #content portal target present on AI-session route ([#9754](https://github.com/windmill-labs/windmill/issues/9754)) ([5e09c50](https://github.com/windmill-labs/windmill/commit/5e09c501713ebbe05b28ce0084eca641f0dbe95c)) +* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23)) +* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5)) +* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb)) +* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37)) + + +### Performance Improvements + +* **audit:** adaptive timestamp floor for S3 audit-log export ([#9752](https://github.com/windmill-labs/windmill/issues/9752)) ([55bed4a](https://github.com/windmill-labs/windmill/commit/55bed4abcfce2a611b16054573980d2eb613ccb3)) +* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d)) + +## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23) + + +### Features + +* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc)) +* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b)) + + +### Bug Fixes + +* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a)) +* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e)) +* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9)) +* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57)) +* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7)) +* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3)) +* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6)) + + +### Performance Improvements + +* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10)) + +## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23) + + +### Features + +* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904)) + + +### Bug Fixes + +* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911)) +* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6)) +* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a)) +* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39)) +* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd)) + + +### Performance Improvements + +* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011)) + +## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23) + + +### Features + +* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd)) + + +### Bug Fixes + +* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72)) +* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c)) +* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76)) +* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9)) +* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6)) + +## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22) + + +### Features + +* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2)) +* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039)) +* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f)) +* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb)) + + +### Bug Fixes + +* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d)) +* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2)) +* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f)) +* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126)) +* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654)) +* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8)) + +## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20) + + +### Features + +* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed)) + + +### Bug Fixes + +* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0)) + ## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19) diff --git a/Dockerfile b/Dockerfile index d327c9b394..a86fb60d2f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,7 +1,7 @@ -ARG DEBIAN_IMAGE=debian:bookworm-slim -ARG RUST_IMAGE=rust:1.93-slim-bookworm +ARG DEBIAN_IMAGE=debian:trixie-slim +ARG RUST_IMAGE=rust:1.93-slim-trixie -FROM debian:bookworm-slim AS nsjail +FROM debian:trixie-slim AS nsjail WORKDIR /nsjail @@ -9,12 +9,12 @@ RUN apt-get -y update \ && apt-get install -y \ bison=2:3.8.* \ flex=2.6.* \ - g++=4:12.2.* \ - gcc=4:12.2.* \ - git=1:2.39.* \ + g++=4:14.2.* \ + gcc=4:14.2.* \ + git=1:2.47.* \ libprotobuf-dev=3.21.* \ libnl-route-3-dev=3.7.* \ - make=4.3-4.1 \ + make=4.4.* \ pkg-config=1.8.* \ protobuf-compiler=3.21.* @@ -44,7 +44,7 @@ FROM rust_base AS windmill_duckdb_ffi_internal_builder WORKDIR /windmill-duckdb-ffi-internal -RUN apt-get update && apt-get install -y clang=1:14.0-55.* libclang-dev=1:14.0-55.* cmake=3.25.* && \ +RUN apt-get update && apt-get install -y clang=1:19.0* libclang-dev=1:19.0* cmake=3.31.* && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* @@ -98,7 +98,7 @@ ARG features="" COPY --from=planner /windmill/recipe.json recipe.json -RUN apt-get update && apt-get install -y libxml2-dev=2.9.* libxmlsec1-dev=1.2.* libkrb5-dev libsasl2-dev libcurl4-openssl-dev clang=1:14.0-55.* libclang-dev=1:14.0-55.* cmake=3.25.* && \ +RUN apt-get update && apt-get install -y libxml2-dev=2.12.* libxmlsec1-dev=1.2.* libkrb5-dev libsasl2-dev libcurl4-openssl-dev clang=1:19.0* libclang-dev=1:19.0* cmake=3.31.* && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* @@ -135,9 +135,8 @@ FROM ${DEBIAN_IMAGE} ARG TARGETPLATFORM ARG POWERSHELL_VERSION=7.5.0 -ARG POWERSHELL_DEB_VERSION=7.5.0-1 -ARG KUBECTL_VERSION=1.28.7 -ARG HELM_VERSION=3.14.3 +ARG KUBECTL_VERSION=1.36.2 +ARG HELM_VERSION=3.21.2 # NOTE: If changing, also change go version in workspace dependencies template at WorkspaceDependenciesEditor.svelte ARG GO_VERSION=1.26.0 ARG APP=/usr/src/app @@ -163,14 +162,14 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH RUN apt-get update \ - && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \ + && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 tini gnupg libargon2-1 \ && if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* # Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \ - && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \ + && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release; echo "$VERSION_CODENAME")-pgdg main" > /etc/apt/sources.list.d/pgdg.list \ && apt-get update \ && apt-get install -y --no-install-recommends postgresql-client \ && apt-get clean \ @@ -183,12 +182,14 @@ RUN if [ "$WITH_GIT" = "true" ]; then \ && rm -rf /var/lib/apt/lists/*; \ else echo 'Building the image without git'; fi; +# PowerShell ships as a tarball: the upstream .deb depends on libicu<=74 which no longer exists in trixie RUN if [ "$WITH_POWERSHELL" = "true" ]; then \ - if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \ - && rm -rf /var/lib/apt/lists/* && \ - dpkg --install 'pwsh.deb' && \ - rm 'pwsh.deb'; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \ + case "$TARGETPLATFORM" in \ + "linux/amd64") pwsh_arch=x64 ;; \ + "linux/arm64") pwsh_arch=arm64 ;; \ + *) pwsh_arch="" ;; \ + esac; \ + if [ -n "$pwsh_arch" ]; then apt-get update -y && apt install libicu76 -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-${pwsh_arch}.tar.gz" && apt-get clean \ && rm -rf /var/lib/apt/lists/* && \ mkdir -p /opt/microsoft/powershell/7 && \ tar zxf powershell.tar.gz -C /opt/microsoft/powershell/7 && \ @@ -233,7 +234,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # Install UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv # Preinstall python runtimes to temp build location (will copy with world-writable perms later) # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run @@ -292,7 +293,7 @@ RUN bun install -g windmill-cli \ RUN curl -fsSL https://claude.ai/install.sh | bash \ && cp /root/.local/share/claude/versions/* /usr/bin/claude -COPY --from=php:8.3.30-cli-bookworm /usr/local/bin/php /usr/bin/php +COPY --from=php:8.3.30-cli-trixie /usr/local/bin/php /usr/bin/php COPY --from=composer:2.9.5 /usr/bin/composer /usr/bin/composer # add the docker client to call docker from a worker if enabled @@ -303,13 +304,13 @@ ENV CARGO_HOME="/tmp/windmill/cache/cargo" ENV LD_LIBRARY_PATH="." # nsjail runtime deps and binary -RUN apt-get update && apt-get install -y --no-install-recommends libprotobuf32 libnl-route-3-200 libnl-3-200 \ +RUN apt-get update && apt-get install -y --no-install-recommends libprotobuf32t64 libnl-route-3-200 libnl-3-200 \ && apt-get clean && rm -rf /var/lib/apt/lists/* COPY --from=nsjail /nsjail/nsjail /bin/nsjail # crane: pulls + flattens images for the sandboxed container runtime (`# sandbox `). # Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md. -ARG CRANE_VERSION=v0.20.6 +ARG CRANE_VERSION=v0.21.7 RUN arch="$(dpkg --print-architecture)"; \ case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \ wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \ diff --git a/ai_evals/AGENTS.md b/ai_evals/AGENTS.md index af5427abae..6e63c4037e 100644 --- a/ai_evals/AGENTS.md +++ b/ai_evals/AGENTS.md @@ -1,208 +1,14 @@ -# AI Evals Authoring Guide +# AI Evals -This folder contains black-box benchmark cases for: +Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`, +`script`, `cli`, `global`). -- `flow` -- `app` -- `script` -- `cli` -- `global` +**Authoring and running cases is documented in the `ai-evals` skill** — load it +before adding/changing a case or running a benchmark. Claude Code reads +`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read +`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in +Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`. -The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape. +For AI chat / copilot changes that these evals measure, see the `ai-chat` skill. -## Core rules - -1. Write prompts like a real user request. -2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details. -3. Keep deterministic validation narrow and hard. -4. Put semantic expectations in `judgeChecklist`. -5. Use `expected` fixtures only when exact structure really matters. - -## Prompt writing - -Prompts should sound like something a user would naturally ask. - -Good: - -- "Create a flow that routes support requests based on customer tier." -- "Add a reset button that sets the counter back to 0." -- "Create a flow that reuses the existing greeting script instead of duplicating the logic." - -Bad: - -- "Use `branchone` with 3 branches and a default branch." -- "Create a `rawscript` step with this exact topology." -- "This is a benchmark harness." - -Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow. - -## Flow-specific rules - -This is the main principle you asked for: - -- flow prompts should read like requests from a user who does not know the product internals -- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs - -That means: - -- creation cases should describe the business behavior and expected result -- modification cases may mention existing step names, because the user can see the current flow -- only mention special Windmill constructs when the case is explicitly about those constructs - -Examples: - -- acceptable creation prompt: - "Create a purchase approval flow that pauses for approval and asks the approver for a comment." -- avoid: - "Create a suspend step with one required event and a resume form." - -For flow cases, do not fail a case just because the model chose a different valid topology. - -## App-specific rules - -App prompts should focus on user-visible behavior: - -- what the UI should let the user do -- what should persist -- what backend behavior is needed - -Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app. - -## CLI-specific rules - -CLI prompts can be more explicit about paths and file names because real CLI users often do specify them. - -Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction. - -When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior. - -## Global-specific rules - -Global prompts should exercise workspace-level drafting behavior: - -- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant -- writing AI drafts rather than saving or deploying by default -- producing coherent multi-artifact changes when the request crosses artifact boundaries - -Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them. - -Datatable cases should set `skipJudge: true` and validate through tool-use -(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions -(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`, -`['update', 'insert into']`). Two reasons the judge is unreliable here: - -- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` - produce no drafts, and the global judge only sees the drafts artifact — it - scores a no-draft conversational answer as empty (same as the - `askUserQuestion` cases). -- Even a case that *does* produce a draft (a script reading the data table via - `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK - reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the - SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']` - plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from - runtime SDK use). - -`stringIncludesAnyOf` is existential over calls (at least one matching call), so a -mutation case still passes when the model mixes its UPDATE/INSERT with -verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful -within a case — writes persist, so a model that re-queries to verify its -CREATE/UPDATE sees the change and does not loop. But the engine is best-effort -(SELECT returns all rows of the referenced/first table with no WHERE/projection), -so still never assert specific returned row values. Seed data via -`workspace.datatables` in the `initial` fixture (see README). - -## Deterministic validation - -Use deterministic validation only for hard failures such as: - -- missing required files -- unexpected extra files when the prompt says not to create them -- syntax errors -- unresolved flow refs -- missing required special modules or suspend config -- obvious artifact corruption - -Do not use deterministic validation to enforce one preferred implementation for broad creation tasks. - -Examples of bad hard checks: - -- exact step topology for a creation flow -- exact branch structure when the prompt only asked for routing behavior -- exact input shape when multiple reasonable shapes are acceptable - -## Judge checklist - -Every non-trivial case should have a `judgeChecklist`. - -The checklist should capture: - -- the user-visible behavior that must be present -- important constraints -- key completion criteria - -The checklist should not duplicate low-level implementation details unless they are truly required by the task. - -Good checklist items: - -- "the flow calculates the order total with 8% tax" -- "the app persists recipes appropriately for a raw Windmill app" -- "the flow reuses the existing workspace script instead of rewriting the logic" - -Bad checklist items: - -- "uses `branchone`" -- "contains a `rawscript` node" - -## When to use `expected` - -Use `expected` fixtures when the case is structure-sensitive, for example: - -- exact file creation -- exact script content -- modification cases where a specific file must change in a specific way -- cases where preserving an existing structure is part of the requirement - -Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass. - -## When to use `initial` - -Use `initial` when the benchmark is about: - -- editing an existing artifact -- reusing existing workspace assets -- preserving existing behavior while adding a change - -If the case is greenfield, prefer no `initial`. - -## Case design ladder - -Prefer suites that get gradually harder: - -1. trivial create case -2. realistic create case -3. reuse-existing-assets case -4. modification case -5. refactor case -6. edge-case or niche product behavior - -The last cases in a suite should cover unusual or product-specific behavior. - -## Anti-patterns - -Avoid these: - -- benchmark framing in prompts -- over-specified internal topology for creation tasks -- judge checklists that just restate implementation details -- deterministic validation that encodes one preferred solution -- fixtures that are so minimal or brittle that they create false negatives - -## Before adding a case - -Ask: - -1. Would a real user plausibly write this prompt? -2. If the model solves it in a different valid way, would the case still pass? -3. Are the hard deterministic checks only catching objectively broken output? -4. Does the `judgeChecklist` describe the real success criteria? -5. If this case fails, will the reason be understandable from the saved artifacts? +The full case format, fields, and fixture details remain in `ai_evals/README.md`. diff --git a/ai_evals/README.md b/ai_evals/README.md index 88825f4ee7..33bb47ae46 100644 --- a/ai_evals/README.md +++ b/ai_evals/README.md @@ -75,6 +75,8 @@ Public CLI surface: - `--model `: choose the model under test - `--models `: run the same cases sequentially against several model aliases - `--verbose`: stream assistant output for frontend runs +- `--skip-judge`: skip LLM judge scoring for the run +- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring - `--record`: append a compact tracked summary line to `ai_evals/history/.jsonl` for full-suite runs only - `--backend-validation `: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals @@ -99,7 +101,7 @@ Notes: - the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5` - frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases - `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there -- the judge model is separate and currently defaults to `claude-sonnet-4-6` +- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs ## Case Format diff --git a/ai_evals/adapters/frontend/benchmarkRunner.ts b/ai_evals/adapters/frontend/benchmarkRunner.ts index 32107eadf1..b45f880699 100644 --- a/ai_evals/adapters/frontend/benchmarkRunner.ts +++ b/ai_evals/adapters/frontend/benchmarkRunner.ts @@ -25,6 +25,12 @@ export async function runFrontendBenchmarkFromEnv(): Promise ); const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1"; const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1"; + const executionOnly = + process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1"; + const judgeModel = + process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly + ? null + : DEFAULT_JUDGE_MODEL; const model = resolveEvalModel( mode, process.env.WMILL_FRONTEND_AI_EVAL_MODEL, @@ -48,7 +54,8 @@ export async function runFrontendBenchmarkFromEnv(): Promise cases: selectedCases, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, + executionOnly, concurrency: verbose ? 1 : undefined, verbose, onProgress: emitProgress @@ -60,7 +67,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise mode, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, caseResults, }); } diff --git a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts index a6d78da36b..e553d15f56 100644 --- a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts @@ -50,6 +50,20 @@ export interface GlobalLiveEditorDraftFixture { value?: unknown; } +// Identity the global system prompt builds paths from. Production reads +// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs +// in, so without this the prompt sees an empty username (`u//...`) and no +// path-selection case is meaningful. Seeded per-case via the initial fixture and +// passed straight to `prepareGlobalSystemMessage` (no global-store mutation). +export interface GlobalUserFixture { + username: string; + is_admin?: boolean; + /** Folders the user can write to (the writable set whoami returns). */ + folders?: string[]; + /** Folders the user can read; read-only folders = folders_read \ folders. */ + folders_read?: string[]; +} + export interface GlobalEvalResult { success: boolean; state: GlobalDraftState; @@ -65,6 +79,7 @@ export interface GlobalEvalResult { export interface GlobalEvalOptions { workspaceFixtures?: BenchmarkWorkspaceRunnables; liveEditorDrafts?: GlobalLiveEditorDraftFixture[]; + user?: GlobalUserFixture; model?: string; maxIterations?: number; provider?: AIProvider; @@ -90,9 +105,11 @@ export async function runGlobalEval( const model = options.model ?? "claude-haiku-4-5-20251001"; const injectActiveEditorContext = process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1"; + // Pass the seeded identity straight to the prompt builder rather than mutating + // the process-global `userStore`, so concurrent cases never race on it. const rawResult = await runEval({ userPrompt, - systemMessage: prepareGlobalSystemMessage(), + systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }), userMessage: prepareGlobalUserMessage( userPrompt, [], diff --git a/ai_evals/adapters/frontend/mockBackend.ts b/ai_evals/adapters/frontend/mockBackend.ts index b78351512d..f9023449a2 100644 --- a/ai_evals/adapters/frontend/mockBackend.ts +++ b/ai_evals/adapters/frontend/mockBackend.ts @@ -11,6 +11,7 @@ import type { DataTableTables, DataTableTableSchema, GetDraftForUserResponse, + GetOwnDraftResponse, ListDraftsResponse, ScriptLang, UpdateDraftResponse, @@ -90,6 +91,13 @@ export function resetBenchmarkMockBackend(): void { benchmarkDrafts.clear() } +// Stand-in for FolderService.createFolder so the global create_folder tool runs in +// memory instead of mutating the real backend. Folders aren't otherwise modelled +// (no folder-listing in evals), so this just echoes the created name. +export function createBenchmarkFolder(_workspace: string, name: string): string { + return name +} + export function registerBenchmarkWorkspace(workspace: string): void { benchmarkWorkspaces.add(workspace) } @@ -287,8 +295,8 @@ export function getBenchmarkJobLogs(workspace: string, jobId: string): string { /** * In-memory stand-in for the per-user draft backend (`DraftService`). The global * AI chat now persists and reads drafts through the backend DB instead of an - * in-tab `UserDraft` cell, so the eval mocks the three draft endpoints it - * exercises (`updateDraft` / `getDraftForUser` / `listDrafts`) and keeps the + * in-tab `UserDraft` cell, so the eval mocks the draft endpoints it exercises + * (`updateDraft` / `getOwnDraft` / `getDraftForUser` / `listDrafts`) and keeps the * saved values here, keyed by workspace + draft kind + storage path. Mirrors the * semantics of the production unit test's mock in * `frontend/src/lib/components/copilot/chat/global/core.test.ts`. @@ -372,6 +380,20 @@ export function getBenchmarkDraftForUser(input: { return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP } } +/** Mirror `DraftService.getOwnDraft`: `null` (200) when absent — unlike + * `getDraftForUser`, absence is not an error on this route. */ +export function getBenchmarkOwnDraft(input: { + workspace: string + kind: UserDraftItemKind + path: string +}): GetOwnDraftResponse { + const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path)) + if (!entry) { + return null + } + return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP } +} + /** Mirror `DraftService.listDrafts`: metadata rows (no value) for a workspace. */ export function listBenchmarkDrafts(workspace: string): ListDraftsResponse { return [...benchmarkDrafts.values()] diff --git a/ai_evals/adapters/frontend/mockBackendDrafts.test.ts b/ai_evals/adapters/frontend/mockBackendDrafts.test.ts index a720de5e43..0ab79d216e 100644 --- a/ai_evals/adapters/frontend/mockBackendDrafts.test.ts +++ b/ai_evals/adapters/frontend/mockBackendDrafts.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'bun:test' import { clearBenchmarkDrafts, getBenchmarkDraftForUser, + getBenchmarkOwnDraft, listBenchmarkDrafts, resetBenchmarkMockBackend, seedBenchmarkDraft, @@ -55,6 +56,27 @@ describe('mockBackend drafts', () => { expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow() }) + it('returns null from getOwnDraft when no draft exists', () => { + expect( + getBenchmarkOwnDraft({ workspace: WORKSPACE, kind: 'trigger_schedule', path: 'u/evals/missing' }) + ).toBeNull() + }) + + // The global chat hydrates drawer-kind drafts (schedule/trigger/resource/variable) + // through getOwnDraft — getDraftForUser rejects those kinds as private. + it('hydrates a saved drawer-kind draft through getOwnDraft', () => { + const value = { path: 'u/evals/nightly', schedule: '0 0 9 * * *' } + updateBenchmarkDraft({ + workspace: WORKSPACE, + kind: 'trigger_schedule', + path: 'u/evals/nightly', + requestBody: { value } + }) + expect( + getBenchmarkOwnDraft({ workspace: WORKSPACE, kind: 'trigger_schedule', path: 'u/evals/nightly' })?.value + ).toEqual(value) + }) + it('throws a 404-shaped error when no draft exists', () => { try { getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' }) diff --git a/ai_evals/adapters/frontend/runtime.ts b/ai_evals/adapters/frontend/runtime.ts index 347e15191c..9a54e86084 100644 --- a/ai_evals/adapters/frontend/runtime.ts +++ b/ai_evals/adapters/frontend/runtime.ts @@ -24,6 +24,8 @@ export async function runFrontendBenchmarkAdapter(input: { runs: number; model?: string; verbose?: boolean; + skipJudge?: boolean; + executionOnly?: boolean; backendValidation?: string; }): Promise { const tempDir = await mkdtemp( @@ -40,6 +42,9 @@ export async function runFrontendBenchmarkAdapter(input: { WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "", WMILL_FRONTEND_AI_EVAL_PROGRESS: "1", WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0", + WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE: + input.skipJudge || input.executionOnly ? "1" : "0", + WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0", WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "", }; diff --git a/ai_evals/adapters/frontend/vitestAdapter.test.ts b/ai_evals/adapters/frontend/vitestAdapter.test.ts index 621ecaefcd..92e33414df 100644 --- a/ai_evals/adapters/frontend/vitestAdapter.test.ts +++ b/ai_evals/adapters/frontend/vitestAdapter.test.ts @@ -40,6 +40,7 @@ vi.mock('$lib/gen', async () => { getBenchmarkDraftForUser, getBenchmarkFlowByPath, getBenchmarkJobLogs, + getBenchmarkOwnDraft, getBenchmarkScriptByHash, getBenchmarkScriptByPath, hasBenchmarkWorkspace, @@ -49,6 +50,7 @@ vi.mock('$lib/gen', async () => { listBenchmarkFlows, listBenchmarkJobs, listBenchmarkScripts, + createBenchmarkFolder, createBenchmarkHttpTrigger, createBenchmarkSchedule, previewBenchmarkSchedule, @@ -85,11 +87,21 @@ vi.mock('$lib/gen', async () => { hasBenchmarkWorkspace(data.workspace) ? getBenchmarkDraftForUser(data) : actual.DraftService.getDraftForUser(data), + getOwnDraft: async (data: { workspace: string; kind: any; path: string }) => + hasBenchmarkWorkspace(data.workspace) + ? getBenchmarkOwnDraft(data) + : actual.DraftService.getOwnDraft(data), listDrafts: async (data: { workspace: string }) => hasBenchmarkWorkspace(data.workspace) ? listBenchmarkDrafts(data.workspace) : actual.DraftService.listDrafts(data) }), + FolderService: wrapService(actual.FolderService, { + createFolder: async (data: { workspace: string; requestBody: { name: string } }) => + hasBenchmarkWorkspace(data.workspace) + ? createBenchmarkFolder(data.workspace, data.requestBody.name) + : actual.FolderService.createFolder(data) + }), ScriptService: wrapService(actual.ScriptService, { listScripts: async (data: { workspace: string }) => hasBenchmarkWorkspace(data.workspace) diff --git a/ai_evals/cases/flow.yaml b/ai_evals/cases/flow.yaml index 5f4abafc48..a21ae81f17 100644 --- a/ai_evals/cases/flow.yaml +++ b/ai_evals/cases/flow.yaml @@ -359,6 +359,8 @@ - request_approval - finalize_purchase topLevelStepTypes: + - id: request_approval + type: [rawscript, script] - id: finalize_purchase type: rawscript schemaRequiredPaths: @@ -373,6 +375,7 @@ judgeChecklist: - "the flow includes an approval step named `request_approval`" - "`request_approval` pauses the flow and asks the approver for a comment" + - "`request_approval` is a real script step that generates approval/resume URLs (e.g. via `getResumeUrls`) so approvers receive an actionable link, not a no-op passthrough (identity) step" - one approval is enough to continue - "the flow includes a final step named `finalize_purchase`" - "`finalize_purchase` returns an approved status object after approval" diff --git a/ai_evals/cases/global.yaml b/ai_evals/cases/global.yaml index 319d13b272..0f7f2a4717 100644 --- a/ai_evals/cases/global.yaml +++ b/ai_evals/cases/global.yaml @@ -3,8 +3,9 @@ Create a draft Bun script at `f/evals/global/greet_user`. It should take a string `name` input and return `Hello, ${name}!`. Leave it as an AI draft only; do not deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json runtime: - maxTurns: 8 + maxTurns: 10 validate: draftCountExactly: 1 requiredDrafts: @@ -871,6 +872,207 @@ - fetches the logs for the requested job id - explains the failure from the returned logs (connection refused to the upstream API) +# --- Page navigation (open_page) --- +# The assistant should take the user to a Windmill page (Runs/Schedules) with the +# right filters via open_page, rather than describing where to click or dumping the +# data. No draft is produced, so the global judge is skipped and we validate the +# tool call and its arguments. + +- id: global-openpage1-runs-failed-of-script + prompt: |- + Take me to the failed runs of the script at f/evals/global/greet_user so I can see what's going wrong. + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - open_page + forbiddenToolsUsed: + - write_script + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + - tool: open_page + field: page + stringIncludesAnyOf: + - runs + - tool: open_page + field: status + stringIncludesAnyOf: + - failure + - tool: open_page + field: path + stringIncludesAnyOf: + - f/evals/global/greet_user + skipJudge: true + judgeChecklist: + - opens the Runs page filtered to the failed runs of f/evals/global/greet_user + - applies both the failure status and the script path as filters + - does not write, deploy, or delete anything + +- id: global-openpage2-runs-of-schedule + prompt: |- + Open the runs page filtered to the jobs triggered by the schedule f/evals/global/nightly_digest. + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - open_page + forbiddenToolsUsed: + - write_script + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + - tool: open_page + field: page + stringIncludesAnyOf: + - runs + - tool: open_page + field: schedule_path + stringIncludesAnyOf: + - f/evals/global/nightly_digest + skipJudge: true + judgeChecklist: + - opens the Runs page filtered to jobs triggered by the f/evals/global/nightly_digest schedule + - passes the schedule path as the filter + - does not write, deploy, or delete anything + +- id: global-openpage3-open-schedule + prompt: |- + Open the schedule f/evals/global/nightly_digest so I can review and edit it. + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - open_page + forbiddenToolsUsed: + - write_schedule + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + - tool: open_page + field: page + stringIncludesAnyOf: + - schedules + - tool: open_page + field: open + stringIncludesAnyOf: + - f/evals/global/nightly_digest + skipJudge: true + judgeChecklist: + - opens the Schedules page and targets the f/evals/global/nightly_digest schedule for editing + - passes the schedule path so the editor opens on it + - does not write, deploy, or delete anything + +- id: global-openpage4-workspace-settings-tab + prompt: |- + Take me to the Git sync configuration for this workspace. + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - open_page + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + - tool: open_page + field: page + stringIncludesAnyOf: + - workspace_settings + - tool: open_page + field: tab + stringIncludesAnyOf: + - git_sync + skipJudge: true + judgeChecklist: + - opens the Workspace settings page on the git_sync tab + - does not write, deploy, or delete anything + +- id: global-openpage5-audit-logs-user + prompt: |- + Open the audit logs filtered to actions performed by the user admin. + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - open_page + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + - tool: open_page + field: page + stringIncludesAnyOf: + - audit_logs + - tool: open_page + field: username + stringIncludesAnyOf: + - admin + skipJudge: true + judgeChecklist: + - opens the Audit logs page filtered to the admin user + - does not write, deploy, or delete anything + +- id: global-openpage6-triggers-kind + prompt: |- + Take me to the Kafka triggers for this workspace. + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - open_page + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + - tool: open_page + field: page + stringIncludesAnyOf: + - triggers + - tool: open_page + field: trigger_kind + stringIncludesAnyOf: + - kafka + skipJudge: true + judgeChecklist: + - opens the Kafka triggers page + - does not write, deploy, or delete anything + +- id: global-closepage1-close-runs-tab + prompt: |- + You just opened the runs page for me in the side panel. Close that tab, I'm done looking at it. + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - close_page + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + toolCallArgs: + - tool: close_page + field: match + stringIncludesAnyOf: + - runs + skipJudge: true + judgeChecklist: + - closes the runs preview tab in the side panel + - does not write, deploy, or delete anything + # --- Documentation search (search_docs) --- # Pure product-knowledge questions: the assistant should consult the docs via # search_docs and answer conversationally, not draft or mutate anything. No @@ -1113,3 +1315,197 @@ - renames the formatCurrency definition, imports, and all call sites to formatMoney - leaves the unrelated formatCurrencyPrecise helper unchanged - leaves the result as an AI draft only + +# --- Path selection (u/ vs f/) --- +# These cases assert how the assistant picks a workspace path when the user gives +# none: a bare name defaults to the personal scope `u//`, an existing folder +# whose purpose matches is used, a non-admin targets a writable folder and never a +# read-only one, and shared intent with no matching folder asks rather than invents. +# Each depends on the seeded `user` fixture (username / is_admin / folders / +# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed. + +- id: global-path1-bare-name-defaults-to-personal + prompt: |- + Stage a quick draft helper that takes a string and returns it trimmed of + leading and trailing whitespace. Just keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json + runtime: + maxTurns: 8 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + pathStartsWith: u/admin/ + toolExpect: + requiredToolsUsed: + - write_script + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - stages a single script draft for a trim helper + - defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given + - does not invent an f/ path + - leaves the result as a draft only + +- id: global-path2-match-existing-folder + prompt: |- + Draft a flow for the marketing team's weekly campaign report. + It should take a week number and return a short summary string. + Keep it as a draft only. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: flow + pathStartsWith: f/marketing/ + toolExpect: + requiredToolsUsed: + - write_flow + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - drafts a flow for the marketing campaign report + - places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder + - leaves the result as a draft only + +- id: global-path3-shared-intent-unknown-folder-asks + prompt: |- + Put together a draft onboarding checklist flow for the People Ops team to use + when a new hire joins. Keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json + runtime: + maxTurns: 8 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - askUserQuestion + forbiddenToolsUsed: + - write_flow + - write_script + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops) + - asks which folder to use instead of guessing or inventing one + - does not create a draft until the folder is known + +- id: global-path4-nonadmin-avoids-readonly-folder + prompt: |- + Draft a small flow that returns today's date as an ISO string, and stage it in + one of our shared team folders. Keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: flow + pathStartsWith: f/team_a/ + forbiddenDrafts: + - type: flow + pathStartsWith: f/team_b/ + toolExpect: + requiredToolsUsed: + - write_flow + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - drafts a flow that returns the current date as an ISO string + - places it in team_a (writable by this non-admin user) and not team_b (read-only) + - leaves the result as a draft only + +- id: global-test-pipeline-create-node + prompt: |- + Set up the first step of a data pipeline at `f/evals/global/orders_ingest`. + On a schedule, it should pull raw orders and land them in a managed DuckLake + table so later steps can build on it. Keep it as an AI draft only — don't + deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + path: f/evals/global/orders_ingest + valueIncludes: + - pipeline + forbiddenDrafts: + - type: flow + pathStartsWith: f/evals/global/ + toolExpect: + requiredToolsUsed: + - write_script + forbiddenToolsUsed: + - write_flow + - deploy_workspace_item + - delete_workspace_item + judgeChecklist: + - builds a data pipeline node as a script (not a flow) + - marks the script as a pipeline member with the pipeline annotation in the script's comment syntax (`-- pipeline` for a DuckDB/SQL node, not `// pipeline`) + - declares a schedule trigger and writes its output to a managed DuckLake table + - leaves the result as an AI draft and does not deploy or save it + +- id: global-test-pipeline-two-node-chain + prompt: |- + Build a small data pipeline in the `f/evals/global` folder: one step that + ingests orders into a DuckLake table, and a second step that reads that table + and writes a daily order-count rollup table. Wire the second step to run off + the first step's output. Keep everything as drafts — don't deploy. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json + runtime: + maxTurns: 14 + validate: + draftCountAtLeast: 2 + forbiddenDrafts: + - type: flow + pathStartsWith: f/evals/global/ + toolExpect: + requiredToolsUsed: + - write_script + forbiddenToolsUsed: + - write_flow + - deploy_workspace_item + - delete_workspace_item + judgeChecklist: + - creates two data pipeline nodes as scripts (not a flow) in f/evals/global + - both scripts carry the pipeline annotation in their comment syntax (`-- pipeline` for DuckDB/SQL nodes, not `// pipeline`) + - the first ingests orders into a DuckLake table + - the second reads that same table and writes a daily rollup, wired to the first step's output asset + - leaves both as AI drafts without deploying + +- id: global-path5-create-folder-then-draft + prompt: |- + Create a new shared folder called "analytics" for our data work, then draft a + script in it that returns the current timestamp as an ISO string. Keep the + script as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + pathStartsWith: f/analytics/ + toolExpect: + requiredToolsUsed: + - create_folder + - write_script + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - creates a new shared folder named "analytics" via create_folder + - drafts a script placed in that folder (f/analytics/...) returning an ISO timestamp + - leaves the script as a draft only diff --git a/ai_evals/cli/index.ts b/ai_evals/cli/index.ts index f92d6d7027..504b8a8d13 100644 --- a/ai_evals/cli/index.ts +++ b/ai_evals/cli/index.ts @@ -25,7 +25,9 @@ import { import { runSuite } from "../core/runSuite"; import { EVAL_MODES, type EvalMode } from "../core/types"; import { DEFAULT_JUDGE_MODEL } from "../core/judge"; -import { createCliModeRunner } from "../modes/cli"; +// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes +// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps +// (e.g. @cliffy/*) just to load this entrypoint. import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime"; import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings"; import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend"; @@ -97,6 +99,11 @@ async function main() { "comma-separated model aliases to run sequentially", ) .option("--verbose", "stream assistant output during frontend runs") + .option("--skip-judge", "skip LLM judge scoring for this run") + .option( + "--execution-only", + "only require the model/proxy/frontend loop to complete", + ) .option( "--record", "append a compact summary line to ai_evals/history/.jsonl", @@ -115,6 +122,8 @@ async function main() { model?: string; models?: string; verbose?: boolean; + skipJudge?: boolean; + executionOnly?: boolean; record?: boolean; backendValidation?: string; }, @@ -127,6 +136,8 @@ async function main() { model: options.model, models: options.models, verbose: options.verbose ?? false, + skipJudge: options.skipJudge ?? false, + executionOnly: options.executionOnly ?? false, record: options.record ?? false, backendValidation: options.backendValidation, }); @@ -175,6 +186,8 @@ async function handleRun(input: { model?: string; models?: string; verbose: boolean; + skipJudge: boolean; + executionOnly: boolean; record: boolean; backendValidation?: string; }) { @@ -230,6 +243,8 @@ async function handleRun(input: { input.runs, getCliEvalModel(model), runModel, + input.skipJudge, + input.executionOnly, ) : await runFrontendBenchmarkAdapter({ mode: input.mode, @@ -237,6 +252,8 @@ async function handleRun(input: { runs: input.runs, model: model.id, verbose: input.verbose, + skipJudge: input.skipJudge, + executionOnly: input.executionOnly, backendValidation, }); @@ -278,20 +295,25 @@ async function runCliBenchmark( runs: number, model: ReturnType, runModel: string, + skipJudge: boolean, + executionOnly: boolean, ) { + const { createCliModeRunner } = await import("../modes/cli"); + const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL; const caseResults = await runSuite({ modeRunner: createCliModeRunner(model), cases, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, + executionOnly, }); return buildRunResult({ mode: "cli", runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, caseResults, }); } diff --git a/ai_evals/core/cases.test.ts b/ai_evals/core/cases.test.ts index 9955a73fa9..5d6e3245db 100644 --- a/ai_evals/core/cases.test.ts +++ b/ai_evals/core/cases.test.ts @@ -212,6 +212,9 @@ describe("loadCases", () => { }, ], }); + expect(caseEntry?.initialPath).toContain( + "ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json" + ); expect(caseEntry?.toolExpect).toMatchObject({ requiredToolsUsed: ["write_script"], forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"], diff --git a/ai_evals/core/runSuite.test.ts b/ai_evals/core/runSuite.test.ts new file mode 100644 index 0000000000..26f300a5aa --- /dev/null +++ b/ai_evals/core/runSuite.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from "bun:test"; +import { runSuite } from "./runSuite"; +import type { ModeRunner } from "./types"; + +const modeRunner: ModeRunner = { + mode: "global", + concurrency: 1, + loadInitial: async () => undefined, + loadExpected: async () => undefined, + run: async () => ({ + success: true, + actual: { ok: true }, + assistantMessageCount: 1, + toolCallCount: 0, + toolsUsed: [], + skillsInvoked: [], + tokenUsage: null, + }), + validate: () => [], +}; + +describe("runSuite", () => { + it("skips judge checks when the run disables judge scoring", async () => { + const [caseResult] = await runSuite({ + modeRunner, + cases: [ + { + id: "case-1", + prompt: "Create a draft script", + judgeChecklist: ["the output satisfies the prompt"], + }, + ], + runs: 1, + runModel: "model-under-test", + judgeModel: null, + }); + + const [attempt] = caseResult.attempts; + expect(attempt.passed).toBe(true); + expect(attempt.judgeScore).toBeNull(); + expect(attempt.judgeSummary).toBeNull(); + expect(attempt.checks.map((check) => check.name)).toEqual([ + "run succeeded", + ]); + }); + + it("only requires run success when execution-only is enabled", async () => { + let loadExpectedCalls = 0; + let validateCalls = 0; + let backendValidateCalls = 0; + + const executionOnlyRunner: ModeRunner< + undefined, + undefined, + { ok: boolean } + > = { + ...modeRunner, + loadExpected: async () => { + loadExpectedCalls++; + return undefined; + }, + validate: () => { + validateCalls++; + return [{ name: "validator failed", passed: false }]; + }, + backendValidate: async () => { + backendValidateCalls++; + return { + checks: [{ name: "backend validation failed", passed: false }], + }; + }, + }; + + const [caseResult] = await runSuite({ + modeRunner: executionOnlyRunner, + cases: [ + { + id: "case-1", + prompt: "Create a draft script", + expectedPath: "fixtures/expected.json", + toolExpect: { requiredToolsUsed: ["write_script"] }, + judgeChecklist: ["the output satisfies the prompt"], + }, + ], + runs: 1, + runModel: "model-under-test", + judgeModel: "judge-model", + executionOnly: true, + }); + + const [attempt] = caseResult.attempts; + expect(attempt.passed).toBe(true); + expect(attempt.judgeScore).toBeNull(); + expect(attempt.judgeSummary).toBeNull(); + expect(attempt.checks.map((check) => check.name)).toEqual([ + "run succeeded", + ]); + expect(loadExpectedCalls).toBe(0); + expect(validateCalls).toBe(0); + expect(backendValidateCalls).toBe(0); + }); +}); diff --git a/ai_evals/core/runSuite.ts b/ai_evals/core/runSuite.ts index 1438749a11..4a8c9dab7b 100644 --- a/ai_evals/core/runSuite.ts +++ b/ai_evals/core/runSuite.ts @@ -15,11 +15,13 @@ export async function runSuite(input: { runs: number; runModel: string | null; judgeModel?: string | null; + executionOnly?: boolean; concurrency?: number; verbose?: boolean; onProgress?: (event: FrontendBenchmarkProgressEvent) => void; }): Promise { - const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL; + const judgeModel = + input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel; const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency); const results = new Array(input.cases.length); let cursor = 0; @@ -52,6 +54,7 @@ export async function runSuite(input: { runs: input.runs, judgeModel, judgeThreshold: input.modeRunner.judgeThreshold ?? 80, + executionOnly: input.executionOnly ?? false, modeRunner: input.modeRunner, totalCases: input.cases.length, verbose: input.verbose ?? false, @@ -72,8 +75,9 @@ async function runCaseAttempts(input: { caseIndex: number; evalCase: EvalCase; runs: number; - judgeModel: string; + judgeModel: string | null; judgeThreshold: number; + executionOnly: boolean; modeRunner: ModeRunner; totalCases: number; verbose: boolean; @@ -99,7 +103,9 @@ async function runCaseAttempts(input: { try { const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath); - const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath); + const expected = input.executionOnly + ? undefined + : await input.modeRunner.loadExpected(input.evalCase.expectedPath); const run = await input.modeRunner.run(input.evalCase.prompt, initial, { evalCase: input.evalCase, caseId: input.evalCase.id, @@ -162,22 +168,30 @@ async function runCaseAttempts(input: { }); const checks: BenchmarkCheck[] = [ buildCheck("run succeeded", run.success, run.error), - ...input.modeRunner.validate({ - evalCase: input.evalCase, - prompt: input.evalCase.prompt, - initial, - expected, - actual: run.actual, - run, - }), - ...validateToolExpectations({ - run, - toolExpect: input.evalCase.toolExpect, - }), ]; + if (!input.executionOnly) { + checks.push( + ...input.modeRunner.validate({ + evalCase: input.evalCase, + prompt: input.evalCase.prompt, + initial, + expected, + actual: run.actual, + run, + }), + ...validateToolExpectations({ + run, + toolExpect: input.evalCase.toolExpect, + }) + ); + } const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? []; - if (run.success && input.modeRunner.backendValidate) { + if ( + run.success && + !input.executionOnly && + input.modeRunner.backendValidate + ) { try { const backendValidation = await input.modeRunner.backendValidate({ evalCase: input.evalCase, @@ -218,7 +232,12 @@ async function runCaseAttempts(input: { let judgeScore: number | null = null; let judgeSummary: string | null = null; - if (run.success && !input.evalCase.skipJudge) { + if ( + run.success && + !input.executionOnly && + input.judgeModel !== null && + !input.evalCase.skipJudge + ) { const judge = await judgeOutput({ mode: input.modeRunner.mode, prompt: input.evalCase.prompt, diff --git a/ai_evals/core/types.ts b/ai_evals/core/types.ts index 52667fa321..f142bc8d36 100644 --- a/ai_evals/core/types.ts +++ b/ai_evals/core/types.ts @@ -47,7 +47,7 @@ export interface FlowValidationSpec { }>; topLevelStepTypes?: Array<{ id: string; - type: string; + type: string | string[]; }>; moduleRules?: Array<{ id: string; diff --git a/ai_evals/core/validators.ts b/ai_evals/core/validators.ts index 23a6709f9b..7570ccd12d 100644 --- a/ai_evals/core/validators.ts +++ b/ai_evals/core/validators.ts @@ -1378,11 +1378,14 @@ function validateFlowRequirements( continue; } + const allowedTypes = Array.isArray(requiredStep.type) + ? requiredStep.type + : [requiredStep.type]; checks.push( check( `${requiredStep.id} type matches required`, - getModuleType(module) === requiredStep.type, - `expected ${requiredStep.type}, got ${getModuleType(module) ?? "(missing)"}` + allowedTypes.includes(getModuleType(module) ?? ""), + `expected ${allowedTypes.join(" or ")}, got ${getModuleType(module) ?? "(missing)"}` ) ); } diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json b/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json new file mode 100644 index 0000000000..8c81142fb0 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json @@ -0,0 +1,6 @@ +{ + "user": { + "username": "admin", + "is_admin": true + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json new file mode 100644 index 0000000000..236f091bc2 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "admin", + "is_admin": true, + "folders": ["evals"], + "folders_read": ["evals"] + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json b/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json new file mode 100644 index 0000000000..7f7e237f36 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "admin", + "is_admin": true, + "folders": ["marketing", "data_engineering", "shared_utils"], + "folders_read": ["marketing", "data_engineering", "shared_utils"] + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json b/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json new file mode 100644 index 0000000000..6320cef4d3 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "bob", + "is_admin": false, + "folders": ["team_a"], + "folders_read": ["team_a", "team_b"] + } +} diff --git a/ai_evals/modes/global.ts b/ai_evals/modes/global.ts index 2cf5413f59..050a4caad9 100644 --- a/ai_evals/modes/global.ts +++ b/ai_evals/modes/global.ts @@ -4,6 +4,7 @@ import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureL import { runGlobalEval, type GlobalLiveEditorDraftFixture, + type GlobalUserFixture, } from "../adapters/frontend/core/global/globalEvalRunner"; import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend"; import type { FrontendEvalModelConfig } from "../core/models"; @@ -15,6 +16,7 @@ import { getFrontendApiKey } from "./frontendCommon"; export interface GlobalInitialFixture { workspace?: BenchmarkWorkspaceRunnables; liveEditorDrafts?: GlobalLiveEditorDraftFixture[]; + user?: GlobalUserFixture; } export function createGlobalModeRunner( @@ -38,6 +40,7 @@ export function createGlobalModeRunner( { workspaceFixtures: initial?.workspace, liveEditorDrafts: initial?.liveEditorDrafts, + user: initial?.user, maxIterations: context.evalCase?.runtime?.maxTurns, provider: modelConfig.provider, model: modelConfig.model, @@ -104,6 +107,7 @@ async function loadGlobalInitialFixture(path: string): Promise max_n\n )", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "VarcharArray", - "VarcharArray", - { - "Custom": { - "name": "asset_kind[]", - "kind": { - "Array": { - "Custom": { - "name": "asset_kind", - "kind": { - "Enum": [ - "s3object", - "resource", - "variable", - "ducklake", - "datatable", - "volume" - ] - } - } - } - } - } - }, - "Int4Array" - ] - }, - "nullable": [] - }, - "hash": "02e526146f3584cd599dec708e1be48db3b0cd1c74adbfa2e4039377daa016f0" -} diff --git a/backend/.sqlx/query-0621faf69b1ef866a95f6310c9651875df409a2d2d72ada629bd71e8abdbbf8b.json b/backend/.sqlx/query-0621faf69b1ef866a95f6310c9651875df409a2d2d72ada629bd71e8abdbbf8b.json new file mode 100644 index 0000000000..036e8e4b57 --- /dev/null +++ b/backend/.sqlx/query-0621faf69b1ef866a95f6310c9651875df409a2d2d72ada629bd71e8abdbbf8b.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings SET deploy_to = $1 WHERE deploy_to = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "0621faf69b1ef866a95f6310c9651875df409a2d2d72ada629bd71e8abdbbf8b" +} diff --git a/backend/.sqlx/query-07168aaf14cb6beff0ad4274b441f7f387f5055c47f493271d26731336257384.json b/backend/.sqlx/query-07168aaf14cb6beff0ad4274b441f7f387f5055c47f493271d26731336257384.json index e7ed0aee65..d29a18c691 100644 --- a/backend/.sqlx/query-07168aaf14cb6beff0ad4274b441f7f387f5055c47f493271d26731336257384.json +++ b/backend/.sqlx/query-07168aaf14cb6beff0ad4274b441f7f387f5055c47f493271d26731336257384.json @@ -46,11 +46,11 @@ ] }, "nullable": [ - false, - false, - false, - false, - false, + true, + true, + true, + true, + true, true, true ] diff --git a/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json b/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json new file mode 100644 index 0000000000..173fe9d05c --- /dev/null +++ b/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT SUM(pg_database_size(datname))::BIGINT AS \"v!\" FROM pg_database", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c" +} diff --git a/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json b/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json index a779aa0e95..6efb66005d 100644 --- a/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json +++ b/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json @@ -35,7 +35,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca.json b/backend/.sqlx/query-0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca.json new file mode 100644 index 0000000000..ee50840b07 --- /dev/null +++ b/backend/.sqlx/query-0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT args->>$2 FROM v2_job WHERE id = ANY($1)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "UuidArray", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca" +} diff --git a/backend/.sqlx/query-0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7.json b/backend/.sqlx/query-0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7.json new file mode 100644 index 0000000000..906c710fdf --- /dev/null +++ b/backend/.sqlx/query-0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7" +} diff --git a/backend/.sqlx/query-0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285.json b/backend/.sqlx/query-0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285.json new file mode 100644 index 0000000000..238522a3ff --- /dev/null +++ b/backend/.sqlx/query-0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285.json @@ -0,0 +1,77 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (mp.asset_kind, mp.asset_path)\n mp.asset_kind AS \"asset_kind: AssetKind\", mp.asset_path,\n mp.snapshot_id AS \"snapshot_id!\", mp.partition\n FROM materialized_partition mp\n JOIN unnest($2::ASSET_KIND[], $3::text[]) AS u(kind, path)\n ON mp.asset_kind = u.kind AND mp.asset_path = u.path\n WHERE mp.workspace_id = $1\n AND mp.status = 'materialized' AND mp.snapshot_id IS NOT NULL\n ORDER BY mp.asset_kind, mp.asset_path, mp.snapshot_id DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_kind: AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "asset_path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "snapshot_id!", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "partition", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind[]", + "kind": { + "Array": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + } + } + }, + "TextArray" + ] + }, + "nullable": [ + false, + false, + true, + false + ] + }, + "hash": "0ee4a4dc8c6f09e7027bd99f98d8d1d346f9d67056ef272314dd9b80bee9f285" +} diff --git a/backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json b/backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json new file mode 100644 index 0000000000..0dbcf04c6e --- /dev/null +++ b/backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n schedule.path, t.jobs FROM schedule,\n LATERAL(SELECT ARRAY(\n SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = schedule.path\n AND c.workspace_id = $1\n AND j.workspace_id = $1\n AND parent_job IS NULL AND runnable_path = schedule.script_path\n AND status <> 'skipped'\n ORDER BY completed_at DESC\n LIMIT 20\n ) AS jobs) t\n WHERE workspace_id = $1 AND NOT starts_with(schedule.path, $4)\n ORDER BY edited_at DESC\n LIMIT $2 OFFSET $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "jobs", + "type_info": "JsonArray" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8", + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null + ] + }, + "hash": "0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946" +} diff --git a/backend/.sqlx/query-0ff0b7abad7717d7de398f629853b22cb831e91026dcd59d362a6d4382dc240b.json b/backend/.sqlx/query-0ff0b7abad7717d7de398f629853b22cb831e91026dcd59d362a6d4382dc240b.json new file mode 100644 index 0000000000..c43b9e53a1 --- /dev/null +++ b/backend/.sqlx/query-0ff0b7abad7717d7de398f629853b22cb831e91026dcd59d362a6d4382dc240b.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (workspace_id, path, hash, content, summary, description, language, created_by, created_at, archived, schema_validation, ws_error_handler_muted, deleted, extra_perms)\n VALUES ('test-workspace', 'f/restricted/item', 314159, 'def main(): return 1', '', '', 'python3', 'test-user', NOW(), false, false, false, false, '{}'::jsonb)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "0ff0b7abad7717d7de398f629853b22cb831e91026dcd59d362a6d4382dc240b" +} diff --git a/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json new file mode 100644 index 0000000000..cef272c219 --- /dev/null +++ b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "language!: _", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + null + ] + }, + "hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8" +} diff --git a/backend/.sqlx/query-12dec9fa086fd231cdba2ecaa4585b62e55f43d448332ff88d8f036443b1edd7.json b/backend/.sqlx/query-12dec9fa086fd231cdba2ecaa4585b62e55f43d448332ff88d8f036443b1edd7.json new file mode 100644 index 0000000000..f2faa9f48a --- /dev/null +++ b/backend/.sqlx/query-12dec9fa086fd231cdba2ecaa4585b62e55f43d448332ff88d8f036443b1edd7.json @@ -0,0 +1,46 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, params, body, is_table_macro, provider_path FROM macro_definition WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "params", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "body", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "is_table_macro", + "type_info": "Bool" + }, + { + "ordinal": 4, + "name": "provider_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + false + ] + }, + "hash": "12dec9fa086fd231cdba2ecaa4585b62e55f43d448332ff88d8f036443b1edd7" +} diff --git a/backend/.sqlx/query-142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb.json b/backend/.sqlx/query-142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb.json new file mode 100644 index 0000000000..c19df789d4 --- /dev/null +++ b/backend/.sqlx/query-142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO schedule (\n workspace_id, path, edited_by, edited_at, schedule, enabled, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n )\n SELECT\n $1, path, edited_by, edited_at, schedule, FALSE, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n FROM schedule WHERE workspace_id = $2 AND path NOT LIKE $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb" +} diff --git a/backend/.sqlx/query-1437b432d2c23e30eb05443e83069cdb049f65ec299b0778ce14677728cf6346.json b/backend/.sqlx/query-1437b432d2c23e30eb05443e83069cdb049f65ec299b0778ce14677728cf6346.json deleted file mode 100644 index 911d6c3b07..0000000000 --- a/backend/.sqlx/query-1437b432d2c23e30eb05443e83069cdb049f65ec299b0778ce14677728cf6346.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n CASE WHEN q.running\n THEN $3::text::jsonb\n ELSE $4::text::jsonb\n END,\n s.flow_status,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n WHERE q.id = $1\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id = $1\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n VALUES ($5, $1, $2)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "x", - "type_info": "Int4" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar", - "Text", - "Text", - "Text" - ] - }, - "nullable": [ - null - ] - }, - "hash": "1437b432d2c23e30eb05443e83069cdb049f65ec299b0778ce14677728cf6346" -} diff --git a/backend/.sqlx/query-15c0584e9eb078442f6928adb894324c47e010e3554c14e755a7e045453745bb.json b/backend/.sqlx/query-15c0584e9eb078442f6928adb894324c47e010e3554c14e755a7e045453745bb.json new file mode 100644 index 0000000000..47bfec9e5c --- /dev/null +++ b/backend/.sqlx/query-15c0584e9eb078442f6928adb894324c47e010e3554c14e755a7e045453745bb.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "15c0584e9eb078442f6928adb894324c47e010e3554c14e755a7e045453745bb" +} diff --git a/backend/.sqlx/query-165abf847773c7fe718fa6c832b259bf2dc531bc468d2ab27f76c9c653be0b9a.json b/backend/.sqlx/query-165abf847773c7fe718fa6c832b259bf2dc531bc468d2ab27f76c9c653be0b9a.json new file mode 100644 index 0000000000..f548727abb --- /dev/null +++ b/backend/.sqlx/query-165abf847773c7fe718fa6c832b259bf2dc531bc468d2ab27f76c9c653be0b9a.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, name FROM workspace WHERE parent_workspace_id = $1 AND is_dev_workspace AND deleted = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "165abf847773c7fe718fa6c832b259bf2dc531bc468d2ab27f76c9c653be0b9a" +} diff --git a/backend/.sqlx/query-16a67b92dbd32024838983184e6974f2ce577b78decd6b821096c9f2f252ae8b.json b/backend/.sqlx/query-16a67b92dbd32024838983184e6974f2ce577b78decd6b821096c9f2f252ae8b.json deleted file mode 100644 index d099d97bd3..0000000000 --- a/backend/.sqlx/query-16a67b92dbd32024838983184e6974f2ce577b78decd6b821096c9f2f252ae8b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n SELECT ws.datatable->'datatables' AS datatable_name\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "datatable_name", - "type_info": "Jsonb" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - null - ] - }, - "hash": "16a67b92dbd32024838983184e6974f2ce577b78decd6b821096c9f2f252ae8b" -} diff --git a/backend/.sqlx/query-16c7838ecfcea5fd231f2a4766f691a9a11ca7bb9797b81444419d6a72883531.json b/backend/.sqlx/query-16c7838ecfcea5fd231f2a4766f691a9a11ca7bb9797b81444419d6a72883531.json new file mode 100644 index 0000000000..058018a8ee --- /dev/null +++ b/backend/.sqlx/query-16c7838ecfcea5fd231f2a4766f691a9a11ca7bb9797b81444419d6a72883531.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (parent_workspace_id IS NOT NULL) AS \"has_parent!\", is_dev_workspace\n FROM workspace WHERE id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "has_parent!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "is_dev_workspace", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null, + false + ] + }, + "hash": "16c7838ecfcea5fd231f2a4766f691a9a11ca7bb9797b81444419d6a72883531" +} diff --git a/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json b/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json new file mode 100644 index 0000000000..e576cf8fae --- /dev/null +++ b/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "kind!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "parent_job", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "runnable_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + null, + true, + true + ] + }, + "hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4" +} diff --git a/backend/.sqlx/query-1b05728b33decc39766ccacca50464b5ff9f34e43fbcc38154939461aadfca9f.json b/backend/.sqlx/query-1b05728b33decc39766ccacca50464b5ff9f34e43fbcc38154939461aadfca9f.json new file mode 100644 index 0000000000..708922a0cb --- /dev/null +++ b/backend/.sqlx/query-1b05728b33decc39766ccacca50464b5ff9f34e43fbcc38154939461aadfca9f.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_multipart_inflight\n (workspace_id, upload_id, storage, inflight_bytes, target_existing_size)\n VALUES ($1, $2, $3, $4, $5)\n ON CONFLICT (workspace_id, upload_id)\n DO UPDATE SET inflight_bytes = $4", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "1b05728b33decc39766ccacca50464b5ff9f34e43fbcc38154939461aadfca9f" +} diff --git a/backend/.sqlx/query-1c5caaaa86e3488549cad179e992172315be5d53dcc266d713da01fd27f310b6.json b/backend/.sqlx/query-1c5caaaa86e3488549cad179e992172315be5d53dcc266d713da01fd27f310b6.json deleted file mode 100644 index 75957a4f2f..0000000000 --- a/backend/.sqlx/query-1c5caaaa86e3488549cad179e992172315be5d53dcc266d713da01fd27f310b6.json +++ /dev/null @@ -1,27 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - { - "Custom": { - "name": "asset_usage_kind", - "kind": { - "Enum": [ - "script", - "flow", - "job" - ] - } - } - } - ] - }, - "nullable": [] - }, - "hash": "1c5caaaa86e3488549cad179e992172315be5d53dcc266d713da01fd27f310b6" -} diff --git a/backend/.sqlx/query-1cd7c77e7a6a5c13c4ca521098bb07c1d805d21899fe0ebac22132b248ffd242.json b/backend/.sqlx/query-1cd7c77e7a6a5c13c4ca521098bb07c1d805d21899fe0ebac22132b248ffd242.json new file mode 100644 index 0000000000..b84cd8a14e --- /dev/null +++ b/backend/.sqlx/query-1cd7c77e7a6a5c13c4ca521098bb07c1d805d21899fe0ebac22132b248ffd242.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO workspace_protection_rule (workspace_id, name, rules, bypass_groups, bypass_users)\n VALUES ($1, $2, $3, $4, $5)\n ON CONFLICT (workspace_id, name)\n DO UPDATE SET rules = EXCLUDED.rules,\n bypass_groups = EXCLUDED.bypass_groups,\n bypass_users = EXCLUDED.bypass_users\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int4", + "TextArray", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "1cd7c77e7a6a5c13c4ca521098bb07c1d805d21899fe0ebac22132b248ffd242" +} diff --git a/backend/.sqlx/query-21242859ee6b72466c6bfb9d826a7159de4488adccb9349cddfbdd572b4005e9.json b/backend/.sqlx/query-21242859ee6b72466c6bfb9d826a7159de4488adccb9349cddfbdd572b4005e9.json new file mode 100644 index 0000000000..1a3d7ef0e4 --- /dev/null +++ b/backend/.sqlx/query-21242859ee6b72466c6bfb9d826a7159de4488adccb9349cddfbdd572b4005e9.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT consumer_path AS \"consumer_path!\", macro_name AS \"macro_name!\"\n FROM macro_usage\n WHERE workspace_id = $1\n AND ($2::text IS NULL OR consumer_path LIKE $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "consumer_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "macro_name!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "21242859ee6b72466c6bfb9d826a7159de4488adccb9349cddfbdd572b4005e9" +} diff --git a/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json b/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json index a53d131a3f..704883d4f1 100644 --- a/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json +++ b/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json @@ -38,7 +38,9 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset", + "freshness" ] } } @@ -75,7 +77,9 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b.json b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json similarity index 54% rename from backend/.sqlx/query-de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b.json rename to backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json index 2ba317edc6..e3f72d9c3a 100644 --- a/backend/.sqlx/query-de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b.json +++ b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json @@ -1,12 +1,23 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT\n kind AS \"kind!: AssetKind\",\n path AS \"path!\"\n FROM asset\n WHERE workspace_id = $1\n AND usage_kind = 'script'\n AND usage_path = $2\n AND usage_access_type IN ('w', 'rw')\n ", + "query": "SELECT version, columns AS \"columns: Json>\"\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC\n LIMIT 1", "describe": { "columns": [ { "ordinal": 0, - "name": "kind!: AssetKind", - "type_info": { + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "columns: Json>", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + { "Custom": { "name": "asset_kind", "kind": { @@ -20,17 +31,7 @@ ] } } - } - }, - { - "ordinal": 1, - "name": "path!", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Text", + }, "Text" ] }, @@ -39,5 +40,5 @@ false ] }, - "hash": "de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b" + "hash": "231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7" } diff --git a/backend/.sqlx/query-246e302feda95892d5eb1a43a29d4d4739fd663cf34f00d947f24b47c481f035.json b/backend/.sqlx/query-246e302feda95892d5eb1a43a29d4d4739fd663cf34f00d947f24b47c481f035.json new file mode 100644 index 0000000000..f828b906cd --- /dev/null +++ b/backend/.sqlx/query-246e302feda95892d5eb1a43a29d4d4739fd663cf34f00d947f24b47c481f035.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM macro_definition WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "246e302feda95892d5eb1a43a29d4d4739fd663cf34f00d947f24b47c481f035" +} diff --git a/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json b/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json new file mode 100644 index 0000000000..3e617e3f0c --- /dev/null +++ b/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM native_retry_attempt nra WHERE NOT EXISTS (SELECT 1 FROM v2_job WHERE id = nra.job_id)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd" +} diff --git a/backend/.sqlx/query-256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46.json b/backend/.sqlx/query-256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46.json new file mode 100644 index 0000000000..5f74d57491 --- /dev/null +++ b/backend/.sqlx/query-256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script_trigger (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s)\n SELECT $2, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s\n FROM script_trigger WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46" +} diff --git a/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json new file mode 100644 index 0000000000..36ea3d7fb2 --- /dev/null +++ b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab" +} diff --git a/backend/.sqlx/query-27a131537cee699ae53088cfd710b70eb81b4e6b8f79d57db9d1d12d7f3ffde2.json b/backend/.sqlx/query-27a131537cee699ae53088cfd710b70eb81b4e6b8f79d57db9d1d12d7f3ffde2.json new file mode 100644 index 0000000000..508eb32ed6 --- /dev/null +++ b/backend/.sqlx/query-27a131537cee699ae53088cfd710b70eb81b4e6b8f79d57db9d1d12d7f3ffde2.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH RECURSIVE chain AS (\n SELECT id, parent_workspace_id, 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, chain.depth + 1\n FROM workspace w\n JOIN chain ON w.id = chain.parent_workspace_id\n WHERE chain.depth < 20\n )\n SELECT COALESCE(MAX(depth), 0)::bigint AS \"depth!\" FROM chain\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "depth!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "27a131537cee699ae53088cfd710b70eb81b4e6b8f79d57db9d1d12d7f3ffde2" +} diff --git a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json similarity index 59% rename from backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json rename to backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json index 0769d083d6..4e5f9f6ed7 100644 --- a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json +++ b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json @@ -1,12 +1,12 @@ { "db_name": "PostgreSQL", - "query": "SELECT 1", + "query": "SELECT NOT pg_is_in_recovery()", "describe": { "columns": [ { "ordinal": 0, "name": "?column?", - "type_info": "Int4" + "type_info": "Bool" } ], "parameters": { @@ -16,5 +16,5 @@ null ] }, - "hash": "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5" + "hash": "282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf" } diff --git a/backend/.sqlx/query-28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23.json b/backend/.sqlx/query-28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23.json new file mode 100644 index 0000000000..a1e3edbbe2 --- /dev/null +++ b/backend/.sqlx/query-28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND path = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23" +} diff --git a/backend/.sqlx/query-29eb2c40e13d6e1ff7c37a05ab107829242f015a331bf15600986be3963878ee.json b/backend/.sqlx/query-29eb2c40e13d6e1ff7c37a05ab107829242f015a331bf15600986be3963878ee.json new file mode 100644 index 0000000000..3ad3666413 --- /dev/null +++ b/backend/.sqlx/query-29eb2c40e13d6e1ff7c37a05ab107829242f015a331bf15600986be3963878ee.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id, is_dev_workspace)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END, $5\n FROM workspace WHERE id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Bool", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "29eb2c40e13d6e1ff7c37a05ab107829242f015a331bf15600986be3963878ee" +} diff --git a/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json new file mode 100644 index 0000000000..f940f25b4a --- /dev/null +++ b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json @@ -0,0 +1,62 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT version, columns AS \"columns: Json>\",\n snapshot_id, job_id, captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "columns: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "snapshot_id", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "job_id", + "type_info": "Uuid" + }, + { + "ordinal": 4, + "name": "captured_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false, + true, + true, + false + ] + }, + "hash": "2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb" +} diff --git a/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json b/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json index 7e13b3aa44..241778d359 100644 --- a/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json +++ b/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json @@ -42,7 +42,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-2c543583bcf7bcaf2f422638ce0741a2e193f18ffa11d08e8ca49cef5c3b850c.json b/backend/.sqlx/query-2c543583bcf7bcaf2f422638ce0741a2e193f18ffa11d08e8ca49cef5c3b850c.json new file mode 100644 index 0000000000..2f1ad395b6 --- /dev/null +++ b/backend/.sqlx/query-2c543583bcf7bcaf2f422638ce0741a2e193f18ffa11d08e8ca49cef5c3b850c.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE datatable_migrations SET datatable = $3 WHERE workspace_id = $1 AND datatable = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "2c543583bcf7bcaf2f422638ce0741a2e193f18ffa11d08e8ca49cef5c3b850c" +} diff --git a/backend/.sqlx/query-2eb746c4cc5c65e277d4c3a12e0bc9b6a776a4941e68d8e2b31904c677493e2d.json b/backend/.sqlx/query-2eb746c4cc5c65e277d4c3a12e0bc9b6a776a4941e68d8e2b31904c677493e2d.json new file mode 100644 index 0000000000..44d9f8f36d --- /dev/null +++ b/backend/.sqlx/query-2eb746c4cc5c65e277d4c3a12e0bc9b6a776a4941e68d8e2b31904c677493e2d.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH RECURSIVE tree AS (\n SELECT id, deleted, 0 AS depth FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.deleted, tree.depth + 1 FROM workspace w\n JOIN tree ON w.parent_workspace_id = tree.id\n WHERE tree.depth < 20\n )\n SELECT COALESCE(MAX(depth) FILTER (WHERE NOT deleted), 0)::bigint AS \"height!\" FROM tree\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "height!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "2eb746c4cc5c65e277d4c3a12e0bc9b6a776a4941e68d8e2b31904c677493e2d" +} diff --git a/backend/.sqlx/query-0b8e5fe95f4a2855678ca041b50405b698a368626da42dd9f4ce9d0681d016a1.json b/backend/.sqlx/query-2ec9f88ad80d192a2066764222fdfed7c553de4df8e876ff8589738dea93d0cf.json similarity index 71% rename from backend/.sqlx/query-0b8e5fe95f4a2855678ca041b50405b698a368626da42dd9f4ce9d0681d016a1.json rename to backend/.sqlx/query-2ec9f88ad80d192a2066764222fdfed7c553de4df8e876ff8589738dea93d0cf.json index 50ed7549e8..6163a0f9fb 100644 --- a/backend/.sqlx/query-0b8e5fe95f4a2855678ca041b50405b698a368626da42dd9f4ce9d0681d016a1.json +++ b/backend/.sqlx/query-2ec9f88ad80d192a2066764222fdfed7c553de4df8e876ff8589738dea93d0cf.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork FROM workspace_diff\n WHERE source_workspace_id = $1 AND fork_workspace_id = $2", + "query": "SELECT path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork FROM workspace_diff\n WHERE source_workspace_id = $1 AND fork_workspace_id = $2\n AND NOT EXISTS (\n SELECT 1 FROM ws_specific ws\n WHERE ws.path = workspace_diff.path\n AND ws.item_kind = workspace_diff.kind\n AND ws.workspace_id IN (workspace_diff.source_workspace_id, workspace_diff.fork_workspace_id)\n )", "describe": { "columns": [ { @@ -55,5 +55,5 @@ true ] }, - "hash": "0b8e5fe95f4a2855678ca041b50405b698a368626da42dd9f4ce9d0681d016a1" + "hash": "2ec9f88ad80d192a2066764222fdfed7c553de4df8e876ff8589738dea93d0cf" } diff --git a/backend/.sqlx/query-3020d5477b4822f1b0e3b2e4f2947e24754b919f7ee1aa2e7c1cb8c36e9e94b1.json b/backend/.sqlx/query-3020d5477b4822f1b0e3b2e4f2947e24754b919f7ee1aa2e7c1cb8c36e9e94b1.json new file mode 100644 index 0000000000..71615c6466 --- /dev/null +++ b/backend/.sqlx/query-3020d5477b4822f1b0e3b2e4f2947e24754b919f7ee1aa2e7c1cb8c36e9e94b1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO ws_specific (workspace_id, item_kind, path)\n SELECT $1::varchar, 'resource', $2::varchar\n WHERE EXISTS (SELECT 1 FROM resource WHERE workspace_id = $1::varchar AND path = $2::varchar)\n ON CONFLICT DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "3020d5477b4822f1b0e3b2e4f2947e24754b919f7ee1aa2e7c1cb8c36e9e94b1" +} diff --git a/backend/.sqlx/query-3211c7631f46975fc0127f106e84676237a3b37ffa59ff6c9134df79eed3b680.json b/backend/.sqlx/query-3211c7631f46975fc0127f106e84676237a3b37ffa59ff6c9134df79eed3b680.json new file mode 100644 index 0000000000..0624b86e0c --- /dev/null +++ b/backend/.sqlx/query-3211c7631f46975fc0127f106e84676237a3b37ffa59ff6c9134df79eed3b680.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff\n (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork)\n VALUES ('test-workspace', 'wm-fork-guard-test', 'f/restricted/item', 'script', 1, 0, true, true, true)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "3211c7631f46975fc0127f106e84676237a3b37ffa59ff6c9134df79eed3b680" +} diff --git a/backend/.sqlx/query-dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9.json b/backend/.sqlx/query-333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba.json similarity index 59% rename from backend/.sqlx/query-dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9.json rename to backend/.sqlx/query-333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba.json index 6ae3f60e49..dfbc2b75e6 100644 --- a/backend/.sqlx/query-dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9.json +++ b/backend/.sqlx/query-333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba.json @@ -1,11 +1,11 @@ { "db_name": "PostgreSQL", - "query": "SELECT email FROM usr WHERE workspace_id = $1 AND username = $2", + "query": "SELECT path FROM schedule WHERE workspace_id = $1 AND path LIKE $2", "describe": { "columns": [ { "ordinal": 0, - "name": "email", + "name": "path", "type_info": "Varchar" } ], @@ -19,5 +19,5 @@ false ] }, - "hash": "dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9" + "hash": "333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba" } diff --git a/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json b/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json new file mode 100644 index 0000000000..b475d068d5 --- /dev/null +++ b/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = $1) AS \"exists!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a" +} diff --git a/backend/.sqlx/query-34b94da001dafbbe66b3b945e71e33128bbc0dcdba9850ccbe4dc278836513bd.json b/backend/.sqlx/query-34b94da001dafbbe66b3b945e71e33128bbc0dcdba9850ccbe4dc278836513bd.json new file mode 100644 index 0000000000..1ca539c46d --- /dev/null +++ b/backend/.sqlx/query-34b94da001dafbbe66b3b945e71e33128bbc0dcdba9850ccbe4dc278836513bd.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (parent_workspace_id IS NOT NULL) AS \"is_fork!\" FROM workspace WHERE id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "is_fork!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "34b94da001dafbbe66b3b945e71e33128bbc0dcdba9850ccbe4dc278836513bd" +} diff --git a/backend/.sqlx/query-3535965aed37a1cc853b229e062d4aac2dbb2ef7596d819db222fb87187d9c4b.json b/backend/.sqlx/query-3535965aed37a1cc853b229e062d4aac2dbb2ef7596d819db222fb87187d9c4b.json new file mode 100644 index 0000000000..b814575234 --- /dev/null +++ b/backend/.sqlx/query-3535965aed37a1cc853b229e062d4aac2dbb2ef7596d819db222fb87187d9c4b.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM skip_workspace_diff_tally WHERE workspace_id IN ('test-workspace', 'wm-fork-guard-test')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "3535965aed37a1cc853b229e062d4aac2dbb2ef7596d819db222fb87187d9c4b" +} diff --git a/backend/.sqlx/query-3571fb1e1aee51850d5789e28025a055b6972eb00e4dc1818f7a68bcf04c1ba7.json b/backend/.sqlx/query-3571fb1e1aee51850d5789e28025a055b6972eb00e4dc1818f7a68bcf04c1ba7.json new file mode 100644 index 0000000000..16063b9a3f --- /dev/null +++ b/backend/.sqlx/query-3571fb1e1aee51850d5789e28025a055b6972eb00e4dc1818f7a68bcf04c1ba7.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT target_existing_size FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND upload_id = $2 LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "target_existing_size", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "3571fb1e1aee51850d5789e28025a055b6972eb00e4dc1818f7a68bcf04c1ba7" +} diff --git a/backend/.sqlx/query-36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419.json b/backend/.sqlx/query-36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419.json new file mode 100644 index 0000000000..d05abd4f3a --- /dev/null +++ b/backend/.sqlx/query-36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT path AS \"asset_path!\", usage_path AS \"producer_path!\"\n FROM asset\n WHERE workspace_id = $1 AND kind = 'ducklake' AND path = ANY($2)\n AND usage_kind = 'script' AND usage_access_type IN ('w', 'rw')", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "producer_path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419" +} diff --git a/backend/.sqlx/query-36fb69f46ccfb559cd5a25db5fb35bd5a777effb1e61f530f3c1868b407e74e4.json b/backend/.sqlx/query-36fb69f46ccfb559cd5a25db5fb35bd5a777effb1e61f530f3c1868b407e74e4.json new file mode 100644 index 0000000000..d95cc6dc75 --- /dev/null +++ b/backend/.sqlx/query-36fb69f46ccfb559cd5a25db5fb35bd5a777effb1e61f530f3c1868b407e74e4.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO macro_usage (workspace_id, consumer_path, macro_name)\n SELECT $2, consumer_path, macro_name\n FROM macro_usage WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "36fb69f46ccfb559cd5a25db5fb35bd5a777effb1e61f530f3c1868b407e74e4" +} diff --git a/backend/.sqlx/query-3703847d0cef9df3a480af8b6a5c21b6af282c68f06ce037a11f703284d8980d.json b/backend/.sqlx/query-3703847d0cef9df3a480af8b6a5c21b6af282c68f06ce037a11f703284d8980d.json new file mode 100644 index 0000000000..3a784490ef --- /dev/null +++ b/backend/.sqlx/query-3703847d0cef9df3a480af8b6a5c21b6af282c68f06ce037a11f703284d8980d.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT trigger_ref AS \"trigger_ref!\"\n FROM script_trigger\n WHERE workspace_id = $1\n AND runnable_path = $2\n AND trigger_kind = 'asset'\n AND runnable_kind = 'script'\n ORDER BY trigger_ref", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "trigger_ref!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "3703847d0cef9df3a480af8b6a5c21b6af282c68f06ce037a11f703284d8980d" +} diff --git a/backend/.sqlx/query-3740ea8dfc666c8b098e55784af185331f8b03eec083fa80c007f6f23019a5d2.json b/backend/.sqlx/query-3740ea8dfc666c8b098e55784af185331f8b03eec083fa80c007f6f23019a5d2.json new file mode 100644 index 0000000000..dcb81f26b8 --- /dev/null +++ b/backend/.sqlx/query-3740ea8dfc666c8b098e55784af185331f8b03eec083fa80c007f6f23019a5d2.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM macro_usage WHERE workspace_id = $1 AND (consumer_path = $2 OR macro_name IN (SELECT name FROM macro_definition WHERE workspace_id = $1 AND provider_path = $2))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "3740ea8dfc666c8b098e55784af185331f8b03eec083fa80c007f6f23019a5d2" +} diff --git a/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json b/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json new file mode 100644 index 0000000000..7d46bd5a66 --- /dev/null +++ b/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4" + ] + }, + "nullable": [] + }, + "hash": "39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91" +} diff --git a/backend/.sqlx/query-5190528997a879981a87420ddf3d28c978c8a5876f5c1ac1613391e86ffb550f.json b/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json similarity index 52% rename from backend/.sqlx/query-5190528997a879981a87420ddf3d28c978c8a5876f5c1ac1613391e86ffb550f.json rename to backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json index d165e78ad2..dd243cc2bb 100644 --- a/backend/.sqlx/query-5190528997a879981a87420ddf3d28c978c8a5876f5c1ac1613391e86ffb550f.json +++ b/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "DELETE FROM v2_job WHERE id = $1", + "query": "UPDATE v2_job_queue SET running = true WHERE id = $1", "describe": { "columns": [], "parameters": { @@ -10,5 +10,5 @@ }, "nullable": [] }, - "hash": "5190528997a879981a87420ddf3d28c978c8a5876f5c1ac1613391e86ffb550f" + "hash": "3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1" } diff --git a/backend/.sqlx/query-3a20f8f159b7185940639716b2ae0d5d1c3769d095352002daf8c11d10eac57b.json b/backend/.sqlx/query-3a20f8f159b7185940639716b2ae0d5d1c3769d095352002daf8c11d10eac57b.json new file mode 100644 index 0000000000..291dc00310 --- /dev/null +++ b/backend/.sqlx/query-3a20f8f159b7185940639716b2ae0d5d1c3769d095352002daf8c11d10eac57b.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n (SELECT COALESCE(SUM(GREATEST(inflight_bytes - target_existing_size, 0)), 0)\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND upload_id <> $2 AND created_at > now() - ($3::text)::interval)::bigint as \"other_reserved!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "committed!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "other_reserved!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "3a20f8f159b7185940639716b2ae0d5d1c3769d095352002daf8c11d10eac57b" +} diff --git a/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json b/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json new file mode 100644 index 0000000000..8a79de3aa4 --- /dev/null +++ b/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) as \"c!\" FROM v2_job_debounce_batch", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2" +} diff --git a/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json new file mode 100644 index 0000000000..eabe7f9bf6 --- /dev/null +++ b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::INT AS \"v!\" FROM pg_stat_activity", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84" +} diff --git a/backend/.sqlx/query-3c057e0284f1219865b8f7dc6eb3c1293c4b72d0b52abeefd1e954617984a2d8.json b/backend/.sqlx/query-3c057e0284f1219865b8f7dc6eb3c1293c4b72d0b52abeefd1e954617984a2d8.json new file mode 100644 index 0000000000..3206ba4433 --- /dev/null +++ b/backend/.sqlx/query-3c057e0284f1219865b8f7dc6eb3c1293c4b72d0b52abeefd1e954617984a2d8.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3 RETURNING name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + false + ] + }, + "hash": "3c057e0284f1219865b8f7dc6eb3c1293c4b72d0b52abeefd1e954617984a2d8" +} diff --git a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json index 3568d1723e..f5ee767768 100644 --- a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json +++ b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json @@ -128,7 +128,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-3d4b21ca3f6dce2141b0a943d3b1bdb31f26e82fb0dc97bef7114d85959b6129.json b/backend/.sqlx/query-3d4b21ca3f6dce2141b0a943d3b1bdb31f26e82fb0dc97bef7114d85959b6129.json new file mode 100644 index 0000000000..137af7638b --- /dev/null +++ b/backend/.sqlx/query-3d4b21ca3f6dce2141b0a943d3b1bdb31f26e82fb0dc97bef7114d85959b6129.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n COALESCE(\n (SELECT MIN(computed_at) FROM workspace_storage_usage WHERE workspace_id = $1) < now() - interval '10 minutes',\n true) as \"stale!\",\n (SELECT COALESCE(SUM(GREATEST(inflight_bytes - target_existing_size, 0)), 0)\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND created_at > now() - ($2::text)::interval)::bigint as \"reserved!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "committed!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "stale!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "reserved!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "3d4b21ca3f6dce2141b0a943d3b1bdb31f26e82fb0dc97bef7114d85959b6129" +} diff --git a/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json new file mode 100644 index 0000000000..8d9c34ed2b --- /dev/null +++ b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM script WHERE archived = false AND deleted = false AND auto_kind = 'wac'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79" +} diff --git a/backend/.sqlx/query-3e0c2268afa4ce21ae6da69d2a496e034c48d960930a2c0c340b28b5c44fe87f.json b/backend/.sqlx/query-3e0c2268afa4ce21ae6da69d2a496e034c48d960930a2c0c340b28b5c44fe87f.json new file mode 100644 index 0000000000..667907e290 --- /dev/null +++ b/backend/.sqlx/query-3e0c2268afa4ce21ae6da69d2a496e034c48d960930a2c0c340b28b5c44fe87f.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO macro_definition (workspace_id, name, provider_path, params, body, is_table_macro)\n SELECT $2, name, provider_path, params, body, is_table_macro\n FROM macro_definition WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "3e0c2268afa4ce21ae6da69d2a496e034c48d960930a2c0c340b28b5c44fe87f" +} diff --git a/backend/.sqlx/query-3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed.json b/backend/.sqlx/query-3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed.json new file mode 100644 index 0000000000..60dbee76ab --- /dev/null +++ b/backend/.sqlx/query-3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT p.path AS \"path!\",\n (SELECT c.completed_at\n FROM v2_job j\n JOIN v2_job_completed c ON c.id = j.id\n WHERE j.workspace_id = $1\n AND j.runnable_path = p.path\n AND j.parent_job IS NULL\n -- No 'singlestepflow': flows may share a script's path, and\n -- a same-path flow run must not read as the script being\n -- fresh (false-fresh). Script retries land as native\n -- 'script' jobs; only the rare flow-wrapper fallback is\n -- missed, which errs stale. Kept in lockstep with the\n -- freshness watchdog's queries (freshness_watchdog_ee).\n AND j.kind IN ('script', 'preview')\n AND c.status = 'success'\n ORDER BY j.created_at DESC\n LIMIT 1) AS last_success_at\n FROM unnest($2::text[]) AS p(path)\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "last_success_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed" +} diff --git a/backend/.sqlx/query-3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9.json b/backend/.sqlx/query-3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9.json new file mode 100644 index 0000000000..4599073796 --- /dev/null +++ b/backend/.sqlx/query-3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ducklake_name AS \"ducklake_name!\", metadata_schema AS \"metadata_schema!\",\n catalog AS \"catalog!\", storage AS \"storage!\",\n storage_ref AS \"storage_ref!\", data_path AS \"data_path!\",\n schema_dropped AS \"schema_dropped!\"\n FROM fork_ducklake_namespace WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ducklake_name!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "metadata_schema!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "catalog!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "storage!", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "storage_ref!", + "type_info": "Text" + }, + { + "ordinal": 5, + "name": "data_path!", + "type_info": "Text" + }, + { + "ordinal": 6, + "name": "schema_dropped!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + false, + false + ] + }, + "hash": "3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9" +} diff --git a/backend/.sqlx/query-e947340c7d40f6b9536e7a24fa84bee393e24f6f39f67e0b9e20e9cb8f04244c.json b/backend/.sqlx/query-41ce08f45b09532cbab6fb039703f08d6476d00d85a02ea3b19aa559b9ec1056.json similarity index 57% rename from backend/.sqlx/query-e947340c7d40f6b9536e7a24fa84bee393e24f6f39f67e0b9e20e9cb8f04244c.json rename to backend/.sqlx/query-41ce08f45b09532cbab6fb039703f08d6476d00d85a02ea3b19aa559b9ec1056.json index 4fe298370b..c12e0cdac4 100644 --- a/backend/.sqlx/query-e947340c7d40f6b9536e7a24fa84bee393e24f6f39f67e0b9e20e9cb8f04244c.json +++ b/backend/.sqlx/query-41ce08f45b09532cbab6fb039703f08d6476d00d85a02ea3b19aa559b9ec1056.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO workspace\n (id, name, owner, parent_workspace_id)\n VALUES ($1, $2, $3, $4)", + "query": "INSERT INTO workspace\n (id, name, owner, parent_workspace_id, is_dev_workspace)\n VALUES ($1, $2, $3, $4, $5)", "describe": { "columns": [], "parameters": { @@ -8,10 +8,11 @@ "Varchar", "Varchar", "Varchar", - "Varchar" + "Varchar", + "Bool" ] }, "nullable": [] }, - "hash": "e947340c7d40f6b9536e7a24fa84bee393e24f6f39f67e0b9e20e9cb8f04244c" + "hash": "41ce08f45b09532cbab6fb039703f08d6476d00d85a02ea3b19aa559b9ec1056" } diff --git a/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json new file mode 100644 index 0000000000..a6ab79cd7c --- /dev/null +++ b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"\n FROM workspace f\n JOIN workspace p ON p.id = f.parent_workspace_id\n WHERE f.id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "deleted!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688" +} diff --git a/backend/.sqlx/query-1a9f2ed5045016a3953db335957b26f41efc8a3cad7af7bc8fe97df6a5bf5078.json b/backend/.sqlx/query-42a0ba479ff164cc190c350927e13902ed94816142faf15446b4b9f19c3097d7.json similarity index 76% rename from backend/.sqlx/query-1a9f2ed5045016a3953db335957b26f41efc8a3cad7af7bc8fe97df6a5bf5078.json rename to backend/.sqlx/query-42a0ba479ff164cc190c350927e13902ed94816142faf15446b4b9f19c3097d7.json index 0265d7d2b5..76eab1d117 100644 --- a/backend/.sqlx/query-1a9f2ed5045016a3953db335957b26f41efc8a3cad7af7bc8fe97df6a5bf5078.json +++ b/backend/.sqlx/query-42a0ba479ff164cc190c350927e13902ed94816142faf15446b4b9f19c3097d7.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO usr\n (workspace_id, email, username, is_admin)\n SELECT $1, email, username, is_admin FROM usr\n WHERE workspace_id = $3 AND email = $2\n ", + "query": "INSERT INTO usr\n (workspace_id, email, username, is_admin)\n SELECT $1, email, username, is_admin FROM usr\n WHERE workspace_id = $3 AND email = $2\n ON CONFLICT DO NOTHING\n ", "describe": { "columns": [], "parameters": { @@ -12,5 +12,5 @@ }, "nullable": [] }, - "hash": "1a9f2ed5045016a3953db335957b26f41efc8a3cad7af7bc8fe97df6a5bf5078" + "hash": "42a0ba479ff164cc190c350927e13902ed94816142faf15446b4b9f19c3097d7" } diff --git a/backend/.sqlx/query-431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722.json b/backend/.sqlx/query-431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722.json new file mode 100644 index 0000000000..b554ef8b55 --- /dev/null +++ b/backend/.sqlx/query-431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (path) path, content\n FROM script\n WHERE workspace_id = $1 AND path = ANY($2)\n AND archived = false AND deleted = false\n ORDER BY path, created_at DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722" +} diff --git a/backend/.sqlx/query-447dd940e6ee86be8eab6982c22a7fd0fe229643573509bd1705a691ff7c4ba2.json b/backend/.sqlx/query-447dd940e6ee86be8eab6982c22a7fd0fe229643573509bd1705a691ff7c4ba2.json new file mode 100644 index 0000000000..1146134a34 --- /dev/null +++ b/backend/.sqlx/query-447dd940e6ee86be8eab6982c22a7fd0fe229643573509bd1705a691ff7c4ba2.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO macro_definition (workspace_id, name, provider_path, params, body, is_table_macro) VALUES ($1, $2, $3, $4, $5, $6)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Text", + "Text", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "447dd940e6ee86be8eab6982c22a7fd0fe229643573509bd1705a691ff7c4ba2" +} diff --git a/backend/.sqlx/query-44c4e9848bc97b66a49d5454d30d40dfe1ffe51839884f3e9e9360d9e17b5afd.json b/backend/.sqlx/query-44c4e9848bc97b66a49d5454d30d40dfe1ffe51839884f3e9e9360d9e17b5afd.json new file mode 100644 index 0000000000..1e1c5f20c3 --- /dev/null +++ b/backend/.sqlx/query-44c4e9848bc97b66a49d5454d30d40dfe1ffe51839884f3e9e9360d9e17b5afd.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "code_up", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "code_down", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "44c4e9848bc97b66a49d5454d30d40dfe1ffe51839884f3e9e9360d9e17b5afd" +} diff --git a/backend/.sqlx/query-453c8dc05f4946a30fbbc517b5beaaee42e954e0b4ed0bd5ba6a5b308894d6ba.json b/backend/.sqlx/query-453c8dc05f4946a30fbbc517b5beaaee42e954e0b4ed0bd5ba6a5b308894d6ba.json new file mode 100644 index 0000000000..cb6bb20397 --- /dev/null +++ b/backend/.sqlx/query-453c8dc05f4946a30fbbc517b5beaaee42e954e0b4ed0bd5ba6a5b308894d6ba.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH RECURSIVE tree AS (\n SELECT id, deleted, 0 AS depth FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.deleted, tree.depth + 1 FROM workspace w\n JOIN tree ON w.parent_workspace_id = tree.id\n WHERE tree.depth < 20\n )\n SELECT COUNT(DISTINCT id) AS \"count!\" FROM tree WHERE id != $1 AND NOT deleted\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "453c8dc05f4946a30fbbc517b5beaaee42e954e0b4ed0bd5ba6a5b308894d6ba" +} diff --git a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json b/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json deleted file mode 100644 index 2f62420f87..0000000000 --- a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_result_stream_v2\n WHERE job_id NOT IN (SELECT id FROM v2_job_queue)\n AND job_id NOT IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at > NOW() - INTERVAL '60 seconds'\n )\n RETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51" -} diff --git a/backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json b/backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json new file mode 100644 index 0000000000..e70052886f --- /dev/null +++ b/backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n COUNT(*) FILTER (WHERE worker LIKE $1) as \"live_workers!\",\n COUNT(DISTINCT worker_instance) FILTER (WHERE worker LIKE $1) as \"live_instances!\",\n COUNT(*) FILTER (WHERE worker LIKE $2) as \"live_agent_workers!\"\n FROM worker_ping\n WHERE ping_at > now() - interval '30 seconds'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "live_workers!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "live_instances!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "live_agent_workers!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83" +} diff --git a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json b/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json deleted file mode 100644 index f63afbc986..0000000000 --- a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8", - "UuidArray" - ] - }, - "nullable": [ - false - ] - }, - "hash": "45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f" -} diff --git a/backend/.sqlx/query-45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd.json b/backend/.sqlx/query-45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd.json new file mode 100644 index 0000000000..5b3e70acbf --- /dev/null +++ b/backend/.sqlx/query-45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT EXISTS(\n SELECT 1 FROM v2_job_queue q\n JOIN v2_job j ON j.id = q.id\n WHERE q.workspace_id = $1\n AND j.runnable_path = $2\n AND j.parent_job IS NULL\n AND j.kind IN ('script', 'preview')\n AND (q.running = true OR q.scheduled_for <= now())\n ) AS \"in_flight!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "in_flight!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd" +} diff --git a/backend/.sqlx/query-45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f.json b/backend/.sqlx/query-45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f.json new file mode 100644 index 0000000000..6dc5fdb9a6 --- /dev/null +++ b/backend/.sqlx/query-45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE pipeline_freshness_state\n SET attempts = attempts + 1,\n last_push_at = now(),\n next_attempt_at = now()\n + (LEAST($3::bigint, $4::bigint * (1::bigint << LEAST(attempts + 1, 20)))::text\n || ' seconds')::interval\n WHERE workspace_id = $1 AND script_path = $2 AND next_attempt_at <= now()\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f" +} diff --git a/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json b/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json deleted file mode 100644 index 3995bffd64..0000000000 --- a/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618" -} diff --git a/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json b/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json new file mode 100644 index 0000000000..0c4d90b073 --- /dev/null +++ b/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM flow_conversation_message m\n USING flow_conversation c\n WHERE m.conversation_id = c.id AND c.workspace_id = $1 AND m.job_id = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c" +} diff --git a/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json b/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json index b90ba150d9..c269f7f340 100644 --- a/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json +++ b/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json @@ -41,7 +41,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-472d351f0bba2dc3404d83aec131cd50b297f80e5392715f54f32fab4bf346fc.json b/backend/.sqlx/query-472d351f0bba2dc3404d83aec131cd50b297f80e5392715f54f32fab4bf346fc.json new file mode 100644 index 0000000000..18af751ca0 --- /dev/null +++ b/backend/.sqlx/query-472d351f0bba2dc3404d83aec131cd50b297f80e5392715f54f32fab4bf346fc.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext('workspace_multipart_inflight'), hashtext($1))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "472d351f0bba2dc3404d83aec131cd50b297f80e5392715f54f32fab4bf346fc" +} diff --git a/backend/.sqlx/query-4865e22673a7886ff84cfe5ee1114d66b1678bfe3028df89dff4e343e2f4ab44.json b/backend/.sqlx/query-4865e22673a7886ff84cfe5ee1114d66b1678bfe3028df89dff4e343e2f4ab44.json new file mode 100644 index 0000000000..5f114c0156 --- /dev/null +++ b/backend/.sqlx/query-4865e22673a7886ff84cfe5ee1114d66b1678bfe3028df89dff4e343e2f4ab44.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH potential AS (\n SELECT email, operator FROM usr WHERE is_service_account IS false\n UNION\n SELECT email, operator FROM workspace_invite\n ),\n per_user AS (\n SELECT email, bool_and(operator) AS only_operator FROM potential GROUP BY email\n )\n SELECT\n COUNT(*) FILTER (WHERE NOT only_operator) AS \"authors!\",\n COUNT(*) FILTER (WHERE only_operator) AS \"operators!\"\n FROM per_user\n WHERE email NOT IN (SELECT email FROM password WHERE disabled IS true)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "authors!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "operators!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "4865e22673a7886ff84cfe5ee1114d66b1678bfe3028df89dff4e343e2f4ab44" +} diff --git a/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json new file mode 100644 index 0000000000..43f527b3aa --- /dev/null +++ b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_database_size(current_database())::BIGINT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf" +} diff --git a/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json new file mode 100644 index 0000000000..e99432f11f --- /dev/null +++ b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('max_connections')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617" +} diff --git a/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json b/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json index 7d950f6d8f..7474818bf3 100644 --- a/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json +++ b/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json @@ -80,7 +80,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-4add520575717d449a3758326777fee03012a3fa0fb943239e004cc4a7067501.json b/backend/.sqlx/query-4add520575717d449a3758326777fee03012a3fa0fb943239e004cc4a7067501.json new file mode 100644 index 0000000000..64330e91cf --- /dev/null +++ b/backend/.sqlx/query-4add520575717d449a3758326777fee03012a3fa0fb943239e004cc4a7067501.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND (content LIKE '%' || $2 || '%' OR path = ANY($3))\n ORDER BY path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "4add520575717d449a3758326777fee03012a3fa0fb943239e004cc4a7067501" +} diff --git a/backend/.sqlx/query-4b03b55595bbc02e63fc920cbaf4a1503df8811d2d212f1a97ffd00599aeb4c7.json b/backend/.sqlx/query-4b03b55595bbc02e63fc920cbaf4a1503df8811d2d212f1a97ffd00599aeb4c7.json new file mode 100644 index 0000000000..06b8754d5f --- /dev/null +++ b/backend/.sqlx/query-4b03b55595bbc02e63fc920cbaf4a1503df8811d2d212f1a97ffd00599aeb4c7.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO notify_event (channel, payload) VALUES ('notify_macro_registry_change', $1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "4b03b55595bbc02e63fc920cbaf4a1503df8811d2d212f1a97ffd00599aeb4c7" +} diff --git a/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json b/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json new file mode 100644 index 0000000000..c2c041e131 --- /dev/null +++ b/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext($1))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f" +} diff --git a/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json b/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json new file mode 100644 index 0000000000..07d1753aa0 --- /dev/null +++ b/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301" +} diff --git a/backend/.sqlx/query-4e4efea2b8d3b0bd2717b27d1895b00e8dba07aa817ee9bfbe2271d41c9b411a.json b/backend/.sqlx/query-4e4efea2b8d3b0bd2717b27d1895b00e8dba07aa817ee9bfbe2271d41c9b411a.json new file mode 100644 index 0000000000..8e9e6a8713 --- /dev/null +++ b/backend/.sqlx/query-4e4efea2b8d3b0bd2717b27d1895b00e8dba07aa817ee9bfbe2271d41c9b411a.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET is_dev_workspace = false WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "4e4efea2b8d3b0bd2717b27d1895b00e8dba07aa817ee9bfbe2271d41c9b411a" +} diff --git a/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json b/backend/.sqlx/query-4efaa4c3c14987fe95a508cc0cbac91ee793ad6f5ceed5b3eda9b48617c5434d.json similarity index 63% rename from backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json rename to backend/.sqlx/query-4efaa4c3c14987fe95a508cc0cbac91ee793ad6f5ceed5b3eda9b48617c5434d.json index 4bb89660ec..bdb39d134f 100644 --- a/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json +++ b/backend/.sqlx/query-4efaa4c3c14987fe95a508cc0cbac91ee793ad6f5ceed5b3eda9b48617c5434d.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) as \"username?\",\n d.created_at as \"draft_saved_at!\"\n FROM draft d\n LEFT JOIN usr u\n ON u.workspace_id = d.workspace_id\n AND u.email = d.email\n WHERE d.workspace_id = $1\n AND d.path = $2\n AND d.typ = $3\n AND (d.email IS NULL OR d.email <> $4)\n ORDER BY d.email NULLS LAST", + "query": "SELECT COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN d.email END) as \"username?\",\n d.created_at as \"draft_saved_at!\"\n FROM draft d\n LEFT JOIN usr u\n ON u.workspace_id = d.workspace_id\n AND u.email = d.email\n LEFT JOIN password p\n ON p.email = d.email\n AND p.super_admin = true\n WHERE d.workspace_id = $1\n AND d.path = $2\n AND d.typ = $3\n AND (d.email IS NULL OR d.email <> $4)\n ORDER BY d.email NULLS LAST", "describe": { "columns": [ { @@ -46,7 +46,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } @@ -59,5 +60,5 @@ false ] }, - "hash": "6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b" + "hash": "4efaa4c3c14987fe95a508cc0cbac91ee793ad6f5ceed5b3eda9b48617c5434d" } diff --git a/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json new file mode 100644 index 0000000000..5bd3cf80a7 --- /dev/null +++ b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM flow WHERE archived = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265" +} diff --git a/backend/.sqlx/query-530a797e67ff352471f1b34f260dd530a653081619e6d3132bf07996520b1e25.json b/backend/.sqlx/query-530a797e67ff352471f1b34f260dd530a653081619e6d3132bf07996520b1e25.json new file mode 100644 index 0000000000..465ee41633 --- /dev/null +++ b/backend/.sqlx/query-530a797e67ff352471f1b34f260dd530a653081619e6d3132bf07996520b1e25.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace\n (id, name, owner, parent_workspace_id, is_dev_workspace, dev_workspace_label)\n VALUES ($1, $2, $3, $4, $5, $6)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Bool", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "530a797e67ff352471f1b34f260dd530a653081619e6d3132bf07996520b1e25" +} diff --git a/backend/.sqlx/query-9c85ba8d41bedbcb5466f44a7d4cf6b4946e1fd337f00d243f518283783833c9.json b/backend/.sqlx/query-5312b8db714139a94d7ff1c0794af063c36ac17e9d331cd9980b91b28d713c72.json similarity index 57% rename from backend/.sqlx/query-9c85ba8d41bedbcb5466f44a7d4cf6b4946e1fd337f00d243f518283783833c9.json rename to backend/.sqlx/query-5312b8db714139a94d7ff1c0794af063c36ac17e9d331cd9980b91b28d713c72.json index fce125c6d5..35b70a239c 100644 --- a/backend/.sqlx/query-9c85ba8d41bedbcb5466f44a7d4cf6b4946e1fd337f00d243f518283783833c9.json +++ b/backend/.sqlx/query-5312b8db714139a94d7ff1c0794af063c36ac17e9d331cd9980b91b28d713c72.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT bool_and(operator) FROM (\n SELECT operator FROM usr WHERE email = $1\n UNION ALL\n SELECT operator FROM workspace_invite WHERE email = $1\n ) t", + "query": "SELECT bool_and(operator) FROM (\n SELECT operator FROM usr WHERE email = $1 AND is_service_account IS false\n UNION ALL\n SELECT operator FROM workspace_invite WHERE email = $1\n ) t", "describe": { "columns": [ { @@ -18,5 +18,5 @@ null ] }, - "hash": "9c85ba8d41bedbcb5466f44a7d4cf6b4946e1fd337f00d243f518283783833c9" + "hash": "5312b8db714139a94d7ff1c0794af063c36ac17e9d331cd9980b91b28d713c72" } diff --git a/backend/.sqlx/query-5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe.json b/backend/.sqlx/query-5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe.json new file mode 100644 index 0000000000..a8facb9d1b --- /dev/null +++ b/backend/.sqlx/query-5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO schedule (\n workspace_id, path, schedule, timezone, edited_by, script_path,\n is_flow, enabled, email, permissioned_as, summary, tag, cron_version\n ) VALUES ($1, $2, $3, 'Etc/UTC', $4, $2, false, true, $5, $6, $7, 'duckdb', 'v2')\n ON CONFLICT (workspace_id, path) DO UPDATE SET\n schedule = EXCLUDED.schedule,\n timezone = EXCLUDED.timezone,\n edited_by = EXCLUDED.edited_by,\n edited_at = now(),\n script_path = EXCLUDED.script_path,\n is_flow = false,\n enabled = true,\n email = EXCLUDED.email,\n permissioned_as = EXCLUDED.permissioned_as,\n summary = EXCLUDED.summary,\n tag = EXCLUDED.tag,\n cron_version = EXCLUDED.cron_version,\n error = NULL", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe" +} diff --git a/backend/.sqlx/query-53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109.json b/backend/.sqlx/query-53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109.json new file mode 100644 index 0000000000..8bce508e75 --- /dev/null +++ b/backend/.sqlx/query-53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND path = ANY($2) AND path != ALL($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "TextArray", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109" +} diff --git a/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json b/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json index 2e189af5e8..d81d6ebd94 100644 --- a/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json +++ b/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json @@ -35,7 +35,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json b/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json new file mode 100644 index 0000000000..8c46b8250c --- /dev/null +++ b/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT attempt FROM native_retry_attempt WHERE job_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "attempt", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + false + ] + }, + "hash": "54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c" +} diff --git a/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json b/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json index 0191cf8bfd..91c941593f 100644 --- a/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json +++ b/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json @@ -42,7 +42,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-564f4bdf43135c603518ef35a3e95bde7f479877018980cffe1c1e9d34aad59e.json b/backend/.sqlx/query-564f4bdf43135c603518ef35a3e95bde7f479877018980cffe1c1e9d34aad59e.json new file mode 100644 index 0000000000..20ac675432 --- /dev/null +++ b/backend/.sqlx/query-564f4bdf43135c603518ef35a3e95bde7f479877018980cffe1c1e9d34aad59e.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 RETURNING timestamp, name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "timestamp", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "564f4bdf43135c603518ef35a3e95bde7f479877018980cffe1c1e9d34aad59e" +} diff --git a/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json new file mode 100644 index 0000000000..711970ff32 --- /dev/null +++ b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('server_version_num')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee" +} diff --git a/backend/.sqlx/query-572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89.json b/backend/.sqlx/query-572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89.json new file mode 100644 index 0000000000..580666f07d --- /dev/null +++ b/backend/.sqlx/query-572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT EXISTS(\n SELECT 1 FROM v2_job j\n JOIN v2_job_completed c ON c.id = j.id\n WHERE j.workspace_id = $1\n AND j.runnable_path = $2\n AND j.parent_job IS NULL\n AND j.kind IN ('script', 'preview')\n AND c.status = 'success'\n AND c.completed_at > now() - ($3::bigint::text || ' seconds')::interval\n ) AS \"fresh!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "fresh!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89" +} diff --git a/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json b/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json index bb326fab87..0c445c1ae5 100644 --- a/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json +++ b/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json @@ -35,7 +35,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-59c1137f718d442c4e99fc07365ebbe13e6298749b19570f9f8c4a0c7958b0ce.json b/backend/.sqlx/query-59c1137f718d442c4e99fc07365ebbe13e6298749b19570f9f8c4a0c7958b0ce.json new file mode 100644 index 0000000000..cd2dd23319 --- /dev/null +++ b/backend/.sqlx/query-59c1137f718d442c4e99fc07365ebbe13e6298749b19570f9f8c4a0c7958b0ce.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n COALESCE(\n (SELECT MIN(computed_at) FROM workspace_storage_usage WHERE workspace_id = $1) < now() - interval '10 minutes',\n true) as \"stale!\",\n (SELECT COALESCE(SUM(GREATEST(inflight_bytes - target_existing_size, 0)), 0)\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND created_at > now() - ($2::text)::interval\n AND ($3::text IS NULL OR upload_id <> $3))::bigint as \"reserved!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "committed!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "stale!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "reserved!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "59c1137f718d442c4e99fc07365ebbe13e6298749b19570f9f8c4a0c7958b0ce" +} diff --git a/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json b/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json new file mode 100644 index 0000000000..0535031aac --- /dev/null +++ b/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dispatch_event WHERE workspace_id = $1 AND producer_job_id = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5" +} diff --git a/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json b/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json new file mode 100644 index 0000000000..28c09ca528 --- /dev/null +++ b/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\",\n result AS \"result: Json>\",\n started_at AS \"started_at!\"FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "success!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "result: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "started_at!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Uuid", + "Int8" + ] + }, + "nullable": [ + null, + true, + true + ] + }, + "hash": "5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391" +} diff --git a/backend/.sqlx/query-5b67c3af6477d7029d118ac259a7535d4d962c328a060f33c07191ac3e033e82.json b/backend/.sqlx/query-5b67c3af6477d7029d118ac259a7535d4d962c328a060f33c07191ac3e033e82.json new file mode 100644 index 0000000000..12628305cc --- /dev/null +++ b/backend/.sqlx/query-5b67c3af6477d7029d118ac259a7535d4d962c328a060f33c07191ac3e033e82.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "5b67c3af6477d7029d118ac259a7535d4d962c328a060f33c07191ac3e033e82" +} diff --git a/backend/.sqlx/query-5cca3509374b1ddd8707c930bc382734e4ca6cad2d849f75a8f8a249f83df83f.json b/backend/.sqlx/query-5cca3509374b1ddd8707c930bc382734e4ca6cad2d849f75a8f8a249f83df83f.json new file mode 100644 index 0000000000..14b085db2e --- /dev/null +++ b/backend/.sqlx/query-5cca3509374b1ddd8707c930bc382734e4ca6cad2d849f75a8f8a249f83df83f.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ws.datatable->'datatables'->$2 FROM workspace_settings ws WHERE ws.workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "5cca3509374b1ddd8707c930bc382734e4ca6cad2d849f75a8f8a249f83df83f" +} diff --git a/backend/.sqlx/query-5cd47ddc6a0181c8e23998adae103ff9c96d613fb2d9b698361f4ede8031e5d2.json b/backend/.sqlx/query-5cd47ddc6a0181c8e23998adae103ff9c96d613fb2d9b698361f4ede8031e5d2.json new file mode 100644 index 0000000000..be54b4a25d --- /dev/null +++ b/backend/.sqlx/query-5cd47ddc6a0181c8e23998adae103ff9c96d613fb2d9b698361f4ede8031e5d2.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM workspace_multipart_inflight WHERE workspace_id = $1 AND upload_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "5cd47ddc6a0181c8e23998adae103ff9c96d613fb2d9b698361f4ede8031e5d2" +} diff --git a/backend/.sqlx/query-5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c.json b/backend/.sqlx/query-5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c.json new file mode 100644 index 0000000000..18e0936d3a --- /dev/null +++ b/backend/.sqlx/query-5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO pipeline_freshness_state\n (workspace_id, script_path, attempts, last_push_at, next_attempt_at)\n VALUES ($1, $2, 1, now(),\n now() + (LEAST($3::bigint, $4::bigint * 2)::text || ' seconds')::interval)\n ON CONFLICT (workspace_id, script_path) DO NOTHING\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c" +} diff --git a/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json new file mode 100644 index 0000000000..20c5c58c40 --- /dev/null +++ b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n COUNT(*) FILTER (WHERE av.raw_app = false)::BIGINT AS \"low_code!\",\n COUNT(*) FILTER (WHERE av.raw_app = true)::BIGINT AS \"raw!\"\n FROM app a\n JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "low_code!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "raw!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174" +} diff --git a/backend/.sqlx/query-63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541.json b/backend/.sqlx/query-63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541.json new file mode 100644 index 0000000000..1bcb7e6e53 --- /dev/null +++ b/backend/.sqlx/query-63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH ancestor AS (\n SELECT wid, ord FROM unnest($1::text[]) WITH ORDINALITY AS a(wid, ord)\n ), anc_mat AS (\n -- Per table, the nearest ancestor (lowest ord) that materialized it.\n SELECT DISTINCT ON (mp.asset_path) mp.asset_path, a.wid, a.ord\n FROM materialized_partition mp\n JOIN ancestor a ON a.wid = mp.workspace_id\n WHERE mp.asset_kind = 'ducklake' AND mp.status = 'materialized'\n AND split_part(mp.asset_path, '/', 1) = $3 AND mp.asset_path LIKE '%/%'\n ORDER BY mp.asset_path, a.ord\n ), fork_mat AS (\n -- Fork-OWNED assets: anything whose physical table exists in the fork\n -- namespace, not just clean materializations. A committed write whose data\n -- tests failed afterwards records status='failed' WITH a snapshot — its table\n -- is real, and a defer view emitted over it would silently yield to it\n -- (CREATE VIEW IF NOT EXISTS) while claiming the read defers to the parent.\n SELECT DISTINCT asset_path FROM materialized_partition\n WHERE workspace_id = $2 AND asset_kind = 'ducklake'\n AND (status = 'materialized' OR snapshot_id IS NOT NULL)\n ), latest_schema AS (\n SELECT DISTINCT ON (workspace_id, asset_path) workspace_id, asset_path, columns\n FROM materialized_asset_schema\n WHERE workspace_id = ANY($1) AND asset_kind = 'ducklake'\n ORDER BY workspace_id, asset_path, version DESC\n )\n SELECT am.asset_path AS \"asset_path!\",\n am.ord AS \"ord!\",\n COALESCE(EXISTS (\n SELECT 1 FROM jsonb_array_elements(ls.columns) e\n WHERE e->>'name' = 'is_current'\n ), false) AS \"has_current!\"\n FROM anc_mat am\n LEFT JOIN latest_schema ls\n ON ls.asset_path = am.asset_path AND ls.workspace_id = am.wid\n WHERE am.asset_path NOT IN (SELECT asset_path FROM fork_mat)\n ORDER BY am.asset_path\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "ord!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "has_current!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "TextArray", + "Text", + "Text" + ] + }, + "nullable": [ + false, + null, + null + ] + }, + "hash": "63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541" +} diff --git a/backend/.sqlx/query-c095a9658c542efc9f0255a1b536d2fd8a25fe4cd57c223db7d744493f8470c6.json b/backend/.sqlx/query-63d323be5cacb7a02283d7d82c79bc408b9a33a228b80e88ec3f6432944a7c19.json similarity index 63% rename from backend/.sqlx/query-c095a9658c542efc9f0255a1b536d2fd8a25fe4cd57c223db7d744493f8470c6.json rename to backend/.sqlx/query-63d323be5cacb7a02283d7d82c79bc408b9a33a228b80e88ec3f6432944a7c19.json index bb80e8d19a..a9cdc35dce 100644 --- a/backend/.sqlx/query-c095a9658c542efc9f0255a1b536d2fd8a25fe4cd57c223db7d744493f8470c6.json +++ b/backend/.sqlx/query-63d323be5cacb7a02283d7d82c79bc408b9a33a228b80e88ec3f6432944a7c19.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT workspace.id, workspace.name, usr.username, workspace_settings.color, workspace.parent_workspace_id,\n CASE WHEN usr.operator THEN workspace_settings.operator_settings ELSE NULL END as operator_settings,\n usr.disabled\n FROM workspace\n JOIN usr ON usr.workspace_id = workspace.id\n JOIN workspace_settings ON workspace_settings.workspace_id = workspace.id\n WHERE usr.email = $1 AND workspace.deleted = false", + "query": "SELECT workspace.id, workspace.name, usr.username, workspace_settings.color, workspace.parent_workspace_id,\n workspace.is_dev_workspace,\n CASE WHEN usr.operator THEN workspace_settings.operator_settings ELSE NULL END as operator_settings,\n usr.disabled\n FROM workspace\n JOIN usr ON usr.workspace_id = workspace.id\n JOIN workspace_settings ON workspace_settings.workspace_id = workspace.id\n WHERE usr.email = $1 AND workspace.deleted = false", "describe": { "columns": [ { @@ -30,11 +30,16 @@ }, { "ordinal": 5, + "name": "is_dev_workspace", + "type_info": "Bool" + }, + { + "ordinal": 6, "name": "operator_settings", "type_info": "Jsonb" }, { - "ordinal": 6, + "ordinal": 7, "name": "disabled", "type_info": "Bool" } @@ -50,9 +55,10 @@ false, true, true, + false, null, false ] }, - "hash": "c095a9658c542efc9f0255a1b536d2fd8a25fe4cd57c223db7d744493f8470c6" + "hash": "63d323be5cacb7a02283d7d82c79bc408b9a33a228b80e88ec3f6432944a7c19" } diff --git a/backend/.sqlx/query-63d6d968905cf82fb3bb0577d41a29a8f88010891fcf3eaf4761a57461f97703.json b/backend/.sqlx/query-63d6d968905cf82fb3bb0577d41a29a8f88010891fcf3eaf4761a57461f97703.json new file mode 100644 index 0000000000..3959f4e8cc --- /dev/null +++ b/backend/.sqlx/query-63d6d968905cf82fb3bb0577d41a29a8f88010891fcf3eaf4761a57461f97703.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET dev_workspace_label = $1 WHERE id = $2 AND is_dev_workspace RETURNING id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "63d6d968905cf82fb3bb0577d41a29a8f88010891fcf3eaf4761a57461f97703" +} diff --git a/backend/.sqlx/query-651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b.json b/backend/.sqlx/query-651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b.json deleted file mode 100644 index cb20ec2ffb..0000000000 --- a/backend/.sqlx/query-651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE v2_job\n SET args = CASE\n WHEN args ? 'partition'\n THEN $1 || jsonb_build_object('partition', args -> 'partition')\n ELSE $1\n END,\n preprocessed = TRUE\n WHERE id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Jsonb", - "Uuid" - ] - }, - "nullable": [] - }, - "hash": "651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b" -} diff --git a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json b/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json index 08ebe6bba5..fcc16e9a7c 100644 --- a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json +++ b/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json @@ -161,7 +161,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-689c85d30f898f237c8a3a1e883440276ea6faee8589ff0febedf6917977a40e.json b/backend/.sqlx/query-689c85d30f898f237c8a3a1e883440276ea6faee8589ff0febedf6917977a40e.json new file mode 100644 index 0000000000..83e2711cf5 --- /dev/null +++ b/backend/.sqlx/query-689c85d30f898f237c8a3a1e883440276ea6faee8589ff0febedf6917977a40e.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name AS \"name!\", provider_path AS \"provider_path!\",\n params AS \"params!\", is_table_macro AS \"is_table_macro!\"\n FROM macro_definition\n WHERE workspace_id = $1\n ORDER BY provider_path, name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "provider_path!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "params!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "is_table_macro!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false + ] + }, + "hash": "689c85d30f898f237c8a3a1e883440276ea6faee8589ff0febedf6917977a40e" +} diff --git a/backend/.sqlx/query-6939fbbaaacd5d5a34016ed9eb295ba3ef329059366632dd94262f2e1bdfed97.json b/backend/.sqlx/query-6939fbbaaacd5d5a34016ed9eb295ba3ef329059366632dd94262f2e1bdfed97.json new file mode 100644 index 0000000000..4dfdcaab97 --- /dev/null +++ b/backend/.sqlx/query-6939fbbaaacd5d5a34016ed9eb295ba3ef329059366632dd94262f2e1bdfed97.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name FROM macro_definition WHERE workspace_id = $1 AND provider_path != $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "6939fbbaaacd5d5a34016ed9eb295ba3ef329059366632dd94262f2e1bdfed97" +} diff --git a/backend/.sqlx/query-6ac00d65b3b7707cba9456171d4332c696e12b52066a5ae2ab10fe3b5bd0f3d0.json b/backend/.sqlx/query-6ac00d65b3b7707cba9456171d4332c696e12b52066a5ae2ab10fe3b5bd0f3d0.json new file mode 100644 index 0000000000..3a1dc9da28 --- /dev/null +++ b/backend/.sqlx/query-6ac00d65b3b7707cba9456171d4332c696e12b52066a5ae2ab10fe3b5bd0f3d0.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, code_down FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "code_down", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + false, + true + ] + }, + "hash": "6ac00d65b3b7707cba9456171d4332c696e12b52066a5ae2ab10fe3b5bd0f3d0" +} diff --git a/backend/.sqlx/query-6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7.json b/backend/.sqlx/query-6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7.json new file mode 100644 index 0000000000..9a0bbf46ef --- /dev/null +++ b/backend/.sqlx/query-6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings\n SET ducklake = jsonb_set(ducklake, ARRAY['ducklakes', $2, 'fork_behavior'], '\"shared\"')\n WHERE workspace_id = $1 AND ducklake->'ducklakes' ? $2\n RETURNING 1 AS \"one!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "one!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7" +} diff --git a/backend/.sqlx/query-6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f.json b/backend/.sqlx/query-6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f.json new file mode 100644 index 0000000000..4e8c7acee2 --- /dev/null +++ b/backend/.sqlx/query-6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE pipeline_freshness_state SET next_attempt_at = now() - interval '1 second'\n WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f" +} diff --git a/backend/.sqlx/query-71ee328f016bac75a9050dc3c13dc898c0b40fe6f1c4e5c631b6f784f7e73a3d.json b/backend/.sqlx/query-71ee328f016bac75a9050dc3c13dc898c0b40fe6f1c4e5c631b6f784f7e73a3d.json new file mode 100644 index 0000000000..acfa08d0bf --- /dev/null +++ b/backend/.sqlx/query-71ee328f016bac75a9050dc3c13dc898c0b40fe6f1c4e5c631b6f784f7e73a3d.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH RECURSIVE tree AS (\n SELECT id, 0 AS depth FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, tree.depth + 1 FROM workspace w\n JOIN tree ON w.parent_workspace_id = tree.id\n WHERE tree.depth < 20\n )\n SELECT id AS \"id!\" FROM tree WHERE id != $1\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "71ee328f016bac75a9050dc3c13dc898c0b40fe6f1c4e5c631b6f784f7e73a3d" +} diff --git a/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json b/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json new file mode 100644 index 0000000000..d55722ed24 --- /dev/null +++ b/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)\n VALUES ($1, $2, $3, $4, now(), $5)\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET description = EXCLUDED.description,\n instructions = EXCLUDED.instructions,\n edited_at = now(),\n edited_by = EXCLUDED.edited_by", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8" +} diff --git a/backend/.sqlx/query-754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc.json b/backend/.sqlx/query-754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc.json deleted file mode 100644 index c5266c66b5..0000000000 --- a/backend/.sqlx/query-754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT runnable_path AS \"runnable_path!\", kind::text AS \"kind!\"\n FROM v2_job\n WHERE workspace_id = $1 AND trigger_kind = 'asset'\n ORDER BY runnable_path", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "runnable_path!", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "kind!", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - true, - null - ] - }, - "hash": "754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc" -} diff --git a/backend/.sqlx/query-760908f44cafb500e4ba2515d1b030390d9510e73e9b7df347b697d6dc7aefe6.json b/backend/.sqlx/query-760908f44cafb500e4ba2515d1b030390d9510e73e9b7df347b697d6dc7aefe6.json new file mode 100644 index 0000000000..384686eb35 --- /dev/null +++ b/backend/.sqlx/query-760908f44cafb500e4ba2515d1b030390d9510e73e9b7df347b697d6dc7aefe6.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down) VALUES ($1, $2, $3, $4, $5, $6)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "760908f44cafb500e4ba2515d1b030390d9510e73e9b7df347b697d6dc7aefe6" +} diff --git a/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json b/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json new file mode 100644 index 0000000000..694ed1887f --- /dev/null +++ b/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM debounce_key WHERE key = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5" +} diff --git a/backend/.sqlx/query-76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb.json b/backend/.sqlx/query-76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb.json new file mode 100644 index 0000000000..b6b0f06be5 --- /dev/null +++ b/backend/.sqlx/query-76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ducklake->'ducklakes' FROM workspace_settings WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb" +} diff --git a/backend/.sqlx/query-7853a596a01884070455e68bbf8ab2afa79a5b5b8521f68ab19f0470f0265c7c.json b/backend/.sqlx/query-7853a596a01884070455e68bbf8ab2afa79a5b5b8521f68ab19f0470f0265c7c.json new file mode 100644 index 0000000000..f425d1eebd --- /dev/null +++ b/backend/.sqlx/query-7853a596a01884070455e68bbf8ab2afa79a5b5b8521f68ab19f0470f0265c7c.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (parent_workspace_id IS NOT NULL) AS \"has_parent!\" FROM workspace WHERE id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "has_parent!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "7853a596a01884070455e68bbf8ab2afa79a5b5b8521f68ab19f0470f0265c7c" +} diff --git a/backend/.sqlx/query-78bb75578a880715fb482445883e0e762f289b7695f29bfd44fa23323c7e8523.json b/backend/.sqlx/query-78bb75578a880715fb482445883e0e762f289b7695f29bfd44fa23323c7e8523.json deleted file mode 100644 index f7d3361d99..0000000000 --- a/backend/.sqlx/query-78bb75578a880715fb482445883e0e762f289b7695f29bfd44fa23323c7e8523.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script' AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Int8" - ] - }, - "nullable": [] - }, - "hash": "78bb75578a880715fb482445883e0e762f289b7695f29bfd44fa23323c7e8523" -} diff --git a/backend/.sqlx/query-798dc8ce1c80b5ebd8120f55b6c9f909bf4babb29910514753cb822dde4e7ca9.json b/backend/.sqlx/query-798dc8ce1c80b5ebd8120f55b6c9f909bf4babb29910514753cb822dde4e7ca9.json new file mode 100644 index 0000000000..f7ff5d527a --- /dev/null +++ b/backend/.sqlx/query-798dc8ce1c80b5ebd8120f55b6c9f909bf4babb29910514753cb822dde4e7ca9.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (ws.datatable->'datatables'->$2->>'migrations_enabled')::boolean FROM workspace_settings ws WHERE ws.workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "bool", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "798dc8ce1c80b5ebd8120f55b6c9f909bf4babb29910514753cb822dde4e7ca9" +} diff --git a/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json new file mode 100644 index 0000000000..f909a33b4c --- /dev/null +++ b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"x\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "x", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5" +} diff --git a/backend/.sqlx/query-7b950d49cb1cc7f9bf8032c9e7655c49027337a9eca1f1ded13dbae6475e3376.json b/backend/.sqlx/query-7b950d49cb1cc7f9bf8032c9e7655c49027337a9eca1f1ded13dbae6475e3376.json new file mode 100644 index 0000000000..e1cf9bc17f --- /dev/null +++ b/backend/.sqlx/query-7b950d49cb1cc7f9bf8032c9e7655c49027337a9eca1f1ded13dbae6475e3376.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT parent_workspace_id FROM workspace WHERE id = $1 AND is_dev_workspace", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "parent_workspace_id", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "7b950d49cb1cc7f9bf8032c9e7655c49027337a9eca1f1ded13dbae6475e3376" +} diff --git a/backend/.sqlx/query-7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a.json b/backend/.sqlx/query-7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a.json new file mode 100644 index 0000000000..5c8ab6fb5b --- /dev/null +++ b/backend/.sqlx/query-7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (workspace_id, path)\n workspace_id AS \"workspace_id!\", path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND content ILIKE '%freshness%'\n -- Workspace archival stops all execution but leaves script rows\n -- intact for unarchival; without this the watchdog would keep\n -- resurrecting runs in a workspace the admin shut down.\n AND EXISTS (SELECT 1 FROM workspace w\n WHERE w.id = script.workspace_id AND w.deleted = false)\n ORDER BY workspace_id, path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "workspace_id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "content!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a" +} diff --git a/backend/.sqlx/query-7e3d7f7516167c320c688c2406076aeebb9081dd36b45f609823dfbb4d3b8e9d.json b/backend/.sqlx/query-7e3d7f7516167c320c688c2406076aeebb9081dd36b45f609823dfbb4d3b8e9d.json new file mode 100644 index 0000000000..f5fec50340 --- /dev/null +++ b/backend/.sqlx/query-7e3d7f7516167c320c688c2406076aeebb9081dd36b45f609823dfbb4d3b8e9d.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE macro_usage SET workspace_id = $1 WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "7e3d7f7516167c320c688c2406076aeebb9081dd36b45f609823dfbb4d3b8e9d" +} diff --git a/backend/.sqlx/query-7fa8b53615a7cb678f9d5caedbc44cfa928469ce3c1a53ec273ecc997f6e61f8.json b/backend/.sqlx/query-7fa8b53615a7cb678f9d5caedbc44cfa928469ce3c1a53ec273ecc997f6e61f8.json new file mode 100644 index 0000000000..e171d44af5 --- /dev/null +++ b/backend/.sqlx/query-7fa8b53615a7cb678f9d5caedbc44cfa928469ce3c1a53ec273ecc997f6e61f8.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND archived = false ORDER BY created_at DESC LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "hash", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "7fa8b53615a7cb678f9d5caedbc44cfa928469ce3c1a53ec273ecc997f6e61f8" +} diff --git a/backend/.sqlx/query-40bcbfdcae9842c7919eb6dcfe44d844508304700b292059b24c3f74454a7cca.json b/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json similarity index 53% rename from backend/.sqlx/query-40bcbfdcae9842c7919eb6dcfe44d844508304700b292059b24c3f74454a7cca.json rename to backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json index 31c0ab4982..82dcf60cc1 100644 --- a/backend/.sqlx/query-40bcbfdcae9842c7919eb6dcfe44d844508304700b292059b24c3f74454a7cca.json +++ b/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )", + "query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )", "describe": { "columns": [], "parameters": { @@ -10,5 +10,5 @@ }, "nullable": [] }, - "hash": "40bcbfdcae9842c7919eb6dcfe44d844508304700b292059b24c3f74454a7cca" + "hash": "80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684" } diff --git a/backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json b/backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json new file mode 100644 index 0000000000..bdba1e899a --- /dev/null +++ b/backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT setting::bigint as \"v!\" FROM pg_settings WHERE name = 'superuser_reserved_connections'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f" +} diff --git a/backend/.sqlx/query-80f526a401d5f26021b5eb424e7481e28c360367f144f172b746cd11abfa67a2.json b/backend/.sqlx/query-80f526a401d5f26021b5eb424e7481e28c360367f144f172b746cd11abfa67a2.json new file mode 100644 index 0000000000..384ea81940 --- /dev/null +++ b/backend/.sqlx/query-80f526a401d5f26021b5eb424e7481e28c360367f144f172b746cd11abfa67a2.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms, summary, created_by)\n VALUES ('test-workspace', 'restricted', 'restricted', ARRAY['u/test-user']::varchar[], '{\"u/test-user\": true}'::jsonb, '', 'test-user')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "80f526a401d5f26021b5eb424e7481e28c360367f144f172b746cd11abfa67a2" +} diff --git a/backend/.sqlx/query-8192986cd6106ed060b3d68dbc21e5bb34f5e68a5c6ac455a9d423188af77b23.json b/backend/.sqlx/query-8192986cd6106ed060b3d68dbc21e5bb34f5e68a5c6ac455a9d423188af77b23.json deleted file mode 100644 index 7a0c5578b0..0000000000 --- a/backend/.sqlx/query-8192986cd6106ed060b3d68dbc21e5bb34f5e68a5c6ac455a9d423188af77b23.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO background_task_state (name, value)\n SELECT $1, jsonb_build_object(\n 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint,\n 'last_ts', '1970-01-01T00:00:00+00:00')\n WHERE NOT EXISTS (SELECT 1 FROM global_settings WHERE name = $2)\n ON CONFLICT (name) DO NOTHING", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "8192986cd6106ed060b3d68dbc21e5bb34f5e68a5c6ac455a9d423188af77b23" -} diff --git a/backend/.sqlx/query-81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b.json b/backend/.sqlx/query-81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b.json new file mode 100644 index 0000000000..3f6a4a8136 --- /dev/null +++ b/backend/.sqlx/query-81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "\n DELETE FROM pipeline_freshness_state s\n WHERE NOT EXISTS (\n SELECT 1 FROM unnest($1::text[], $2::text[]) AS w(workspace_id, script_path)\n WHERE w.workspace_id = s.workspace_id AND w.script_path = s.script_path\n )\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "TextArray", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b" +} diff --git a/backend/.sqlx/query-82185eb02e03e3dd1a4b5a3f22c3b60169989703ee33d14ab348301885c9d745.json b/backend/.sqlx/query-82185eb02e03e3dd1a4b5a3f22c3b60169989703ee33d14ab348301885c9d745.json deleted file mode 100644 index c0573a8fab..0000000000 --- a/backend/.sqlx/query-82185eb02e03e3dd1a4b5a3f22c3b60169989703ee33d14ab348301885c9d745.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO workspace SELECT $1, $2, owner, false, premium FROM workspace WHERE id = $3", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Text" - ] - }, - "nullable": [] - }, - "hash": "82185eb02e03e3dd1a4b5a3f22c3b60169989703ee33d14ab348301885c9d745" -} diff --git a/backend/.sqlx/query-8288b3916022adcbb28da2a0e4329cba84759b8a590546aea8f0ddab4f861c38.json b/backend/.sqlx/query-8288b3916022adcbb28da2a0e4329cba84759b8a590546aea8f0ddab4f861c38.json new file mode 100644 index 0000000000..673becc2ff --- /dev/null +++ b/backend/.sqlx/query-8288b3916022adcbb28da2a0e4329cba84759b8a590546aea8f0ddab4f861c38.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT parent_workspace_id, deleted FROM workspace WHERE id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "parent_workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "deleted", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + false + ] + }, + "hash": "8288b3916022adcbb28da2a0e4329cba84759b8a590546aea8f0ddab4f861c38" +} diff --git a/backend/.sqlx/query-82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2.json b/backend/.sqlx/query-82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2.json deleted file mode 100644 index 96587ed1d0..0000000000 --- a/backend/.sqlx/query-82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT q.runnable_settings_handle\n FROM v2_job j JOIN v2_job_queue q ON q.id = j.id\n WHERE j.workspace_id = $1 AND j.runnable_path = $2\n AND j.trigger_kind = 'asset'", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "runnable_settings_handle", - "type_info": "Int8" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - true - ] - }, - "hash": "82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2" -} diff --git a/backend/.sqlx/query-ebbbd069e0f33be9609604025d159fe1ecbefc2e9c11f7c4900b7121d4367e01.json b/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json similarity index 56% rename from backend/.sqlx/query-ebbbd069e0f33be9609604025d159fe1ecbefc2e9c11f7c4900b7121d4367e01.json rename to backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json index 146c7f05b3..79b43245ef 100644 --- a/backend/.sqlx/query-ebbbd069e0f33be9609604025d159fe1ecbefc2e9c11f7c4900b7121d4367e01.json +++ b/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT concurrency_settings, debouncing_settings FROM runnable_settings WHERE hash = $1", + "query": "SELECT concurrency_settings, debouncing_settings, retry_settings FROM runnable_settings WHERE hash = $1", "describe": { "columns": [ { @@ -12,6 +12,11 @@ "ordinal": 1, "name": "debouncing_settings", "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "retry_settings", + "type_info": "Int8" } ], "parameters": { @@ -20,9 +25,10 @@ ] }, "nullable": [ + true, true, true ] }, - "hash": "ebbbd069e0f33be9609604025d159fe1ecbefc2e9c11f7c4900b7121d4367e01" + "hash": "82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496" } diff --git a/backend/.sqlx/query-84cbf9623a989dc16be1f8681c95eed9ae3e8d9c1552f396b6773767087bcfec.json b/backend/.sqlx/query-84cbf9623a989dc16be1f8681c95eed9ae3e8d9c1552f396b6773767087bcfec.json new file mode 100644 index 0000000000..0859d3e071 --- /dev/null +++ b/backend/.sqlx/query-84cbf9623a989dc16be1f8681c95eed9ae3e8d9c1552f396b6773767087bcfec.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET parent_workspace_id = $1, is_dev_workspace = true WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "84cbf9623a989dc16be1f8681c95eed9ae3e8d9c1552f396b6773767087bcfec" +} diff --git a/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json b/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json new file mode 100644 index 0000000000..03fc734037 --- /dev/null +++ b/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dispatch_event WHERE producer_job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab" +} diff --git a/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json b/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json new file mode 100644 index 0000000000..5aacf1a295 --- /dev/null +++ b/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json @@ -0,0 +1,47 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO materialized_partition\n (workspace_id, asset_kind, asset_path, partition, status,\n snapshot_id, row_count, job_id, materialized_at, error)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)\n ON CONFLICT (workspace_id, asset_kind, asset_path, partition)\n DO UPDATE SET status = EXCLUDED.status,\n -- A failed run records no snapshot, but must not erase the last\n -- committed one: a physical table from an earlier commit (or from a\n -- committed write whose data tests then failed) still exists, and\n -- fork defer/graph state keys on that evidence.\n snapshot_id = COALESCE(EXCLUDED.snapshot_id, materialized_partition.snapshot_id),\n row_count = EXCLUDED.row_count,\n job_id = EXCLUDED.job_id,\n materialized_at = now(),\n error = EXCLUDED.error", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Varchar", + "Text", + { + "Custom": { + "name": "materialization_status", + "kind": { + "Enum": [ + "running", + "materialized", + "failed" + ] + } + } + }, + "Int8", + "Int8", + "Uuid", + "Text" + ] + }, + "nullable": [] + }, + "hash": "853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424" +} diff --git a/backend/.sqlx/query-854c2e57362df2b6907082c660f01d9c16a9d4b5b422159fc239ff5502a7b71f.json b/backend/.sqlx/query-854c2e57362df2b6907082c660f01d9c16a9d4b5b422159fc239ff5502a7b71f.json new file mode 100644 index 0000000000..7250aff598 --- /dev/null +++ b/backend/.sqlx/query-854c2e57362df2b6907082c660f01d9c16a9d4b5b422159fc239ff5502a7b71f.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND created_at < now() - ($2::text)::interval", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "854c2e57362df2b6907082c660f01d9c16a9d4b5b422159fc239ff5502a7b71f" +} diff --git a/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json b/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json deleted file mode 100644 index 96e3439612..0000000000 --- a/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b" -} diff --git a/backend/.sqlx/query-868985685d95197efc534bb2f3e0c956bea94dffb46b3b838c096f04b66d6c52.json b/backend/.sqlx/query-868985685d95197efc534bb2f3e0c956bea94dffb46b3b838c096f04b66d6c52.json new file mode 100644 index 0000000000..34d99a7e61 --- /dev/null +++ b/backend/.sqlx/query-868985685d95197efc534bb2f3e0c956bea94dffb46b3b838c096f04b66d6c52.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, name, dev_workspace_label FROM workspace WHERE parent_workspace_id = $1 AND is_dev_workspace AND deleted = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "dev_workspace_label", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "868985685d95197efc534bb2f3e0c956bea94dffb46b3b838c096f04b66d6c52" +} diff --git a/backend/.sqlx/query-8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f.json b/backend/.sqlx/query-8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f.json new file mode 100644 index 0000000000..5741a1aa5b --- /dev/null +++ b/backend/.sqlx/query-8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n schedule.path, t.jobs FROM schedule,\n LATERAL(SELECT ARRAY(\n SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = schedule.path\n AND c.workspace_id = $1\n AND j.workspace_id = $1\n AND parent_job IS NULL AND runnable_path = schedule.script_path\n AND status <> 'skipped'\n ORDER BY completed_at DESC\n LIMIT 20\n ) AS jobs) t\n WHERE workspace_id = $1 AND schedule.path NOT LIKE $4\n ORDER BY edited_at DESC\n LIMIT $2 OFFSET $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "jobs", + "type_info": "JsonArray" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8", + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null + ] + }, + "hash": "8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f" +} diff --git a/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json new file mode 100644 index 0000000000..ffce4491e3 --- /dev/null +++ b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json @@ -0,0 +1,67 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\",\n language AS \"language!: windmill_common::scripts::ScriptLang\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "language!: windmill_common::scripts::ScriptLang", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d" +} diff --git a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json similarity index 68% rename from backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json rename to backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json index 56b887e8b9..f8f8511709 100644 --- a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json +++ b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json @@ -1,16 +1,17 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()", + "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()\n WHERE (background_task_state.value->>'last_xmin')::bigint <= $4", "describe": { "columns": [], "parameters": { "Left": [ "Text", "Jsonb", - "Text" + "Text", + "Int8" ] }, "nullable": [] }, - "hash": "8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d" + "hash": "881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829" } diff --git a/backend/.sqlx/query-8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e.json b/backend/.sqlx/query-8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e.json new file mode 100644 index 0000000000..829083e0ec --- /dev/null +++ b/backend/.sqlx/query-8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT ws.ducklake->'ducklakes' AS ducklake_name\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ducklake_name", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e" +} diff --git a/backend/.sqlx/query-8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3.json b/backend/.sqlx/query-8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3.json new file mode 100644 index 0000000000..9fc8cf1336 --- /dev/null +++ b/backend/.sqlx/query-8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM schedule WHERE workspace_id = $1 AND starts_with(path, $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3" +} diff --git a/backend/.sqlx/query-8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444.json b/backend/.sqlx/query-8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444.json new file mode 100644 index 0000000000..efd12114f3 --- /dev/null +++ b/backend/.sqlx/query-8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "\n -- Fork rows also count when a snapshot ever committed (a failed run\n -- preserves it): the physical table exists, so reads hit the FORK's\n -- data — showing 'deferred' would misstate what a query returns.\n -- Ancestor rows still require a clean materialization.\n SELECT DISTINCT asset_path AS \"asset_path!\", workspace_id AS \"workspace_id!\"\n FROM materialized_partition\n WHERE (workspace_id = $1 OR workspace_id = ANY($2))\n AND asset_kind = 'ducklake'\n AND (status = 'materialized'\n OR (workspace_id = $1 AND snapshot_id IS NOT NULL))\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "workspace_id!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444" +} diff --git a/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json new file mode 100644 index 0000000000..eac4547361 --- /dev/null +++ b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO materialized_asset_schema\n (workspace_id, asset_kind, asset_path, version, columns,\n snapshot_id, job_id, captured_at)\n VALUES ($1, $2, $3, $4, $5, $6, $7, now())", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Varchar", + "Int8", + "Jsonb", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0" +} diff --git a/backend/.sqlx/query-8ed229e88dc49b0ba7328d48f991493fcdbab4ae91cae271ea911f5b14ecf0d4.json b/backend/.sqlx/query-8ed229e88dc49b0ba7328d48f991493fcdbab4ae91cae271ea911f5b14ecf0d4.json new file mode 100644 index 0000000000..3f8deb8835 --- /dev/null +++ b/backend/.sqlx/query-8ed229e88dc49b0ba7328d48f991493fcdbab4ae91cae271ea911f5b14ecf0d4.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id, is_dev_workspace, dev_workspace_label)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END, $5,\n CASE WHEN $5 THEN dev_workspace_label ELSE NULL END\n FROM workspace WHERE id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Bool", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "8ed229e88dc49b0ba7328d48f991493fcdbab4ae91cae271ea911f5b14ecf0d4" +} diff --git a/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json b/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json index 13402c8387..90031c14d6 100644 --- a/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json +++ b/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json @@ -35,7 +35,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-8fef781ecef8bb7e4f5172adbab1d1eefa5cf05ff6521ee605af7ac2b90fa60b.json b/backend/.sqlx/query-8fef781ecef8bb7e4f5172adbab1d1eefa5cf05ff6521ee605af7ac2b90fa60b.json deleted file mode 100644 index f65ea6d2d4..0000000000 --- a/backend/.sqlx/query-8fef781ecef8bb7e4f5172adbab1d1eefa5cf05ff6521ee605af7ac2b90fa60b.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT created_at FROM script WHERE workspace_id = $1 AND hash = $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "created_at", - "type_info": "Timestamptz" - } - ], - "parameters": { - "Left": [ - "Text", - "Int8" - ] - }, - "nullable": [ - false - ] - }, - "hash": "8fef781ecef8bb7e4f5172adbab1d1eefa5cf05ff6521ee605af7ac2b90fa60b" -} diff --git a/backend/.sqlx/query-90d21ee2276b41a4dd3dd8ed12f36e00b8d4fc27a2d72b74dff2dace900db75c.json b/backend/.sqlx/query-90d21ee2276b41a4dd3dd8ed12f36e00b8d4fc27a2d72b74dff2dace900db75c.json new file mode 100644 index 0000000000..3f159707b6 --- /dev/null +++ b/backend/.sqlx/query-90d21ee2276b41a4dd3dd8ed12f36e00b8d4fc27a2d72b74dff2dace900db75c.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO ws_specific (workspace_id, item_kind, path)\n SELECT $1::varchar, 'variable', $2::varchar\n WHERE EXISTS (SELECT 1 FROM variable WHERE workspace_id = $1::varchar AND path = $2::varchar)\n ON CONFLICT DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "90d21ee2276b41a4dd3dd8ed12f36e00b8d4fc27a2d72b74dff2dace900db75c" +} diff --git a/backend/.sqlx/query-910b9b8afb3df5e437e43ff4adc97d0a03f6c26f0b14fc92900f7600e70a7a8b.json b/backend/.sqlx/query-910b9b8afb3df5e437e43ff4adc97d0a03f6c26f0b14fc92900f7600e70a7a8b.json deleted file mode 100644 index 569a5122ba..0000000000 --- a/backend/.sqlx/query-910b9b8afb3df5e437e43ff4adc97d0a03f6c26f0b14fc92900f7600e70a7a8b.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT status = 'success' AS \"success!\"\n FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "success!", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Text", - "Text", - "Text", - "Uuid", - "Int8" - ] - }, - "nullable": [ - null - ] - }, - "hash": "910b9b8afb3df5e437e43ff4adc97d0a03f6c26f0b14fc92900f7600e70a7a8b" -} diff --git a/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json new file mode 100644 index 0000000000..0f75ec3883 --- /dev/null +++ b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM workspace WHERE deleted = false AND id NOT LIKE 'wm-fork%'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf" +} diff --git a/backend/.sqlx/query-919bee5afcaba361f23760b5f64ce48f63e4d6d3e2d4fa9247b50963112f47ad.json b/backend/.sqlx/query-919bee5afcaba361f23760b5f64ce48f63e4d6d3e2d4fa9247b50963112f47ad.json new file mode 100644 index 0000000000..34c18e2633 --- /dev/null +++ b/backend/.sqlx/query-919bee5afcaba361f23760b5f64ce48f63e4d6d3e2d4fa9247b50963112f47ad.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT tag FROM flow WHERE workspace_id = $1 AND path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "tag", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "919bee5afcaba361f23760b5f64ce48f63e4d6d3e2d4fa9247b50963112f47ad" +} diff --git a/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json b/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json new file mode 100644 index 0000000000..794f62e49d --- /dev/null +++ b/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1 AND id = ANY($2))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f" +} diff --git a/backend/.sqlx/query-93d95d25c6b2398faa416646f5daaff40a634fc5530e202f65fad6a241f2671a.json b/backend/.sqlx/query-93d95d25c6b2398faa416646f5daaff40a634fc5530e202f65fad6a241f2671a.json new file mode 100644 index 0000000000..c0207b6bc7 --- /dev/null +++ b/backend/.sqlx/query-93d95d25c6b2398faa416646f5daaff40a634fc5530e202f65fad6a241f2671a.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_multipart_inflight\n (workspace_id, upload_id, part_id, storage, part_bytes, target_existing_size)\n VALUES ($1, $2, $3, $4, $5, $6)\n ON CONFLICT (workspace_id, upload_id, part_id)\n DO UPDATE SET part_bytes = $5", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "93d95d25c6b2398faa416646f5daaff40a634fc5530e202f65fad6a241f2671a" +} diff --git a/backend/.sqlx/query-954ead4f28726b31e2cafbe04d3ed3e9625f54cd89cfd595665e4473c5f3ea6c.json b/backend/.sqlx/query-954ead4f28726b31e2cafbe04d3ed3e9625f54cd89cfd595665e4473c5f3ea6c.json new file mode 100644 index 0000000000..253fe79753 --- /dev/null +++ b/backend/.sqlx/query-954ead4f28726b31e2cafbe04d3ed3e9625f54cd89cfd595665e4473c5f3ea6c.json @@ -0,0 +1,37 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n -- reservation of every OTHER in-flight upload\n (SELECT COALESCE(SUM(GREATEST(t.total - t.existing, 0)), 0)\n FROM (SELECT SUM(part_bytes) as total, MAX(target_existing_size) as existing\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND upload_id <> $2\n AND created_at > now() - ($4::text)::interval\n GROUP BY upload_id) t)::bigint as \"other_reserved!\",\n -- this upload's already-recorded parts, excluding the part being (re)uploaded\n (SELECT COALESCE(SUM(part_bytes), 0)\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND upload_id = $2 AND part_id <> $3)::bigint as \"this_other_parts!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "committed!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "other_reserved!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "this_other_parts!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "954ead4f28726b31e2cafbe04d3ed3e9625f54cd89cfd595665e4473c5f3ea6c" +} diff --git a/backend/.sqlx/query-95e370a82ef46d9310f77a99b437a20a5773113e290d69a016363543067baf14.json b/backend/.sqlx/query-95e370a82ef46d9310f77a99b437a20a5773113e290d69a016363543067baf14.json new file mode 100644 index 0000000000..eb94fe6ab5 --- /dev/null +++ b/backend/.sqlx/query-95e370a82ef46d9310f77a99b437a20a5773113e290d69a016363543067baf14.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT MAX(timestamp) FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "max", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "95e370a82ef46d9310f77a99b437a20a5773113e290d69a016363543067baf14" +} diff --git a/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json b/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json new file mode 100644 index 0000000000..e2132d7737 --- /dev/null +++ b/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) as \"c!\" FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d" +} diff --git a/backend/.sqlx/query-9cfebb5bfee0d7d78f4e71c106a6aa1cb9aae18a6c3a1e5d6b36697ed063b50c.json b/backend/.sqlx/query-9cfebb5bfee0d7d78f4e71c106a6aa1cb9aae18a6c3a1e5d6b36697ed063b50c.json new file mode 100644 index 0000000000..07a14677e4 --- /dev/null +++ b/backend/.sqlx/query-9cfebb5bfee0d7d78f4e71c106a6aa1cb9aae18a6c3a1e5d6b36697ed063b50c.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM macro_definition WHERE workspace_id = $1 AND provider_path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "9cfebb5bfee0d7d78f4e71c106a6aa1cb9aae18a6c3a1e5d6b36697ed063b50c" +} diff --git a/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json b/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json new file mode 100644 index 0000000000..58630c5356 --- /dev/null +++ b/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE consumed_at IS NOT NULL", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96" +} diff --git a/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json b/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json index ad9e57801e..ab01730c02 100644 --- a/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json +++ b/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json @@ -35,7 +35,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-9ee6263223bd415d7ac21328a8ebc731d43e7331b9fd83282a87f8e2fd8fa770.json b/backend/.sqlx/query-9ee6263223bd415d7ac21328a8ebc731d43e7331b9fd83282a87f8e2fd8fa770.json new file mode 100644 index 0000000000..64dfe64198 --- /dev/null +++ b/backend/.sqlx/query-9ee6263223bd415d7ac21328a8ebc731d43e7331b9fd83282a87f8e2fd8fa770.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM password WHERE username IS NOT NULL)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9ee6263223bd415d7ac21328a8ebc731d43e7331b9fd83282a87f8e2fd8fa770" +} diff --git a/backend/.sqlx/query-9eea74f68a0c99d5ce6601fd854c1d18df0ab30f528ef4ce7910c43dd44f4cfb.json b/backend/.sqlx/query-9eea74f68a0c99d5ce6601fd854c1d18df0ab30f528ef4ce7910c43dd44f4cfb.json new file mode 100644 index 0000000000..5ac3174af6 --- /dev/null +++ b/backend/.sqlx/query-9eea74f68a0c99d5ce6601fd854c1d18df0ab30f528ef4ce7910c43dd44f4cfb.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings SET datatable = jsonb_set(datatable, ARRAY['datatables', $2::text, 'migrations_enabled'], 'false'::jsonb) WHERE workspace_id = $1 AND jsonb_exists(datatable->'datatables', $2) RETURNING 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "9eea74f68a0c99d5ce6601fd854c1d18df0ab30f528ef4ce7910c43dd44f4cfb" +} diff --git a/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json b/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json new file mode 100644 index 0000000000..6a39c40970 --- /dev/null +++ b/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77" +} diff --git a/backend/.sqlx/query-9f567f04f67ce3b197eaa641eaf2d0bbe1f5aff27187505778b6abe2c87a5d01.json b/backend/.sqlx/query-9f567f04f67ce3b197eaa641eaf2d0bbe1f5aff27187505778b6abe2c87a5d01.json new file mode 100644 index 0000000000..a29b5f0924 --- /dev/null +++ b/backend/.sqlx/query-9f567f04f67ce3b197eaa641eaf2d0bbe1f5aff27187505778b6abe2c87a5d01.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET parent_workspace_id = $1, is_dev_workspace = true, dev_workspace_label = $3 WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "9f567f04f67ce3b197eaa641eaf2d0bbe1f5aff27187505778b6abe2c87a5d01" +} diff --git a/backend/.sqlx/query-a08e1e22ec6cc25bb8939e488a1c7cf162f2fa7c5f67f8a9e0071e5b7f238dcb.json b/backend/.sqlx/query-a08e1e22ec6cc25bb8939e488a1c7cf162f2fa7c5f67f8a9e0071e5b7f238dcb.json new file mode 100644 index 0000000000..9ddc929622 --- /dev/null +++ b/backend/.sqlx/query-a08e1e22ec6cc25bb8939e488a1c7cf162f2fa7c5f67f8a9e0071e5b7f238dcb.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, provider_path FROM macro_definition WHERE workspace_id = $1 AND name = ANY($2) LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "provider_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "a08e1e22ec6cc25bb8939e488a1c7cf162f2fa7c5f67f8a9e0071e5b7f238dcb" +} diff --git a/backend/.sqlx/query-a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d.json b/backend/.sqlx/query-a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d.json new file mode 100644 index 0000000000..e3e8e3f2ad --- /dev/null +++ b/backend/.sqlx/query-a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND path LIKE $2 AND path != ALL($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d" +} diff --git a/backend/.sqlx/query-a22611ed3b7a6caa76a218454494c96e1371956bb303eb8d3e3268139cd8fce0.json b/backend/.sqlx/query-a22611ed3b7a6caa76a218454494c96e1371956bb303eb8d3e3268139cd8fce0.json new file mode 100644 index 0000000000..2b9b0619a1 --- /dev/null +++ b/backend/.sqlx/query-a22611ed3b7a6caa76a218454494c96e1371956bb303eb8d3e3268139cd8fce0.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id FROM workspace WHERE parent_workspace_id = $1 AND is_dev_workspace", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "a22611ed3b7a6caa76a218454494c96e1371956bb303eb8d3e3268139cd8fce0" +} diff --git a/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json new file mode 100644 index 0000000000..24e387a783 --- /dev/null +++ b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($3::timestamptz IS NULL OR completed_at >= $3)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614" +} diff --git a/backend/.sqlx/query-a488e5a6492562bbc5fa86ab348b610b028e88d5d2dbbea44377bf9dc416cd03.json b/backend/.sqlx/query-a488e5a6492562bbc5fa86ab348b610b028e88d5d2dbbea44377bf9dc416cd03.json new file mode 100644 index 0000000000..0c3fdb8d0a --- /dev/null +++ b/backend/.sqlx/query-a488e5a6492562bbc5fa86ab348b610b028e88d5d2dbbea44377bf9dc416cd03.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_try_advisory_xact_lock(hashtext('workspace_storage_usage'), hashtext($1))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_try_advisory_xact_lock", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "a488e5a6492562bbc5fa86ab348b610b028e88d5d2dbbea44377bf9dc416cd03" +} diff --git a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json index 9a21f228ea..405904604a 100644 --- a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json +++ b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json @@ -191,7 +191,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json b/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json new file mode 100644 index 0000000000..a8dc380f40 --- /dev/null +++ b/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH mine AS (\n SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1\n ), claimed AS (\n -- Claim the whole batch in ONE update so concurrent same-batch\n -- survivors lock rows in identical scan order (no lock-ordering\n -- deadlock); each re-evaluates `consumed_at IS NULL` under EvalPlanQual\n -- and skips rows the other already took. A claim therefore consumes\n -- every still-unclaimed row of the batch atomically.\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE debounce_batch = (SELECT debounce_batch FROM mine)\n AND consumed_at IS NULL\n RETURNING id\n )\n SELECT\n EXISTS (SELECT 1 FROM mine) AS \"had_row!\",\n (SELECT consumed_by FROM mine) AS prev_consumed_by,\n ARRAY(SELECT id FROM claimed) AS \"claimed_ids!\",\n EXISTS (SELECT 1 FROM claimed WHERE id = $1) AS \"claimed_self!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "had_row!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "prev_consumed_by", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "claimed_ids!", + "type_info": "UuidArray" + }, + { + "ordinal": 3, + "name": "claimed_self!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be" +} diff --git a/backend/.sqlx/query-a6d54bb1e2ad153e0c89a4c71ade07eab23e2b35e58f812c69ba29de2d8e2fa5.json b/backend/.sqlx/query-a6d54bb1e2ad153e0c89a4c71ade07eab23e2b35e58f812c69ba29de2d8e2fa5.json new file mode 100644 index 0000000000..986638f649 --- /dev/null +++ b/backend/.sqlx/query-a6d54bb1e2ad153e0c89a4c71ade07eab23e2b35e58f812c69ba29de2d8e2fa5.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM macro_usage WHERE workspace_id = $1 AND (consumer_path = $2 OR consumer_path = $3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "a6d54bb1e2ad153e0c89a4c71ade07eab23e2b35e58f812c69ba29de2d8e2fa5" +} diff --git a/backend/.sqlx/query-a7017ba623279e614ef31a3026c5274749d7a436a753813da098758514a7493d.json b/backend/.sqlx/query-a7017ba623279e614ef31a3026c5274749d7a436a753813da098758514a7493d.json new file mode 100644 index 0000000000..691016a610 --- /dev/null +++ b/backend/.sqlx/query-a7017ba623279e614ef31a3026c5274749d7a436a753813da098758514a7493d.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT jsonb_object_keys(large_file_storage->'secondary_storage') as \"key!\"\n FROM workspace_settings WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "key!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "a7017ba623279e614ef31a3026c5274749d7a436a753813da098758514a7493d" +} diff --git a/backend/.sqlx/query-a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9.json b/backend/.sqlx/query-a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9.json new file mode 100644 index 0000000000..21e0f4613b --- /dev/null +++ b/backend/.sqlx/query-a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND starts_with(path, $2) AND path != ALL($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9" +} diff --git a/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json b/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json new file mode 100644 index 0000000000..c3cee83397 --- /dev/null +++ b/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b" +} diff --git a/backend/.sqlx/query-a8019c47b15baa9497ae44935456fea272ac289fc17ea9fc5aa237f77778b58f.json b/backend/.sqlx/query-a8019c47b15baa9497ae44935456fea272ac289fc17ea9fc5aa237f77778b58f.json new file mode 100644 index 0000000000..0f1bcc2271 --- /dev/null +++ b/backend/.sqlx/query-a8019c47b15baa9497ae44935456fea272ac289fc17ea9fc5aa237f77778b58f.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT inflight_bytes, target_existing_size FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND upload_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "inflight_bytes", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "target_existing_size", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "a8019c47b15baa9497ae44935456fea272ac289fc17ea9fc5aa237f77778b58f" +} diff --git a/backend/.sqlx/query-a8cec061756c7be61af829f4a5207ba6333ad5feb4557a9341427e9e68608025.json b/backend/.sqlx/query-a8cec061756c7be61af829f4a5207ba6333ad5feb4557a9341427e9e68608025.json new file mode 100644 index 0000000000..c262f3d1f6 --- /dev/null +++ b/backend/.sqlx/query-a8cec061756c7be61af829f4a5207ba6333ad5feb4557a9341427e9e68608025.json @@ -0,0 +1,46 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT datatable, timestamp, name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 ORDER BY datatable, timestamp ASC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "datatable", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "timestamp", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "code_up", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "code_down", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + true + ] + }, + "hash": "a8cec061756c7be61af829f4a5207ba6333ad5feb4557a9341427e9e68608025" +} diff --git a/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json new file mode 100644 index 0000000000..a604c85831 --- /dev/null +++ b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'aurora_version') AS \"aurora!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'rds_superuser') AS \"rds!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'cloudsqlsuperuser') AS \"cloudsql!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname IN ('azure_pg_admin', 'azuresu')) AS \"azure!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "aurora!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "rds!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "cloudsql!", + "type_info": "Bool" + }, + { + "ordinal": 3, + "name": "azure!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db" +} diff --git a/backend/.sqlx/query-a981f9b6424969e5fd72fb18c20e2910a138a848f3f6674dec58053572682077.json b/backend/.sqlx/query-a981f9b6424969e5fd72fb18c20e2910a138a848f3f6674dec58053572682077.json new file mode 100644 index 0000000000..5d20f83f9d --- /dev/null +++ b/backend/.sqlx/query-a981f9b6424969e5fd72fb18c20e2910a138a848f3f6674dec58053572682077.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id FROM workspace WHERE parent_workspace_id = $1 AND is_dev_workspace AND deleted = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "a981f9b6424969e5fd72fb18c20e2910a138a848f3f6674dec58053572682077" +} diff --git a/backend/.sqlx/query-a999a5cf7b481d852222311a34959f38114926063718668907587d1b80dfc75e.json b/backend/.sqlx/query-a999a5cf7b481d852222311a34959f38114926063718668907587d1b80dfc75e.json new file mode 100644 index 0000000000..b022524bbb --- /dev/null +++ b/backend/.sqlx/query-a999a5cf7b481d852222311a34959f38114926063718668907587d1b80dfc75e.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO resource\n (workspace_id, path, value, description, resource_type, created_by, edited_at, labels)\n VALUES ($1, $2, $3, $4, $5, $6, now(), $7) ON CONFLICT (workspace_id, path) DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Jsonb", + "Text", + "Varchar", + "Varchar", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "a999a5cf7b481d852222311a34959f38114926063718668907587d1b80dfc75e" +} diff --git a/backend/.sqlx/query-a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b.json b/backend/.sqlx/query-a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b.json deleted file mode 100644 index d12d305b92..0000000000 --- a/backend/.sqlx/query-a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n SELECT path AS \"path!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "path!", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - false - ] - }, - "hash": "a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b" -} diff --git a/backend/.sqlx/query-ab0dca3f021243d71222643165548af40192832c07c1c7049def3a80057bb683.json b/backend/.sqlx/query-ab0dca3f021243d71222643165548af40192832c07c1c7049def3a80057bb683.json new file mode 100644 index 0000000000..1148679230 --- /dev/null +++ b/backend/.sqlx/query-ab0dca3f021243d71222643165548af40192832c07c1c7049def3a80057bb683.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COALESCE(SUM(bytes), 0)::bigint as \"total!\",\n COALESCE(MIN(computed_at) < now() - interval '10 minutes', true) as \"stale!\"\n FROM workspace_storage_usage WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "total!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "stale!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "ab0dca3f021243d71222643165548af40192832c07c1c7049def3a80057bb683" +} diff --git a/backend/.sqlx/query-ad07e40790b947d1e8da17dd4ae30365210bba4bff0b7e32d06c53ed13572360.json b/backend/.sqlx/query-ad07e40790b947d1e8da17dd4ae30365210bba4bff0b7e32d06c53ed13572360.json new file mode 100644 index 0000000000..e543db2a30 --- /dev/null +++ b/backend/.sqlx/query-ad07e40790b947d1e8da17dd4ae30365210bba4bff0b7e32d06c53ed13572360.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT 1 FROM flow WHERE workspace_id = $1 AND path = $2 LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ad07e40790b947d1e8da17dd4ae30365210bba4bff0b7e32d06c53ed13572360" +} diff --git a/backend/.sqlx/query-ae2293c7d9d2b5dad61b27db3b29d5f87fc6b8d53fe6a073eef488533cbd3ed3.json b/backend/.sqlx/query-ae2293c7d9d2b5dad61b27db3b29d5f87fc6b8d53fe6a073eef488533cbd3ed3.json new file mode 100644 index 0000000000..1b4082b18a --- /dev/null +++ b/backend/.sqlx/query-ae2293c7d9d2b5dad61b27db3b29d5f87fc6b8d53fe6a073eef488533cbd3ed3.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COALESCE(\n (SELECT email FROM usr WHERE workspace_id = $1 AND username = $2),\n (SELECT email FROM password WHERE (username = $2 OR email = $2) AND super_admin = true)\n )", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "coalesce", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ae2293c7d9d2b5dad61b27db3b29d5f87fc6b8d53fe6a073eef488533cbd3ed3" +} diff --git a/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json b/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json new file mode 100644 index 0000000000..3413ddc87f --- /dev/null +++ b/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM v2_job_debounce_batch WHERE id = $1) as \"e!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "e!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c" +} diff --git a/backend/.sqlx/query-aef927110ef4cff51d3faabb0c389730dd215f4e90d105f0e86025f0ac42f020.json b/backend/.sqlx/query-aef927110ef4cff51d3faabb0c389730dd215f4e90d105f0e86025f0ac42f020.json new file mode 100644 index 0000000000..2a90d12ad0 --- /dev/null +++ b/backend/.sqlx/query-aef927110ef4cff51d3faabb0c389730dd215f4e90d105f0e86025f0ac42f020.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down)\n SELECT $2, datatable, timestamp, name, code_up, code_down\n FROM datatable_migrations WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "aef927110ef4cff51d3faabb0c389730dd215f4e90d105f0e86025f0ac42f020" +} diff --git a/backend/.sqlx/query-af19b9e3deb4f5c9e6ba77963a5da5c60aa6878b2c77f6028a68d64f797c3322.json b/backend/.sqlx/query-af19b9e3deb4f5c9e6ba77963a5da5c60aa6878b2c77f6028a68d64f797c3322.json new file mode 100644 index 0000000000..c21a21163e --- /dev/null +++ b/backend/.sqlx/query-af19b9e3deb4f5c9e6ba77963a5da5c60aa6878b2c77f6028a68d64f797c3322.json @@ -0,0 +1,70 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT workspace.id, workspace.name, usr.username, workspace_settings.color, workspace.parent_workspace_id,\n workspace.is_dev_workspace, workspace.dev_workspace_label,\n CASE WHEN usr.operator THEN workspace_settings.operator_settings ELSE NULL END as operator_settings,\n usr.disabled\n FROM workspace\n JOIN usr ON usr.workspace_id = workspace.id\n JOIN workspace_settings ON workspace_settings.workspace_id = workspace.id\n WHERE usr.email = $1 AND workspace.deleted = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "username", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "color", + "type_info": "Varchar" + }, + { + "ordinal": 4, + "name": "parent_workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "is_dev_workspace", + "type_info": "Bool" + }, + { + "ordinal": 6, + "name": "dev_workspace_label", + "type_info": "Varchar" + }, + { + "ordinal": 7, + "name": "operator_settings", + "type_info": "Jsonb" + }, + { + "ordinal": 8, + "name": "disabled", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + true, + true, + false, + true, + null, + false + ] + }, + "hash": "af19b9e3deb4f5c9e6ba77963a5da5c60aa6878b2c77f6028a68d64f797c3322" +} diff --git a/backend/.sqlx/query-451d9cde90d14071e21ffb5f615052b7ba7fc315fc301ed5c0ff50d9a3ab0d4a.json b/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json similarity index 50% rename from backend/.sqlx/query-451d9cde90d14071e21ffb5f615052b7ba7fc315fc301ed5c0ff50d9a3ab0d4a.json rename to backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json index 9e7ffed082..406cbdb176 100644 --- a/backend/.sqlx/query-451d9cde90d14071e21ffb5f615052b7ba7fc315fc301ed5c0ff50d9a3ab0d4a.json +++ b/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json @@ -1,10 +1,11 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings)\n VALUES ($1, $2, $3)\n ON CONFLICT (hash)\n DO NOTHING", + "query": "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings, retry_settings)\n VALUES ($1, $2, $3, $4)\n ON CONFLICT (hash)\n DO NOTHING", "describe": { "columns": [], "parameters": { "Left": [ + "Int8", "Int8", "Int8", "Int8" @@ -12,5 +13,5 @@ }, "nullable": [] }, - "hash": "451d9cde90d14071e21ffb5f615052b7ba7fc315fc301ed5c0ff50d9a3ab0d4a" + "hash": "afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42" } diff --git a/backend/.sqlx/query-b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74.json b/backend/.sqlx/query-b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74.json new file mode 100644 index 0000000000..d5b10b13b9 --- /dev/null +++ b/backend/.sqlx/query-b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT attempts FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "attempts", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74" +} diff --git a/backend/.sqlx/query-b3200181380df2f645bfce270dc1cf520939fa4fa4b65371d11db7416ba0b14c.json b/backend/.sqlx/query-b3200181380df2f645bfce270dc1cf520939fa4fa4b65371d11db7416ba0b14c.json new file mode 100644 index 0000000000..f3cc23c5b8 --- /dev/null +++ b/backend/.sqlx/query-b3200181380df2f645bfce270dc1cf520939fa4fa4b65371d11db7416ba0b14c.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext($1), hashtext($2))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b3200181380df2f645bfce270dc1cf520939fa4fa4b65371d11db7416ba0b14c" +} diff --git a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json b/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json index 3efa843923..092d15e592 100644 --- a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json +++ b/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json @@ -166,7 +166,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-b3a9ef177aecfbcb31abba9f25e1292668a687ca01ae1c14c0ed3b5cf10fcb03.json b/backend/.sqlx/query-b3a9ef177aecfbcb31abba9f25e1292668a687ca01ae1c14c0ed3b5cf10fcb03.json new file mode 100644 index 0000000000..545a9c161b --- /dev/null +++ b/backend/.sqlx/query-b3a9ef177aecfbcb31abba9f25e1292668a687ca01ae1c14c0ed3b5cf10fcb03.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM macro_definition WHERE workspace_id = $1 AND (provider_path = $2 OR provider_path = $3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "b3a9ef177aecfbcb31abba9f25e1292668a687ca01ae1c14c0ed3b5cf10fcb03" +} diff --git a/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json new file mode 100644 index 0000000000..dc82db7591 --- /dev/null +++ b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"cutoff?\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "cutoff?", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e" +} diff --git a/backend/.sqlx/query-b55b0f11b8d73ff9fd19da3bc5555e8365dbd4766f273ef3f68e734972a96cd6.json b/backend/.sqlx/query-b55b0f11b8d73ff9fd19da3bc5555e8365dbd4766f273ef3f68e734972a96cd6.json new file mode 100644 index 0000000000..c6022d3013 --- /dev/null +++ b/backend/.sqlx/query-b55b0f11b8d73ff9fd19da3bc5555e8365dbd4766f273ef3f68e734972a96cd6.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff\n (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork)\n VALUES\n ('test-workspace', 'wm-fork-test-workspace', 'f/rt/ghost', 'http_trigger', 1, 0, true, false, true),\n ('test-workspace', 'wm-fork-test-workspace', 'f/rt/ghost_behind', 'http_trigger', 0, 1, true, true, false)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "b55b0f11b8d73ff9fd19da3bc5555e8365dbd4766f273ef3f68e734972a96cd6" +} diff --git a/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json b/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json index 52e1ced19a..da7e3d5787 100644 --- a/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json +++ b/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json @@ -46,7 +46,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json b/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json new file mode 100644 index 0000000000..5b0f0b319c --- /dev/null +++ b/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtextextended($1, 0::int8))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238" +} diff --git a/backend/.sqlx/query-b96c1a720654ae8ae52e8098cc18d887920beaec97696a17da020a2adfb052f0.json b/backend/.sqlx/query-b96c1a720654ae8ae52e8098cc18d887920beaec97696a17da020a2adfb052f0.json new file mode 100644 index 0000000000..b3861c0111 --- /dev/null +++ b/backend/.sqlx/query-b96c1a720654ae8ae52e8098cc18d887920beaec97696a17da020a2adfb052f0.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down) VALUES ($1, $2, $3, 'initial', $4, NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Text" + ] + }, + "nullable": [] + }, + "hash": "b96c1a720654ae8ae52e8098cc18d887920beaec97696a17da020a2adfb052f0" +} diff --git a/backend/.sqlx/query-b98844926ff127c528ed3e7bc63bf1ebed0192be267983e8fbd18f79997e6142.json b/backend/.sqlx/query-b98844926ff127c528ed3e7bc63bf1ebed0192be267983e8fbd18f79997e6142.json new file mode 100644 index 0000000000..94cf77ebc5 --- /dev/null +++ b/backend/.sqlx/query-b98844926ff127c528ed3e7bc63bf1ebed0192be267983e8fbd18f79997e6142.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO usr (workspace_id, username, email, is_admin, created_at, operator, disabled, role, is_service_account, added_via)\n SELECT $1, username, email, is_admin, created_at, operator, disabled, role, is_service_account, added_via\n FROM usr WHERE workspace_id = $2\n ON CONFLICT DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "b98844926ff127c528ed3e7bc63bf1ebed0192be267983e8fbd18f79997e6142" +} diff --git a/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json b/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json new file mode 100644 index 0000000000..fc002b7074 --- /dev/null +++ b/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json @@ -0,0 +1,25 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH target AS (\n SELECT id FROM v2_job_queue WHERE id = $1 AND NOT running FOR UPDATE\n ), completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n $3::text::jsonb,\n s.flow_status,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n WHERE q.id IN (SELECT id FROM target)\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id IN (SELECT id FROM target)\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n SELECT $4, $1, $2 WHERE EXISTS (SELECT 1 FROM target)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "x", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135" +} diff --git a/backend/.sqlx/query-bb8318ddc8e2235dce5c832ff2b0ebf972bcc7f821fbf1e0171266efd8d46616.json b/backend/.sqlx/query-bb8318ddc8e2235dce5c832ff2b0ebf972bcc7f821fbf1e0171266efd8d46616.json new file mode 100644 index 0000000000..96608e1bc2 --- /dev/null +++ b/backend/.sqlx/query-bb8318ddc8e2235dce5c832ff2b0ebf972bcc7f821fbf1e0171266efd8d46616.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_storage_usage (workspace_id, storage, bytes, computed_at)\n VALUES ($1, $2, GREATEST($3::bigint, 0), to_timestamp(0))\n ON CONFLICT (workspace_id, storage)\n DO UPDATE SET bytes = GREATEST(workspace_storage_usage.bytes + $3::bigint, 0)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "bb8318ddc8e2235dce5c832ff2b0ebf972bcc7f821fbf1e0171266efd8d46616" +} diff --git a/backend/.sqlx/query-bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43.json b/backend/.sqlx/query-bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43.json new file mode 100644 index 0000000000..ce77275faf --- /dev/null +++ b/backend/.sqlx/query-bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO pipeline_freshness_state (workspace_id, script_path, attempts) VALUES ($1, $2, 3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43" +} diff --git a/backend/.sqlx/query-bcb36b5d38a07dbdfed71983aec1af05c5a6882f988ba5e0ec1f63828353987b.json b/backend/.sqlx/query-bcb36b5d38a07dbdfed71983aec1af05c5a6882f988ba5e0ec1f63828353987b.json new file mode 100644 index 0000000000..514024eb8d --- /dev/null +++ b/backend/.sqlx/query-bcb36b5d38a07dbdfed71983aec1af05c5a6882f988ba5e0ec1f63828353987b.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH RECURSIVE chain AS (\n SELECT id, parent_workspace_id, 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, chain.depth + 1\n FROM workspace w\n JOIN chain ON w.id = chain.parent_workspace_id\n WHERE chain.depth < 20\n )\n SELECT id AS \"id!\" FROM chain WHERE parent_workspace_id IS NULL LIMIT 1\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "bcb36b5d38a07dbdfed71983aec1af05c5a6882f988ba5e0ec1f63828353987b" +} diff --git a/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json b/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json index f6ff25a4bf..419ab26383 100644 --- a/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json +++ b/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json @@ -80,7 +80,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json index 8dc66064dc..b9d33a6b5f 100644 --- a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json +++ b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json @@ -111,7 +111,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json b/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json new file mode 100644 index 0000000000..22618ad999 --- /dev/null +++ b/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354" +} diff --git a/backend/.sqlx/query-bf6e1abbab6bdf0e67eefa5f807a26b06d24a7b3e292ae4be97fbbb197d468db.json b/backend/.sqlx/query-bf6e1abbab6bdf0e67eefa5f807a26b06d24a7b3e292ae4be97fbbb197d468db.json new file mode 100644 index 0000000000..35d3ace40e --- /dev/null +++ b/backend/.sqlx/query-bf6e1abbab6bdf0e67eefa5f807a26b06d24a7b3e292ae4be97fbbb197d468db.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_diff SET has_changes = NULL\n WHERE path = $2 AND kind = $3\n AND ($1 IN (source_workspace_id, fork_workspace_id))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "bf6e1abbab6bdf0e67eefa5f807a26b06d24a7b3e292ae4be97fbbb197d468db" +} diff --git a/backend/.sqlx/query-bfa4fb5998dec9baf89a2d950186e2dc9c0aecf7831ce66082a40f17cc5ac0de.json b/backend/.sqlx/query-bfa4fb5998dec9baf89a2d950186e2dc9c0aecf7831ce66082a40f17cc5ac0de.json new file mode 100644 index 0000000000..a1f7783645 --- /dev/null +++ b/backend/.sqlx/query-bfa4fb5998dec9baf89a2d950186e2dc9c0aecf7831ce66082a40f17cc5ac0de.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(\n SELECT 1 FROM workspace\n WHERE id = $1 AND parent_workspace_id = $2 AND is_dev_workspace\n )", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "bfa4fb5998dec9baf89a2d950186e2dc9c0aecf7831ce66082a40f17cc5ac0de" +} diff --git a/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json b/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json new file mode 100644 index 0000000000..f9fbc7a58b --- /dev/null +++ b/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM flow_conversation_message WHERE job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636" +} diff --git a/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json new file mode 100644 index 0000000000..b2e218e511 --- /dev/null +++ b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "UuidArray", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75" +} diff --git a/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json b/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json new file mode 100644 index 0000000000..2d1e21019e --- /dev/null +++ b/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_path FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba" +} diff --git a/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json b/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json new file mode 100644 index 0000000000..7981dbc983 --- /dev/null +++ b/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json @@ -0,0 +1,111 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT asset_kind AS \"asset_kind: AssetKind\", asset_path, partition,\n status AS \"status: MaterializationStatus\", snapshot_id,\n row_count, job_id, materialized_at, error\n FROM materialized_partition\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY partition DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_kind: AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "asset_path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "partition", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "status: MaterializationStatus", + "type_info": { + "Custom": { + "name": "materialization_status", + "kind": { + "Enum": [ + "running", + "materialized", + "failed" + ] + } + } + } + }, + { + "ordinal": 4, + "name": "snapshot_id", + "type_info": "Int8" + }, + { + "ordinal": 5, + "name": "row_count", + "type_info": "Int8" + }, + { + "ordinal": 6, + "name": "job_id", + "type_info": "Uuid" + }, + { + "ordinal": 7, + "name": "materialized_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 8, + "name": "error", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + true, + true, + true, + false, + true + ] + }, + "hash": "c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3" +} diff --git a/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json b/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json new file mode 100644 index 0000000000..876465fb8d --- /dev/null +++ b/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT j.runnable_path AS \"runnable_path!\", j.kind::text AS \"kind!\",\n q.runnable_settings_handle\n FROM v2_job j JOIN v2_job_queue q ON q.id = j.id\n WHERE j.workspace_id = $1 AND j.trigger_kind = 'asset'\n ORDER BY j.runnable_path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "kind!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "runnable_settings_handle", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + null, + true + ] + }, + "hash": "c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59" +} diff --git a/backend/.sqlx/query-c5639d48c92d6863f16f97de91eae590d0ed2f4b7831956bb429a47c478f7c1f.json b/backend/.sqlx/query-c5639d48c92d6863f16f97de91eae590d0ed2f4b7831956bb429a47c478f7c1f.json new file mode 100644 index 0000000000..fa1396235f --- /dev/null +++ b/backend/.sqlx/query-c5639d48c92d6863f16f97de91eae590d0ed2f4b7831956bb429a47c478f7c1f.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down) VALUES ($1, $2, $3, $4, $5, $6) ON CONFLICT (workspace_id, datatable, timestamp) DO UPDATE SET name = EXCLUDED.name, code_up = EXCLUDED.code_up, code_down = EXCLUDED.code_down", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "c5639d48c92d6863f16f97de91eae590d0ed2f4b7831956bb429a47c478f7c1f" +} diff --git a/backend/.sqlx/query-c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896.json b/backend/.sqlx/query-c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896.json new file mode 100644 index 0000000000..7cfc1f109f --- /dev/null +++ b/backend/.sqlx/query-c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (asset_path)\n asset_path, version, columns AS \"columns: Json>\", captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = 'ducklake' AND asset_path = ANY($2)\n ORDER BY asset_path, version DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "columns: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "captured_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false, + false, + false + ] + }, + "hash": "c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896" +} diff --git a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json b/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json deleted file mode 100644 index a68387f905..0000000000 --- a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_perms\nWHERE job_id NOT IN (SELECT id FROM v2_job_queue)\nRETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b" -} diff --git a/backend/.sqlx/query-c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0.json b/backend/.sqlx/query-c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0.json new file mode 100644 index 0000000000..d316c16fff --- /dev/null +++ b/backend/.sqlx/query-c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO schedule (\n workspace_id, path, edited_by, edited_at, schedule, enabled, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n )\n SELECT\n $1, path, edited_by, edited_at, schedule, FALSE, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n FROM schedule WHERE workspace_id = $2 AND NOT starts_with(path, $3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0" +} diff --git a/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json b/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json new file mode 100644 index 0000000000..ca39442f59 --- /dev/null +++ b/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "description", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b" +} diff --git a/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json b/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json index 4629a9bd0a..161b9d36f9 100644 --- a/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json +++ b/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json @@ -42,7 +42,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-c961131f349c68b91b0aea9805fd592276fa08da3f13f9c74aeedb666a55df05.json b/backend/.sqlx/query-c961131f349c68b91b0aea9805fd592276fa08da3f13f9c74aeedb666a55df05.json new file mode 100644 index 0000000000..c1e5495204 --- /dev/null +++ b/backend/.sqlx/query-c961131f349c68b91b0aea9805fd592276fa08da3f13f9c74aeedb666a55df05.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM macro_usage WHERE workspace_id = $1 AND macro_name = ANY($2) AND consumer_path != $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "TextArray", + "Text" + ] + }, + "nullable": [] + }, + "hash": "c961131f349c68b91b0aea9805fd592276fa08da3f13f9c74aeedb666a55df05" +} diff --git a/backend/.sqlx/query-c9d9c08505e69790154876f50f90d503a92e44e291a76a53ef09ded619edfacb.json b/backend/.sqlx/query-c9d9c08505e69790154876f50f90d503a92e44e291a76a53ef09ded619edfacb.json new file mode 100644 index 0000000000..b089e466cf --- /dev/null +++ b/backend/.sqlx/query-c9d9c08505e69790154876f50f90d503a92e44e291a76a53ef09ded619edfacb.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ws.datatable->'datatables' FROM workspace_settings ws WHERE ws.workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "c9d9c08505e69790154876f50f90d503a92e44e291a76a53ef09ded619edfacb" +} diff --git a/backend/.sqlx/query-cb04b72b25b7bd1163316be6a4e25e934a5a7677a9fdc7ea62f9dadbace15ec3.json b/backend/.sqlx/query-cb04b72b25b7bd1163316be6a4e25e934a5a7677a9fdc7ea62f9dadbace15ec3.json new file mode 100644 index 0000000000..8f0841180b --- /dev/null +++ b/backend/.sqlx/query-cb04b72b25b7bd1163316be6a4e25e934a5a7677a9fdc7ea62f9dadbace15ec3.json @@ -0,0 +1,46 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT m.name AS \"name!\", m.params AS \"params!\", m.body AS \"body!\",\n m.is_table_macro AS \"is_table_macro!\", m.provider_path AS \"provider_path!\"\n FROM macro_definition m\n WHERE m.workspace_id = $1\n AND EXISTS (\n SELECT 1 FROM script s\n WHERE s.workspace_id = m.workspace_id\n AND s.path = m.provider_path\n AND s.archived = false\n AND s.deleted = false\n )\n ORDER BY m.provider_path, m.name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "params!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "body!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "is_table_macro!", + "type_info": "Bool" + }, + { + "ordinal": 4, + "name": "provider_path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + false + ] + }, + "hash": "cb04b72b25b7bd1163316be6a4e25e934a5a7677a9fdc7ea62f9dadbace15ec3" +} diff --git a/backend/.sqlx/query-cb1b8adfe43c193d8b97afc3b04b6ebebd50e943263d6884bcdd322faeaa8345.json b/backend/.sqlx/query-cb1b8adfe43c193d8b97afc3b04b6ebebd50e943263d6884bcdd322faeaa8345.json new file mode 100644 index 0000000000..1419a1a20c --- /dev/null +++ b/backend/.sqlx/query-cb1b8adfe43c193d8b97afc3b04b6ebebd50e943263d6884bcdd322faeaa8345.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE macro_definition SET workspace_id = $1 WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "cb1b8adfe43c193d8b97afc3b04b6ebebd50e943263d6884bcdd322faeaa8345" +} diff --git a/backend/.sqlx/query-cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f.json b/backend/.sqlx/query-cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f.json new file mode 100644 index 0000000000..0fac858572 --- /dev/null +++ b/backend/.sqlx/query-cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH RECURSIVE chain AS (\n SELECT id, parent_workspace_id, 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, chain.depth + 1\n FROM workspace w\n JOIN chain ON w.id = chain.parent_workspace_id\n WHERE chain.depth < 20\n )\n SELECT id AS \"id!\" FROM chain WHERE depth > 0 ORDER BY depth\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f" +} diff --git a/backend/.sqlx/query-ccff2c556f1171bd10b09b1aa1d2d8ced2d4a75eef905cb40cb5be6eb8bd5e2b.json b/backend/.sqlx/query-ccff2c556f1171bd10b09b1aa1d2d8ced2d4a75eef905cb40cb5be6eb8bd5e2b.json new file mode 100644 index 0000000000..abb128f463 --- /dev/null +++ b/backend/.sqlx/query-ccff2c556f1171bd10b09b1aa1d2d8ced2d4a75eef905cb40cb5be6eb8bd5e2b.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT storage, bytes, computed_at FROM workspace_storage_usage\n WHERE workspace_id = $1 ORDER BY storage", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "storage", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "bytes", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "computed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "ccff2c556f1171bd10b09b1aa1d2d8ced2d4a75eef905cb40cb5be6eb8bd5e2b" +} diff --git a/backend/.sqlx/query-cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3.json b/backend/.sqlx/query-cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3.json new file mode 100644 index 0000000000..83f7394250 --- /dev/null +++ b/backend/.sqlx/query-cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE fork_ducklake_namespace SET schema_dropped = true\n WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3\n AND storage = $4 AND storage_ref = $5 AND data_path = $6", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3" +} diff --git a/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json new file mode 100644 index 0000000000..2c775af0b1 --- /dev/null +++ b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO background_task_state (name, value)\n SELECT $1, jsonb_build_object(\n 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint,\n 'last_ts', now(),\n 'last_oldest_inflight_ts',\n COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))\n WHERE NOT EXISTS (SELECT 1 FROM global_settings WHERE name = $2)\n ON CONFLICT (name) DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1" +} diff --git a/backend/.sqlx/query-cf2e74dcd0992f22eb3b62995fd0099ee5f46dafba8f323cf002e329ac69d1ac.json b/backend/.sqlx/query-cf2e74dcd0992f22eb3b62995fd0099ee5f46dafba8f323cf002e329ac69d1ac.json new file mode 100644 index 0000000000..83c306f4d8 --- /dev/null +++ b/backend/.sqlx/query-cf2e74dcd0992f22eb3b62995fd0099ee5f46dafba8f323cf002e329ac69d1ac.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT datatable, timestamp FROM datatable_migrations WHERE workspace_id = $1 AND datatable = ANY($2) AND timestamp = ANY($3)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "datatable", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "timestamp", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray", + "Int8Array" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "cf2e74dcd0992f22eb3b62995fd0099ee5f46dafba8f323cf002e329ac69d1ac" +} diff --git a/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json new file mode 100644 index 0000000000..4364da5dd0 --- /dev/null +++ b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_result_stream_v2\n WHERE ctid IN (\n SELECT jrs.ctid FROM job_result_stream_v2 jrs\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)\n AND NOT EXISTS (\n SELECT 1 FROM v2_job_completed c\n WHERE c.id = jrs.job_id\n AND c.completed_at > NOW() - INTERVAL '60 seconds'\n )\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230" +} diff --git a/backend/.sqlx/query-d0e826043e5a129ae6768c274c67b6254ff6c5fd450ecdab886a3183a894d266.json b/backend/.sqlx/query-d0e826043e5a129ae6768c274c67b6254ff6c5fd450ecdab886a3183a894d266.json deleted file mode 100644 index e900b0f9e0..0000000000 --- a/backend/.sqlx/query-d0e826043e5a129ae6768c274c67b6254ff6c5fd450ecdab886a3183a894d266.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "WITH ids AS (\n SELECT id as job_id FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )\n ) SELECT args->>$2 FROM ids LEFT JOIN v2_job ON v2_job.id = ids.job_id\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "?column?", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Text" - ] - }, - "nullable": [ - null - ] - }, - "hash": "d0e826043e5a129ae6768c274c67b6254ff6c5fd450ecdab886a3183a894d266" -} diff --git a/backend/.sqlx/query-d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7.json b/backend/.sqlx/query-d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7.json new file mode 100644 index 0000000000..f705dbc337 --- /dev/null +++ b/backend/.sqlx/query-d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings\n SET ducklake = jsonb_set(ducklake, '{ducklakes}', (\n SELECT COALESCE(jsonb_object_agg(key, value - 'fork_behavior'), '{}'::jsonb)\n FROM jsonb_each(ducklake->'ducklakes')\n ))\n WHERE workspace_id = $1 AND jsonb_typeof(ducklake->'ducklakes') = 'object'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7" +} diff --git a/backend/.sqlx/query-d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706.json b/backend/.sqlx/query-d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706.json new file mode 100644 index 0000000000..cbf96dad52 --- /dev/null +++ b/backend/.sqlx/query-d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO pipeline_freshness_state (workspace_id, script_path) VALUES ($1, $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706" +} diff --git a/backend/.sqlx/query-d3334846d5928e9b260d17e7ca99f6f24e83e019d6cd1590c42b14524c00f4d3.json b/backend/.sqlx/query-d3334846d5928e9b260d17e7ca99f6f24e83e019d6cd1590c42b14524c00f4d3.json new file mode 100644 index 0000000000..fe92251342 --- /dev/null +++ b/backend/.sqlx/query-d3334846d5928e9b260d17e7ca99f6f24e83e019d6cd1590c42b14524c00f4d3.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO usr (workspace_id, email, username, is_admin, role)\n VALUES ('wm-fork-guard-test', 'test2@windmill.dev', 'test-user-2', true, 'Admin')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "d3334846d5928e9b260d17e7ca99f6f24e83e019d6cd1590c42b14524c00f4d3" +} diff --git a/backend/.sqlx/query-d350eb5358c62526e70eba991fdd817a998bef51ac27e83942afc548640fb785.json b/backend/.sqlx/query-d350eb5358c62526e70eba991fdd817a998bef51ac27e83942afc548640fb785.json new file mode 100644 index 0000000000..deed0f1cdb --- /dev/null +++ b/backend/.sqlx/query-d350eb5358c62526e70eba991fdd817a998bef51ac27e83942afc548640fb785.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM macro_usage WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "d350eb5358c62526e70eba991fdd817a998bef51ac27e83942afc548640fb785" +} diff --git a/backend/.sqlx/query-d38e25120ec7bfcbd210d967ef2406d1773657c62d67e55dac666f9469d50e6a.json b/backend/.sqlx/query-d38e25120ec7bfcbd210d967ef2406d1773657c62d67e55dac666f9469d50e6a.json new file mode 100644 index 0000000000..1fd1fbaece --- /dev/null +++ b/backend/.sqlx/query-d38e25120ec7bfcbd210d967ef2406d1773657c62d67e55dac666f9469d50e6a.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM workspace_storage_usage WHERE workspace_id = $1 AND storage != ALL($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "d38e25120ec7bfcbd210d967ef2406d1773657c62d67e55dac666f9469d50e6a" +} diff --git a/backend/.sqlx/query-d3ce4e7f3dd10548197734a0cf38c644f70be809dcfe7d973147962530c601d5.json b/backend/.sqlx/query-d3ce4e7f3dd10548197734a0cf38c644f70be809dcfe7d973147962530c601d5.json new file mode 100644 index 0000000000..9f8b86b4e6 --- /dev/null +++ b/backend/.sqlx/query-d3ce4e7f3dd10548197734a0cf38c644f70be809dcfe7d973147962530c601d5.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM workspace_protection_rule WHERE workspace_id = $1 AND name = $2\n AND NOT EXISTS (\n SELECT 1 FROM workspace\n WHERE parent_workspace_id = $1 AND is_dev_workspace AND deleted = false\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "d3ce4e7f3dd10548197734a0cf38c644f70be809dcfe7d973147962530c601d5" +} diff --git a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json index d5365ffe94..d97c02d26b 100644 --- a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json +++ b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json @@ -111,7 +111,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json b/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json index 33a5534b42..052d83fcd9 100644 --- a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json +++ b/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json @@ -251,7 +251,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-d4923137cf9b6bf06e0e21ba907e7c80dd0ef92d78224792bec4c3323e14023c.json b/backend/.sqlx/query-d4923137cf9b6bf06e0e21ba907e7c80dd0ef92d78224792bec4c3323e14023c.json new file mode 100644 index 0000000000..5609adaad8 --- /dev/null +++ b/backend/.sqlx/query-d4923137cf9b6bf06e0e21ba907e7c80dd0ef92d78224792bec4c3323e14023c.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n COALESCE(\n (SELECT MIN(computed_at) FROM workspace_storage_usage WHERE workspace_id = $1) < now() - interval '10 minutes',\n true) as \"stale!\",\n (SELECT COALESCE(SUM(GREATEST(t.total - t.existing, 0)), 0)\n FROM (SELECT SUM(part_bytes) as total, MAX(target_existing_size) as existing\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND created_at > now() - ($2::text)::interval\n AND ($3::text IS NULL OR upload_id <> $3)\n GROUP BY upload_id) t)::bigint as \"reserved!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "committed!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "stale!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "reserved!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "d4923137cf9b6bf06e0e21ba907e7c80dd0ef92d78224792bec4c3323e14023c" +} diff --git a/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json b/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json index 0eed069163..e60f5cc187 100644 --- a/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json +++ b/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json @@ -47,7 +47,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-d4f047b0a672d5bfe9fd7db60f11563ef05db7cc54a958a72384b08a69eb784e.json b/backend/.sqlx/query-d4f047b0a672d5bfe9fd7db60f11563ef05db7cc54a958a72384b08a69eb784e.json new file mode 100644 index 0000000000..3a5531b9a6 --- /dev/null +++ b/backend/.sqlx/query-d4f047b0a672d5bfe9fd7db60f11563ef05db7cc54a958a72384b08a69eb784e.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT usage_path FROM asset\n WHERE workspace_id = $1\n AND kind = 'ducklake'\n AND path = $2\n AND usage_kind = 'script'\n AND usage_access_type IN ('w', 'rw')\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "usage_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "d4f047b0a672d5bfe9fd7db60f11563ef05db7cc54a958a72384b08a69eb784e" +} diff --git a/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json b/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json new file mode 100644 index 0000000000..8c2a7716fd --- /dev/null +++ b/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\"\n FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "success!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Uuid", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994" +} diff --git a/backend/.sqlx/query-d74dfaa8cb9fca89c2e21f810cd45ff7b595c7c50f5e6ff2733eded6ef544996.json b/backend/.sqlx/query-d74dfaa8cb9fca89c2e21f810cd45ff7b595c7c50f5e6ff2733eded6ef544996.json new file mode 100644 index 0000000000..4a511c2d6e --- /dev/null +++ b/backend/.sqlx/query-d74dfaa8cb9fca89c2e21f810cd45ff7b595c7c50f5e6ff2733eded6ef544996.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT is_admin FROM usr WHERE workspace_id = $1 AND email = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "is_admin", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "d74dfaa8cb9fca89c2e21f810cd45ff7b595c7c50f5e6ff2733eded6ef544996" +} diff --git a/backend/.sqlx/query-d8f234765e2de89f780937a505c236c5be9b36b4df95735fedb44ed944606ac8.json b/backend/.sqlx/query-d8f234765e2de89f780937a505c236c5be9b36b4df95735fedb44ed944606ac8.json new file mode 100644 index 0000000000..48a5c693ac --- /dev/null +++ b/backend/.sqlx/query-d8f234765e2de89f780937a505c236c5be9b36b4df95735fedb44ed944606ac8.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2 RETURNING id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "d8f234765e2de89f780937a505c236c5be9b36b4df95735fedb44ed944606ac8" +} diff --git a/backend/.sqlx/query-caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439.json b/backend/.sqlx/query-dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8.json similarity index 53% rename from backend/.sqlx/query-caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439.json rename to backend/.sqlx/query-dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8.json index f2398bf143..0d36677337 100644 --- a/backend/.sqlx/query-caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439.json +++ b/backend/.sqlx/query-dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8.json @@ -1,17 +1,16 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT ws.datatable->'datatables'->$2 AS config\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ", + "query": "\n SELECT ws.datatable->'datatables' AS datatables\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ", "describe": { "columns": [ { "ordinal": 0, - "name": "config", + "name": "datatables", "type_info": "Jsonb" } ], "parameters": { "Left": [ - "Text", "Text" ] }, @@ -19,5 +18,5 @@ null ] }, - "hash": "caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439" + "hash": "dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8" } diff --git a/backend/.sqlx/query-deac41298e8b0d0870e314fef0813c24dd55d63bda78a0a5f35ed6f22bea6bef.json b/backend/.sqlx/query-deac41298e8b0d0870e314fef0813c24dd55d63bda78a0a5f35ed6f22bea6bef.json deleted file mode 100644 index 67ce9d8719..0000000000 --- a/backend/.sqlx/query-deac41298e8b0d0870e314fef0813c24dd55d63bda78a0a5f35ed6f22bea6bef.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO resource\n (workspace_id, path, value, description, resource_type, created_by, edited_at, labels)\n VALUES ($1, $2, $3, $4, $5, $6, now(), $7) ON CONFLICT (workspace_id, path)\n DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description, resource_type = EXCLUDED.resource_type, edited_at = now(), labels = EXCLUDED.labels", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Jsonb", - "Text", - "Varchar", - "Varchar", - "TextArray" - ] - }, - "nullable": [] - }, - "hash": "deac41298e8b0d0870e314fef0813c24dd55d63bda78a0a5f35ed6f22bea6bef" -} diff --git a/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json b/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json new file mode 100644 index 0000000000..87cd891113 --- /dev/null +++ b/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf" +} diff --git a/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json new file mode 100644 index 0000000000..20ae255eb1 --- /dev/null +++ b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.path = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + false + ] + }, + "hash": "e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd" +} diff --git a/backend/.sqlx/query-e17d7fb386ec45386401a20d97b50d224da5beb74d289fe17ea960887aa50f1a.json b/backend/.sqlx/query-e17d7fb386ec45386401a20d97b50d224da5beb74d289fe17ea960887aa50f1a.json new file mode 100644 index 0000000000..c3db56398e --- /dev/null +++ b/backend/.sqlx/query-e17d7fb386ec45386401a20d97b50d224da5beb74d289fe17ea960887aa50f1a.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE workspace_id = $1\n AND language = 'duckdb'::script_lang\n AND archived = false\n AND deleted = false\n AND path != $2\n ORDER BY path, created_at DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "e17d7fb386ec45386401a20d97b50d224da5beb74d289fe17ea960887aa50f1a" +} diff --git a/backend/.sqlx/query-e1ada31c1625b453c2ff85edbcd7ad51a4cd5cbdc2fa34038530070d6a579455.json b/backend/.sqlx/query-e1ada31c1625b453c2ff85edbcd7ad51a4cd5cbdc2fa34038530070d6a579455.json deleted file mode 100644 index f38c023cb3..0000000000 --- a/backend/.sqlx/query-e1ada31c1625b453c2ff85edbcd7ad51a4cd5cbdc2fa34038530070d6a579455.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "WITH potential AS (\n SELECT email, operator FROM usr\n UNION\n SELECT email, operator FROM workspace_invite\n ),\n per_user AS (\n SELECT email, bool_and(operator) AS only_operator FROM potential GROUP BY email\n )\n SELECT\n COUNT(*) FILTER (WHERE NOT only_operator) AS \"authors!\",\n COUNT(*) FILTER (WHERE only_operator) AS \"operators!\"\n FROM per_user", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "authors!", - "type_info": "Int8" - }, - { - "ordinal": 1, - "name": "operators!", - "type_info": "Int8" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - null, - null - ] - }, - "hash": "e1ada31c1625b453c2ff85edbcd7ad51a4cd5cbdc2fa34038530070d6a579455" -} diff --git a/backend/.sqlx/query-e26401ac6ecefe4e11306f2c714165d9a755102d1df51ffe63abf5900bcdb42c.json b/backend/.sqlx/query-e26401ac6ecefe4e11306f2c714165d9a755102d1df51ffe63abf5900bcdb42c.json new file mode 100644 index 0000000000..4609e7adf4 --- /dev/null +++ b/backend/.sqlx/query-e26401ac6ecefe4e11306f2c714165d9a755102d1df51ffe63abf5900bcdb42c.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM macro_definition WHERE workspace_id = $1 AND provider_path = $2) AS \"e!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "e!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "e26401ac6ecefe4e11306f2c714165d9a755102d1df51ffe63abf5900bcdb42c" +} diff --git a/backend/.sqlx/query-e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd.json b/backend/.sqlx/query-e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd.json new file mode 100644 index 0000000000..37bd25eb98 --- /dev/null +++ b/backend/.sqlx/query-e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO fork_ducklake_namespace\n (workspace_id, ducklake_name, metadata_schema, catalog, storage, storage_ref, data_path)\n VALUES ($1, $2, $3, $4, $5, $6, $7)\n ON CONFLICT (workspace_id, ducklake_name, catalog, storage, storage_ref, data_path)\n DO UPDATE SET schema_dropped = false", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd" +} diff --git a/backend/.sqlx/query-e4a46d47aee96473a2bd0f1dbdc819520a2ba987fd29859845c2177563ab8cfb.json b/backend/.sqlx/query-e4a46d47aee96473a2bd0f1dbdc819520a2ba987fd29859845c2177563ab8cfb.json new file mode 100644 index 0000000000..18c75f5722 --- /dev/null +++ b/backend/.sqlx/query-e4a46d47aee96473a2bd0f1dbdc819520a2ba987fd29859845c2177563ab8cfb.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, value FROM variable\n WHERE workspace_id = $1 AND is_secret = true AND value != ''", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "value", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "e4a46d47aee96473a2bd0f1dbdc819520a2ba987fd29859845c2177563ab8cfb" +} diff --git a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json index 470c651020..a35373a959 100644 --- a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json +++ b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json @@ -191,7 +191,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json b/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json new file mode 100644 index 0000000000..121d7fe04a --- /dev/null +++ b/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "description", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "instructions", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d" +} diff --git a/backend/.sqlx/query-e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a.json b/backend/.sqlx/query-e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a.json new file mode 100644 index 0000000000..8c80c14ea4 --- /dev/null +++ b/backend/.sqlx/query-e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_path AS \"runnable_path!\", created_by AS \"created_by!\",\n args AS \"args: sqlx::types::Json\"\n FROM v2_job\n WHERE workspace_id = $1 AND trigger_kind = 'freshness'\n ORDER BY created_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "created_by!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "args: sqlx::types::Json", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + false, + true + ] + }, + "hash": "e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a" +} diff --git a/backend/.sqlx/query-e5d8d46b9431033fa7572a880ee0e74e14afde3c02000a88e696d7b6adcad0d6.json b/backend/.sqlx/query-e5d8d46b9431033fa7572a880ee0e74e14afde3c02000a88e696d7b6adcad0d6.json new file mode 100644 index 0000000000..d6c6a5c02f --- /dev/null +++ b/backend/.sqlx/query-e5d8d46b9431033fa7572a880ee0e74e14afde3c02000a88e696d7b6adcad0d6.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT timestamp, name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 ORDER BY timestamp ASC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "timestamp", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "code_up", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "code_down", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false, + true + ] + }, + "hash": "e5d8d46b9431033fa7572a880ee0e74e14afde3c02000a88e696d7b6adcad0d6" +} diff --git a/backend/.sqlx/query-e6a8ddfd74ebab55ede5989fd7d09b246b6f84e49c659bc8e2c7b66cfec6d976.json b/backend/.sqlx/query-e6a8ddfd74ebab55ede5989fd7d09b246b6f84e49c659bc8e2c7b66cfec6d976.json deleted file mode 100644 index a1b52e81fd..0000000000 --- a/backend/.sqlx/query-e6a8ddfd74ebab55ede5989fd7d09b246b6f84e49c659bc8e2c7b66cfec6d976.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT status = 'success' AS \"success!\",\n result AS \"result: Json>\",\n started_at AS \"started_at!\"FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "success!", - "type_info": "Bool" - }, - { - "ordinal": 1, - "name": "result: Json>", - "type_info": "Jsonb" - }, - { - "ordinal": 2, - "name": "started_at!", - "type_info": "Timestamptz" - } - ], - "parameters": { - "Left": [ - "Text", - "Text", - "Text", - "Uuid", - "Int8" - ] - }, - "nullable": [ - null, - true, - true - ] - }, - "hash": "e6a8ddfd74ebab55ede5989fd7d09b246b6f84e49c659bc8e2c7b66cfec6d976" -} diff --git a/backend/.sqlx/query-e82d854f3f8736a9ba7e2604ac505b693965f1bca6fef7418b19f98671c9204c.json b/backend/.sqlx/query-e82d854f3f8736a9ba7e2604ac505b693965f1bca6fef7418b19f98671c9204c.json new file mode 100644 index 0000000000..4f0731f5e4 --- /dev/null +++ b/backend/.sqlx/query-e82d854f3f8736a9ba7e2604ac505b693965f1bca6fef7418b19f98671c9204c.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_storage_usage (workspace_id, storage, bytes, computed_at)\n VALUES ($1, $2, $3, now())\n ON CONFLICT (workspace_id, storage) DO UPDATE SET bytes = $3, computed_at = now()", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "e82d854f3f8736a9ba7e2604ac505b693965f1bca6fef7418b19f98671c9204c" +} diff --git a/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json b/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json new file mode 100644 index 0000000000..29ce0c6f2c --- /dev/null +++ b/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61" +} diff --git a/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json b/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json new file mode 100644 index 0000000000..f236c2aae0 --- /dev/null +++ b/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES\n ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'),\n ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'),\n ($3, nextval('debounce_batch_seq'), NULL),\n ($4, nextval('debounce_batch_seq'), now() - interval '20 minutes')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid", + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2" +} diff --git a/backend/.sqlx/query-ebce1633afe3d3f8a1a68bc8d26adcdd88b137bb892437a3ca67fd716772512b.json b/backend/.sqlx/query-ebce1633afe3d3f8a1a68bc8d26adcdd88b137bb892437a3ca67fd716772512b.json new file mode 100644 index 0000000000..1930427b94 --- /dev/null +++ b/backend/.sqlx/query-ebce1633afe3d3f8a1a68bc8d26adcdd88b137bb892437a3ca67fd716772512b.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n COUNT(*) FILTER (WHERE NOT operator AND NOT disabled AND NOT is_service_account) AS \"developers!\",\n COUNT(*) FILTER (WHERE operator AND NOT disabled AND NOT is_service_account) AS \"operators!\"\n FROM usr WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "developers!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "operators!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "ebce1633afe3d3f8a1a68bc8d26adcdd88b137bb892437a3ca67fd716772512b" +} diff --git a/backend/.sqlx/query-ec8e502f9c2926e578e02cd164a7a3e43ef12b97d55353f86a9a368617efccca.json b/backend/.sqlx/query-ec8e502f9c2926e578e02cd164a7a3e43ef12b97d55353f86a9a368617efccca.json new file mode 100644 index 0000000000..fbcf0743be --- /dev/null +++ b/backend/.sqlx/query-ec8e502f9c2926e578e02cd164a7a3e43ef12b97d55353f86a9a368617efccca.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings SET datatable = jsonb_set(datatable, ARRAY['datatables', $2::text, 'migrations_enabled'], 'true'::jsonb) WHERE workspace_id = $1 AND jsonb_exists(datatable->'datatables', $2) RETURNING 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ec8e502f9c2926e578e02cd164a7a3e43ef12b97d55353f86a9a368617efccca" +} diff --git a/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json new file mode 100644 index 0000000000..dd7f4da810 --- /dev/null +++ b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.id = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null, + null, + false + ] + }, + "hash": "ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db" +} diff --git a/backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json b/backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json similarity index 55% rename from backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json rename to backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json index 4b96e7005b..84a2320a80 100644 --- a/backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json +++ b/backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json @@ -1,15 +1,23 @@ { "db_name": "PostgreSQL", - "query": "WITH to_delete AS (\n SELECT id FROM v2_job_queue\n JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = $1\n AND j.workspace_id = $2\n AND flow_step_id IS NULL\n AND running = false\n FOR UPDATE\n ), deleted AS (\n DELETE FROM v2_job_queue\n WHERE id IN (SELECT id FROM to_delete)\n RETURNING id\n ) DELETE FROM v2_job WHERE id IN (SELECT id FROM deleted)", + "query": "WITH to_delete AS (\n SELECT id FROM v2_job_queue\n JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = $1\n AND j.workspace_id = $2\n AND flow_step_id IS NULL\n AND running = false\n FOR UPDATE\n )\n DELETE FROM v2_job_queue\n WHERE id IN (SELECT id FROM to_delete)\n RETURNING id", "describe": { - "columns": [], + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + } + ], "parameters": { "Left": [ "Text", "Text" ] }, - "nullable": [] + "nullable": [ + false + ] }, - "hash": "d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a" + "hash": "f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3" } diff --git a/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json b/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json new file mode 100644 index 0000000000..1039dd3950 --- /dev/null +++ b/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN NULL ELSE debounce_key.job_id END,\n job_id = EXCLUDED.job_id,\n debounced_times = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN 0 ELSE debounce_key.debounced_times + 1 END,\n first_started_at = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN now() ELSE debounce_key.first_started_at END\n RETURNING debounced_times, first_started_at, previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT $1, COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq'))\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "debounced_times", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "first_started_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "job_id_to_debounce", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2" +} diff --git a/backend/.sqlx/query-f1f79711f131ff4116489153db1b71e528c66fab1508e5c41b63bc5b9077a08c.json b/backend/.sqlx/query-f1f79711f131ff4116489153db1b71e528c66fab1508e5c41b63bc5b9077a08c.json new file mode 100644 index 0000000000..844dfb3e47 --- /dev/null +++ b/backend/.sqlx/query-f1f79711f131ff4116489153db1b71e528c66fab1508e5c41b63bc5b9077a08c.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO resource\n (workspace_id, path, value, description, resource_type, created_by, edited_at, labels)\n VALUES ($1, $2, $3, $4, $5, $6, now(), $7) ON CONFLICT (workspace_id, path)\n DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description, resource_type = EXCLUDED.resource_type, edited_at = now(), labels = EXCLUDED.labels", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Jsonb", + "Text", + "Varchar", + "Varchar", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "f1f79711f131ff4116489153db1b71e528c66fab1508e5c41b63bc5b9077a08c" +} diff --git a/backend/.sqlx/query-f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472.json b/backend/.sqlx/query-f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472.json new file mode 100644 index 0000000000..8257722c5c --- /dev/null +++ b/backend/.sqlx/query-f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) AS \"count!\" FROM pipeline_freshness_state WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472" +} diff --git a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json similarity index 64% rename from backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json rename to backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json index af4ada67f2..84eb4caba9 100644 --- a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json +++ b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = 'anonymous'\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", + "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = $4\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", "describe": { "columns": [ { @@ -11,6 +11,7 @@ ], "parameters": { "Left": [ + "Text", "Text", "Text", "Text" @@ -20,5 +21,5 @@ null ] }, - "hash": "a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d" + "hash": "f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7" } diff --git a/backend/.sqlx/query-f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1.json b/backend/.sqlx/query-f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1.json new file mode 100644 index 0000000000..7049fafe37 --- /dev/null +++ b/backend/.sqlx/query-f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n SELECT $2, path, kind, usage_access_type, usage_path, usage_kind, columns\n FROM asset WHERE workspace_id = $1 AND usage_kind IN ('script', 'flow')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1" +} diff --git a/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json b/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json new file mode 100644 index 0000000000..81b34464f4 --- /dev/null +++ b/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN NULL ELSE debounce_key.job_id END,\n job_id = EXCLUDED.job_id,\n debounced_times = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN 0 ELSE debounce_key.debounced_times + 1 END,\n first_started_at = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN now() ELSE debounce_key.first_started_at END\n RETURNING debounced_times, first_started_at, previous_job_id AS job_id_to_debounce\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "debounced_times", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "first_started_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "job_id_to_debounce", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5" +} diff --git a/backend/.sqlx/query-f629c27daf692d7055fa299edaae110a5d6dcdc65aa293b1a15d67dee5308ddc.json b/backend/.sqlx/query-f629c27daf692d7055fa299edaae110a5d6dcdc65aa293b1a15d67dee5308ddc.json new file mode 100644 index 0000000000..9e32600de9 --- /dev/null +++ b/backend/.sqlx/query-f629c27daf692d7055fa299edaae110a5d6dcdc65aa293b1a15d67dee5308ddc.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO macro_usage (workspace_id, consumer_path, macro_name) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "f629c27daf692d7055fa299edaae110a5d6dcdc65aa293b1a15d67dee5308ddc" +} diff --git a/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json b/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json new file mode 100644 index 0000000000..132d1da5a1 --- /dev/null +++ b/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH ids AS (SELECT id FROM v2_job WHERE workspace_id = $1),\n _de AS (DELETE FROM dispatch_event WHERE workspace_id = $1),\n _fc AS (DELETE FROM flow_conversation_message WHERE job_id IN (SELECT id FROM ids))\n DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM ids)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645" +} diff --git a/backend/.sqlx/query-f89b024949f29eb5142744635914b94fd84c1bb64e9659f798df2888848894bd.json b/backend/.sqlx/query-f89b024949f29eb5142744635914b94fd84c1bb64e9659f798df2888848894bd.json new file mode 100644 index 0000000000..8a6d989157 --- /dev/null +++ b/backend/.sqlx/query-f89b024949f29eb5142744635914b94fd84c1bb64e9659f798df2888848894bd.json @@ -0,0 +1,46 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT datatable, timestamp, name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 ORDER BY datatable, timestamp", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "datatable", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "timestamp", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "code_up", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "code_down", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + true + ] + }, + "hash": "f89b024949f29eb5142744635914b94fd84c1bb64e9659f798df2888848894bd" +} diff --git a/backend/.sqlx/query-fa04b3660f1f90c39c3d12f39c51df8f25a3bdae60aeaed313612cb858040c8d.json b/backend/.sqlx/query-fa04b3660f1f90c39c3d12f39c51df8f25a3bdae60aeaed313612cb858040c8d.json new file mode 100644 index 0000000000..7f967b60a5 --- /dev/null +++ b/backend/.sqlx/query-fa04b3660f1f90c39c3d12f39c51df8f25a3bdae60aeaed313612cb858040c8d.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM ws_specific WHERE workspace_id = $1 AND item_kind = $2 AND path = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "fa04b3660f1f90c39c3d12f39c51df8f25a3bdae60aeaed313612cb858040c8d" +} diff --git a/backend/.sqlx/query-fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588.json b/backend/.sqlx/query-fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588.json new file mode 100644 index 0000000000..ad17a60330 --- /dev/null +++ b/backend/.sqlx/query-fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM fork_ducklake_namespace\n WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3\n AND storage = $4 AND storage_ref = $5 AND data_path = $6", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588" +} diff --git a/backend/.sqlx/query-fc1663ecda7167dc12c0fd5a206cbddf85e55f6c7ef226cdd66e025c002bac98.json b/backend/.sqlx/query-fc1663ecda7167dc12c0fd5a206cbddf85e55f6c7ef226cdd66e025c002bac98.json new file mode 100644 index 0000000000..626c541556 --- /dev/null +++ b/backend/.sqlx/query-fc1663ecda7167dc12c0fd5a206cbddf85e55f6c7ef226cdd66e025c002bac98.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) FROM workspace WHERE id != 'admins' AND deleted = true", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "fc1663ecda7167dc12c0fd5a206cbddf85e55f6c7ef226cdd66e025c002bac98" +} diff --git a/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json new file mode 100644 index 0000000000..484dd23e2a --- /dev/null +++ b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT req.id AS \"id!\",\n (CASE\n WHEN usr.email IS NULL THEN 'deleted'\n WHEN workspace.deleted THEN 'archived'\n ELSE 'active'\n END) AS \"status!\"\n FROM unnest($1::text[]) AS req(id)\n LEFT JOIN workspace ON workspace.id = req.id\n LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "status!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "TextArray", + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441" +} diff --git a/backend/.sqlx/query-fc59d9b911529de75c466593252358581f5716b3b58dc2e6b9b9282c50b1b66b.json b/backend/.sqlx/query-fc59d9b911529de75c466593252358581f5716b3b58dc2e6b9b9282c50b1b66b.json new file mode 100644 index 0000000000..1efaaa32e8 --- /dev/null +++ b/backend/.sqlx/query-fc59d9b911529de75c466593252358581f5716b3b58dc2e6b9b9282c50b1b66b.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job SET labels = (\n SELECT array_agg(DISTINCT l)\n FROM unnest(coalesce(labels, ARRAY[]::TEXT[]) || $2) l\n ) WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "fc59d9b911529de75c466593252358581f5716b3b58dc2e6b9b9282c50b1b66b" +} diff --git a/backend/.sqlx/query-fcb0cf55f5c9047066a6fdd62bf6f85238cbf189ccf18191493d6357e269d12d.json b/backend/.sqlx/query-fcb0cf55f5c9047066a6fdd62bf6f85238cbf189ccf18191493d6357e269d12d.json new file mode 100644 index 0000000000..c630f5dd05 --- /dev/null +++ b/backend/.sqlx/query-fcb0cf55f5c9047066a6fdd62bf6f85238cbf189ccf18191493d6357e269d12d.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT timestamp, name, code_up FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 ORDER BY timestamp ASC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "timestamp", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "code_up", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "fcb0cf55f5c9047066a6fdd62bf6f85238cbf189ccf18191493d6357e269d12d" +} diff --git a/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json new file mode 100644 index 0000000000..5ac11c6a5e --- /dev/null +++ b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT to_char(timestamp AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS \"day!\",\n id AS \"id!\",\n timestamp AS \"ts!\",\n row_to_json(r)::text AS \"line!\"\n FROM (\n SELECT workspace_id, id, timestamp, username, operation,\n action_kind::text AS action_kind, resource, parameters, email, span\n FROM audit_partitioned\n WHERE timestamp >= $1 AND timestamp < $2\n AND (timestamp, id) > ($3, $4)\n ORDER BY timestamp, id\n LIMIT $5\n ) r\n ORDER BY timestamp, id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "day!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "id!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "ts!", + "type_info": "Timestamptz" + }, + { + "ordinal": 3, + "name": "line!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Timestamptz", + "Timestamptz", + "Timestamptz", + "Int8", + "Int8" + ] + }, + "nullable": [ + null, + false, + false, + null + ] + }, + "hash": "fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c" +} diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 9ff2c0be90..24cf2cb837 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -6,3 +6,57 @@ - **DB schema**: `backend/summarized_schema.txt` - **API routes entry point**: `windmill-api/src/lib.rs` - **OpenAPI spec**: `windmill-api/openapi.yaml` +- **DuckDB local jobs**: build the dynamic FFI library before running DuckDB scripts locally: + ```bash + cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh + ``` + Re-run after clean builds or when `target/debug/libwindmill_duckdb_ffi_internal.*` is missing. + The bundled DuckDB compile (~2min) is cached in a per-user dir shared across + worktrees (keyed by the crate's `Cargo.lock`), so a fresh worktree reuses it and + the build is near-instant — you don't pay the full compile per worktree. Editing + the FFI crate's own source falls back to an isolated per-worktree `./target`. +- **Running data pipelines (DuckLake) from source**: see the section below — a plain build + advertises the `duckdb` tag but cannot execute DuckDB scripts and has no working S3 proxy. + +## Running data pipelines (DuckLake) from source + +DuckLake pipelines need **both** the right cargo features **and** the prebuilt DuckDB FFI. A +plain `cargo run` (or `cargo run --features quickjs`) does **not** suffice, and the failure modes +are silent-ish, so agents lose time. Verify feature names against `backend/Cargo.toml` `[features]`. + +**Feature sets** (run from `backend/`): + +| Goal | Command | +|---|---| +| CE DuckLake (DuckDB scripts + S3 proxy) | `cargo run --features quickjs,duckdb,parquet,private` | +| + Python scripts | add `,python` | +| EE features (WAP, partitioning, forks, …) | add `,enterprise,license` | + +`enterprise` already pulls in `license`, but list both when you want the license-gated paths. +`quickjs` is for JS eval, not DuckLake per se — keep it if your baseline build had it. + +**Before running any DuckDB script**, build the FFI (see the bullet above): +`cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. + +**Two gotchas that a wrong feature set produces:** + +1. **`duckdb` tag advertised, feature missing.** The `duckdb` worker tag is in the *unconditional* + default tag list (`windmill-common/src/worker.rs`, `DEFAULT_TAGS`), so a worker advertises it even + without the `duckdb` feature. Jobs then dispatch but fail at execution with + `"Duck DB requires the duckdb feature to be enabled"` (`windmill-worker/src/worker.rs`). Fix: + compile with `--features duckdb`. +2. **DuckLake writes 404 (no S3 proxy).** The workspace S3 proxy (`/w/{ws}/s3_proxy/*`) that + DuckLake uses for reads/writes only mounts the real service under + `#[cfg(all(feature = "private", feature = "parquet"))]` (`windmill-api/src/s3_proxy_oss.rs`); + otherwise it's an empty router and every proxied request 404s. Fix: compile with **both** + `private` and `parquet`. + +## Cloud vs self-hosted gating + +The `cloud` cargo feature is compiled into **all** EE builds, so `#[cfg(feature = "cloud")]` is **not** a "cloud-only" runtime gate — it only means the code is present. The real gate for behavior specific to the managed cloud (app.windmill.dev) is the runtime flag `*CLOUD_HOSTED` (`windmill_common::worker::CLOUD_HOSTED`, from the `CLOUD_HOSTED` env var; note it's loaded from `.env` via `dotenv`, so it won't show in `/proc//environ` — check the running behavior, not the exec env). + +Cloud-only logic must be behind `if *CLOUD_HOSTED { ... }`: feature-gate the helper so it compiles, then **runtime-gate the call**. `#[cfg(feature = "cloud")]` on its own is only sufficient for: +- pure helper/struct definitions (they only run when a gated caller invokes them), +- code already inside an `if *CLOUD_HOSTED { ... }` block, +- handlers that early-return on `!*CLOUD_HOSTED`, +- idempotent no-ops that are harmless off-cloud (e.g. cache invalidation). diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 367c17e912..8a6ff9ad53 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -180,9 +180,9 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.102" +version = "1.0.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" +checksum = "2a4385e2e34eb35d6b3efe798b9eb88096925d87726c0798709bf56d9ed84af3" [[package]] name = "ar_archive_writer" @@ -195,9 +195,9 @@ dependencies = [ [[package]] name = "arc-swap" -version = "1.9.1" +version = "1.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207" +checksum = "c049c0be4daef0b145cb3555416b3b8ef5b7888a38aea1a3a155801fe7b0810b" dependencies = [ "rustversion", ] @@ -237,9 +237,9 @@ checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" [[package]] name = "arrayvec" -version = "0.7.6" +version = "0.7.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" +checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" [[package]] name = "arrow" @@ -517,7 +517,7 @@ dependencies = [ "futures-core", "libc", "portable-atomic", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "tokio", "tokio-stream", "xattr", @@ -775,9 +775,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.17.0" +version = "1.17.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00" +checksum = "4342d8937fc7e5dd9b1c60292261c0670c882a2cd1719cfc11b1af41731e32ad" dependencies = [ "aws-lc-sys", "zeroize", @@ -785,14 +785,15 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.41.0" +version = "0.42.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4" +checksum = "6d9ceb1da931507a12f4fccea479dccd00da1943e1b4ae72d8e502d707361444" dependencies = [ "cc", "cmake", "dunce", "fs_extra", + "pkg-config", ] [[package]] @@ -1530,7 +1531,7 @@ dependencies = [ "proc-macro2", "quote", "regex", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "shlex 1.3.0", "syn 2.0.118", ] @@ -1550,7 +1551,7 @@ dependencies = [ "proc-macro2", "quote", "regex", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "shlex 1.3.0", "syn 2.0.118", ] @@ -1826,13 +1827,13 @@ dependencies = [ [[package]] name = "bstr" -version = "1.12.1" +version = "1.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab" +checksum = "5cee35f73844aa3014bb606320a6c1f010249dbdf43342fe54b5a4f6a8ed4b79" dependencies = [ "memchr", "regex-automata", - "serde", + "serde_core", ] [[package]] @@ -1855,9 +1856,9 @@ dependencies = [ [[package]] name = "byte-unit" -version = "5.2.3" +version = "5.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37bcaa4a0975bed4a760af3efe4368825098ce5f9d37a30c5a021d635dc63d8f" +checksum = "4a813de7f2bbedb7dce265b64f1cf5908ebe4d56281ece8d847e98113788b9b0" dependencies = [ "rust_decimal", "schemars 1.2.1", @@ -1915,9 +1916,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" -version = "1.12.0" +version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ae3f5d315924270530207e2a68396c3cc547f6dca3fbdca317cfb1a51edb593" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" dependencies = [ "serde", ] @@ -2056,9 +2057,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.64" +version = "1.2.66" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dad887fd958be91b5098c0248def011f4523ab786cd411be668777e55063501f" +checksum = "f5d6cac793997bd970000024b2934968efe83b382de4fdcf4fcb46b6ee4ad996" dependencies = [ "find-msvc-tools", "jobserver", @@ -2101,9 +2102,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" [[package]] name = "chacha20" -version = "0.10.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" dependencies = [ "cfg-if", "cpufeatures 0.3.0", @@ -2445,9 +2446,9 @@ dependencies = [ [[package]] name = "crc-any" -version = "2.5.0" +version = "2.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a62ec9ff5f7965e4d7280bd5482acd20aadb50d632cf6c1d74493856b011fa73" +checksum = "46db9f663dfb869b80fcf59e32d7a80fc6c464a4f6328f3f06a00f5e36d05f8c" dependencies = [ "debug-helper", ] @@ -2490,18 +2491,18 @@ dependencies = [ [[package]] name = "crossbeam-channel" -version = "0.5.15" +version = "0.5.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" +checksum = "d85363c37faeca707aef026efa9f3b34d077bce547e48f770770625c6013679e" dependencies = [ "crossbeam-utils", ] [[package]] name = "crossbeam-deque" -version = "0.8.6" +version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51" +checksum = "5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb" dependencies = [ "crossbeam-epoch", "crossbeam-utils", @@ -2509,27 +2510,27 @@ dependencies = [ [[package]] name = "crossbeam-epoch" -version = "0.9.18" +version = "0.9.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f" dependencies = [ "crossbeam-utils", ] [[package]] name = "crossbeam-queue" -version = "0.3.12" +version = "0.3.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0f58bbc28f91df819d0aa2a2c00cd19754769c2fad90579b3592b1c9ba7a3115" +checksum = "803d13fb3b09d88be9f4dbc29062c66b19bf7170867ceb746d2a8689bf6c7a26" dependencies = [ "crossbeam-utils", ] [[package]] name = "crossbeam-utils" -version = "0.8.21" +version = "0.8.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" +checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17" [[package]] name = "crossterm_winapi" @@ -2601,9 +2602,9 @@ dependencies = [ [[package]] name = "curl-sys" -version = "0.4.89+curl-8.20.0" +version = "0.4.90+curl-8.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d680779285438f2d0927485973ab45b212ea990bddb80de8a55a1e3c1d9ba22" +checksum = "97799a0d220bfb3361e0fe4936966ff8c4b24d65c3f06dfc70d7b680b44e7897" dependencies = [ "cc", "libc", @@ -3412,9 +3413,9 @@ checksum = "c286de4e81ea2590afc24d754e0f83810c566f50a1388fa75ebd57928c0d9745" [[package]] name = "debug-helper" -version = "0.3.13" +version = "0.3.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f578e8e2c440e7297e008bb5486a3a8a194775224bbc23729b0dbdfaeebf162e" +checksum = "80a4af69c60438a1a82af89d362f4729fd38db7b73f305a237636fad31ceb2bf" [[package]] name = "debugid" @@ -4211,7 +4212,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf592ae6a864437e98ef9c6ae7936b822077e9d038a3a48ee081ab92313afad4" dependencies = [ "num-bigint", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "swc_atoms", "swc_common", "swc_ecma_ast", @@ -4371,18 +4372,18 @@ dependencies = [ [[package]] name = "enum-ordinalize" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a1091a7bb1f8f2c4b28f1fe2cef4980ca2d410a3d727d67ecc3178c9b0800f0" +checksum = "07f808d588c10e464ea6f7d3eaed500049eff30aaac103460f61828c2d65b3eb" dependencies = [ "enum-ordinalize-derive", ] [[package]] name = "enum-ordinalize-derive" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631" +checksum = "42e528e2d34ba8a67a1a650b86beae8ef69fc5fdb638016f386b973226590432" dependencies = [ "proc-macro2", "quote", @@ -5071,9 +5072,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if", + "js-sys", "libc", "r-efi 6.0.0", "rand_core 0.10.1", + "wasm-bindgen", ] [[package]] @@ -5233,9 +5236,9 @@ dependencies = [ [[package]] name = "gosyn" -version = "0.2.10" +version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c99c1502d84229dc7ddb6af755f40ebe80e7e932fa78ecef979cedcf9999ba93" +checksum = "fed1657682b1c3f63ece1fe5b60fc6c5f5923612a20d19f6af38ce79eaf361e3" dependencies = [ "anyhow", "strum", @@ -5552,7 +5555,7 @@ dependencies = [ "hashbrown 0.14.5", "new_debug_unreachable", "once_cell", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "triomphe", ] @@ -5660,9 +5663,9 @@ dependencies = [ [[package]] name = "humantime" -version = "2.3.0" +version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" +checksum = "15cdd26707701c53297e2fa6afb323d55fbc1d0810c3aec078ae3ef0424c3c15" [[package]] name = "hyper" @@ -5712,9 +5715,9 @@ dependencies = [ [[package]] name = "hyper-http-proxy" -version = "1.1.0" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ad4b0a1e37510028bc4ba81d0e38d239c39671b0f0ce9e02dfa93a8133f7c08" +checksum = "8021e0ae20c08eadc94d0bdafdeda66d4f0858541c146ae6e46b219bfe58497e" dependencies = [ "bytes", "futures-util", @@ -5726,7 +5729,6 @@ dependencies = [ "hyper-util", "native-tls", "pin-project-lite", - "rustls-native-certs 0.7.3", "tokio", "tokio-native-tls", "tokio-rustls 0.26.4", @@ -6086,9 +6088,9 @@ dependencies = [ [[package]] name = "io-uring" -version = "0.7.12" +version = "0.7.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4d09b98f7eace8982db770e4408e7470b028ce513ac28fecdc6bf4c30fe92b62" +checksum = "9080b15e63775b9a2ac7dca720f7050a8b955e092ea0f6020a4a80f69998cdc0" dependencies = [ "bitflags 2.13.0", "cfg-if", @@ -6235,11 +6237,11 @@ dependencies = [ [[package]] name = "jobserver" -version = "0.1.34" +version = "0.1.35" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" dependencies = [ - "getrandom 0.3.4", + "getrandom 0.4.3", "libc", ] @@ -6627,14 +6629,14 @@ dependencies = [ [[package]] name = "libredox" -version = "0.1.17" +version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f02ab6bace2054fb888a3c16f990117b579d14a3088e472d63c6011fa185c9d3" +checksum = "c943259e342f1e06ff2da7a83eabdfe7f92ce10262688dbf1895ff0b3e6e4652" dependencies = [ "bitflags 2.13.0", "libc", "plain", - "redox_syscall 0.8.1", + "redox_syscall 0.9.0", ] [[package]] @@ -6715,9 +6717,9 @@ dependencies = [ [[package]] name = "log" -version = "0.4.32" +version = "0.4.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" [[package]] name = "loom" @@ -7040,15 +7042,15 @@ dependencies = [ [[package]] name = "memchr" -version = "2.8.2" +version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" [[package]] name = "memmap2" -version = "0.9.10" +version = "0.9.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3" +checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" dependencies = [ "libc", "stable_deref_trait", @@ -7246,7 +7248,7 @@ dependencies = [ "native-tls", "pem 3.0.6", "percent-encoding", - "rand 0.10.1", + "rand 0.10.2", "serde", "socket2 0.6.4", "thiserror 2.0.18", @@ -7563,9 +7565,9 @@ dependencies = [ [[package]] name = "num-bigint" -version = "0.4.6" +version = "0.4.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" dependencies = [ "num-integer", "num-traits", @@ -7626,11 +7628,10 @@ dependencies = [ [[package]] name = "num-iter" -version = "0.1.45" +version = "0.1.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf" +checksum = "c92800bd69a1eac91786bcfe9da64a897eb72911b8dc3095decbd07429e8048b" dependencies = [ - "autocfg", "num-integer", "num-traits", ] @@ -8361,9 +8362,9 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] name = "pest" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0848c601009d37dfa3430c4666e147e49cdcf1b92ecd3e63657d8a5f19da662" +checksum = "47627dd7305c6a2d6c8c6bcd24c5a4c17dbbf425f4f9c5313e724b38fc9782e9" dependencies = [ "memchr", "ucd-trie", @@ -8371,9 +8372,9 @@ dependencies = [ [[package]] name = "pest_derive" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11f486f1ea21e6c10ed15d5a7c77165d0ee443402f0780849d1768e7d9d6fe77" +checksum = "4b4254325ecad416ab689e27ba51da03ba01a9632bc6e108f5fe7c3c4ad29d58" dependencies = [ "pest", "pest_generator", @@ -8381,9 +8382,9 @@ dependencies = [ [[package]] name = "pest_generator" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8040c4647b13b210a963c1ed407c1ff4fdfa01c31d6d2a098218702e6664f94f" +checksum = "6c4c0e91ead7a8f7acecbca6f003fc2e8282b1dbe2dd9c9d2f16aba42995e0a7" dependencies = [ "pest", "pest_meta", @@ -8394,12 +8395,11 @@ dependencies = [ [[package]] name = "pest_meta" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89815c69d36021a140146f26659a81d6c2afa33d216d736dd4be5381a7362220" +checksum = "f9744bc48116fee06334924bb5f2bad41eed5e89bd26e29b0b799f9a3f82c210" dependencies = [ "pest", - "sha2 0.10.9", ] [[package]] @@ -8914,9 +8914,9 @@ dependencies = [ [[package]] name = "pulp" -version = "0.22.2" +version = "0.22.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e205bb30d5b916c55e584c22201771bcf2bad9aabd5d4127f38387140c38632" +checksum = "046aa45b989642ec2e4717c8e72d677b13edd831a4d3b6cf37d9a3e54912496a" dependencies = [ "bytemuck", "cfg-if", @@ -8931,9 +8931,9 @@ dependencies = [ [[package]] name = "pulp-wasm-simd-flag" -version = "0.1.0" +version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40e24eee682d89fb193496edf918a7f407d30175b2e785fe057e4392dfd182e0" +checksum = "1d8f70e07b9c3962945a74e59ca1c511bba65b6419468acc217c457d93f3c740" [[package]] name = "pure-rust-locales" @@ -8963,28 +8963,28 @@ dependencies = [ [[package]] name = "quick_cache" -version = "0.6.23" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a3db184a8b66cfe87f0263a1de147a6b554c864d1767c6f7fa4eb0e5497b565" +checksum = "403c1a912fec895cafb223201e368234842acb9220aaf08ab042ae89ba5f135c" dependencies = [ - "ahash 0.8.12", "equivalent", - "hashbrown 0.16.1", + "foldhash 0.2.0", + "hashbrown 0.17.1", "parking_lot", ] [[package]] name = "quinn" -version = "0.11.9" +version = "0.11.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20" +checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8" dependencies = [ "bytes", "cfg_aliases", "pin-project-lite", "quinn-proto", "quinn-udp", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "rustls 0.23.35", "socket2 0.6.4", "thiserror 2.0.18", @@ -8995,17 +8995,18 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.14" +version = "0.11.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" +checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" dependencies = [ "aws-lc-rs", "bytes", - "getrandom 0.3.4", + "getrandom 0.4.3", "lru-slab", - "rand 0.9.0", + "rand 0.10.2", + "rand_pcg", "ring 0.17.14", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "rustls 0.23.35", "rustls-pki-types", "slab", @@ -9017,23 +9018,23 @@ dependencies = [ [[package]] name = "quinn-udp" -version = "0.5.14" +version = "0.5.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd" +checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694" dependencies = [ "cfg_aliases", "libc", "once_cell", "socket2 0.6.4", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] name = "quote" -version = "1.0.45" +version = "1.0.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" dependencies = [ "proc-macro2", ] @@ -9093,9 +9094,9 @@ dependencies = [ [[package]] name = "rand" -version = "0.10.1" +version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" dependencies = [ "chacha20", "getrandom 0.4.3", @@ -9184,6 +9185,15 @@ dependencies = [ "rand_core 0.5.1", ] +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + [[package]] name = "raw-cpuid" version = "11.6.0" @@ -9309,9 +9319,9 @@ dependencies = [ [[package]] name = "redox_syscall" -version = "0.8.1" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b44b894f2a6e36457d665d1e08c3866add6ed5e70050c1b4ba8a8ddedb02ce7" +checksum = "c5102a6aaa05aa011a238e178e6bca86d2cb56fc9f586d37cb80f5bca6e07759" dependencies = [ "bitflags 2.13.0", ] @@ -9559,7 +9569,7 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc05fbf560421a0357a750cbe78c7ca19d4923918490daabba313d5dbc871e47" dependencies = [ - "rand 0.10.1", + "rand 0.10.2", ] [[package]] @@ -9831,9 +9841,9 @@ dependencies = [ [[package]] name = "rustc-demangle" -version = "0.1.27" +version = "0.1.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b50b8869d9fc858ce7266cce0194bd74df58b9d0e3f6df3a9fc8eb470d95c09d" +checksum = "b74b56ffa8bb2830709a538c2cbcae9aa062db0d2a42563bfb09bdaae44020eb" [[package]] name = "rustc-hash" @@ -9843,9 +9853,9 @@ checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" [[package]] name = "rustc-hash" -version = "2.1.2" +version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" [[package]] name = "rustc_version" @@ -9999,9 +10009,9 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.14.1" +version = "1.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9" +checksum = "764899a24af3980067ee14bc143654f297b22eaebfe3c7b6b211920a5a59b046" dependencies = [ "web-time", "zeroize", @@ -10138,9 +10148,9 @@ dependencies = [ [[package]] name = "rustversion" -version = "1.0.22" +version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" [[package]] name = "ryu" @@ -10889,7 +10899,7 @@ dependencies = [ "data-encoding", "debugid", "if_chain", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "serde_json", "unicode-id-start", @@ -11180,9 +11190,9 @@ dependencies = [ [[package]] name = "sse-stream" -version = "0.2.3" +version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3962b63f038885f15bce2c6e02c0e7925c072f1ac86bb60fd44c5c6b762fb72" +checksum = "39f24a9b78c40b90817bbcd1821c74ddfd74916aadd29403d001532a9195532d" dependencies = [ "bytes", "futures-util", @@ -11322,7 +11332,7 @@ dependencies = [ "allocator-api2", "bumpalo", "hashbrown 0.14.5", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", ] [[package]] @@ -11351,7 +11361,7 @@ dependencies = [ "new_debug_unreachable", "num-bigint", "once_cell", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "siphasher 0.3.11", "swc_atoms", @@ -11400,7 +11410,7 @@ dependencies = [ "num-bigint", "once_cell", "phf 0.11.3", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "string_enum", "swc_atoms", @@ -11421,7 +11431,7 @@ dependencies = [ "num-bigint", "once_cell", "regex", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "ryu-js", "serde", "swc_allocator", @@ -11455,7 +11465,7 @@ dependencies = [ "either", "num-bigint", "phf 0.11.3", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "seq-macro", "serde", "smallvec", @@ -11475,7 +11485,7 @@ checksum = "c675d14700c92f12585049b22b02356f1e142f4b0c32a4d0eb4b7a968a4c0c1e" dependencies = [ "anyhow", "pathdiff", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "swc_atoms", "swc_common", @@ -11509,7 +11519,7 @@ dependencies = [ "once_cell", "par-core", "phf 0.11.3", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "swc_atoms", "swc_common", @@ -11552,7 +11562,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39b3b34f6a28348416174912009d09994ab71c867682ec78d641a9feb3a96b4e" dependencies = [ "either", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "swc_atoms", "swc_common", @@ -11573,7 +11583,7 @@ dependencies = [ "bytes-str", "indexmap 2.14.0", "once_cell", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "sha1", "string_enum", @@ -11594,7 +11604,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3872c006ccfdcc19f1cf5c01c15915a69964ba7982c9f581cdb7e727e77b9a2c" dependencies = [ "bytes-str", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "swc_atoms", "swc_common", @@ -11615,7 +11625,7 @@ dependencies = [ "num_cpus", "once_cell", "par-core", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "ryu-js", "swc_atoms", "swc_common", @@ -11673,7 +11683,7 @@ dependencies = [ "data-encoding", "debugid", "if_chain", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "serde_json", "unicode-id-start", @@ -11870,7 +11880,7 @@ dependencies = [ "rayon", "regex", "rust-stemmers", - "rustc-hash 2.1.2", + "rustc-hash 2.1.3", "serde", "serde_json", "sketches-ddsketch", @@ -12189,9 +12199,9 @@ dependencies = [ [[package]] name = "time" -version = "0.3.49" +version = "0.3.53" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "711a53c2d47bbd818258c498c8dbfe186a2526c631495cfe7e078567f86b8469" +checksum = "18dfaaeddcb932337b5e7866ee7d0ce9b76d2fd092997146f187ec09b4558a50" dependencies = [ "deranged", "num-conv", @@ -12209,9 +12219,9 @@ checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" [[package]] name = "time-macros" -version = "0.2.29" +version = "0.2.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71c652a3727a9cbb9a02f707f530b618ce00d0ccd762009c8c23bd191df3c17d" +checksum = "c431b87111666e491a90baa837f914fb45cd5dc3c268591b0220ff5057f2085f" dependencies = [ "num-conv", "time-core", @@ -12923,9 +12933,9 @@ dependencies = [ [[package]] name = "triomphe" -version = "0.1.15" +version = "0.1.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dd69c5aa8f924c7519d6372789a74eac5b94fb0f8fcf0d4a97eb0bfc3e785f39" +checksum = "b40688ea6389c8171614b25491f71d4a27946e0c7ce2da1c6de27e25abf1a0ae" dependencies = [ "serde", "stable_deref_trait", @@ -13322,9 +13332,9 @@ checksum = "7fcfc827f90e53a02eaef5e535ee14266c1d569214c6aa70133a624d8a3164ba" [[package]] name = "utf8-width" -version = "0.1.8" +version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1292c0d970b54115d14f2492fe0170adf21d68a1de108eebc51c1df4f346a091" +checksum = "159a7cadce548703edd50d24069bc294c5415ecab0a480e0cd1ca06d112dc94a" [[package]] name = "utf8_iter" @@ -13340,9 +13350,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.23.3" +version = "1.23.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" +checksum = "bf80a72845275afea99e7f2b434723d3bc7e38470fcd1c7ed39a599c73319a53" dependencies = [ "getrandom 0.4.3", "js-sys", @@ -13735,7 +13745,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-nats", @@ -13817,7 +13827,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.733.1" +version = "1.753.0" dependencies = [ "async-stream", "async-trait", @@ -13850,7 +13860,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13863,7 +13873,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "argon2", @@ -14001,7 +14011,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14024,20 +14034,22 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", "serde", "serde_json", "sqlx", + "tracing", "windmill-api-auth", "windmill-common", + "windmill-parser-sql-asset", ] [[package]] name = "windmill-api-auth" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14063,7 +14075,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.733.1" +version = "1.753.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14073,7 +14085,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14090,7 +14102,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14112,7 +14124,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14135,7 +14147,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14151,7 +14163,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14172,7 +14184,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14193,7 +14205,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14207,7 +14219,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-nats", @@ -14242,7 +14254,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14267,7 +14279,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14285,7 +14297,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14307,7 +14319,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14327,7 +14339,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14354,6 +14366,7 @@ dependencies = [ "windmill-parser", "windmill-parser-py", "windmill-parser-py-asset", + "windmill-parser-sql", "windmill-parser-sql-asset", "windmill-parser-ts", "windmill-parser-ts-asset", @@ -14363,7 +14376,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14391,7 +14404,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.733.1" +version = "1.753.0" dependencies = [ "lazy_static", "serde", @@ -14403,7 +14416,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.733.1" +version = "1.753.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14428,7 +14441,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14442,10 +14455,11 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.733.1" +version = "1.753.0" dependencies = [ "axum 0.8.9", "chrono", + "futures", "hex", "http 1.4.2", "hyper 1.10.1", @@ -14458,6 +14472,7 @@ dependencies = [ "sqlx", "strum", "tokio", + "tokio-postgres", "tracing", "uuid", "windmill-api-auth", @@ -14475,7 +14490,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.733.1" +version = "1.753.0" dependencies = [ "chrono", "lazy_static", @@ -14489,7 +14504,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14508,7 +14523,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.733.1" +version = "1.753.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14610,7 +14625,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.733.1" +version = "1.753.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14629,7 +14644,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.733.1" +version = "1.753.0" dependencies = [ "regex", "serde", @@ -14644,7 +14659,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14668,7 +14683,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "futures", @@ -14685,7 +14700,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.733.1" +version = "1.753.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14701,7 +14716,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -14722,7 +14737,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -14753,7 +14768,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "arc-swap", @@ -14778,7 +14793,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-stream", @@ -14812,7 +14827,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "futures", @@ -14830,7 +14845,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.733.1" +version = "1.753.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14839,7 +14854,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -14851,7 +14866,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde_json", @@ -14863,7 +14878,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "gosyn", @@ -14875,7 +14890,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -14887,7 +14902,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde_json", @@ -14899,7 +14914,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "nu-parser", @@ -14910,7 +14925,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14921,7 +14936,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14933,7 +14948,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14944,7 +14959,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-recursion", @@ -14966,7 +14981,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde_json", @@ -14978,7 +14993,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -14992,7 +15007,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15009,7 +15024,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -15022,7 +15037,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde", @@ -15034,7 +15049,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -15052,7 +15067,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15068,7 +15083,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15084,7 +15099,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde", @@ -15095,7 +15110,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-recursion", @@ -15129,11 +15144,12 @@ dependencies = [ "uuid", "windmill-audit", "windmill-common", + "windmill-jseval", ] [[package]] name = "windmill-runtime-nativets" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "const_format", @@ -15172,7 +15188,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.733.1" +version = "1.753.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15183,17 +15199,19 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-recursion", "axum 0.8.9", + "base64 0.22.1", "chrono", "futures", "hex", "http 1.4.2", "hyper 1.10.1", "lazy_static", + "magic-crypt", "quick_cache", "reqwest 0.13.1", "serde", @@ -15215,7 +15233,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15239,7 +15257,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15272,7 +15290,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15305,7 +15323,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15325,7 +15343,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15359,7 +15377,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15395,7 +15413,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15418,7 +15436,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15442,7 +15460,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-nats", @@ -15466,7 +15484,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15501,7 +15519,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15529,7 +15547,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-trait", @@ -15554,7 +15572,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15573,7 +15591,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-once-cell", @@ -15683,7 +15701,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.733.1" +version = "1.753.0" dependencies = [ "bytes", "futures", @@ -16409,18 +16427,18 @@ dependencies = [ [[package]] name = "zerocopy" -version = "0.8.52" +version = "0.8.53" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce1022995ff5ff5d841ad7d994facc23098cd40152f2c1d11cd607c6f530653f" +checksum = "75726053136156d419e285b9b7eddaaea9e3fea6ce32eed44a89901f0bd98de1" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.52" +version = "0.8.53" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930" +checksum = "4714fd92cf900833d49538023a9b3915155210801d1c1169eba513b2addefd71" dependencies = [ "proc-macro2", "quote", @@ -16501,9 +16519,9 @@ dependencies = [ [[package]] name = "zlib-rs" -version = "0.6.3" +version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" +checksum = "5431d5661c32445236631278f27946e444ddafe4684cac70b185272d4f9c52d5" [[package]] name = "zmij" diff --git a/backend/Cargo.toml b/backend/Cargo.toml index 5dd2d8f76d..12ec704c32 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.733.1" +version = "1.753.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.733.1" +version = "1.753.0" authors = ["Ruben Fiszel "] edition = "2021" @@ -112,7 +112,7 @@ strip = "none" default = [] private = ["windmill-api/private", "windmill-api-agent-workers?/private", "windmill-autoscaling/private", "windmill-common/private", "windmill-dep-map/private", "windmill-object-store/private", "windmill-git-sync/private", "windmill-indexer/private", "windmill-operator?/private", "windmill-queue/private", "windmill-worker/private", "windmill-test-utils/private"] agent_worker_server = ["windmill-api/agent_worker_server", "dep:windmill-api-agent-workers", "windmill-test-utils/agent_worker_server"] -enterprise = ["windmill-worker/enterprise", "windmill-queue/enterprise", "windmill-api/enterprise", "windmill-api-agent-workers?/enterprise", "dep:windmill-autoscaling", "windmill-autoscaling/enterprise", "windmill-git-sync/enterprise", "windmill-common/prometheus", "windmill-common/enterprise", "windmill-object-store/enterprise"] +enterprise = ["windmill-worker/enterprise", "windmill-queue/enterprise", "windmill-api/enterprise", "windmill-api-agent-workers?/enterprise", "dep:windmill-autoscaling", "windmill-autoscaling/enterprise", "windmill-git-sync/enterprise", "windmill-common/prometheus", "windmill-common/enterprise", "windmill-object-store/enterprise", "license"] local_reports = ["windmill-common/local_reports"] enterprise_saml = ["windmill-api/enterprise_saml", "oauth2"] stripe = ["windmill-api/stripe"] @@ -121,6 +121,7 @@ embedding = ["windmill-api/embedding"] parquet = ["windmill-api/parquet", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker/parquet"] prometheus = ["windmill-common/prometheus", "windmill-api/prometheus", "windmill-worker/prometheus", "windmill-queue/prometheus", "dep:prometheus"] flow_testing = ["windmill-worker/flow_testing"] +failpoints = ["windmill-worker/failpoints", "windmill-queue/failpoints"] quickjs = ["windmill-worker/quickjs", "windmill-api/quickjs"] openidconnect = ["windmill-api/openidconnect", "windmill-common/openidconnect", "windmill-object-store/openidconnect"] cloud = ["windmill-queue/cloud", "windmill-worker/cloud", "windmill-common/cloud", "windmill-api/cloud"] diff --git a/backend/THREAT_MODEL.md b/backend/THREAT_MODEL.md index dc98a5a625..9501b5bf0d 100644 --- a/backend/THREAT_MODEL.md +++ b/backend/THREAT_MODEL.md @@ -88,7 +88,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | EP12 Stored-content rendering | App builder HTML component, markdown, S3 download response headers | stored user content → admin browser (same origin) | Admin session, account takeover | | EP13 Log/file reading & export endpoints | `service_logs`, `jobs_u/getupdate` log file read (symlinks), workspace/tarball export | authed/unauth request → arbitrary file or admin-only config | Arbitrary files, global settings | | EP14 Secret-value & resource-value caches | In-memory caches in `windmill-store` keyed (historically un-keyed) by path | cache lookup crossing identity/folder boundary | Secret variables, resource creds | -| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS=false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | +| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS` now defaults to `true`; can still be overridden to `false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | | EP16 Supply chain | Cached hub scripts, GitHub workflow actions, vendored deps, Docker base image | build/update-time input → host & build integrity | Worker host, build integrity | | EP17 Token lifecycle | Token create/rescope/refresh, script-issued JWTs | scoped caller → broader privilege | Tokens, accounts, isolation | @@ -103,7 +103,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | T5 | Worker compromise & cross-tenant access via weak-by-default isolation (nsjail off by default → user code runs with only PID-ns `unshare`); sandbox escape where nsjail/dind/podman is enabled | remote_auth | EP9, EP15 | Worker host, isolation, downstream | critical | likely | unmitigated | nsjail off by default everywhere (`DISABLE_NSJAIL=true`); shipped compose gives PID-ns `unshare` only (`FAVOR_UNSHARE_PID=true`), bare installs get no isolation. Where nsjail enabled: read-only remounts, jail-tmp refusal, podman socket gating | GHSA-6qr8-xhg4-453q, GHSA-3vpp-vf62-wqp6, f8467f38c8, df5aec0f5d, f1b6746e0e | | T6 | Disclosure of secrets, resource credentials, and workspace encryption keys across the authorization boundary (AI proxy, MCP, caches, export); database read additionally yields plaintext instance-level `global_settings` secrets | remote_auth | EP6, EP14, EP13 | Secret variables, encryption keys, resource creds, global settings | critical | likely | partially_mitigated | RLS on `$var:`, cache scoping by caller, admin checks on export; per-workspace secret *variables* encrypted at rest, but `global_settings` is plaintext under the default DB secret backend | GHSA-jwg4-v3cj-rvfm, GHSA-8m2p-2crh-9h3w, GHSA-6635-6fch-v8px, GHSA-437f-725p-7w84, GHSA-f27g-j463-q85w (CVE-2026-26964), GHSA-j679-v6vj-jfxc, GHSA-6vrr-fq33-qpfp, 0ba128afe7, 7836a4e733, ff8e39c69b | | T7 | Full instance compromise from insecure deployment defaults (dind control, default admin/`changeme`, exposed Postgres, publicly readable SUPERADMIN_SECRET) | remote_unauth | EP15 | All assets | critical | likely | partially_mitigated | first-time-setup warning on default admin; docs recommend hardening | GHSA-3vpp-vf62-wqp6, GHSA-24fr-44f8-fqwg (CVE-2026-29059), GHSA-6q36-5p3h-766j | -| T8 | Unauthenticated RCE via the Debugger WebSocket in the default `windmill_extra` configuration | remote_unauth | EP15 | Worker host, all assets | critical | possible | unmitigated | `REQUIRE_SIGNED_DEBUG_REQUESTS` exists but defaults to false | GHSA-725h-99vx-9xr4 | +| T8 | Unauthenticated RCE via the Debugger WebSocket: `/ws_debug/*` exposed by the gateway/ingress with the debugger service as the auth boundary; signature gate was bypassable via `program`-mode launches (read+exec an arbitrary server-side file path, never signed) even with signing on, and the WS handshake had no Origin check (CSWSH) | remote_unauth | EP15 | Worker host, all assets | critical | possible | partially_mitigated | `program`-mode launches now rejected when `REQUIRE_SIGNED_DEBUG_REQUESTS` is on (signing covers every launch, not just inline `code`); shipped `docker-compose` now defaults `REQUIRE_SIGNED_DEBUG_REQUESTS=true`; opt-in `DEBUG_ALLOWED_ORIGINS` allowlist rejects cross-origin handshakes. Residual: code default is secure but operators can still set `=false`; origin allowlist is opt-in | GHSA-725h-99vx-9xr4 | | T9 | Supply-chain compromise via cached hub scripts, GitHub workflow command injection, or vulnerable base-image deps | supply_chain | EP16 | Worker host, build integrity | critical | possible | partially_mitigated | hub-script re-pin to patched versions; HUB_BASE_URL override | GHSA-w2m9-q5f7-3gpq, edf340c4d4, GHSA-8rq7-w7g6-8wvr, GHSA-vch9-39v5-4wg7 (CVE-2024-37371) | | T10 | Unauthenticated disclosure of job results, args, logs, and admin config via missing-authz public endpoints | remote_unauth | EP2, EP13 | Job results/args/logs, global settings, scripts | high | likely | partially_mitigated | anonymous-job checks, log-endpoint authz hardening | GHSA-qfg7-x243-5hg4, GHSA-v448-fmm4-52fp, 108a88a180, bb90f4ce83 | | T11 | Stored XSS leading to admin/account takeover via app HTML component, markdown, or S3 download content-type | remote_auth | EP12 | Admin session, accounts | high | likely | partially_mitigated | DOMPurify markdown sanitization, `X-Content-Type-Options: nosniff` + CSP sandbox on downloads | GHSA-9c5c-hh3c-r9mc, GHSA-qxj7-hpx3-r892, GHSA-cf2x-rg8c-v63v, bb78b1c06d, 625b67dff0 | diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index fc292591e6..3975aa3922 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -b2712bc1411b29ec1c95466831bf4c7982447fd8 \ No newline at end of file +a61da8ad2de819f6675314efa59371105ad974cb \ No newline at end of file diff --git a/backend/migrations/20260617081932_datatable_migrations.down.sql b/backend/migrations/20260617081932_datatable_migrations.down.sql new file mode 100644 index 0000000000..26efc2bdcc --- /dev/null +++ b/backend/migrations/20260617081932_datatable_migrations.down.sql @@ -0,0 +1 @@ +DROP TABLE datatable_migrations; diff --git a/backend/migrations/20260617081932_datatable_migrations.up.sql b/backend/migrations/20260617081932_datatable_migrations.up.sql new file mode 100644 index 0000000000..72c7dc5ec6 --- /dev/null +++ b/backend/migrations/20260617081932_datatable_migrations.up.sql @@ -0,0 +1,21 @@ +-- SQL migrations defined per data table within a workspace. +-- `datatable` is the target data table name, `name` is the migration name +-- (e.g. add_index_to_customers), and `timestamp` is the migration version +-- (YYYYMMDDHHMMSS), recorded as `version` in the data table's `_wm_migrations` +-- table once applied. +CREATE TABLE datatable_migrations ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE, + datatable VARCHAR(255) NOT NULL, + timestamp BIGINT NOT NULL, + name VARCHAR(255) NOT NULL, + code_up TEXT NOT NULL, + code_down TEXT, + PRIMARY KEY (workspace_id, datatable, timestamp) +); + +-- No standalone index: the (workspace_id, datatable, timestamp) primary-key btree +-- already serves both `WHERE workspace_id = $1` and `WHERE workspace_id = $1 AND +-- datatable = $2` lookups via its leading columns. + +GRANT ALL ON datatable_migrations TO windmill_user; +GRANT ALL ON datatable_migrations TO windmill_admin; diff --git a/backend/migrations/20260617144417_add_ai_skill.down.sql b/backend/migrations/20260617144417_add_ai_skill.down.sql new file mode 100644 index 0000000000..8fd6e1606d --- /dev/null +++ b/backend/migrations/20260617144417_add_ai_skill.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS ai_skill; diff --git a/backend/migrations/20260617144417_add_ai_skill.up.sql b/backend/migrations/20260617144417_add_ai_skill.up.sql new file mode 100644 index 0000000000..b9a5003572 --- /dev/null +++ b/backend/migrations/20260617144417_add_ai_skill.up.sql @@ -0,0 +1,15 @@ +-- Workspace-scoped AI chat skills (Claude/Codex-style SKILL.md instructions). +-- `name` is the skill folder slug; `description` is advertised in the AI chat +-- system prompt, `instructions` is the SKILL.md body fetched on demand. +CREATE TABLE ai_skill ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE, + name VARCHAR(255) NOT NULL, + description TEXT NOT NULL, + instructions TEXT NOT NULL, + edited_at TIMESTAMPTZ NOT NULL DEFAULT now(), + edited_by VARCHAR(255) NOT NULL DEFAULT '', + PRIMARY KEY (workspace_id, name) +); + +GRANT ALL ON ai_skill TO windmill_user; +GRANT ALL ON ai_skill TO windmill_admin; diff --git a/backend/migrations/20260619150718_native_retry_settings.down.sql b/backend/migrations/20260619150718_native_retry_settings.down.sql new file mode 100644 index 0000000000..459ebe4eaf --- /dev/null +++ b/backend/migrations/20260619150718_native_retry_settings.down.sql @@ -0,0 +1,4 @@ +ALTER TABLE runnable_settings +DROP COLUMN IF EXISTS retry_settings; + +DROP TABLE IF EXISTS retry_settings; diff --git a/backend/migrations/20260619150718_native_retry_settings.up.sql b/backend/migrations/20260619150718_native_retry_settings.up.sql new file mode 100644 index 0000000000..e03192c710 --- /dev/null +++ b/backend/migrations/20260619150718_native_retry_settings.up.sql @@ -0,0 +1,16 @@ +CREATE TABLE IF NOT EXISTS retry_settings( + hash BIGINT PRIMARY KEY, + constant_attempts INTEGER, + constant_seconds INTEGER, + exponential_attempts INTEGER, + exponential_multiplier INTEGER, + exponential_seconds INTEGER, + exponential_random_factor INTEGER, + retry_if_expr TEXT +); + +ALTER TABLE runnable_settings +ADD COLUMN IF NOT EXISTS retry_settings BIGINT DEFAULT NULL; + +GRANT ALL ON retry_settings TO windmill_admin; +GRANT ALL ON retry_settings TO windmill_user; diff --git a/backend/migrations/20260619170118_add_materialized_partition.down.sql b/backend/migrations/20260619170118_add_materialized_partition.down.sql new file mode 100644 index 0000000000..4932338956 --- /dev/null +++ b/backend/migrations/20260619170118_add_materialized_partition.down.sql @@ -0,0 +1,3 @@ +DROP INDEX IF EXISTS idx_materialized_partition_asset_status; +DROP TABLE IF EXISTS materialized_partition; +DROP TYPE IF EXISTS MATERIALIZATION_STATUS; diff --git a/backend/migrations/20260619170118_add_materialized_partition.up.sql b/backend/migrations/20260619170118_add_materialized_partition.up.sql new file mode 100644 index 0000000000..da5f806e4f --- /dev/null +++ b/backend/migrations/20260619170118_add_materialized_partition.up.sql @@ -0,0 +1,29 @@ +-- Per-partition materialization state for managed `// materialize` assets. +-- One row per (asset, partition): the latest materialization of that slice. +-- Drives: the partition-status grid (CE observability), run-stale/gap +-- detection, and the EE backfill worklist (missing/failed partitions). The +-- `partition` column uses '' as the sentinel for an unpartitioned (whole-table) +-- materialization, since partition is part of the primary key and cannot be +-- NULL. +CREATE TYPE MATERIALIZATION_STATUS AS ENUM ('running', 'materialized', 'failed'); + +CREATE TABLE IF NOT EXISTS materialized_partition ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + asset_kind ASSET_KIND NOT NULL, + asset_path VARCHAR(255) NOT NULL, + partition TEXT NOT NULL DEFAULT '', + status MATERIALIZATION_STATUS NOT NULL, + -- DuckLake snapshot id produced by the write; NULL while running / on + -- failure. The pin that makes downstream reads reproducible. + snapshot_id BIGINT, + row_count BIGINT, + job_id UUID, + materialized_at TIMESTAMPTZ NOT NULL DEFAULT now(), + error TEXT, + PRIMARY KEY (workspace_id, asset_kind, asset_path, partition) +); + +-- Backfill enumeration / grid "show only gaps": filter an asset's partitions +-- by status without scanning the whole table. +CREATE INDEX IF NOT EXISTS idx_materialized_partition_asset_status + ON materialized_partition (workspace_id, asset_kind, asset_path, status); diff --git a/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql b/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql new file mode 100644 index 0000000000..3d532de9cb --- /dev/null +++ b/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql @@ -0,0 +1,3 @@ +-- No-op: this migration only re-pins the function's search_path. Reverting would +-- mean restoring the hardcoded `SET search_path = public`, which is the very bug +-- this repairs, so there is nothing to undo. diff --git a/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql b/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql new file mode 100644 index 0000000000..f8e448891f --- /dev/null +++ b/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql @@ -0,0 +1,22 @@ +-- Repair instances that already applied the folder-labels migrations while the +-- function hardcoded `SET search_path = public`. On a non-public schema (PG_SCHEMA) +-- the function was pinned to `public`, so at runtime it read the wrong `folder` +-- table (or a stray public.folder) instead of the workspace's real one. +-- +-- `FROM CURRENT` snapshots the migration connection's search_path (the actual +-- Windmill schema) into the function, keeping the SECURITY DEFINER injection +-- hardening. On public-schema installs this re-pins to `public`, i.e. a no-op. +-- Idempotent: redefining with the same body is harmless on already-correct installs. +CREATE OR REPLACE FUNCTION folder_labels(w_id text, item_path text) RETURNS text[] +LANGUAGE sql STABLE SECURITY DEFINER SET search_path FROM CURRENT AS $$ + SELECT ( + SELECT array_agg(l ORDER BY first_ord) + FROM ( + SELECT u.l, min(u.ord) AS first_ord + FROM unnest(f.labels) WITH ORDINALITY AS u(l, ord) + GROUP BY u.l + ) deduped + ) + FROM folder f + WHERE f.workspace_id = w_id AND item_path LIKE 'f/%' AND f.name = split_part(item_path, '/', 2) +$$; diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql new file mode 100644 index 0000000000..3c69d46fc8 --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql @@ -0,0 +1,3 @@ +-- Irreversible data backfill: once a raw app's draft is retyped to 'raw_app' it +-- is indistinguishable from one saved as 'raw_app' by the per-kind code, so the +-- original typ='app' state cannot be reconstructed. No-op on revert. diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql new file mode 100644 index 0000000000..d4b068a80c --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql @@ -0,0 +1,35 @@ +-- The pre-per-user `DRAFT_TYPE` enum had only ('script','flow','app'): a raw +-- app's draft was therefore stored as typ='app'. The new model splits app vs +-- raw_app into distinct draft kinds chosen from the deployed app's `raw_app` +-- flag, so a raw app's pre-migration draft is invisible to the per-kind lookups +-- (editor overlay, migrate-legacy, get-for-user), which all query typ='raw_app'. +-- Realign every such draft (any owner, including the legacy NULL-email row) to +-- 'raw_app' when the deployed app at that path is a raw app. + +-- Drop, don't retype, a stale 'app' row when a 'raw_app' draft already exists +-- for the same owner (the newer 'raw_app' row, saved with the per-kind code, is +-- authoritative) — retyping would collide on the draft_pkey_with_user / +-- draft_pkey_legacy partial unique indexes over (workspace_id, path, typ, email). +DELETE FROM draft d +USING app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE + AND EXISTS ( + SELECT 1 FROM draft d2 + WHERE d2.workspace_id = d.workspace_id + AND d2.path = d.path + AND d2.typ = 'raw_app' + AND d2.email IS NOT DISTINCT FROM d.email + ); + +UPDATE draft d +SET typ = 'raw_app' +FROM app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE; diff --git a/backend/migrations/20260624161218_dev_workspace.down.sql b/backend/migrations/20260624161218_dev_workspace.down.sql new file mode 100644 index 0000000000..45b8e65f67 --- /dev/null +++ b/backend/migrations/20260624161218_dev_workspace.down.sql @@ -0,0 +1,3 @@ +ALTER TABLE workspace DROP CONSTRAINT IF EXISTS workspace_dev_requires_parent; +DROP INDEX IF EXISTS workspace_canonical_dev_idx; +ALTER TABLE workspace DROP COLUMN is_dev_workspace; diff --git a/backend/migrations/20260624161218_dev_workspace.up.sql b/backend/migrations/20260624161218_dev_workspace.up.sql new file mode 100644 index 0000000000..a0a252eb7c --- /dev/null +++ b/backend/migrations/20260624161218_dev_workspace.up.sql @@ -0,0 +1,14 @@ +-- A dev workspace is a fork (parent_workspace_id set) that is the standing editable +-- environment paired with its parent ("prod"), as opposed to a throwaway fork. +ALTER TABLE workspace ADD COLUMN is_dev_workspace BOOLEAN NOT NULL DEFAULT false; + +-- At most one active canonical dev workspace per parent (one editable source per prod). +-- Excludes soft-deleted (archived) workspaces so a new dev can replace an archived one. +CREATE UNIQUE INDEX workspace_canonical_dev_idx ON workspace (parent_workspace_id) + WHERE is_dev_workspace AND deleted = false; + +-- A dev workspace is a fork, so it must have a parent. Enforce the invariant at the schema level so +-- no path (or manual write) can persist a "root dev workspace". No backfill needed: the column is +-- added above with default false, so no existing row can violate this at creation time. +ALTER TABLE workspace ADD CONSTRAINT workspace_dev_requires_parent + CHECK (NOT is_dev_workspace OR parent_workspace_id IS NOT NULL); diff --git a/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql b/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql new file mode 100644 index 0000000000..0a09f0f42a --- /dev/null +++ b/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS native_retry_attempt; diff --git a/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql b/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql new file mode 100644 index 0000000000..e193fb8b8f --- /dev/null +++ b/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql @@ -0,0 +1,18 @@ +-- Sparse marker: one row per native retry attempt (a re-pushed Script job that +-- gained a retry). Presence = "this job is a native retry attempt"; `attempt` is +-- the chain position (1-based). Lets consumers — asset-cascade dispatch, the +-- per-occurrence schedule-handler counting, and the run-page chain — identify +-- retries explicitly instead of inferring from incidental fields (parent_job + +-- runnable + flow_innermost), which collides with schedule handlers and WAC +-- inline children. Sparse: only failed-and-retried jobs under a retry policy +-- produce rows. Lifecycle: removed with their job in the retention sweep (no FK, +-- to keep the bulk job delete cheap). +CREATE TABLE IF NOT EXISTS native_retry_attempt ( + job_id UUID PRIMARY KEY, + -- `integer` matches the retry policy's i32 attempt count; avoids any narrowing + -- on the maybe_enqueue read/write path. + attempt INTEGER NOT NULL +); + +GRANT ALL ON native_retry_attempt TO windmill_admin; +GRANT ALL ON native_retry_attempt TO windmill_user; diff --git a/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql new file mode 100644 index 0000000000..9399c18a6b --- /dev/null +++ b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql @@ -0,0 +1,16 @@ +-- Re-establish the ON DELETE CASCADE foreign keys. Once the cascades were gone the explicit +-- delete paths may have left orphan rows (or none were created); purge any orphans first so +-- the constraints can be validated. +DELETE FROM dispatch_event WHERE producer_job_id NOT IN (SELECT id FROM v2_job); +DELETE FROM flow_conversation_message WHERE job_id IS NOT NULL AND job_id NOT IN (SELECT id FROM v2_job); +DELETE FROM zombie_job_counter WHERE job_id NOT IN (SELECT id FROM v2_job); + +ALTER TABLE dispatch_event + ADD CONSTRAINT dispatch_event_producer_job_id_fkey + FOREIGN KEY (producer_job_id) REFERENCES v2_job(id) ON DELETE CASCADE; +ALTER TABLE flow_conversation_message + ADD CONSTRAINT flow_conversation_message_job_id_fkey + FOREIGN KEY (job_id) REFERENCES v2_job(id) ON DELETE CASCADE; +ALTER TABLE zombie_job_counter + ADD CONSTRAINT zombie_job_counter_job_id_fkey + FOREIGN KEY (job_id) REFERENCES v2_job(id) ON DELETE CASCADE; diff --git a/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql new file mode 100644 index 0000000000..ebaa555b0e --- /dev/null +++ b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql @@ -0,0 +1,11 @@ +-- Drop the ON DELETE CASCADE foreign keys from v2_job's sparse side tables. +-- These cascades made bulk retention deletes (DELETE FROM v2_job WHERE id = ANY(...)) +-- fire a per-row RI trigger for each FK; for flow_conversation_message, whose job_id +-- column is unindexed, that meant a sequential scan of the whole table per deleted row +-- (benchmarked at ~14x the base delete time). Deletion of these tables is now handled +-- explicitly by windmill_common::jobs::delete_jobs and the workspace/export delete paths, +-- following the existing no-FK precedent of job_logs / job_stats / native_retry_attempt. + +ALTER TABLE dispatch_event DROP CONSTRAINT IF EXISTS dispatch_event_producer_job_id_fkey; +ALTER TABLE flow_conversation_message DROP CONSTRAINT IF EXISTS flow_conversation_message_job_id_fkey; +ALTER TABLE zombie_job_counter DROP CONSTRAINT IF EXISTS zombie_job_counter_job_id_fkey; diff --git a/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql b/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql new file mode 100644 index 0000000000..0e90a4b693 --- /dev/null +++ b/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql @@ -0,0 +1 @@ +DROP INDEX IF EXISTS ix_resume_job_flow; diff --git a/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql b/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql new file mode 100644 index 0000000000..7edded95e3 --- /dev/null +++ b/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql @@ -0,0 +1,5 @@ +-- Index resume_job's FK column to v2_job_queue. Without it, every per-job-completion +-- DELETE FROM v2_job_queue (the system's hottest delete path) cascades into a sequential +-- scan of resume_job. The table is small (only currently-suspended flows), so the scan is +-- cheap today, but the index makes the cascade an index probe and removes the footgun. +CREATE INDEX IF NOT EXISTS ix_resume_job_flow ON resume_job (flow); diff --git a/backend/migrations/20260625135355_debounce_batch_consumed.down.sql b/backend/migrations/20260625135355_debounce_batch_consumed.down.sql new file mode 100644 index 0000000000..8ce64f87ac --- /dev/null +++ b/backend/migrations/20260625135355_debounce_batch_consumed.down.sql @@ -0,0 +1,4 @@ +DROP INDEX IF EXISTS idx_v2_job_debounce_batch_consumed_at; +ALTER TABLE v2_job_debounce_batch + DROP COLUMN IF EXISTS consumed_at, + DROP COLUMN IF EXISTS consumed_by; diff --git a/backend/migrations/20260625135355_debounce_batch_consumed.up.sql b/backend/migrations/20260625135355_debounce_batch_consumed.up.sql new file mode 100644 index 0000000000..2559cbb74e --- /dev/null +++ b/backend/migrations/20260625135355_debounce_batch_consumed.up.sql @@ -0,0 +1,17 @@ +-- Claim-based, exactly-once consumption of debounce batches. +-- A batch row is "claimed" by the survivor that accumulates its args. Stamping the +-- row consumed (instead of deleting it) lets a later-pulled survivor of the same +-- batch tell "my contribution was already processed" (consumed_at set -> no-op) +-- apart from "I was never batched" (no row at all; CE / legacy -> run my own args). +-- consumed_at doubles as the GC timestamp. NULL = not yet consumed. +-- consumed_by records which job claimed the row, so a job that is re-pulled (e.g. +-- crash recovery) can tell its own prior claim (keep its accumulated args) apart from +-- a sibling survivor having swept it in (run empty). +ALTER TABLE v2_job_debounce_batch + ADD COLUMN IF NOT EXISTS consumed_at TIMESTAMP WITH TIME ZONE, + ADD COLUMN IF NOT EXISTS consumed_by UUID; + +-- Keeps the GC sweep (delete consumed rows past a grace period) cheap. +CREATE INDEX IF NOT EXISTS idx_v2_job_debounce_batch_consumed_at + ON v2_job_debounce_batch (consumed_at) + WHERE consumed_at IS NOT NULL; diff --git a/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql b/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql new file mode 100644 index 0000000000..8a653a9842 --- /dev/null +++ b/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS materialized_asset_schema; diff --git a/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql b/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql new file mode 100644 index 0000000000..9c548aded7 --- /dev/null +++ b/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql @@ -0,0 +1,37 @@ +-- Captured output schema of a managed `// materialize` asset (gap #2a). +-- After a managed materialize commits, the worker DESCRIBEs the written table +-- and records its column list here as asset-level metadata. Schema is a +-- property of the asset/table, not of a partition slice, so it lives in its own +-- table keyed by (workspace, asset_kind, asset_path) rather than as a column on +-- materialized_partition (which would duplicate the identical schema across +-- every partition row). This is the producer-side capture that #2b (save-time +-- consumer-ref contract enforcement) reads back. +-- +-- Versioning across re-materializations: a new `version` row is inserted only +-- when the captured column set differs from the latest stored version for the +-- asset; an unchanged re-materialize re-affirms the latest row in place. So the +-- table is a compact schema-evolution history and MAX(version) is the current +-- contract. +CREATE TABLE IF NOT EXISTS materialized_asset_schema ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + asset_kind ASSET_KIND NOT NULL, + asset_path VARCHAR(255) NOT NULL, + -- Monotonic per (workspace, asset_kind, asset_path), starting at 1; only + -- bumped when the schema actually changes. + version BIGINT NOT NULL, + -- The captured columns, ordered as the table presents them: + -- [{"name": "...", "type": "..."}, ...]. + columns JSONB NOT NULL, + -- DuckLake snapshot the schema was captured from (NULL for non-ducklake / + -- substrates without snapshots). + snapshot_id BIGINT, + job_id UUID, + captured_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, asset_kind, asset_path, version) +); + +-- Default privileges (migration 20250205131523) only apply to objects created +-- by the role that set them, so grant explicitly — the API reads/writes this +-- table as the invoking role (same fix as script_trigger in 20260619112847). +GRANT ALL ON materialized_asset_schema TO windmill_user; +GRANT ALL ON materialized_asset_schema TO windmill_admin; diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql new file mode 100644 index 0000000000..1ce2c852b7 --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql @@ -0,0 +1,21 @@ +-- Restore the previous anchor: epoch sentinel + preserve-cursor (DO NOTHING). +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', '1970-01-01T00:00:00+00:00' + ) + ) + ON CONFLICT (name) DO NOTHING; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP FUNCTION IF EXISTS audit_logs_s3_oldest_inflight_ts(); diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql new file mode 100644 index 0000000000..ff2ed110c6 --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql @@ -0,0 +1,86 @@ +-- Anchors the audit→object-store export cursor when the setting is enabled. +-- +-- `last_ts`/`last_oldest_inflight_ts` must be a *recent* floor, not epoch: the +-- export's `timestamp >= floor` predicate is the only partition-pruning bound (the +-- `age(xmin)` cursor is unindexable), so an epoch floor would scan the whole +-- `audit_partitioned` table on the first run and never finish under a +-- `statement_timeout`. The floor must be at or below the timestamp of any row whose +-- xid >= this snapshot xmin; the oldest in-flight `xact_start` is that bound when +-- stats are visible (no restricted role / prepared 2PC txn), else a bounded 7-day +-- window. +-- +-- `ON CONFLICT DO UPDATE ... WHERE last_xmin <` keeps the cursor monotonic: a +-- re-enable re-anchors it forward (so the export resumes from ~now rather than +-- rescanning the disabled gap — that gap is the backfill's job), but it never moves +-- backwards, so it is HA-safe and can't be regressed by a slower concurrent writer. +-- +-- The task name literal must match +-- `windmill_common::global_settings::AUDIT_LOGS_S3_EXPORT_TASK`. + +-- Oldest in-flight `xact_start` when this role can observe *all* sessions (so the +-- min is a true cluster-wide bound), else NULL — callers substitute a conservative +-- window. The stats-visibility check goes through `is_superuser` (a preset GUC, no +-- catalog read) first, then a best-effort `pg_has_role` probe guarded by EXCEPTION: +-- `pg_has_role` reads `pg_authid`, which some managed providers (e.g. Cloud SQL) +-- forbid even to read from an elevated context, raising "Modifying pg_authid or +-- pg_auth_members is not allowed in elevated context". The EXCEPTION block runs in +-- its own subtransaction, so a failure there returns NULL (→ conservative fallback) +-- without aborting the caller — the migration-time UPDATE below, the trigger, or the +-- export task, none of which must fail just because the optimization is unavailable. +CREATE OR REPLACE FUNCTION audit_logs_s3_oldest_inflight_ts() +RETURNS timestamptz AS $$ +DECLARE + v_can_read_all_stats boolean := current_setting('is_superuser') = 'on'; +BEGIN + IF NOT v_can_read_all_stats THEN + BEGIN + v_can_read_all_stats := pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'); + EXCEPTION WHEN OTHERS THEN + v_can_read_all_stats := false; + END; + END IF; + IF v_can_read_all_stats AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) THEN + RETURN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL); + END IF; + RETURN NULL; +END; +$$ LANGUAGE plpgsql; + +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', now(), + 'last_oldest_inflight_ts', + COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days') + ) + ) + ON CONFLICT (name) DO UPDATE + SET value = EXCLUDED.value + WHERE (background_task_state.value->>'last_xmin')::bigint + < (EXCLUDED.value->>'last_xmin')::bigint; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +-- Recovery for a legacy epoch-sentinel checkpoint (`last_ts = epoch`, never +-- drained). It cannot be safely resumed: its un-drained backlog can be arbitrarily +-- old, so stamping a recent floor over the old xmin would prune the older rows while +-- the cursor advanced past them (silent loss), and keeping the epoch floor would +-- reintroduce the full scan. Re-anchor it to now like a fresh enable; the pre-anchor +-- window is recoverable via the opt-in backfill, not silently dropped. +UPDATE background_task_state +SET value = jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', to_jsonb(now()), + 'last_oldest_inflight_ts', + to_jsonb(COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))) +WHERE name = 'audit_logs_s3_export' + AND (value->>'last_ts')::timestamptz <= 'epoch'::timestamptz; diff --git a/backend/migrations/20260701080313_grant_dispatch_event_to_windmill_roles.down.sql b/backend/migrations/20260701080313_grant_dispatch_event_to_windmill_roles.down.sql new file mode 100644 index 0000000000..2278fbfb2e --- /dev/null +++ b/backend/migrations/20260701080313_grant_dispatch_event_to_windmill_roles.down.sql @@ -0,0 +1,4 @@ +REVOKE ALL ON dispatch_event FROM windmill_user; +REVOKE ALL ON dispatch_event FROM windmill_admin; +REVOKE ALL ON SEQUENCE dispatch_event_id_seq FROM windmill_user; +REVOKE ALL ON SEQUENCE dispatch_event_id_seq FROM windmill_admin; diff --git a/backend/migrations/20260701080313_grant_dispatch_event_to_windmill_roles.up.sql b/backend/migrations/20260701080313_grant_dispatch_event_to_windmill_roles.up.sql new file mode 100644 index 0000000000..5d0b4650ce --- /dev/null +++ b/backend/migrations/20260701080313_grant_dispatch_event_to_windmill_roles.up.sql @@ -0,0 +1,15 @@ +-- The dispatch_event table (migration 20260523055641_dispatch_event) was +-- created relying on ALTER DEFAULT PRIVILEGES to grant access to windmill_user +-- and windmill_admin. Those default privileges only apply to objects created by +-- the role that set them (migration 20250205131523), so deployments whose +-- migration runner is a different role leave dispatch_event ungranted. Direct +-- writes run as the invoking role -- the dispatcher insert (asset_dispatch.rs) +-- and the DELETE in delete_jobs (windmill-common/src/jobs.rs), reached whenever +-- a job's side rows are reaped, e.g. on schedule disable -- and fail with +-- "permission denied for table dispatch_event". Grant explicitly to guarantee +-- access regardless of who ran the migrations (same fix as notify_event in +-- 20260619091631 and script_trigger in 20260619112847). +GRANT ALL ON dispatch_event TO windmill_user; +GRANT ALL ON dispatch_event TO windmill_admin; +GRANT ALL ON SEQUENCE dispatch_event_id_seq TO windmill_user; +GRANT ALL ON SEQUENCE dispatch_event_id_seq TO windmill_admin; diff --git a/backend/migrations/20260701083047_grant_user_db_tables_to_windmill_roles.down.sql b/backend/migrations/20260701083047_grant_user_db_tables_to_windmill_roles.down.sql new file mode 100644 index 0000000000..b73484e053 --- /dev/null +++ b/backend/migrations/20260701083047_grant_user_db_tables_to_windmill_roles.down.sql @@ -0,0 +1,6 @@ +REVOKE ALL ON workspace_diff FROM windmill_user; +REVOKE ALL ON workspace_diff FROM windmill_admin; +REVOKE ALL ON materialized_partition FROM windmill_user; +REVOKE ALL ON materialized_partition FROM windmill_admin; +REVOKE ALL ON debounce_stale_data FROM windmill_user; +REVOKE ALL ON debounce_stale_data FROM windmill_admin; diff --git a/backend/migrations/20260701083047_grant_user_db_tables_to_windmill_roles.up.sql b/backend/migrations/20260701083047_grant_user_db_tables_to_windmill_roles.up.sql new file mode 100644 index 0000000000..9205934600 --- /dev/null +++ b/backend/migrations/20260701083047_grant_user_db_tables_to_windmill_roles.up.sql @@ -0,0 +1,22 @@ +-- Same grant gap fixed for notify_event (20260619091631), script_trigger +-- (20260619112847), and dispatch_event: tables created after the one-time +-- GRANT ALL in 20250205131523 rely on ALTER DEFAULT PRIVILEGES, which only +-- applies to objects created by the role that set them. On deployments whose +-- migration runner is a different role, these tables end up ungranted, and +-- writes that run under the RLS role (a transaction opened via +-- user_db.begin(&authed) -> SET LOCAL ROLE windmill_user/windmill_admin) fail +-- with "permission denied for table ". +-- +-- Each table below has a confirmed write on a user_db transaction: +-- * workspace_diff -- UPDATE in set_ws_specific (workspaces.rs) +-- * materialized_partition -- INSERT via record_materialization (assets API) +-- * debounce_stale_data -- DELETE in resume_suspended_trigger_jobs +-- (global_handler.rs), the same tx that reaps a +-- job's side rows +-- None has a sequence, so only table grants are needed. +GRANT ALL ON workspace_diff TO windmill_user; +GRANT ALL ON workspace_diff TO windmill_admin; +GRANT ALL ON materialized_partition TO windmill_user; +GRANT ALL ON materialized_partition TO windmill_admin; +GRANT ALL ON debounce_stale_data TO windmill_user; +GRANT ALL ON debounce_stale_data TO windmill_admin; diff --git a/backend/migrations/20260701123436_realign_superadmin_favorites_to_derived_username.down.sql b/backend/migrations/20260701123436_realign_superadmin_favorites_to_derived_username.down.sql new file mode 100644 index 0000000000..2e3e434bcc --- /dev/null +++ b/backend/migrations/20260701123436_realign_superadmin_favorites_to_derived_username.down.sql @@ -0,0 +1,3 @@ +-- Irreversible data backfill: once realigned to the derived username, a favorite +-- is indistinguishable from one legitimately created under that username, so there +-- is nothing safe to revert. No-op. diff --git a/backend/migrations/20260701123436_realign_superadmin_favorites_to_derived_username.up.sql b/backend/migrations/20260701123436_realign_superadmin_favorites_to_derived_username.up.sql new file mode 100644 index 0000000000..b06f63d7af --- /dev/null +++ b/backend/migrations/20260701123436_realign_superadmin_favorites_to_derived_username.up.sql @@ -0,0 +1,19 @@ +-- A superadmin acting in a workspace they are not a member of used to be +-- identified by their raw email (so favorites were stored with usr = email). +-- They are now identified by their instance-derived username (password.username), +-- so realign those pre-existing favorites to keep them visible. Only email-keyed +-- rows are ever a superadmin's (members always store a non-email username), and +-- the anti-join skips rows that would collide with an already-derived favorite. +UPDATE favorite f +SET usr = p.username +FROM password p +WHERE f.usr = p.email + AND p.super_admin = true + AND p.username IS NOT NULL + AND NOT EXISTS ( + SELECT 1 FROM favorite f2 + WHERE f2.workspace_id = f.workspace_id + AND f2.usr = p.username + AND f2.favorite_kind = f.favorite_kind + AND f2.path = f.path + ); diff --git a/backend/migrations/20260702064737_workspace_storage_usage.down.sql b/backend/migrations/20260702064737_workspace_storage_usage.down.sql new file mode 100644 index 0000000000..e32480a602 --- /dev/null +++ b/backend/migrations/20260702064737_workspace_storage_usage.down.sql @@ -0,0 +1 @@ +DROP TABLE workspace_storage_usage; diff --git a/backend/migrations/20260702064737_workspace_storage_usage.up.sql b/backend/migrations/20260702064737_workspace_storage_usage.up.sql new file mode 100644 index 0000000000..d7274e8742 --- /dev/null +++ b/backend/migrations/20260702064737_workspace_storage_usage.up.sql @@ -0,0 +1,17 @@ +-- Cached per-(workspace, storage) byte usage of workspace object storage, +-- refreshed by listing the storage location and adjusted optimistically as +-- uploads complete. Read on every workspace-storage write in CE builds to +-- enforce the storage quota, and by the storage_usage endpoint in all builds. +CREATE TABLE workspace_storage_usage ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace (id) ON DELETE CASCADE, + storage VARCHAR(255) NOT NULL, + bytes BIGINT NOT NULL DEFAULT 0, + computed_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, storage) +); + +-- Tables created after the one-time GRANT ALL in 20250205131523 need explicit +-- grants: ALTER DEFAULT PRIVILEGES only covers objects created by the role +-- that set them (same gap as workspace_diff, notify_event, script_trigger). +GRANT ALL ON workspace_storage_usage TO windmill_user; +GRANT ALL ON workspace_storage_usage TO windmill_admin; diff --git a/backend/migrations/20260702095830_duckdb_macro_registry.down.sql b/backend/migrations/20260702095830_duckdb_macro_registry.down.sql new file mode 100644 index 0000000000..6f48e71d0f --- /dev/null +++ b/backend/migrations/20260702095830_duckdb_macro_registry.down.sql @@ -0,0 +1,2 @@ +DROP TABLE IF EXISTS macro_usage; +DROP TABLE IF EXISTS macro_definition; diff --git a/backend/migrations/20260702095830_duckdb_macro_registry.up.sql b/backend/migrations/20260702095830_duckdb_macro_registry.up.sql new file mode 100644 index 0000000000..ccc4bcc8cc --- /dev/null +++ b/backend/migrations/20260702095830_duckdb_macro_registry.up.sql @@ -0,0 +1,31 @@ +-- Workspace DuckDB macro registry: one row per macro defined by a deployed +-- `// macros` library script. Names are workspace-unique (macros are injected +-- unqualified into consumer jobs). +CREATE TABLE macro_definition ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace (id) ON DELETE CASCADE, + name VARCHAR(255) NOT NULL, + provider_path VARCHAR(510) NOT NULL, + params TEXT NOT NULL, + body TEXT NOT NULL, + is_table_macro BOOLEAN NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, name) +); +CREATE INDEX idx_macro_definition_provider ON macro_definition (workspace_id, provider_path); + +-- Deploy-recorded consumer→macro edges, for the asset graph only (the worker +-- re-detects calls live at job time). +CREATE TABLE macro_usage ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace (id) ON DELETE CASCADE, + consumer_path VARCHAR(510) NOT NULL, + macro_name VARCHAR(255) NOT NULL, + PRIMARY KEY (workspace_id, consumer_path, macro_name) +); +CREATE INDEX idx_macro_usage_name ON macro_usage (workspace_id, macro_name); + +-- Both tables are written on user_db transactions (SET LOCAL ROLE); the +-- one-time GRANT ALL migration predates them, so explicit grants are required. +GRANT ALL ON macro_definition TO windmill_user; +GRANT ALL ON macro_definition TO windmill_admin; +GRANT ALL ON macro_usage TO windmill_user; +GRANT ALL ON macro_usage TO windmill_admin; diff --git a/backend/migrations/20260702213513_workspace_multipart_inflight.down.sql b/backend/migrations/20260702213513_workspace_multipart_inflight.down.sql new file mode 100644 index 0000000000..e72edd0b12 --- /dev/null +++ b/backend/migrations/20260702213513_workspace_multipart_inflight.down.sql @@ -0,0 +1 @@ +DROP TABLE workspace_multipart_inflight; diff --git a/backend/migrations/20260702213513_workspace_multipart_inflight.up.sql b/backend/migrations/20260702213513_workspace_multipart_inflight.up.sql new file mode 100644 index 0000000000..9768eaf842 --- /dev/null +++ b/backend/migrations/20260702213513_workspace_multipart_inflight.up.sql @@ -0,0 +1,32 @@ +-- Reservation for the parts of in-flight (initiated but not yet completed) +-- multipart uploads to workspace object storage. Uncommitted parts occupy +-- object-store capacity but are invisible to the list-based storage recount +-- until completion, so CE folds this reservation into the remaining quota to +-- bound abandoned uploads. One row per uploaded part so a re-uploaded part +-- (same part_id) replaces rather than double-counts; a part is recorded only +-- after its upstream upload succeeds. Rows are removed on successful complete +-- and lazily expired after a TTL (abort/abandon rely on the TTL, which matches +-- when the object store reaps the uncommitted parts). +-- part_id - S3 part number or Azure block id (string) +-- part_bytes - size of that part +-- target_existing_size - size of the object the upload will overwrite (0 if new), +-- credited so an overwrite only reserves the net growth +CREATE TABLE workspace_multipart_inflight ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace (id) ON DELETE CASCADE, + upload_id VARCHAR(512) NOT NULL, + part_id VARCHAR(256) NOT NULL, + storage VARCHAR(255) NOT NULL, + part_bytes BIGINT NOT NULL DEFAULT 0, + target_existing_size BIGINT NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, upload_id, part_id) +); + +CREATE INDEX idx_workspace_multipart_inflight_created_at + ON workspace_multipart_inflight (created_at); + +-- Tables created after the one-time GRANT ALL in 20250205131523 need explicit +-- grants: ALTER DEFAULT PRIVILEGES only covers objects created by the role that +-- set them (same gap as workspace_storage_usage, notify_event, script_trigger). +GRANT ALL ON workspace_multipart_inflight TO windmill_user; +GRANT ALL ON workspace_multipart_inflight TO windmill_admin; diff --git a/backend/migrations/20260703165613_pipeline_freshness_watchdog.down.sql b/backend/migrations/20260703165613_pipeline_freshness_watchdog.down.sql new file mode 100644 index 0000000000..aed2f9d32a --- /dev/null +++ b/backend/migrations/20260703165613_pipeline_freshness_watchdog.down.sql @@ -0,0 +1,5 @@ +-- Postgres has no ALTER TYPE ... DROP VALUE for enums. The 'freshness' value +-- stays even on rollback, consistent with prior job_trigger_kind additions +-- (see 20260510174213_asset_trigger_dispatch). +DROP INDEX IF EXISTS idx_script_pipeline_freshness_scan; +DROP TABLE IF EXISTS pipeline_freshness_state; diff --git a/backend/migrations/20260703165613_pipeline_freshness_watchdog.up.sql b/backend/migrations/20260703165613_pipeline_freshness_watchdog.up.sql new file mode 100644 index 0000000000..acf9545c96 --- /dev/null +++ b/backend/migrations/20260703165613_pipeline_freshness_watchdog.up.sql @@ -0,0 +1,35 @@ +-- Attribution for runs pushed by the pipeline freshness watchdog (the EE +-- background loop that re-runs a `// freshness`-annotated producer whose +-- output aged past its window). +ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'freshness'; + +-- Per-(workspace, script) watchdog state: exponential-backoff bookkeeping so +-- a persistently failing producer isn't re-pushed on every scan tick, and an +-- atomic claim so concurrent servers can't double-push in the same tick +-- (claim = the UPDATE/INSERT that advances next_attempt_at; only the winner +-- pushes). Rows exist only while a script is stale — observing it fresh (or +-- its annotation gone) deletes the row, resetting the backoff. +CREATE TABLE pipeline_freshness_state ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace (id) ON DELETE CASCADE, + script_path VARCHAR(510) NOT NULL, + attempts INTEGER NOT NULL DEFAULT 1, + last_push_at TIMESTAMPTZ NOT NULL DEFAULT now(), + next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, script_path) +); + +-- Written only by the server monitor loop on the raw (non-RLS) pool, but +-- granted like every other app table so a future user-transaction reader +-- doesn't hit the recurring missing-GRANT class of bug. +GRANT ALL ON pipeline_freshness_state TO windmill_user; +GRANT ALL ON pipeline_freshness_state TO windmill_admin; + +-- The watchdog's ~60s candidate scan (latest deployed pipeline members) +-- filters on this exact predicate and orders by (workspace_id, path, +-- created_at DESC); without a matching partial index it seq-scans the whole +-- script-version heap on every tick, on instances that mostly have zero +-- pipeline scripts. (idx_script_pipeline_path is text_pattern_ops for +-- prefix LIKE — it can't serve this ordering.) +CREATE INDEX idx_script_pipeline_freshness_scan + ON script (workspace_id, path, created_at DESC) + WHERE auto_kind = 'pipeline' AND archived = false AND deleted = false; diff --git a/backend/migrations/20260703170745_fork_ducklake_namespace.down.sql b/backend/migrations/20260703170745_fork_ducklake_namespace.down.sql new file mode 100644 index 0000000000..fd722bf414 --- /dev/null +++ b/backend/migrations/20260703170745_fork_ducklake_namespace.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS fork_ducklake_namespace; diff --git a/backend/migrations/20260703170745_fork_ducklake_namespace.up.sql b/backend/migrations/20260703170745_fork_ducklake_namespace.up.sql new file mode 100644 index 0000000000..7762304199 --- /dev/null +++ b/backend/migrations/20260703170745_fork_ducklake_namespace.up.sql @@ -0,0 +1,51 @@ +-- Registry of ducklake namespaces provisioned for fork/dev workspaces. One row per +-- (fork workspace, lake name): records the exact catalog metadata schema and data +-- sub-path the fork's jobs attach to, so fork deletion can drop the pg schema and +-- delete the S3 prefix deterministically (the row is written on first resolution, +-- before any physical state exists). +-- +-- Deliberately NO foreign key to workspace(id): rows are the durable cleanup ledger and +-- must OUTLIVE the workspace row when physical cleanup fails after the delete commits +-- (unreachable catalog, storage outage) — a CASCADE would erase the only record of the +-- orphaned namespace, letting a recreated same-id fork silently reattach stale tables. +-- Rows are deleted explicitly after each successful cleanup; leftover rows for a reused +-- id are retried at fork creation, which refuses to proceed while they cannot be cleaned. +CREATE TABLE fork_ducklake_namespace ( + workspace_id VARCHAR(50) NOT NULL, + ducklake_name VARCHAR(255) NOT NULL, + metadata_schema VARCHAR(63) NOT NULL, + -- Canonical identity of the catalog database the metadata schema lives in + -- (`:`, e.g. `instance:wm_ducklake` or + -- `postgres:u/admin/pg`). Cleanup connects to THIS catalog, not whatever the fork's + -- settings point at by then — a drifted catalog resource must not make cleanup drop a + -- schema in the wrong database and orphan the real one. + catalog TEXT NOT NULL, + -- Named workspace storage holding the fork's data files; '' = the default storage + -- (part of the PK, which cannot hold NULL). + storage TEXT NOT NULL DEFAULT '', + -- The storage's RESOLVED identity at registration time (`:`, e.g. `s3:u/admin/minio` or `filesystem:/data/lfs`; '' = unknown). Cleanup + -- deletes the fork prefix from THIS storage, not whatever the logical name points at by + -- then — repointing a storage after attach must not orphan the original fork data (or + -- delete a colliding prefix from the new one). + storage_ref TEXT NOT NULL DEFAULT '', + -- The fork namespace's data path within that storage (a bucket-root + -- `__wm_forks//…` prefix). + data_path TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + -- Cleanup phase state: true once the metadata schema has been dropped but data files (or + -- the row delete) still failed. Later retries then skip the schema phase entirely — they + -- need NO catalog credentials, which may be gone for good with the deleted fork's + -- resources. Reset to false whenever a live fork re-registers the row (re-attaching + -- recreates the schema). + schema_dropped BOOLEAN NOT NULL DEFAULT false, + -- One row per physical location EVER attached: if the fork's lake settings drift + -- (catalog/storage/path change), later attaches add rows rather than replace them, so + -- cleanup covers every location the fork wrote, not just the first. + PRIMARY KEY (workspace_id, ducklake_name, catalog, storage, storage_ref, data_path) +); + +-- Resolution runs under user_db transactions (SET LOCAL ROLE) in API contexts, so the +-- windmill roles need explicit grants (default privileges don't apply to app-created tables). +GRANT ALL ON fork_ducklake_namespace TO windmill_user; +GRANT ALL ON fork_ducklake_namespace TO windmill_admin; diff --git a/backend/migrations/20260706094033_add_dev_workspace_label.down.sql b/backend/migrations/20260706094033_add_dev_workspace_label.down.sql new file mode 100644 index 0000000000..0b9b81636a --- /dev/null +++ b/backend/migrations/20260706094033_add_dev_workspace_label.down.sql @@ -0,0 +1 @@ +ALTER TABLE workspace DROP COLUMN dev_workspace_label; diff --git a/backend/migrations/20260706094033_add_dev_workspace_label.up.sql b/backend/migrations/20260706094033_add_dev_workspace_label.up.sql new file mode 100644 index 0000000000..951f1ec3c9 --- /dev/null +++ b/backend/migrations/20260706094033_add_dev_workspace_label.up.sql @@ -0,0 +1,4 @@ +-- Cosmetic display label for a dev workspace: NULL/'dev' render as "dev", 'staging' renders as "stg". +-- Only meaningful when is_dev_workspace = true; changes nothing about behavior (locking, promote and +-- compare all key off is_dev_workspace / parent_workspace_id). The value is validated in the handler. +ALTER TABLE workspace ADD COLUMN dev_workspace_label VARCHAR; diff --git a/backend/parsers/windmill-parser-py-asset/src/lib.rs b/backend/parsers/windmill-parser-py-asset/src/lib.rs index e8a0a1cc40..8d2e87a53e 100644 --- a/backend/parsers/windmill-parser-py-asset/src/lib.rs +++ b/backend/parsers/windmill-parser-py-asset/src/lib.rs @@ -262,6 +262,14 @@ impl AssetsFinder { }; match arg_val { + // S3 helpers take an `S3Object` (constructor or dict literal) or an + // `s3://bucket/key` string. Other helpers take a bare resource-path + // string literal. + Some(arg) if matches!(kind, AssetKind::S3Object) => { + let path = s3_object_arg_path(arg).ok_or(())?; + self.assets + .push(ParseAssetsResult { kind, path, access_type, columns: None }); + } Some(Expr::Constant(ExprConstant { value: Constant::Str(value), .. })) => { let path = parse_asset_syntax(&value, false) .map(|(_, p)| p) @@ -282,6 +290,81 @@ impl AssetsFinder { // Positional arguments in python can also be used by their name struct Arg(usize, &'static str); +/// Extract a string-literal keyword argument, e.g. `s3="value"` in a call. +fn keyword_str_value(keywords: &[rustpython_ast::Keyword], name: &str) -> Option { + keywords + .iter() + .find(|kw| kw.arg.as_deref() == Some(name)) + .and_then(|kw| match &kw.value { + Expr::Constant(ExprConstant { value: Constant::Str(s), .. }) => Some(s.clone()), + _ => None, + }) +} + +/// Extract a string-literal value from a dict literal, e.g. `{"s3": "value"}`. +fn dict_str_value(dict: &rustpython_ast::ExprDict, name: &str) -> Option { + dict.keys + .iter() + .zip(dict.values.iter()) + .find_map(|(key, value)| match (key.as_ref()?, value) { + ( + Expr::Constant(ExprConstant { value: Constant::Str(k), .. }), + Expr::Constant(ExprConstant { value: Constant::Str(v), .. }), + ) if k.as_str() == name => Some(v.clone()), + _ => None, + }) +} + +/// Resolve the SDK `S3Object` argument of `load_s3_file`/`load_s3_file_reader`/ +/// `write_s3_file` to a canonical asset path, mirroring `windmill-parser-ts-asset`: +/// `S3Object(s3="", storage=""?)` — or the equivalent dict literal — +/// maps to the URI `s3:///` (empty bucket for default storage, i.e. +/// `s3:///`), and a `"s3://bucket/key"` URI string is passed through. +/// String args mirror the runtime `parse_s3_object` contract exactly: only a +/// `s3:///` URI with a non-empty key is valid — any other +/// string (bare key, `s3://x`, empty key) raises at run time, so recording an +/// edge for it would be a phantom node for a call that can only error. +/// The resulting URI is fed through `parse_asset_syntax` so the stored path +/// matches the TS object form and the `# on s3:///…` trigger form exactly. +fn s3_object_arg_path(expr: &Expr) -> Option { + let uri = match expr { + Expr::Constant(ExprConstant { value: Constant::Str(s), .. }) => { + match s + .strip_prefix("s3://") + .and_then(|rest| rest.split_once('/')) + { + Some((_, key)) if !key.is_empty() => s.clone(), + _ => return None, + } + } + Expr::Call(call) => { + // `S3Object(...)` imported directly or as `wmill.S3Object(...)` + let func_name = call + .func + .as_name_expr() + .map(|n| n.id.as_str()) + .or_else(|| call.func.as_attribute_expr().map(|a| a.attr.as_str()))?; + if func_name != "S3Object" { + return None; + } + let key = keyword_str_value(&call.keywords, "s3")?; + let storage = keyword_str_value(&call.keywords, "storage").unwrap_or_default(); + format!("s3://{storage}/{key}") + } + Expr::Dict(dict) => { + let key = dict_str_value(dict, "s3")?; + let storage = dict_str_value(dict, "storage").unwrap_or_default(); + format!("s3://{storage}/{key}") + } + _ => return None, + }; + Some( + parse_asset_syntax(&uri, false) + .map(|(_, p)| p.to_string()) + .unwrap_or(uri), + ) +} + #[cfg(test)] mod tests { use std::collections::BTreeMap; @@ -300,13 +383,227 @@ def main(): s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/test.csv".to_string(), + path: "test.csv".to_string(), access_type: Some(R), columns: None, },]) ); } + #[test] + fn test_py_asset_parser_bare_string_no_asset() { + // A plain (non-`s3://`) string is rejected by the runtime + // `parse_s3_object`, so the parser must not record a phantom asset + // for a call that can only error. + let input = r#" +import wmill +def main(): + wmill.write_s3_file("pipelines/etl/out.jsonl", b"") +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!(s.map_err(|e| e.to_string()), Ok(vec![])); + } + + #[test] + fn test_py_asset_parser_invalid_uri_no_write_edge() { + // `s3://x` (no key part) and `s3://bucket/` (empty key) are rejected + // by the runtime `parse_s3_object` — same rule for the SDK-arg path: + // no R/W edge. The generic URI-literal scan may still record them as + // ambiguous (`access_type: None`) assets, like any `s3://…` string + // constant anywhere in a script. + let input = r#" +import wmill +def main(): + wmill.write_s3_file("s3://broken", b"") + wmill.write_s3_file("s3://bucket/", b"") +"#; + let assets = parse_assets(input).expect("parse").assets; + assert!( + assets.iter().all(|a| a.access_type.is_none()), + "invalid URIs must not produce R/W edges: {assets:?}" + ); + } + + #[test] + fn test_py_asset_parser_write_s3_object_constructor() { + // The SDK signature is `write_s3_file(s3object: S3Object | str, ...)` and + // its docstring recommends the constructor form with a bare key. It must + // resolve to the same canonical path as the TS object form and a + // `# on s3:///` trigger. + let input = r#" +import wmill +from wmill import S3Object +def main(): + wmill.write_s3_file(S3Object(s3="analytics/x.csv"), b"content") +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "analytics/x.csv".to_string(), + access_type: Some(W), + columns: None, + },]) + ); + } + + #[test] + fn test_py_write_key_matches_duckdb_read_key() { + // Cross-language lineage: this write records `exports/x`, the same path a + // DuckDB `read_csv('s3://exports/x')` resolves to (see + // windmill-parser-sql-asset `test_duckdb_read_key_matches_sdk_write_key`), + // so the producer and consumer connect in the pipeline graph. + let input = r#" +import wmill +from wmill import S3Object +def main(): + wmill.write_s3_file(S3Object(s3="exports/x"), b"content") +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "exports/x".to_string(), + access_type: Some(W), + columns: None, + },]) + ); + } + + #[test] + fn test_py_asset_parser_s3_object_with_storage() { + // `S3Object(s3=..., storage=...)` maps to `s3:///`, + // matching the `s3://bucket/key` string form and the TS object form. + let input = r#" +import wmill +from wmill import S3Object +def main(): + wmill.load_s3_file(S3Object(s3="dir/in.csv", storage="mybucket")) +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "mybucket/dir/in.csv".to_string(), + access_type: Some(R), + columns: None, + },]) + ); + } + + #[test] + fn test_py_asset_parser_s3_object_reader_and_keyword_arg() { + // load_s3_file_reader + passing the S3Object via the `s3object` keyword + // and via the `wmill.S3Object` attribute form. + let input = r#" +import wmill +def main(): + wmill.load_s3_file_reader(s3object=wmill.S3Object(s3="dir/in.csv")) +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "dir/in.csv".to_string(), + access_type: Some(R), + columns: None, + },]) + ); + } + + #[test] + fn test_py_asset_parser_s3_object_dict_literal() { + // `S3Object` subclasses dict, so the SDK also accepts a plain dict + // literal — same resolution as the constructor form. + let input = r#" +import wmill +def main(): + wmill.write_s3_file({"s3": "out.json"}, b"{}") + wmill.load_s3_file({"s3": "dir/in.csv", "storage": "mybucket"}) +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "mybucket/dir/in.csv".to_string(), + access_type: Some(R), + columns: None, + }, + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "out.json".to_string(), + access_type: Some(W), + columns: None, + }, + ]) + ); + } + + #[test] + fn test_py_asset_parser_multiple_s3_object_writes() { + // Several direct constructor-form writes in main() — all outputs must + // be detected (merge_assets returns a deterministic path-sorted order). + let input = r#" +import wmill +from wmill import S3Object +def main(): + wmill.write_s3_file(S3Object(s3="pipelines/km_real/raw_events.json"), b"[]") + wmill.write_s3_file(S3Object(s3="pipelines/km_real/enriched.json"), b"[]") + wmill.write_s3_file(S3Object(s3="pipelines/km_real/summary.json"), b"[]") + wmill.write_s3_file(S3Object(s3="pipelines/km_real/report.json"), b"{}") +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "pipelines/km_real/enriched.json".to_string(), + access_type: Some(W), + columns: None, + }, + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "pipelines/km_real/raw_events.json".to_string(), + access_type: Some(W), + columns: None, + }, + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "pipelines/km_real/report.json".to_string(), + access_type: Some(W), + columns: None, + }, + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "pipelines/km_real/summary.json".to_string(), + access_type: Some(W), + columns: None, + }, + ]) + ); + } + + #[test] + fn test_py_asset_parser_s3_object_dynamic_key_no_false_positive() { + // A computed key can't be resolved statically — must yield nothing + // rather than a bogus path. + let input = r#" +import wmill +from wmill import S3Object +def main(name: str): + wmill.write_s3_file(S3Object(s3=f"pipelines/{name}.json"), b"{}") +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!(s.map_err(|e| e.to_string()), Ok(vec![])); + } + #[test] fn test_py_asset_parser_unused_sql() { let input = r#" diff --git a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs index 812bfa785d..d2ab5b51a7 100644 --- a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs +++ b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs @@ -9,8 +9,9 @@ use sqlparser::{ parser::Parser, }; use windmill_parser::asset_parser::{ - asset_was_used, merge_assets, parse_asset_syntax, parse_pipeline_annotations, AssetKind, - AssetUsageAccessType, ParseAssetsOutput, ParseAssetsResult, + asset_was_used, merge_assets, merge_column_lineage, parse_asset_syntax, + parse_pipeline_annotations, AssetKind, AssetUsageAccessType, ColumnLineage, ColumnRef, + ParseAssetsOutput, ParseAssetsResult, }; use AssetUsageAccessType::*; @@ -33,19 +34,105 @@ pub fn parse_assets(input: &str) -> anyhow::Result { } } - let pipeline = parse_pipeline_annotations(input); - Ok(ParseAssetsOutput::new( - merge_assets(collector.assets), - Vec::new(), - pipeline, - )) + let mut pipeline = parse_pipeline_annotations(input); + // Scope inferred lineage to a single output asset so columns from an + // auxiliary CTAS aren't attributed to the materialized one (the flat list + // has no per-entry output on the wire). The `// materialize` target, when + // declared, IS the output: keep entries tagged with it plus untagged + // top-level-SELECT entries (which describe that target). Without a declared + // target, keep inference only when every tagged entry shares one output + // asset — otherwise it's ambiguous which asset the flat list describes, so + // drop it rather than show false dependencies. + let target = pipeline + .materialize + .as_ref() + .map(|m| (m.target_kind, m.target_path.clone())); + let inferred: Vec = match &target { + Some(t) => collector + .column_lineage + .into_iter() + .filter(|(out, _)| out.as_ref().map_or(true, |o| o == t)) + .map(|(_, cl)| cl) + .collect(), + None => { + let mut first: Option<&(AssetKind, String)> = None; + let mut ambiguous = false; + for (out, _) in &collector.column_lineage { + if let Some(o) = out { + match first { + None => first = Some(o), + Some(f) if f != o => { + ambiguous = true; + break; + } + _ => {} + } + } + } + if ambiguous { + Vec::new() + } else { + collector + .column_lineage + .into_iter() + .map(|(_, cl)| cl) + .collect() + } + } + }; + // Body-inferred column lineage, with `// column` annotations taking + // precedence per output column (explicit declaration overrides inference). + pipeline.column_lineage = merge_column_lineage(inferred, pipeline.column_lineage); + // A bare string literal in query position is only weak read evidence: a + // summary `SELECT 's3:///out.csv' AS target` after `COPY … TO + // 's3:///out.csv'` must not turn the write into rw (which draws a + // spurious read edge and an asset⇄script cycle in the pipeline graph). + // Surface weak reads only for assets with no other recorded usage, so a + // path that is *merely* mentioned still shows up linked to the script. + let mut assets = merge_assets(collector.assets); + for weak in merge_assets(collector.weak_string_reads) { + if !assets + .iter() + .any(|a| a.kind == weak.kind && a.path == weak.path) + { + assets.push(weak); + } + } + assets.sort_by(|a, b| a.path.cmp(&b.path)); + Ok(ParseAssetsOutput::new(assets, Vec::new(), pipeline)) +} + +/// Provenance of the innermost access context. The access type alone can't +/// tell a definitive read apart from a mere mention: a bare string literal in +/// generic query position (`QueryRead`) is only *weak* read evidence — e.g. a +/// summary `SELECT 's3:///out.csv' AS target` echoing a path — while the same +/// literal as a read-function argument or a `COPY` target is definitive. +#[derive(Clone, Copy, PartialEq, Eq)] +enum AccessCtx { + QueryRead, + ReadFn, + CopyWrite, +} + +impl AccessCtx { + fn access_type(self) -> AssetUsageAccessType { + match self { + AccessCtx::QueryRead | AccessCtx::ReadFn => R, + AccessCtx::CopyWrite => W, + } + } } /// Visitor that collects S3 asset literals from SQL statements struct AssetCollector { assets: Vec, - // e.g set to Read when we are inside a SELECT ... FROM ... statement - current_access_type_stack: Vec, + // Bare string literals seen in generic query position — weak read + // evidence, surfaced by `parse_assets` only when the script has no other + // recorded usage of the same asset (a real write must not gain a spurious + // read edge from a mention). + weak_string_reads: Vec, + // e.g set to QueryRead when we are inside a SELECT ... FROM ... statement + current_access_type_stack: Vec, // e.g ATTACH 'ducklake://a' AS dl; => { "dl": (Ducklake, "a") } var_identifiers: BTreeMap, // e.g USE dl; @@ -54,26 +141,46 @@ struct AssetCollector { cte_name_stack: Vec>, // Locally created tables (not attached to an asset) local_table_names: HashSet, + // Inferred column-level lineage: one entry per output column of an + // output-producing query, mapping it to the upstream source columns its + // expression reads. Each is tagged with the *output asset* it belongs to — + // `Some((kind, path))` for a CTAS / CREATE VIEW into a real asset, `None` + // for a top-level managed-materialize SELECT (its output is the `// + // materialize` target, known only in `parse_assets`). `parse_assets` uses + // the tag to scope the flat list to a single output asset so columns from an + // auxiliary output don't get attributed to the materialized one. Best-effort: + // dynamic/raw SQL, INSERT…SELECT, and wildcards are left to annotations. + column_lineage: Vec<(Option<(AssetKind, String)>, ColumnLineage)>, } impl AssetCollector { fn new() -> Self { Self { assets: Vec::new(), + weak_string_reads: Vec::new(), current_access_type_stack: Vec::with_capacity(8), var_identifiers: BTreeMap::new(), currently_used_asset: None, cte_name_stack: Vec::new(), local_table_names: HashSet::new(), + column_lineage: Vec::new(), } } - /// If the name resolves to an attached asset, record it. Otherwise, register it as a local - /// table/view so that subsequent references are not mistakenly attributed to the active asset. - fn track_table_definition(&mut self, name: &ObjectName) { - if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { - self.assets.push(asset); - } else if let Some(simple_name) = get_trivial_obj_name(name) { + /// Record a `CREATE TABLE`/`VIEW` target. A *temporary* table/view is always + /// local — even a one-part name under an active `USE dl`, which would + /// otherwise resolve to an asset (`ducklake://…/tmp`) and then leak as a + /// column source for later references. A non-temp name that resolves to an + /// attached asset is recorded as that asset; anything else is registered + /// local so subsequent references aren't attributed to the active asset. + fn track_table_definition(&mut self, name: &ObjectName, is_temporary: bool) { + if !is_temporary { + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.assets.push(asset); + return; + } + } + if let Some(simple_name) = get_trivial_obj_name(name) { self.local_table_names.insert(simple_name.to_lowercase()); } } @@ -94,7 +201,11 @@ impl AssetCollector { name: &ObjectName, access_type: Option, ) -> Option { - let access_type = access_type.or_else(|| self.current_access_type_stack.last().copied()); + let access_type = access_type.or_else(|| { + self.current_access_type_stack + .last() + .map(|c| c.access_type()) + }); if let Some((kind, path)) = &self.currently_used_asset { // We don't want to infer that any simple identifier refers to an asset if // we are not in a known R/W context @@ -233,12 +344,18 @@ impl AssetCollector { // Check if the string matches our asset syntax patterns if let Some((kind, path)) = parse_asset_syntax(s, false) { if kind == AssetKind::S3Object { - self.assets.push(ParseAssetsResult { + let ctx = self.current_access_type_stack.last().copied(); + let result = ParseAssetsResult { kind, path: path.to_string(), - access_type: self.current_access_type_stack.last().copied(), + access_type: ctx.map(AccessCtx::access_type), columns: None, - }); + }; + if ctx == Some(AccessCtx::QueryRead) { + self.weak_string_reads.push(result); + } else { + self.assets.push(result); + } } } } @@ -246,7 +363,7 @@ impl AssetCollector { fn handle_obj_name_pre(&mut self, name: &ObjectName) { if let Some(fname) = get_trivial_obj_name(name) { if is_read_fn(fname) { - self.current_access_type_stack.push(R); + self.current_access_type_stack.push(AccessCtx::ReadFn); } } if let Some(str_lit) = get_str_lit_from_obj_name(name) { @@ -280,6 +397,22 @@ impl AssetCollector { } } + // Infer the output-column lineage of a query that produces an asset, tagging + // each entry with its `output` asset. Called only for an output-producing + // query — a top-level managed-materialize SELECT (`output: None`, resolved + // to the `// materialize` target later) or a CTAS / CREATE VIEW into a real + // asset (`output: Some`). A CTAS into a local/temp staging table is never + // an output, so it's simply not passed here. + fn infer_query_output( + &mut self, + query: &sqlparser::ast::Query, + output: Option<(AssetKind, String)>, + ) { + if let Some(select) = query.body.as_select() { + self.infer_column_lineage(&select.projection, &select.from, output); + } + } + fn handle_table_with_joins( &mut self, table_with_joins: &sqlparser::ast::TableWithJoins, @@ -302,17 +435,57 @@ impl AssetCollector { } } - // Extract columns from SELECT items and create individual asset results for each column - // Only processes columns that reference known assets to avoid false positives - fn extract_column_assets( - &mut self, - projection: &[SelectItem], + // The alias-map entry (key → asset) for one FROM/JOIN table factor, or + // `None` if it isn't an asset-backed table. The key is its alias, else the + // bare table name; S3 table-functions and string-literal tables are only + // keyed when aliased (an unaliased one is ambiguous). Returns the asset with + // a single matched relation so the caller can attribute qualified columns. + fn table_alias_entry(&self, relation: &TableFactor) -> Option<(String, ParseAssetsResult)> { + let TableFactor::Table { name, alias, args, .. } = relation else { + return None; + }; + let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); + if has_args { + let alias = alias.as_ref()?; + let asset = self.get_s3_asset_from_table_function(relation)?; + return Some((alias.name.value.clone(), asset)); + } + let asset = self + .get_associated_asset_from_obj_name(name, Some(R)) + .or_else(|| self.get_s3_asset_from_str_literal_table(relation))?; + if get_str_lit_from_obj_name(name).is_some() { + // String-literal S3 table: only unambiguous when aliased. + let alias = alias.as_ref()?; + return Some((alias.name.value.clone(), asset)); + } + let key = match alias { + Some(a) => a.name.value.clone(), + None => name + .0 + .last() + .and_then(|id| id.as_ident()) + .map(|id| id.value.clone()) + .unwrap_or_default(), + }; + Some((key, asset)) + } + + // Resolve a query's FROM clause into (single-table asset, alias→asset map). + // `single_table` is `Some` only for an unambiguous one-table FROM with no + // joins (so bare column refs can be attributed); `table_to_asset` keys by + // alias/table name for qualified refs and includes every JOINed table. + // Shared by `extract_column_assets` (read columns) and `infer_column_lineage` + // (output→input edges) so both resolve identically. + fn build_from_maps( + &self, from_tables: &[sqlparser::ast::TableWithJoins], + ) -> ( + Option, + BTreeMap, ) { - // Check if this is a single-table SELECT (to avoid ambiguity). - // For S3 table functions (read_parquet/read_csv/read_json), detect the asset even - // though args are present, since we know the file path from the string literal arg. - let single_table = if from_tables.len() == 1 { + // Single unambiguous table only when there's exactly one FROM entry AND + // it has no joins — otherwise a bare column could belong to any side. + let single_table = if from_tables.len() == 1 && from_tables[0].joins.is_empty() { let relation = &from_tables[0].relation; if let TableFactor::Table { name, args, .. } = relation { let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); @@ -329,51 +502,30 @@ impl AssetCollector { None }; - // Build a map of table aliases/names to assets for multi-table queries. - // For S3 table functions, only aliased references are unambiguous - // (e.g. SELECT t.col1 FROM read_parquet('s3://...') AS t). + // Alias → asset for qualified column refs, across every FROM entry AND + // its JOINed tables (so `c.col` in `FROM a JOIN c` resolves). let mut table_to_asset: BTreeMap = BTreeMap::new(); for table_with_joins in from_tables { - if let TableFactor::Table { name, alias, args, .. } = &table_with_joins.relation { - let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); - if has_args { - // For table functions, only add to the alias map when an alias is present - if let Some(alias) = alias { - if let Some(asset) = - self.get_s3_asset_from_table_function(&table_with_joins.relation) - { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } - } else if let Some(asset) = self - .get_associated_asset_from_obj_name(name, Some(R)) - .or_else(|| { - self.get_s3_asset_from_str_literal_table(&table_with_joins.relation) - }) - { - // For string literal S3 tables (e.g. FROM 's3:///file.parquet'), only add to - // the alias map when an alias is present (to avoid false positives). - // For regular named tables, use alias or table name as key. - let is_str_literal = get_str_lit_from_obj_name(name).is_some(); - if is_str_literal { - if let Some(alias) = alias { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } else { - let table_key = if let Some(alias) = alias { - alias.name.value.clone() - } else { - name.0 - .last() - .and_then(|id| id.as_ident()) - .map(|id| id.value.clone()) - .unwrap_or_default() - }; - table_to_asset.insert(table_key, asset); - } + if let Some((k, a)) = self.table_alias_entry(&table_with_joins.relation) { + table_to_asset.insert(k, a); + } + for join in &table_with_joins.joins { + if let Some((k, a)) = self.table_alias_entry(&join.relation) { + table_to_asset.insert(k, a); } } } + (single_table, table_to_asset) + } + + // Extract columns from SELECT items and create individual asset results for each column + // Only processes columns that reference known assets to avoid false positives + fn extract_column_assets( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); // Process each SELECT item for item in projection { @@ -446,6 +598,136 @@ impl AssetCollector { } } } + + // Infer column-level lineage for an output-producing query's projection: + // each *named* output column → the upstream source columns its expression + // reads. Covers passthroughs (`amount`) and computed columns + // (`amount + tax AS total`) alike. Skipped: wildcards and unaliased + // expressions (no stable output name), and inputs that don't resolve to a + // known asset. A column with no resolved inputs is dropped. + // + // Best-effort and intentionally flat: results from every output query in the + // script accumulate into one list (the graph hangs them off the materialize + // write-edge), with no per-output-table association. This is exact for the + // common single-output member (a managed-materialize SELECT, or one CTAS), + // but a multi-statement script that stages through a TEMP table reports the + // *intermediate* column names (the final SELECT reads the temp table, whose + // columns don't resolve to an asset, so they drop out). A `// column` + // annotation overrides any output column where inference is wrong or coarse. + fn infer_column_lineage( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + output: Option<(AssetKind, String)>, + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); + for item in projection { + let (out_col, expr) = match item { + SelectItem::ExprWithAlias { expr, alias } => (alias.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::Identifier(id)) => (id.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::CompoundIdentifier(parts)) => { + match parts.last() { + Some(last) => (last.value.clone(), expr), + None => continue, + } + } + _ => continue, + }; + let mut collector = ColumnIdentCollector { refs: Vec::new(), query_depth: 0 }; + let _ = expr.visit(&mut collector); + let mut inputs: Vec = Vec::new(); + for parts in &collector.refs { + if let Some(cr) = self.resolve_column_ref(parts, &single_table, &table_to_asset) { + if !inputs.contains(&cr) { + inputs.push(cr); + } + } + } + if !inputs.is_empty() { + self.column_lineage + .push((output.clone(), ColumnLineage { column: out_col, inputs })); + } + } + } + + // Resolve identifier `parts` (e.g. `["t","amount"]` or `["amount"]`) to the + // source asset column it reads, mirroring `extract_column_assets`' + // resolution: a bare ident needs an unambiguous single-table FROM; a + // qualified ident resolves its prefix via the alias map, or (≥3 parts) as a + // db/schema-qualified object name. + fn resolve_column_ref( + &self, + parts: &[String], + single_table: &Option, + table_to_asset: &BTreeMap, + ) -> Option { + let asset_to_ref = |asset: &ParseAssetsResult, col: &str| ColumnRef { + from_kind: asset.kind, + from_path: asset.path.clone(), + from_column: col.to_string(), + }; + match parts { + [col] => single_table.as_ref().map(|a| asset_to_ref(a, col)), + [.., col] => { + let prefix = parts.first()?; + if let Some(asset) = table_to_asset.get(prefix) { + Some(asset_to_ref(asset, col)) + } else if parts.len() >= 3 { + let obj_parts: Vec = parts[..parts.len() - 1] + .iter() + .map(|p| ObjectNamePart::Identifier(sqlparser::ast::Ident::new(p.clone()))) + .collect(); + let asset = + self.get_associated_asset_from_obj_name(&ObjectName(obj_parts), Some(R))?; + Some(asset_to_ref(&asset, col)) + } else { + None + } + } + [] => None, + } + } +} + +// Collects the identifier paths an expression reads, for column-lineage +// inference: `Expr::Identifier(a)` → `["a"]`, `Expr::CompoundIdentifier(t.a)` → +// `["t","a"]`. The derived `Visit` walk recurses through operators, functions, +// casts and CASE, so every leaf identifier of the outer expression is captured. +struct ColumnIdentCollector { + refs: Vec>, + // Depth of nested (sub)queries inside the expression. Identifiers are only + // captured at depth 0: a scalar/correlated subquery's columns belong to ITS + // own FROM, not the outer projection's, so descending would misattribute + // (e.g. `(SELECT x FROM other) AS c FROM orders` must NOT bind `c` to + // `orders.x`). Subquery-derived columns are simply left to annotations. + query_depth: usize, +} + +impl Visitor for ColumnIdentCollector { + type Break = (); + + fn pre_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth += 1; + std::ops::ControlFlow::Continue(()) + } + + fn post_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth = self.query_depth.saturating_sub(1); + std::ops::ControlFlow::Continue(()) + } + + fn pre_visit_expr(&mut self, expr: &Expr) -> std::ops::ControlFlow { + if self.query_depth == 0 { + match expr { + Expr::Identifier(id) => self.refs.push(vec![id.value.clone()]), + Expr::CompoundIdentifier(parts) => self + .refs + .push(parts.iter().map(|id| id.value.clone()).collect()), + _ => {} + } + } + std::ops::ControlFlow::Continue(()) + } } impl Visitor for AssetCollector { @@ -458,9 +740,20 @@ impl Visitor for AssetCollector { match table_factor { TableFactor::Table { name, args, .. } => { if args.is_none() { - // Avoid Table Functions - self.handle_obj_name_pre(name); + // FROM 's3:///…' is a definitive read — record it directly + // so it isn't demoted to a weak in-query mention. + if let Some(asset) = self.get_s3_asset_from_str_literal_table(table_factor) { + self.assets.push(asset); + } } + // For a read-function table factor this pushes ReadFn, making + // every literal inside its arguments a definitive read — the + // direct form (read_csv('s3:///…')) but also list and named + // arguments (read_parquet(['s3:///…'])). Must run for BOTH the + // plain-table and table-function branches: post_visit_table_factor + // pops via handle_obj_name_post unconditionally, so skipping the + // push here would unbalance the stack. + self.handle_obj_name_pre(name); } _ => {} } @@ -486,6 +779,13 @@ impl Visitor for AssetCollector { Expr::Value(ValueWithSpan { value: Value::DoubleQuotedString(s), .. }) => { self.handle_string_literal(s); } + // Read-function call in expression position: its argument literals + // are definitive reads. Balances the pop in `post_visit_expr`. + Expr::Function(func) => { + if get_trivial_obj_name(&func.name).is_some_and(is_read_fn) { + self.current_access_type_stack.push(AccessCtx::ReadFn); + } + } _ => {} } std::ops::ControlFlow::Continue(()) @@ -505,7 +805,11 @@ impl Visitor for AssetCollector { ) -> std::ops::ControlFlow { match statement { sqlparser::ast::Statement::Query(q) => { + // A top-level SELECT is the managed-materialize output, so its + // columns ARE the materialized asset's columns (output resolved + // to the `// materialize` target in `parse_assets`). self.handle_query_reads(q); + self.infer_query_output(q, None); } sqlparser::ast::Statement::Insert(insert) => { @@ -658,18 +962,32 @@ impl Visitor for AssetCollector { } sqlparser::ast::Statement::CreateTable(create_table) => { - self.track_table_definition(&create_table.name); + self.track_table_definition(&create_table.name, create_table.temporary); // `CREATE TABLE x AS SELECT … FROM y` reads y. The AS-query // isn't a `Statement::Query`, so its FROM tables are only - // caught here. + // caught here. Only infer output lineage when `x` is a real + // asset — a CTAS into a local/temp staging table is not the + // materialized output (its columns aren't the asset's). if let Some(query) = &create_table.query { self.handle_query_reads(query); + // Infer only when `x` is a real asset (its output columns + // ARE that asset's), tagged with it so `parse_assets` can + // scope lineage per output. A local/temp staging table is + // not an asset → not inferred. + if let Some(asset) = + self.get_associated_asset_from_obj_name(&create_table.name, Some(W)) + { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } } } - sqlparser::ast::Statement::CreateView { name, query, .. } => { - self.track_table_definition(name); + sqlparser::ast::Statement::CreateView { name, query, temporary, .. } => { + self.track_table_definition(name, *temporary); self.handle_query_reads(query); + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } } // DROP TABLE/VIEW is a write to the dropped object — the @@ -687,13 +1005,15 @@ impl Visitor for AssetCollector { | sqlparser::ast::ObjectType::MaterializedView ) { for name in names { - self.track_table_definition(name); + // DROP is a write to the named object; resolve it as an + // asset if it is one (not a temp-creation context). + self.track_table_definition(name, false); } } } sqlparser::ast::Statement::Copy { target: CopyTarget::File { filename }, .. } => { - self.current_access_type_stack.push(W); + self.current_access_type_stack.push(AccessCtx::CopyWrite); self.handle_string_literal(filename); self.current_access_type_stack.pop(); } @@ -760,7 +1080,7 @@ impl Visitor for AssetCollector { &mut self, query: &sqlparser::ast::Query, ) -> std::ops::ControlFlow { - self.current_access_type_stack.push(R); + self.current_access_type_stack.push(AccessCtx::QueryRead); self.cte_name_stack.push(collect_cte_names(query)); std::ops::ControlFlow::Continue(()) } @@ -825,13 +1145,13 @@ mod tests { Ok(vec![ ParseAssetsResult { kind: AssetKind::S3Object, - path: "/a.parquet".to_string(), + path: "a.parquet".to_string(), access_type: Some(R), columns: None }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/c.parquet".to_string(), + path: "c.parquet".to_string(), access_type: Some(W), columns: None }, @@ -845,6 +1165,151 @@ mod tests { ); } + #[test] + fn test_duckdb_read_key_matches_sdk_write_key() { + // Cross-language lineage: a TS `writeS3File({ s3: "exports/x" })` or + // Python `write_s3_file(S3Object(s3="exports/x"))` records the asset path + // `exports/x` (default storage). A DuckDB reader of the same object must + // resolve to the identical path so the graph connects the producer and + // consumer — both the bare `s3://exports/x` and the triple-slash + // `s3:///exports/x` default-storage form must yield `exports/x`. + for uri in ["s3://exports/x", "s3:///exports/x"] { + let input = format!("SELECT * FROM read_csv('{uri}');"); + let assets = parse_assets(&input).expect("parse").assets; + assert_eq!( + assets, + vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "exports/x".to_string(), + access_type: Some(R), + columns: None + }], + "DuckDB read of {uri} must resolve to the SDK write key" + ); + } + } + + #[test] + fn test_copy_target_echoed_in_select_stays_write_only() { + // The trailing summary SELECT merely mentions the COPY target — it + // must not add a read (rw would draw an asset⇄script cycle). + let input = r#" + COPY (SELECT 1 AS x) TO 's3:///out.csv'; + SELECT 's3:///out.csv' AS target, 42 AS rows_written; + "#; + let s = parse_assets(input).map(|s| s.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "out.csv".to_string(), + access_type: Some(W), + columns: None + }]) + ); + } + + #[test] + fn test_bare_string_mention_without_other_usage_is_a_read() { + let input = r#" + SELECT 's3:///referenced.csv' AS path; + "#; + let s = parse_assets(input).map(|s| s.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "referenced.csv".to_string(), + access_type: Some(R), + columns: None + }]) + ); + } + + #[test] + fn test_self_refresh_read_fn_plus_copy_stays_rw() { + // A definitive read (read_csv) of the same file the script rewrites + // is a real rw — only *bare-literal* mentions are demoted. + let input = r#" + CREATE TABLE tmp AS SELECT * FROM read_csv('s3:///data.csv'); + COPY (SELECT * FROM tmp) TO 's3:///data.csv'; + "#; + let s = parse_assets(input).map(|s| s.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "data.csv".to_string(), + access_type: Some(RW), + columns: None + }]) + ); + } + + #[test] + fn test_read_fn_list_arg_plus_copy_stays_rw() { + // read_parquet's list form is as definitive as the direct literal — + // it must not be demoted to a weak mention when the file is rewritten. + let input = r#" + CREATE TABLE tmp AS SELECT * FROM read_parquet(['s3:///data.parquet']); + COPY (SELECT * FROM tmp) TO 's3:///data.parquet'; + "#; + let s = parse_assets(input).map(|s| s.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "data.parquet".to_string(), + access_type: Some(RW), + columns: None + }]) + ); + } + + #[test] + fn test_read_fn_list_arg_multiple_files_are_reads() { + let input = r#" + SELECT * FROM read_parquet(['s3:///a.parquet', 's3:///b.parquet']); + "#; + let s = parse_assets(input).map(|s| s.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "a.parquet".to_string(), + access_type: Some(R), + columns: None + }, + ParseAssetsResult { + kind: AssetKind::S3Object, + path: "b.parquet".to_string(), + access_type: Some(R), + columns: None + } + ]) + ); + } + + #[test] + fn test_from_string_literal_of_written_file_stays_rw() { + // FROM-position string literal is likewise a definitive read. + let input = r#" + CREATE TABLE tmp AS SELECT * FROM 's3:///data.parquet'; + COPY (SELECT * FROM tmp) TO 's3:///data.parquet'; + "#; + let s = parse_assets(input).map(|s| s.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "data.parquet".to_string(), + access_type: Some(RW), + columns: None + }]) + ); + } + #[test] fn test_sql_asset_parser_attach_no_usage_is_registered_as_unknown() { let input = r#" @@ -1520,7 +1985,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/example_file.parquet"); + assert_eq!(result[0].path, "example_file.parquet"); assert_eq!(result[0].access_type, Some(R)); let columns = result[0].columns.as_ref().expect("Should have columns"); @@ -1551,7 +2016,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/example_file.parquet"); + assert_eq!(result[0].path, "example_file.parquet"); assert!(result[0].columns.is_none()); } @@ -1564,7 +2029,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/example_file.parquet"); + assert_eq!(result[0].path, "example_file.parquet"); let columns = result[0].columns.as_ref().expect("Should have columns"); assert_eq!(columns.get("col1"), Some(&R)); @@ -1583,7 +2048,7 @@ mod tests { assert_eq!(result.len(), 2); assert!(result.iter().any(|a| { - a.path == "/file1.parquet" + a.path == "file1.parquet" && a.columns.as_ref().map_or(false, |c| c.contains_key("col1")) })); assert!(result.iter().any(|a| { @@ -1616,7 +2081,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/test.parquet"); + assert_eq!(result[0].path, "test.parquet"); assert_eq!(result[0].access_type, Some(R)); let columns = result[0].columns.as_ref().expect("Should have columns"); @@ -1920,8 +2385,9 @@ mod ctas_read_tests { assets ); assert!( - assets.iter().any(|a| a.path == "main/exciting_809" - && a.access_type == Some(W)), + assets + .iter() + .any(|a| a.path == "main/exciting_809" && a.access_type == Some(W)), "expected write of main/exciting_809, got {:?}", assets ); @@ -1935,9 +2401,292 @@ mod ctas_read_tests { "#; let assets = parse_assets(input).unwrap().assets; assert!( - assets.iter().any(|a| a.path == "main/fx_rates" && a.access_type == Some(R)), + assets + .iter() + .any(|a| a.path == "main/fx_rates" && a.access_type == Some(R)), "expected read of main/fx_rates, got {:?}", assets ); } + + fn lineage(input: &str) -> Vec { + parse_assets(input).unwrap().column_lineage + } + + #[test] + fn test_infer_lineage_computed_and_passthrough() { + // CTAS with a computed column (amount + tax) and a passthrough (id). + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_bare_column_single_table() { + // Managed-materialize form: a plain top-level SELECT, bare columns + // attributed to the single FROM table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + SELECT amount AS revenue FROM orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "revenue".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_annotation_overrides() { + // The `// column` annotation for `order_total` wins; `id` stays inferred. + let input = r#" + -- column order_total <- datatable://prod/manual.grand_total + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + // Annotation entry is authoritative and listed first. + assert_eq!(got[0].column, "order_total"); + assert_eq!(got[0].inputs[0].from_path, "prod/manual"); + assert_eq!(got[0].inputs[0].from_column, "grand_total"); + // Inferred `id` survives; inferred `order_total` dropped (no dup). + assert!(got.iter().any(|c| c.column == "id")); + assert_eq!(got.iter().filter(|c| c.column == "order_total").count(), 1); + } + + #[test] + fn test_infer_lineage_skips_local_staging_ctas() { + // A CTAS into a TEMP/local table is NOT the materialized output, so its + // columns must not be reported (they'd be anchored to the script's + // `// materialize` target as if they were the final asset's columns). + // The final SELECT reads the local staging table → unresolved → empty. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TEMP TABLE tmp AS SELECT dl.orders.amount AS amt FROM dl.orders; + SELECT amt AS total FROM tmp; + "#; + assert!( + lineage(input).is_empty(), + "staging columns must not be reported as final output; got {:?}", + lineage(input) + ); + } + + #[test] + fn test_infer_lineage_ctas_into_asset_still_inferred() { + // A CTAS whose target IS an asset is the output, so it's still inferred. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT dl.orders.amount AS amt FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "amt".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_temp_table_under_use_is_local() { + // A one-part TEMP table name under an active `USE dl` must NOT resolve to + // an asset (`warehouse/tmp`); it's local, so the final SELECT reading it + // can't invent `warehouse/tmp.amt` as a column source for the output. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + CREATE TEMP TABLE tmp AS SELECT amount AS amt FROM orders; + SELECT amt AS total FROM tmp; + "#; + let got = lineage(input); + assert!( + got.is_empty(), + "temp staging under USE must not leak warehouse/tmp as a source; got {:?}", + got + ); + // And no phantom `warehouse/tmp` asset is recorded. + let assets = parse_assets(input).unwrap().assets; + assert!( + !assets.iter().any(|a| a.path == "warehouse/tmp"), + "temp table must not be recorded as an asset; got {:?}", + assets + ); + } + + #[test] + fn test_infer_lineage_scopes_to_materialize_target() { + // A script with a `// materialize` target plus an AUXILIARY CTAS into a + // different asset: only the materialized target's columns are reported; + // the auxiliary output's columns must not be attributed to it. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.audit AS SELECT dl.orders.id AS aid FROM dl.orders; + SELECT dl.orders.amount AS total FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }], + "auxiliary `audit` columns must not appear on the materialized target" + ); + } + + #[test] + fn test_infer_lineage_drops_ambiguous_multi_output() { + // No `// materialize` target and two real CTAS outputs: which asset the + // flat lineage describes is ambiguous, so inference is dropped rather + // than attributed to an arbitrary one. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.a AS SELECT dl.orders.id AS x FROM dl.orders; + CREATE TABLE dl.b AS SELECT dl.orders.amount AS y FROM dl.orders; + "#; + assert!( + lineage(input).is_empty(), + "ambiguous multi-output must drop inference" + ); + } + + #[test] + fn test_infer_lineage_wildcard_yields_nothing() { + // `SELECT *` has no enumerable output columns → no inferred lineage. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT * FROM dl.orders; + "#; + assert!(lineage(input).is_empty()); + } + + #[test] + fn test_infer_lineage_resolves_joined_inputs() { + // Columns from BOTH sides of an explicit JOIN must resolve, incl. a + // computed column mixing the two. A bare column is dropped (ambiguous + // across the join) rather than misattributed to the first table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT o.id, c.region AS cust_region, o.amount + c.discount AS net + FROM dl.orders o + JOIN dl.customers c ON c.id = o.customer_id; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "cust_region".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "region".to_string(), + }], + }, + ColumnLineage { + column: "net".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "discount".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_does_not_descend_into_subqueries() { + // A scalar subquery's bare column (`amount`) belongs to the subquery's + // own FROM, NOT the outer `dl.orders` — it must not be attributed to the + // outer table. The subquery column is left to annotations; the + // passthrough `id` still resolves. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, (SELECT amount FROM dl.other LIMIT 1) AS c + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }], + "subquery column `c` must be dropped, not misattributed to orders" + ); + } } diff --git a/backend/parsers/windmill-parser-sql/src/lib.rs b/backend/parsers/windmill-parser-sql/src/lib.rs index 931f470d3c..dc094a0a7b 100644 --- a/backend/parsers/windmill-parser-sql/src/lib.rs +++ b/backend/parsers/windmill-parser-sql/src/lib.rs @@ -826,6 +826,29 @@ fn parse_duckdb_file(code: &str) -> anyhow::Result>> { } args.append(&mut parse_sql_sanitized_interpolation(code)); + + // A `// partitioned` script receives its resolved partition as a job arg + // named `partition` (windmill_common::partition::PARTITION_ARG), and duckdb + // binds named parameters only when they appear in the parsed signature — + // so auto-declare it (as `-- $partition (text)` would) to make `$partition` + // usable without a manual declaration. An explicit declaration wins. + // `has_default` keeps the field optional: the platform resolves the value + // at run start when it is not passed explicitly. + if !args.iter().any(|arg| arg.name == "partition") + && windmill_parser::asset_parser::parse_pipeline_annotations(code) + .partition + .is_some() + { + args.push(Arg { + name: "partition".to_string(), + typ: Typ::Str(None), + default: None, + otyp: Some("text".to_string()), + has_default: true, + oidx: None, + otyp_inferred: false, + }); + } Ok(Some(args)) } @@ -1985,4 +2008,63 @@ SELECT x Ok(()) } + + #[test] + fn test_parse_duckdb_partitioned_auto_declares_partition() -> anyhow::Result<()> { + let code = r#"// partitioned daily +// materialize ducklake://main/sales_daily +SELECT $partition AS day, count(*) AS n FROM sales WHERE day = $partition +"#; + let args = parse_duckdb_sig(code)?.args; + assert_eq!( + args, + vec![Arg { + otyp: Some("text".to_string()), + name: "partition".to_string(), + typ: Typ::Str(None), + default: None, + has_default: true, + oidx: None, + otyp_inferred: false, + }] + ); + + // `--`-style annotation headers auto-declare too. + let dash_code = "-- partitioned hourly\nSELECT $partition AS h\n"; + assert_eq!(parse_duckdb_sig(dash_code)?.args, args); + + Ok(()) + } + + #[test] + fn test_parse_duckdb_partitioned_explicit_declaration_wins() -> anyhow::Result<()> { + let code = r#"// partitioned daily +-- $partition (text) +-- $limit (int) = 10 +SELECT * FROM sales WHERE day = $partition LIMIT $limit +"#; + let args = parse_duckdb_sig(code)?.args; + // No duplicate: the explicit (required) declaration is kept as-is. + assert_eq!(args.iter().filter(|a| a.name == "partition").count(), 1); + assert_eq!( + args[0], + Arg { + otyp: Some("text".to_string()), + name: "partition".to_string(), + typ: Typ::Str(None), + default: None, + has_default: false, + oidx: None, + otyp_inferred: false, + } + ); + Ok(()) + } + + #[test] + fn test_parse_duckdb_unpartitioned_does_not_declare_partition() -> anyhow::Result<()> { + let code = "SELECT 1 AS partition_count\n"; + assert_eq!(parse_duckdb_sig(code)?.args, vec![]); + Ok(()) + } } diff --git a/backend/parsers/windmill-parser-ts-asset/src/lib.rs b/backend/parsers/windmill-parser-ts-asset/src/lib.rs index b7fe0fea5c..f273db5cbe 100644 --- a/backend/parsers/windmill-parser-ts-asset/src/lib.rs +++ b/backend/parsers/windmill-parser-ts-asset/src/lib.rs @@ -341,12 +341,25 @@ fn object_str_prop(obj: &ObjectLit, name: &str) -> Option { /// `writeS3File` to a canonical asset path, mirroring the runtime /// `parseS3Object`: an object `{ s3: "", storage?: "" }` maps to /// the URI `s3:///` (empty bucket for default storage, i.e. -/// `s3:///`), and a bare `"s3://bucket/key"` string is passed through. +/// `s3:///`), and a `"s3://bucket/key"` URI string is passed through. +/// String args mirror the runtime `parseS3Object` contract exactly: only a +/// `s3:///` URI with a non-empty key is valid — any other +/// string (bare key, `s3://x`, empty key) throws at run time, so recording an +/// edge for it would be a phantom node for a call that can only error. /// The resulting URI is fed through `parse_asset_syntax` so the stored path /// matches the `// on s3:///…` trigger form exactly. fn s3_object_arg_path(arg: &Expr) -> Option { let uri = match arg { - Expr::Lit(Lit::Str(s)) => s.value.to_string(), + Expr::Lit(Lit::Str(s)) => { + let v = s.value.to_string(); + match v + .strip_prefix("s3://") + .and_then(|rest| rest.split_once('/')) + { + Some((_, key)) if !key.is_empty() => v, + _ => return None, + } + } Expr::Object(obj) => { let key = object_str_prop(obj, "s3")?; let storage = object_str_prop(obj, "storage").unwrap_or_default(); @@ -420,7 +433,7 @@ mod tests { s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/test.csv".to_string(), + path: "test.csv".to_string(), access_type: Some(R), columns: None, },]) @@ -448,7 +461,31 @@ mod tests { s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/raw_events.json".to_string(), + path: "pipelines/km_real/raw_events.json".to_string(), + access_type: Some(W), + columns: None, + },]) + ); + } + + #[test] + fn test_ts_write_key_matches_duckdb_read_key() { + // Cross-language lineage: this write records `exports/x`, the same path a + // DuckDB `read_csv('s3://exports/x')` resolves to (see + // windmill-parser-sql-asset `test_duckdb_read_key_matches_sdk_write_key`), + // so the producer and consumer connect in the pipeline graph. + let input = r#" + import * as wmill from "windmill-client" + export async function main() { + await wmill.writeS3File({ s3: "exports/x" }, "[]") + } + "#; + let s = parse_assets(input); + assert_eq!( + s.map(|r| r.assets).map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "exports/x".to_string(), access_type: Some(W), columns: None, },]) @@ -477,6 +514,42 @@ mod tests { ); } + #[test] + fn test_ts_asset_parser_bare_string_no_asset() { + // A plain (non-`s3://`) string is rejected by the runtime + // `parseS3Object`, so the parser must not record a phantom asset for + // a call that can only error. + let input = r#" + import * as wmill from "windmill-client" + export async function main() { + await wmill.writeS3File("pipelines/etl/out.jsonl", "[]") + } + "#; + let s = parse_assets(input); + assert_eq!(s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![])); + } + + #[test] + fn test_ts_asset_parser_invalid_uri_no_write_edge() { + // `s3://x` (no key part) and `s3://bucket/` (empty key) are rejected + // by the runtime `parseS3Object` — same rule for the SDK-arg path: + // no R/W edge. The generic URI-literal scan may still record them as + // ambiguous (`access_type: None`) assets, like any `s3://…` string + // constant anywhere in a script. + let input = r#" + import * as wmill from "windmill-client" + export async function main() { + await wmill.writeS3File("s3://broken", "[]") + await wmill.writeS3File("s3://bucket/", "[]") + } + "#; + let assets = parse_assets(input).expect("parse").assets; + assert!( + assets.iter().all(|a| a.access_type.is_none()), + "invalid URIs must not produce R/W edges: {assets:?}" + ); + } + #[test] fn test_ts_asset_parser_multiple_s3_object_writes() { // Mirrors the f/km/r_seed shape: several direct object-form writes in @@ -497,25 +570,25 @@ mod tests { Ok(vec![ ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/enriched.json".to_string(), + path: "pipelines/km_real/enriched.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/raw_events.json".to_string(), + path: "pipelines/km_real/raw_events.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/report.json".to_string(), + path: "pipelines/km_real/report.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/summary.json".to_string(), + path: "pipelines/km_real/summary.json".to_string(), access_type: Some(W), columns: None, }, @@ -536,7 +609,7 @@ mod tests { s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/out.json".to_string(), + path: "out.json".to_string(), access_type: Some(W), columns: None, },]) diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 6c00419131..4961d0907a 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.733.1" +version = "1.753.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.733.1" +version = "1.753.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.733.1" +version = "1.753.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.733.1" +version = "1.753.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 4ab52f15a5..5763177b0b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.733.1" +version = "1.753.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/parsers/windmill-parser-yaml/src/lib.rs b/backend/parsers/windmill-parser-yaml/src/lib.rs index 5d9945cadd..f2a1a2b824 100644 --- a/backend/parsers/windmill-parser-yaml/src/lib.rs +++ b/backend/parsers/windmill-parser-yaml/src/lib.rs @@ -269,6 +269,12 @@ pub struct DelegateToGitRepoDetails { pub commit: Option, pub inventories_location: Option, pub vars_location: Option, + /// Path (relative to the cloned repo root) of an `ansible.cfg` to use as the + /// effective config for the run. When set, Windmill points `ANSIBLE_CONFIG` at + /// it so the repo's own settings (roles paths, inventory plugins, callbacks…) + /// apply, and only injects the settings that depend on runtime state it alone + /// controls (temp/home dirs, vault password) on top. + pub ansible_cfg: Option, #[serde(default, skip_serializing_if = "std::ops::Not::not")] pub install_requirements: bool, } @@ -629,6 +635,10 @@ fn extract_delegate_to_git_repo_details(value: &Yaml) -> Option Option, // `// freshness ` — SLA stating outputs must be at most - // `duration` old. Active backstop: when no other trigger has fired the - // script within the window, a watchdog re-runs it. Distinct from - // schedule (which is producer cadence); freshness is consumer SLA and - // applies regardless of which trigger last fired. + // `duration` old. Drives passive monitoring in CE (the asset graph + // colors the node's badge fresh/stale against its last successful run) + // and the enterprise watchdog (windmill-queue `freshness_watchdog`), + // which re-runs a stale unpartitioned producer. Distinct from schedule + // (which is producer cadence); freshness is consumer SLA and applies + // regardless of which trigger last fired. #[serde(skip_serializing_if = "Option::is_none", default)] pub freshness: Option, // `// trigger all` → AND join barrier; default (`any`) = OR (current @@ -107,6 +109,32 @@ pub struct ParseAssetsOutput { // The delay is a raw duration string parsed at deploy (parser-light). #[serde(skip_serializing_if = "Option::is_none", default)] pub retry: Option, + // `// materialize [manual] [append] [key=]` — + // managed-materialization target + its strategy. At most one per script. + // Drives the worker's write-strategy + snapshot capture. + #[serde(skip_serializing_if = "Option::is_none", default)] + pub materialize: Option, + // `// data_test …` — data-quality assertions run against the + // materialized asset after the write commits. Accumulating (multiple + // lines allowed). Drives the worker's post-materialize verifier probes. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub data_tests: Vec, + // `// column <- .[, …]` — declared column-level lineage, + // one entry per output column. Accumulating. Pure metadata: drives the + // column-lineage graph view, executes nothing. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub column_lineage: Vec, + // Bare `// macros` (must be alone on the line, like `// pipeline`) — + // marks this DuckDB script as a workspace *macro library*: its body is + // CREATE [OR REPLACE] MACRO statements (plus plain setup) registered at + // deploy and injected as TEMP macros into consumer jobs at run time. + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub macros: bool, + // `// use ` — force-inject the whole named macro + // library (definitions + its setup statements) into this script's jobs, + // for dynamic SQL that call-detection can't see. Accumulating. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub use_libs: Vec, } #[derive(Serialize, Debug, PartialEq, Clone)] @@ -190,6 +218,40 @@ pub struct PartitionSpec { pub start: Option, } +impl PartitionKind { + /// The `strftime`/chrono format that renders a time grain's identity + /// string. SINGLE SOURCE OF TRUTH: the partition resolver stamps the stored + /// identity with this, and the `wm_partition` materialize macro filters + /// against it — the two must never drift, so both read it from here. + /// `Dynamic` has no wall-clock identity; it falls back to a plain date only + /// where some format is unconditionally required (never for bucketing). + pub fn default_time_format(&self) -> &'static str { + match self { + PartitionKind::Hourly => "%Y-%m-%dT%H", + PartitionKind::Weekly => "%G-W%V", + PartitionKind::Monthly => "%Y-%m", + _ => "%Y-%m-%d", + } + } +} + +impl PartitionSpec { + /// Effective identity format for a TIME partition: the explicit `format=` + /// override, else the per-grain default. `None` for `dynamic` — its + /// identity is a caller-supplied key, not a formatted instant, so there is + /// no `strftime` bucketing expression (and hence no `wm_partition` macro). + pub fn time_strftime_format(&self) -> Option<&str> { + match self.kind { + PartitionKind::Dynamic { .. } => None, + _ => Some( + self.format + .as_deref() + .unwrap_or_else(|| self.kind.default_time_format()), + ), + } + } +} + // Freshness SLA. The duration is kept as a raw string ("1h", "30m", "2d") // and validated downstream — the parser deliberately doesn't bind to a // specific duration crate so the annotation grammar stays parser-light. @@ -209,6 +271,228 @@ pub struct RetrySpec { pub delay: Option, } +// `// materialize [manual] [append] [key=] [history] [track=]` +// — declares that this script produces a *managed* materialization of `` +// (a `ducklake://` table). By default the runtime generates the write DDL around +// the script's single trailing `SELECT` and owns idempotency, partition-state +// and snapshot capture. `manual` is the escape hatch: the script writes its own +// DDL and the runtime only records state (track-only). The reconciliation +// strategy options apply to managed mode: none → DELETE-by-partition + INSERT +// (replace); `key=` → MERGE (dedup within slice, SCD type 1); `append` → +// INSERT-only. `append` wins if both are given (deploy-time warning). +// `key= history` upgrades the merge to SCD type 2: the SELECT is the current +// snapshot (one row per key), and a change to any tracked column (`track=`, +// default all non-key) closes the prior version and opens a new one, keeping full +// history (`valid_from`/`valid_to`/`is_current`). The leading keyword `scd2` is a +// recognized alias for `history`. `deletes=close` (scd2 only) also closes a key +// that disappears from the snapshot; default leaves absent keys current. +// `on_schema_change=ignore` suppresses downstream schema-contract warnings for +// the produced asset (save-time metadata only; default `warn`). +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct MaterializeSpec { + pub target_kind: AssetKind, + pub target_path: String, + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub manual: bool, + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub append: bool, + #[serde(skip_serializing_if = "Option::is_none", default)] + pub unique_key: Option, + // `scd2` managed history mode: the SELECT is the current snapshot (one row + // per `unique_key`), and the runtime maintains a Slowly-Changing-Dimension + // type-2 history (`valid_from`/`valid_to`/`is_current`). `unique_key` (the + // `key=` opt) is the natural key; `track` lists the columns whose change + // opens a new version (empty ⇒ all non-key columns). Managed mode only. + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub scd2: bool, + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub track: Vec, + // scd2 only: `deletes=close` opts into hard-delete-close — a key that + // disappears from the snapshot has its current version closed (dbt's + // `hard_deletes=close`). Default (false) leaves absent keys current. + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub close_deleted: bool, + // `on_schema_change=warn|ignore|fail|sync` governs two orthogonal things: + // • Save-time contract warnings (gap #2b): consumers referencing columns + // the captured schema no longer has warn by default; only `ignore` + // suppresses those warnings (`warn`/`fail`/`sync` all keep them). + // • Run-time write guardrails for the persist-and-mutate strategies + // (partitioned replace, merge, append), where the table schema is fixed + // at first CREATE and the write is positional — a renamed/added/removed + // SELECT column silently lands in the wrong column. `warn` logs the + // drift and proceeds positionally; `fail` aborts before mutating; `sync` + // ALTERs the table to match and writes by name. Whole-table replace and + // scd2 are unaffected. See `sql_materialize.rs`. + #[serde(skip_serializing_if = "OnSchemaChange::is_warn", default)] + pub on_schema_change: OnSchemaChange, +} + +impl MaterializeSpec { + /// The `_current` SCD2 companion view this managed materialize also + /// produces, or `None` when it isn't a managed scd2 target. Managed scd2 + /// creates the base table *and* a `_current` "latest row per key" view + /// each run (see `sql_materialize.rs`); `manual` mode owns its own DDL and + /// short-circuits before that codegen, so it produces no companion. + pub fn scd2_current_target(&self) -> Option<(AssetKind, String)> { + (self.scd2 && !self.manual) + .then(|| (self.target_kind, format!("{}_current", self.target_path))) + } + + /// Every asset this managed materialize produces: the base table, plus — for + /// managed scd2 — the `_current` companion view. The producer's + /// trailing `SELECT` doesn't express these writes (the runtime generates the + /// DDL), so this is the single source of truth every graph surface (deploy + /// asset rows, the CLI `--local` graph, and the frontend live graph) uses to + /// link reads of the base *and* the `_current` view back to this producer. + pub fn write_targets(&self) -> Vec<(AssetKind, String)> { + let mut targets = vec![(self.target_kind, self.target_path.clone())]; + targets.extend(self.scd2_current_target()); + targets + } +} + +// dbt's `on_schema_change`, covering both the save-time contract check and the +// run-time write guardrail for the positional persist-and-mutate strategies: +// • `warn` (default): surface consumer contract warnings; at write time, log +// the drift loudly and proceed with the positional write against the fixed +// table schema. +// • `ignore`: suppress consumer contract warnings; at write time, no guard +// (the pre-guardrail behaviour). +// • `fail`: keep contract warnings; at write time, abort the run before +// mutating when the SELECT's column *set* diverges from the table's. +// • `sync`: keep contract warnings; at write time, ALTER the table to match +// the SELECT (add/drop columns) and INSERT BY NAME. +// `warn`/`fail` drift detection is name-set based (added/removed columns), which +// is what the positional persist-and-mutate INSERT can misalign on. It does NOT +// flag a pure *reorder* of same-named columns: `SELECT b, a` into a `(a, b)` +// table has an identical column set, so `fail` does not abort and the positional +// INSERT swaps the values. Reorder-safety is exactly what `sync` provides +// (INSERT BY NAME maps by name), so a SELECT whose column order is not pinned to +// the table's should use `sync`, not `fail`. (An ordered-list comparison would +// close this, but a false positive there would abort a correctly-aligned write, +// so the guard stays on the set difference.) +// `fail`/`sync` only affect partitioned replace, merge and append; whole-table +// replace already rebuilds each run, and scd2 has no positional write — for an +// scd2 target `sync` degrades to `warn` (no write-time effect; deploy-time +// rejection is out of scope here). +#[derive(Serialize, Debug, PartialEq, Eq, Clone, Copy, Default)] +#[serde(rename_all = "lowercase")] +pub enum OnSchemaChange { + #[default] + Warn, + Ignore, + Fail, + Sync, +} + +impl OnSchemaChange { + pub fn is_warn(&self) -> bool { + matches!(self, OnSchemaChange::Warn) + } +} + +impl MaterializeSpec { + /// Deploy-time validation of the option combination against the script's + /// partitioning, returning a human-facing error for combinations the + /// runtime cannot honor. Called at save (`create_script_internal`) so a + /// misconfigured script is rejected up front, and again in the DuckDB + /// executor as a safety net for preview/test runs that never deploy. Both + /// checks are SCD2-specific and inert for `manual` mode (which owns its DDL + /// and ignores the reconciliation strategy). `partitioned` is whether the + /// script declares `// partitioned`. + pub fn validate(&self, partitioned: bool) -> Result<(), String> { + if self.manual || !self.scd2 { + return Ok(()); + } + // SCD2 needs a natural key to identify an entity across versions. + if self.unique_key.as_deref().map_or(true, str::is_empty) { + return Err( + "materialize scd2: requires a natural key — add `key=` (e.g. \ + `// materialize ducklake:/// key=id history`)" + .to_string(), + ); + } + // SCD2's diff/close/open shape has no partition-scoped form in v1. + if partitioned { + return Err( + "materialize scd2: `// partitioned` is not supported with scd2 in v1 — remove \ + `// partitioned`, or drop `history`/`scd2` to materialize the partition without \ + history" + .to_string(), + ); + } + Ok(()) + } +} + +// `// data_test …` — a data-quality assertion run against the +// freshly-materialized asset (post DELETE+INSERT), failing the run on +// violation. The first extensible annotation family: the parser turns a +// `data_test` line into one of a known *vocabulary* of checks, and the +// runtime turns each check into a SQL "verifier" probe. A sibling annotation +// family (e.g. column-lineage) follows the same shape — a keyword head +// selecting a variant, the rest parsed per-variant — rather than growing a +// new closed list. See `docs/ducklake-materialization.md` §"Extensible +// annotations". Multiple `// data_test` lines accumulate (unlike the +// single-value annotations above, which are first-write-wins). +// +// Built-ins mirror dbt's generic data tests; `Custom` is the escape hatch +// (dbt's singular test): a DuckDB script path whose SELECT returns the +// violating rows. The keyword is `data_test` — NOT `test` — to stay clear +// of the unrelated `// test:` CI-test annotation (see +// `windmill_common::schema::parse_ci_test_annotation`), matching dbt 1.8's +// own `tests:` → `data_tests:` rename. +#[derive(Serialize, Debug, PartialEq, Clone)] +#[serde(tag = "type", rename_all = "snake_case")] +pub enum DataTest { + // `// data_test unique ` — no two non-NULL rows share `column`. + Unique { column: String }, + // `// data_test not_null ` — `column` is never NULL. + NotNull { column: String }, + // `// data_test accepted_values = a,b,c` — every non-NULL value of + // `column` is one of `values` (comma-separated; surrounding quotes stripped). + AcceptedValues { column: String, values: Vec }, + // `// data_test relationships -> .` — referential + // integrity: every non-NULL `column` value exists in `to_path`'s `to_column`. + Relationships { column: String, to_kind: AssetKind, to_path: String, to_column: String }, + // `// data_test ` — escape hatch: a deployed DuckDB script + // whose trailing SELECT returns the violating rows (non-empty ⇒ fail). + Custom { path: String }, +} + +// `// column <- .[, …]` — declared column-level +// lineage: one output column of this script's produced asset and the upstream +// source columns it derives from. A sibling of `DataTest` in the extensible +// annotation family (`docs/pipelines-vs-dbt.md` §3): same parse shape — a head +// token (the output column) then a per-variant tail — but accumulating, one +// line per output column. Unlike `data_test` these are pure metadata: they +// drive the column-lineage graph view, never a runtime probe. +// +// dbt derives column lineage from SQL-AST parsing; Windmill is polyglot +// (Python/TS/Bash/SQL in one DAG), so a uniform AST is not available. The +// annotation is the explicit, language-agnostic declaration — the same +// "annotations are real comments parsed strictly" stance as the rest of the +// pipeline grammar. Body-inferred per-asset column *sets* (`columns` on +// `ParseAssetsResult`) complement it but cannot express column→column edges. +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnLineage { + // The produced asset's output column this line describes. + pub column: String, + // Upstream source columns it derives from (≥1; malformed refs dropped). + pub inputs: Vec, +} + +// One `.` upstream reference inside a `// column` line. The +// asset URI accepts the default-syntax shorthands (like `// materialize` / +// `// data_test relationships`); the column is the segment after the final +// `.` (so a schema-qualified `warehouse/main.orders.amount` keeps `amount`). +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnRef { + pub from_kind: AssetKind, + pub from_path: String, + pub from_column: String, +} + // `// trigger any` (default) vs `// trigger all`. `Any` = OR: any trigger // firing runs the script (current behaviour). `All` = AND: the script // runs only once every partition-bearing input has materialized at the @@ -239,6 +523,21 @@ pub struct PipelineAnnotations { pub debounce_default: Option, pub tag: Option, pub retry: Option, + pub materialize: Option, + pub data_tests: Vec, + pub column_lineage: Vec, + pub macros: bool, + pub use_libs: Vec, + // `// mute ` — suppress the auto-derived cascade edge for a read + // that would otherwise trigger this script (a lookup / slowly-changing + // dimension you read every run but don't want to re-run on). Only Asset + // specs are stored; native trigger kinds are never auto-derived, so + // muting them is meaningless. + pub mute: Vec, + // `// mute all` — opt out of auto-derivation entirely for this script. + // Falls back to explicit-`// on`-only semantics. Explicit `// on` edges + // are unaffected. + pub mute_all: bool, } impl ParseAssetsOutput { @@ -262,10 +561,37 @@ impl ParseAssetsOutput { debounce_default: pipeline.debounce_default, tag: pipeline.tag, retry: pipeline.retry, + materialize: pipeline.materialize, + data_tests: pipeline.data_tests, + column_lineage: pipeline.column_lineage, + macros: pipeline.macros, + use_libs: pipeline.use_libs, } } } +// Combine column lineage inferred from the body (SQL AST) with lineage declared +// via `// column` annotations. The annotation is the *override*: where both +// describe the same output column, the explicit declaration wins and the +// inferred entry is dropped. Inferred entries are also deduped by output column +// among themselves (first wins). Used by the language asset-parsers so a +// `// column` line can correct a mis-inferred edge without disabling inference +// for the rest of the columns. +pub fn merge_column_lineage( + inferred: Vec, + annotated: Vec, +) -> Vec { + let mut seen: std::collections::HashSet = + annotated.iter().map(|c| c.column.clone()).collect(); + let mut out = annotated; + for c in inferred { + if seen.insert(c.column.clone()) { + out.push(c); + } + } + out +} + #[derive(Debug, Clone, Serialize)] pub struct DelegateToGitRepoDetails { pub resource: String, @@ -355,6 +681,24 @@ pub fn parse_asset_syntax(s: &str, enable_default_syntax: bool) -> Option<(Asset for (prefix, kind) in ASSET_KINDS.iter() { if s.starts_with(prefix) { let path = &s[prefix.len()..]; + // Canonicalize S3 keys to a single asset identity. The SDK object + // form (`{ s3: "key" }` / `S3Object(s3="key")`, default storage) + // resolves to `s3:///key`, whose path is `/key`, while DuckDB + // `s3://key` and `// on s3://key` yield the bare `key`. Strip every + // leading slash so the triple-slash default-storage form and the + // `s3://storage/key` form share one path — otherwise a TS/Python + // writer and a DuckDB reader of the same object become disconnected + // nodes in the pipeline graph. Stripping ALL leading slashes (not + // just one) keeps the identity stable through URI reconstruction: + // `trigger_spec_to_row` rebuilds `s3://`, so a canonical path + // must never itself start with `/` or the rebuilt ref would parse + // back to a different key. Only leading slashes are touched, so + // Hive-partition keys (`s3://b/y=2024/f.parquet`) are untouched. + let path = if matches!(kind, AssetKind::S3Object) { + path.trim_start_matches('/') + } else { + path + }; return Some((*kind, path)); } } @@ -459,14 +803,18 @@ fn parse_kv_opts(s: &str) -> BTreeMap { out } -// Scan raw source for pipeline annotations. Language-agnostic: any line -// whose first non-whitespace tokens are a comment prefix (`//`, `#`, or -// `--`) followed by one of the recognized keywords: +// Scan the leading comment header for pipeline annotations. Only the +// contiguous block of comment lines at the top of the file is considered +// (blank lines tolerated, scan stops at the first line of actual code) so +// that ordinary comments in the body can't false-positive as annotations. +// Language-agnostic: any header line whose first non-whitespace tokens are +// a comment prefix (`//`, `#`, or `--`) followed by one of the recognized +// keywords: // - `pipeline` → opt-in marker (must be alone on the line) // - `on ` → asset / native trigger edge (including // the marker-only `on schedule` form) // - `partitioned [opts]` → partition declaration -// - `freshness ` → SLA / active backstop +// - `freshness ` → SLA window (badge + EE watchdog) // - `tag ` → worker-tag override (annotation wins // over UI-set value at deploy) // - `retry []` → cascade-only retry policy @@ -499,6 +847,9 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { for raw_line in code.lines() { let line = raw_line.trim_start(); + if line.is_empty() { + continue; + } let rest = if let Some(r) = line.strip_prefix("//") { r } else if let Some(r) = line.strip_prefix("--") { @@ -506,7 +857,11 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { } else if let Some(r) = line.strip_prefix('#') { r } else { - continue; + // Annotations live in the leading comment header. Stop at the first + // line of actual code so comments inside the body (e.g. a regular + // `# tag ...` prose comment) can't false-positive as annotations. + // Mirrors BashAnnotations::sandbox_image / ssh_target. + break; }; let rest = rest.trim_start(); @@ -519,6 +874,30 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { continue; } + if let Some(after_kw) = consume_keyword(rest, "macros") { + // Strict like `pipeline`: keyword alone on the line, so prose + // such as `// macros are defined below` never false-positives. + if after_kw.trim().is_empty() { + out.macros = true; + } + continue; + } + + // `// use ` — accumulating. The argument must be a + // single whitespace-free token containing `/` (all script paths do), + // so prose like `// use this script to …` is dropped fail-safe. + if let Some(after_kw) = consume_keyword(rest, "use") { + let path = after_kw.trim(); + if !path.is_empty() + && !path.contains(char::is_whitespace) + && path.contains('/') + && !out.use_libs.iter().any(|p| p == path) + { + out.use_libs.push(path.to_string()); + } + continue; + } + if let Some(after_kw) = consume_keyword(rest, "partitioned") { if out.partition.is_none() { if let Some(spec) = parse_partitioned_spec(after_kw.trim()) { @@ -556,7 +935,14 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { if let Some(after_kw) = consume_keyword(rest, "tag") { let name = after_kw.trim(); - if !name.is_empty() && out.tag.is_none() { + // Worker tags are single-word identifiers (e.g. `heavy`, `gpu`). + // A value with whitespace or beyond the `script.tag` column width + // is almost certainly a regular comment starting with "# tag ...". + if !name.is_empty() + && !name.contains(char::is_whitespace) + && name.len() <= 50 + && out.tag.is_none() + { out.tag = Some(name.to_string()); } continue; @@ -571,6 +957,55 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { continue; } + if let Some(after_kw) = consume_keyword(rest, "materialize") { + if out.materialize.is_none() { + if let Some(spec) = parse_materialize_spec(after_kw.trim()) { + out.materialize = Some(spec); + } + } + continue; + } + + // `// mute all` opts out of auto-derived cascade edges entirely; + // `// mute ` suppresses the one edge. Only asset refs are + // muteable — native trigger kinds are never auto-derived. Checked + // before the generic `on`/asset shorthand (a complete word, so + // prose like `// muted for now` never matches). + if let Some(after_kw) = consume_keyword(rest, "mute") { + let arg = after_kw.trim(); + if arg == "all" { + out.mute_all = true; + } else if let Some(spec @ TriggerSpec::Asset { .. }) = parse_trigger_spec(arg) { + if !out.mute.contains(&spec) { + out.mute.push(spec); + } + } + continue; + } + + // `data_test` is checked before `on`/asset shorthands and is a complete + // word (so it never collides with the `// test:` CI annotation, which + // has no whitespace after `test`). Accumulates — every well-formed line + // adds a check; malformed lines are dropped (fail-safe, the missing + // check is then simply absent from the graph + run). + if let Some(after_kw) = consume_keyword(rest, "data_test") { + if let Some(spec) = parse_data_test_spec(after_kw.trim()) { + out.data_tests.push(spec); + } + continue; + } + + // `// column <- .[, …]` — accumulating column lineage. + // A complete word, so it never swallows a body comment that happens to + // start with `column` followed by non-lineage prose (that has no `<-` + // and is dropped fail-safe). Checked before `on`/asset shorthands. + if let Some(after_kw) = consume_keyword(rest, "column") { + if let Some(spec) = parse_column_lineage_spec(after_kw.trim()) { + out.column_lineage.push(spec); + } + continue; + } + if let Some(after_kw) = consume_keyword(rest, "on") { let spec_text = after_kw.trim(); if spec_text.is_empty() { @@ -598,6 +1033,39 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { out } +// Count `// data_test <…>` lines in the leading comment header whose right-hand +// side fails to parse into a check. `parse_pipeline_annotations` drops these +// fail-safe (a malformed line just yields no check), which is the wrong default +// for a data-quality assertion: a typo silently disables the test. The deploy +// path uses this count to warn. Same leading-block boundary as the parser (stop +// at the first non-comment line) so a body comment can't be miscounted, and the +// same `parse_data_test_spec` grammar so "malformed" means exactly what the +// parser rejects — no second grammar to drift. +pub fn count_malformed_data_tests(code: &str) -> usize { + let mut malformed = 0; + for raw_line in code.lines() { + let line = raw_line.trim_start(); + if line.is_empty() { + continue; + } + let rest = if let Some(r) = line.strip_prefix("//") { + r + } else if let Some(r) = line.strip_prefix("--") { + r + } else if let Some(r) = line.strip_prefix('#') { + r + } else { + break; + }; + if let Some(after_kw) = consume_keyword(rest.trim_start(), "data_test") { + if parse_data_test_spec(after_kw.trim()).is_none() { + malformed += 1; + } + } + } + malformed +} + // Parse a `// retry []` right-hand side. `` is a // non-negative decimal; `` is an optional raw duration string left // for `parse_duration_secs` to validate at deploy. A bare zero count (or @@ -618,6 +1086,201 @@ fn parse_retry_spec(s: &str) -> Option { Some(RetrySpec { count, delay }) } +// Parse a `// materialize [manual] [append] [key=] [history] +// [track=]` right-hand side. An optional leading `manual` token opts out +// of managed mode (track-only); a leading `scd2` token is an alias for the +// `history` flag. The next whitespace token is the target asset URI +// (default-syntax shorthands enabled, so `ducklake` → `ducklake://main`); the +// remainder are strategy options — bare `append`, bare `history` (SCD type-2 on +// a keyed merge), `key=` (merge/scd2 key), `track=` (scd2 tracked +// columns), and `deletes=close` (scd2 hard-delete-close) — which apply to managed +// mode only. A missing/empty target yields `None` (the annotation is dropped, +// fail-safe). +fn parse_materialize_spec(s: &str) -> Option { + // One optional leading mode keyword: `manual` (escape hatch, track-only) or + // `scd2` (alias for the `history` flag below). A missing keyword is the + // default managed mode. + fn strip_mode<'a>(s: &'a str, kw: &str) -> Option<&'a str> { + s.strip_prefix(kw) + .filter(|after| after.is_empty() || after.starts_with(char::is_whitespace)) + .map(|after| after.trim_start()) + } + let (manual, scd2_kw, rest) = if let Some(after) = strip_mode(s, "manual") { + (true, false, after) + } else if let Some(after) = strip_mode(s, "scd2") { + (false, true, after) + } else { + (false, false, s) + }; + let mut it = rest.trim().splitn(2, char::is_whitespace); + let asset_tok = it.next()?; + let opts_str = it.next().unwrap_or(""); + let (target_kind, path) = parse_asset_syntax(asset_tok.trim(), true)?; + if path.is_empty() { + return None; + } + let append = opts_str.split_whitespace().any(|t| t == "append"); + // SCD type-2 history mode. The primary spelling is the bare `history` flag on + // a keyed merge (`key=history`) — it reads as "a keyed upsert that keeps + // history"; the leading `scd2` keyword is a recognized alias for the same. + let scd2 = scd2_kw || opts_str.split_whitespace().any(|t| t == "history"); + let opts = parse_kv_opts(opts_str); + let unique_key = opts.get("key").filter(|k| !k.is_empty()).cloned(); + // `track=` (scd2 only): comma-separated columns whose change opens a + // new version. Empty entries dropped; an empty list ⇒ track all non-key cols. + // Like every `=`-option here the value is whitespace-terminated, so the list + // must contain no spaces (`track=a,b`, not `track=a, b` — the rest is dropped). + let track = opts + .get("track") + .map(|v| { + v.split(',') + .map(|c| c.trim().to_string()) + .filter(|c| !c.is_empty()) + .collect::>() + }) + .unwrap_or_default(); + // `deletes=close` (scd2 only) opts into hard-delete-close; any other value + // (or absence) keeps the soft-delete default. + let close_deleted = opts.get("deletes").map(|v| v == "close").unwrap_or(false); + // `on_schema_change=ignore|fail|sync`; any other value (or absence) keeps + // the `warn` default, fail-safe like `deletes=` above (a typo must never + // silently disable the guardrail). + let on_schema_change = match opts.get("on_schema_change").map(String::as_str) { + Some("ignore") => OnSchemaChange::Ignore, + Some("fail") => OnSchemaChange::Fail, + Some("sync") => OnSchemaChange::Sync, + _ => OnSchemaChange::Warn, + }; + Some(MaterializeSpec { + target_kind, + target_path: path.to_string(), + manual, + append, + unique_key, + scd2, + track, + close_deleted, + on_schema_change, + }) +} + +// Parse a `// data_test …` right-hand side into one `DataTest`. The +// leading token selects the variant; the remainder is parsed per-variant. +// Anything not matching a built-in keyword is the `Custom` escape hatch — a +// single script-path token. Returns `None` for malformed input so a typo +// fails safe (the check is dropped, never silently mis-parsed). +// +// This is the extension seam: a new built-in is one match arm + its parser; +// a sibling annotation family (column-lineage) reuses the same head-keyword +// dispatch shape rather than adding a parallel closed list. +fn parse_data_test_spec(s: &str) -> Option { + let s = s.trim(); + if s.is_empty() { + return None; + } + let mut it = s.splitn(2, char::is_whitespace); + let head = it.next()?; + let rest = it.next().unwrap_or("").trim(); + match head { + "unique" => Some(DataTest::Unique { column: single_ident(rest)? }), + "not_null" => Some(DataTest::NotNull { column: single_ident(rest)? }), + "accepted_values" => parse_accepted_values(rest), + "relationships" => parse_relationships(rest), + // Custom escape hatch: the whole right-hand side must be one path token + // (`head` with no trailing content). Trailing content after a + // non-built-in head is a malformed built-in (e.g. `uniq order_id`) and + // is rejected rather than misread as a path. + _ if rest.is_empty() => Some(DataTest::Custom { path: head.to_string() }), + _ => None, + } +} + +// A single bare identifier token (column name). Rejects empty / multi-token +// input. The identifier is double-quoted + escaped at codegen, so any +// character is safe here; we only enforce "exactly one token". +fn single_ident(s: &str) -> Option { + let s = s.trim(); + if s.is_empty() || s.split_whitespace().count() != 1 { + return None; + } + Some(s.to_string()) +} + +// Strip one layer of matching surrounding single or double quotes. +fn unquote(s: &str) -> &str { + let b = s.as_bytes(); + if b.len() >= 2 && (b[0] == b'"' || b[0] == b'\'') && b[b.len() - 1] == b[0] { + &s[1..s.len() - 1] + } else { + s + } +} + +// `= a,b,c` — column, then `=`, then a comma-separated value list. +// Surrounding quotes are stripped per value; empty values are dropped; a +// value may not itself contain a comma (v1 limitation). +fn parse_accepted_values(s: &str) -> Option { + let (col, vals) = s.split_once('=')?; + let column = single_ident(col)?; + let values: Vec = vals + .split(',') + .map(|v| unquote(v.trim()).to_string()) + .filter(|v| !v.is_empty()) + .collect(); + if values.is_empty() { + return None; + } + Some(DataTest::AcceptedValues { column, values }) +} + +// `-> .` — referential integrity. The referenced +// column is the segment after the final `.`; everything before it is the +// asset URI (default-syntax shorthands enabled, like `// materialize`). +fn parse_relationships(s: &str) -> Option { + let (col, target) = s.split_once("->")?; + let column = single_ident(col)?; + let target = target.trim(); + let (asset_uri, ref_col) = target.rsplit_once('.')?; + let to_column = single_ident(ref_col)?; + let (to_kind, to_path) = parse_asset_syntax(asset_uri.trim(), true)?; + if to_path.is_empty() { + return None; + } + Some(DataTest::Relationships { column, to_kind, to_path: to_path.to_string(), to_column }) +} + +// Parse a `// column <- [, …]` right-hand side. The head +// (before `<-`) is the output column; the tail is a comma-separated list of +// `.` upstream references. Mirrors `parse_accepted_values`' +// "drop empties, require ≥1" stance: individually malformed refs are dropped +// and the line is kept iff at least one ref parses; a missing `<-`, a non-ident +// output column, or zero valid refs drops the whole line (fail-safe). +fn parse_column_lineage_spec(s: &str) -> Option { + let (out_col, refs) = s.split_once("<-")?; + let column = single_ident(out_col)?; + let inputs: Vec = refs + .split(',') + .filter_map(|r| parse_column_ref(r.trim())) + .collect(); + if inputs.is_empty() { + return None; + } + Some(ColumnLineage { column, inputs }) +} + +// `.` — the referenced column is the segment after the final +// `.`; everything before it is the asset URI (default-syntax shorthands +// enabled, like `// materialize`). Same shape as `parse_relationships`' target. +fn parse_column_ref(s: &str) -> Option { + let (asset_uri, ref_col) = s.rsplit_once('.')?; + let from_column = single_ident(ref_col)?; + let (from_kind, from_path) = parse_asset_syntax(asset_uri.trim(), true)?; + if from_path.is_empty() { + return None; + } + Some(ColumnRef { from_kind, from_path: from_path.to_string(), from_column }) +} + // Parse a `// partitioned [opts]` right-hand side. Recognized kinds: // `daily`, `hourly`, `weekly`, `monthly` (with optional tz/format/start), // and `dynamic key=""` (plus optional format). @@ -697,6 +1360,83 @@ fn parse_trigger_spec(s: &str) -> Option { mod pipeline_annotation_tests { use super::*; + #[test] + fn s3_key_normalization_unifies_uri_forms() { + // A TS/Python SDK write of `{ s3: "exports/x" }` (default storage) + // resolves to the URI `s3:///exports/x`, while a DuckDB read of + // `s3://exports/x` and the `// on s3://exports/x` trigger form yield the + // bare `exports/x`. All three must canonicalize to one asset key so + // the writer and reader connect in the pipeline graph. + let sdk_write = parse_asset_syntax("s3:///exports/x", false); + let duckdb_read = parse_asset_syntax("s3://exports/x", false); + assert_eq!(sdk_write, Some((AssetKind::S3Object, "exports/x"))); + assert_eq!(duckdb_read, Some((AssetKind::S3Object, "exports/x"))); + assert_eq!(sdk_write, duckdb_read); + + // The `// on` trigger annotation goes through the same function. + assert_eq!( + parse_asset_syntax("s3:///exports/x", true), + parse_asset_syntax("s3://exports/x", true) + ); + + // Explicit-storage form is unaffected (no leading slash to strip). + assert_eq!( + parse_asset_syntax("s3://mybucket/exports/x", false), + Some((AssetKind::S3Object, "mybucket/exports/x")) + ); + + // Hive-partition keys and nested paths under default storage are + // preserved verbatim (only leading slashes are stripped). + assert_eq!( + parse_asset_syntax("s3:///t/year=2024/month=01/f.parquet", false), + Some((AssetKind::S3Object, "t/year=2024/month=01/f.parquet")) + ); + + // Every leading slash is stripped so a canonical S3 path never starts + // with `/`. `S3Object(s3="/x")` resolves to the quad-slash URI + // `s3:////x`; the identity must be the bare `x` (not `/x`) so the ref + // that `trigger_spec_to_row` rebuilds round-trips back to it. + assert_eq!( + parse_asset_syntax("s3:////x", false), + Some((AssetKind::S3Object, "x")) + ); + assert_eq!( + parse_asset_syntax("s3://///deep///", false), + Some((AssetKind::S3Object, "deep///")) + ); + + // Non-S3 kinds keep their leading slash (their paths are workspace- + // relative and the slash is significant). + assert_eq!( + parse_asset_syntax("res://f/foo", false), + Some((AssetKind::Resource, "f/foo")) + ); + assert_eq!( + parse_asset_syntax("ducklake://analytics/orders", false), + Some((AssetKind::Ducklake, "analytics/orders")) + ); + } + + #[test] + fn s3_explicit_storage_aliases_default_storage_nested_key() { + // Accepted tradeoff of one canonical key: the explicit-storage form + // `s3://storage/key` and the default-storage nested-key form + // `s3:///storage/key` collapse to the same node `storage/key`, even + // though they name different objects. This is a best-effort lineage + // graph that does not split the first segment as a storage name; the + // collision only happens when a storage config is named to match a + // default-storage prefix. Pinned so the aliasing is intentional, not a + // latent surprise. + assert_eq!( + parse_asset_syntax("s3://mybucket/x", false), + parse_asset_syntax("s3:///mybucket/x", false) + ); + assert_eq!( + parse_asset_syntax("s3://mybucket/x", false), + Some((AssetKind::S3Object, "mybucket/x")) + ); + } + #[test] fn bare_pipeline_marker() { let out = parse_pipeline_annotations("// pipeline\nconsole.log('hi')"); @@ -723,6 +1463,33 @@ mod pipeline_annotation_tests { assert!(!out.in_pipeline); } + #[test] + fn macros_marker_strict_like_pipeline() { + assert!(parse_pipeline_annotations("// macros\nCREATE MACRO m(a) AS a;").macros); + assert!(parse_pipeline_annotations("-- macros \nSELECT 1;").macros); + // Trailing prose / keyword variants disqualify the line. + assert!(!parse_pipeline_annotations("// macros are defined below\n").macros); + assert!(!parse_pipeline_annotations("// macros_v2\n").macros); + } + + #[test] + fn use_accumulates_dedups_and_rejects_prose() { + let out = parse_pipeline_annotations( + "// use f/lib/stats\n// use f/lib/dates\n// use f/lib/stats\nSELECT 1;", + ); + assert_eq!(out.use_libs, vec!["f/lib/stats", "f/lib/dates"]); + + // Prose, slashless tokens, and multi-token lines are dropped fail-safe. + let out = parse_pipeline_annotations( + "// use this script to compute\n// use standalone\n// use f/lib/ok extra\n", + ); + assert!(out.use_libs.is_empty()); + + // Only the leading comment header is scanned. + let out = parse_pipeline_annotations("SELECT 1;\n-- use f/lib/late\n"); + assert!(out.use_libs.is_empty()); + } + #[test] fn on_schedule_marker() { // `// on schedule` is marker-only — the binding is the schedule row's @@ -1008,6 +1775,56 @@ mod pipeline_annotation_tests { assert!(out.tag.is_none()); } + #[test] + fn tag_with_whitespace_is_skipped() { + // A regular English comment starting with "# tag " must not be + // mistaken for a worker-tag annotation (worker tags are single words). + let out = + parse_pipeline_annotations("# tag this function so we remember to refactor it later"); + assert!(out.tag.is_none()); + } + + #[test] + fn tag_too_long_is_skipped() { + let long = "x".repeat(51); + let out = parse_pipeline_annotations(&format!("// tag {long}")); + assert!(out.tag.is_none()); + } + + #[test] + fn annotations_in_body_are_ignored() { + // Only the leading comment header is scanned. A regular `# tag ...` + // prose comment buried in the body — the WIN-2090 false-positive that + // crashed the `script.tag` INSERT — must not be treated as an + // annotation once real code has started. + let code = concat!( + "import pandas as pd\n", + "\n", + "def main():\n", + " # tag each row with its source so downstream steps can filter\n", + " # on s3://should/not/parse\n", + " return pd.DataFrame()\n", + ); + let out = parse_pipeline_annotations(code); + assert!(out.tag.is_none()); + assert!(out.triggers.is_empty()); + } + + #[test] + fn header_allows_blank_lines_before_code() { + // Blank lines (e.g. after a shebang) don't end the header; the first + // line of real code does. + let code = concat!( + "#!/usr/bin/env python\n", + "\n", + "# tag heavy\n", + "import os\n", + "# tag light\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!(out.tag.as_deref(), Some("heavy")); + } + #[test] fn retry_count_only() { let out = parse_pipeline_annotations("// retry 3"); @@ -1044,6 +1861,257 @@ mod pipeline_annotation_tests { assert!(out.retry.is_none()); } + #[test] + fn materialize_managed_default() { + let out = parse_pipeline_annotations("// materialize ducklake://analytics/orders_daily"); + let m = out.materialize.expect("materialize"); + assert_eq!(m.target_kind, AssetKind::Ducklake); + assert_eq!(m.target_path, "analytics/orders_daily"); + // managed by default; replace strategy (no append / key) + assert!(!m.manual); + assert!(!m.append); + assert_eq!(m.unique_key, None); + } + + #[test] + fn materialize_manual_escape_hatch() { + let out = + parse_pipeline_annotations("// materialize manual ducklake://analytics/orders_daily"); + let m = out.materialize.expect("materialize"); + assert!(m.manual); + assert_eq!(m.target_path, "analytics/orders_daily"); + } + + #[test] + fn materialize_merge_and_append_options() { + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders_daily key=order_id"); + let m = out.materialize.expect("materialize"); + assert_eq!(m.unique_key.as_deref(), Some("order_id")); + assert!(!m.append); + + let out = parse_pipeline_annotations("// materialize ducklake://a/events append"); + let m = out.materialize.expect("materialize"); + assert!(m.append); + assert_eq!(m.unique_key, None); + } + + #[test] + fn materialize_scd2_history_flag_with_key_and_track() { + // Primary spelling: `key=history` on a merge. + let out = parse_pipeline_annotations( + "// materialize ducklake://a/dim key=id history track=name,tier", + ); + let m = out.materialize.expect("materialize"); + assert!(m.scd2); + assert!(!m.manual); + assert_eq!(m.unique_key.as_deref(), Some("id")); + assert_eq!(m.track, vec!["name".to_string(), "tier".to_string()]); + } + + #[test] + fn materialize_scd2_keyword_is_alias_for_history() { + let out = parse_pipeline_annotations("// materialize scd2 ducklake://a/dim key=id"); + let m = out.materialize.expect("materialize"); + assert!(m.scd2); + assert_eq!(m.unique_key.as_deref(), Some("id")); + assert!(m.track.is_empty()); + // soft-delete default + assert!(!m.close_deleted); + } + + #[test] + fn materialize_scd2_write_targets_include_current_view() { + // A managed scd2 materialize produces the base table AND the + // `_current` companion view, so the producer must be recorded as + // writing both (reads of the view otherwise resolve to an orphan asset). + let out = parse_pipeline_annotations( + "// materialize ducklake://main/dim_customers key=id history", + ); + let m = out.materialize.expect("materialize"); + assert_eq!( + m.write_targets(), + vec![ + (AssetKind::Ducklake, "main/dim_customers".to_string()), + ( + AssetKind::Ducklake, + "main/dim_customers_current".to_string() + ), + ] + ); + assert_eq!( + m.scd2_current_target(), + Some(( + AssetKind::Ducklake, + "main/dim_customers_current".to_string() + )) + ); + } + + #[test] + fn materialize_non_scd2_write_targets_are_base_only() { + // A plain merge (no `history`) creates no companion view — only the base. + let out = parse_pipeline_annotations("// materialize ducklake://main/dim_customers key=id"); + let m = out.materialize.expect("materialize"); + assert_eq!( + m.write_targets(), + vec![(AssetKind::Ducklake, "main/dim_customers".to_string())] + ); + assert_eq!(m.scd2_current_target(), None); + } + + #[test] + fn materialize_manual_scd2_has_no_companion_view() { + // `manual` mode owns its own DDL and never creates the `_current` view, + // so registering it would be a false producer edge. + let out = parse_pipeline_annotations( + "// materialize manual ducklake://main/dim_customers key=id history", + ); + let m = out.materialize.expect("materialize"); + assert!(m.manual && m.scd2); + assert_eq!(m.scd2_current_target(), None); + assert_eq!( + m.write_targets(), + vec![(AssetKind::Ducklake, "main/dim_customers".to_string())] + ); + } + + #[test] + fn materialize_scd2_deletes_close_opt() { + let out = parse_pipeline_annotations( + "// materialize ducklake://a/dim key=id history deletes=close", + ); + let m = out.materialize.expect("materialize"); + assert!(m.scd2); + assert!(m.close_deleted); + // any other value keeps the soft-delete default + let out = parse_pipeline_annotations( + "// materialize ducklake://a/dim key=id history deletes=ignore", + ); + assert!(!out.materialize.expect("materialize").close_deleted); + } + + #[test] + fn materialize_validate_scd2_requires_key() { + // scd2 without `key=` is rejected at deploy (was a run-time error). + let m = parse_pipeline_annotations("// materialize ducklake://a/dim history") + .materialize + .expect("materialize"); + assert!(m.scd2 && m.unique_key.is_none()); + let err = m.validate(false).expect_err("scd2 without key must fail"); + assert!(err.contains("requires a natural key")); + // with a key it validates + let m = parse_pipeline_annotations("// materialize ducklake://a/dim key=id history") + .materialize + .expect("materialize"); + assert!(m.validate(false).is_ok()); + } + + #[test] + fn materialize_validate_scd2_rejects_partitioned() { + // scd2 + `// partitioned` has no v1 form — rejected at deploy. + let m = parse_pipeline_annotations("// materialize ducklake://a/dim key=id history") + .materialize + .expect("materialize"); + let err = m.validate(true).expect_err("scd2 + partitioned must fail"); + assert!(err.contains("`// partitioned` is not supported with scd2")); + // unpartitioned scd2 is fine + assert!(m.validate(false).is_ok()); + } + + #[test] + fn materialize_validate_non_scd2_and_manual_are_inert() { + // Non-scd2 strategies are unconstrained by these checks, partitioned or not. + let m = parse_pipeline_annotations("// materialize ducklake://a/orders key=id") + .materialize + .expect("materialize"); + assert!(m.validate(true).is_ok()); + assert!(m.validate(false).is_ok()); + // `manual` owns its DDL and ignores the strategy — never rejected here, + // even with a partitioned scd2-looking combo. + let m = parse_pipeline_annotations("// materialize manual ducklake://a/dim history") + .materialize + .expect("materialize"); + assert!(m.manual && m.scd2); + assert!(m.validate(true).is_ok()); + } + + #[test] + fn materialize_on_schema_change_opt() { + let out = parse_pipeline_annotations( + "// materialize ducklake://a/orders on_schema_change=ignore", + ); + let m = out.materialize.expect("materialize"); + assert_eq!(m.on_schema_change, OnSchemaChange::Ignore); + // default is warn + let out = parse_pipeline_annotations("// materialize ducklake://a/orders"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Warn + ); + // fail + sync parse to their own variants + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=fail"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Fail + ); + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=sync"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Sync + ); + // unknown/junk value keeps the warn default (fail-safe, like `deletes=`) + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=bogus"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Warn + ); + // composes with other opts + let out = parse_pipeline_annotations( + "// materialize ducklake://a/dim key=id history on_schema_change=ignore", + ); + let m = out.materialize.expect("materialize"); + assert!(m.scd2); + assert_eq!(m.on_schema_change, OnSchemaChange::Ignore); + } + + #[test] + fn materialize_key_without_history_is_plain_merge() { + let out = parse_pipeline_annotations("// materialize ducklake://a/dim key=id"); + let m = out.materialize.expect("materialize"); + assert!(!m.scd2, "no history flag ⇒ SCD1 merge, not scd2"); + assert_eq!(m.unique_key.as_deref(), Some("id")); + } + + #[test] + fn materialize_default_syntax_shorthand() { + let out = parse_pipeline_annotations("// materialize ducklake"); + let m = out.materialize.expect("materialize"); + assert_eq!(m.target_kind, AssetKind::Ducklake); + assert_eq!(m.target_path, "main"); + assert!(!m.manual); + } + + #[test] + fn materialize_manual_only_is_dropped() { + // `manual` with no target is not a valid materialization. + let out = parse_pipeline_annotations("// materialize manual"); + assert!(out.materialize.is_none()); + } + + #[test] + fn materialize_first_wins() { + let out = parse_pipeline_annotations( + "// materialize ducklake://a/x\n# materialize manual ducklake://b/y", + ); + let m = out.materialize.expect("materialize"); + assert_eq!(m.target_path, "a/x"); + assert!(!m.manual); + } + #[test] fn combined() { let code = concat!( @@ -1053,7 +2121,8 @@ mod pipeline_annotation_tests { "// partitioned daily tz=\"UTC\"\n", "// freshness 2h\n", "// tag heavy\n", - "// retry 3 5s\n" + "// retry 3 5s\n", + "// materialize ducklake://analytics/orders_daily key=order_id\n" ); let out = parse_pipeline_annotations(code); assert!(out.in_pipeline); @@ -1064,6 +2133,10 @@ mod pipeline_annotation_tests { let r = out.retry.expect("retry"); assert_eq!(r.count, 3); assert_eq!(r.delay.as_deref(), Some("5s")); + let m = out.materialize.expect("materialize"); + assert!(!m.manual); + assert_eq!(m.target_path, "analytics/orders_daily"); + assert_eq!(m.unique_key.as_deref(), Some("order_id")); } #[test] @@ -1108,4 +2181,249 @@ mod pipeline_annotation_tests { assert_eq!(m.get("b").unwrap(), "fine"); assert!(m.get("garbage").is_none()); } + + #[test] + fn data_test_builtins() { + let code = concat!( + "// data_test unique order_id\n", + "// data_test not_null user_id\n", + "// data_test accepted_values status = paid,pending,refunded\n", + "// data_test relationships user_id -> datatable://prod/users.id\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!( + out.data_tests, + vec![ + DataTest::Unique { column: "order_id".to_string() }, + DataTest::NotNull { column: "user_id".to_string() }, + DataTest::AcceptedValues { + column: "status".to_string(), + values: vec![ + "paid".to_string(), + "pending".to_string(), + "refunded".to_string() + ], + }, + DataTest::Relationships { + column: "user_id".to_string(), + to_kind: AssetKind::DataTable, + to_path: "prod/users".to_string(), + to_column: "id".to_string(), + }, + ] + ); + } + + #[test] + fn data_test_accepts_quotes_and_spacing() { + let out = parse_pipeline_annotations("// data_test accepted_values kind = \"a b\", 'c' ,d"); + assert_eq!( + out.data_tests, + vec![DataTest::AcceptedValues { + column: "kind".to_string(), + values: vec!["a b".to_string(), "c".to_string(), "d".to_string()], + }] + ); + } + + #[test] + fn data_test_custom_escape_hatch() { + // A non-built-in single token is a custom script path; default-syntax + // asset shorthands are NOT triggered here (a path is just a path). + let out = parse_pipeline_annotations("// data_test f/tests/orders_amount_sane"); + assert_eq!( + out.data_tests, + vec![DataTest::Custom { path: "f/tests/orders_amount_sane".to_string() }] + ); + } + + #[test] + fn data_test_relationships_ducklake_shorthand() { + let out = parse_pipeline_annotations( + "// data_test relationships sku -> ducklake://warehouse/dim_products.sku", + ); + assert_eq!( + out.data_tests, + vec![DataTest::Relationships { + column: "sku".to_string(), + to_kind: AssetKind::Ducklake, + to_path: "warehouse/dim_products".to_string(), + to_column: "sku".to_string(), + }] + ); + } + + #[test] + fn data_test_malformed_dropped_fail_safe() { + // A misspelled built-in with trailing content is not a valid path token + // → dropped, not misread as a custom test. An empty value list, a + // missing arrow target, and a bare keyword are all dropped too. + let out = parse_pipeline_annotations(concat!( + "// data_test uniq order_id\n", // typo'd built-in + arg + "// data_test accepted_values s =\n", // no values + "// data_test relationships a -> b\n", // no `.refcol` + "// data_test unique\n", // missing column + "// data_test\n", // bare keyword + )); + assert!(out.data_tests.is_empty()); + } + + #[test] + fn data_test_not_confused_with_ci_test_annotation() { + // `// test:` is the unrelated CI-test annotation — it must NOT be + // parsed as a data test (no whitespace after `test`, and the keyword + // is `data_test` anyway). + let out = parse_pipeline_annotations("// test: f/foo/bar\n// data_test unique id"); + assert_eq!( + out.data_tests, + vec![DataTest::Unique { column: "id".to_string() }] + ); + } + + #[test] + fn column_lineage_basic() { + let code = concat!( + "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n", + "// column user_name <- datatable://prod/users.name\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!( + out.column_lineage, + vec![ + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ColumnLineage { + column: "user_name".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/users".to_string(), + from_column: "name".to_string(), + }], + }, + ] + ); + } + + #[test] + fn column_lineage_schema_qualified_keeps_last_dot_as_column() { + // The column is the segment after the FINAL dot, so a schema-qualified + // ducklake table (`main.dim_products`) survives intact. + let out = parse_pipeline_annotations( + "// column sku <- ducklake://warehouse/main.dim_products.sku", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "sku".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/main.dim_products".to_string(), + from_column: "sku".to_string(), + }], + }] + ); + } + + #[test] + fn column_lineage_drops_malformed_refs_keeps_valid() { + // `bad_no_dot` has no `.col` and is dropped; the line survives on its + // one valid ref. Mirrors accepted_values' drop-empties-keep-≥1 stance. + let out = parse_pipeline_annotations( + "// column total <- bad_no_dot, datatable://prod/orders.amount", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn merge_column_lineage_annotation_overrides_inferred() { + let inferred = vec![ + ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "amount".to_string(), + }], + }, + ColumnLineage { + column: "qty".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "qty".to_string(), + }], + }, + ]; + // Annotation redefines `total` (wins) and leaves `qty` to inference. + let annotated = vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/x".to_string(), + from_column: "grand_total".to_string(), + }], + }]; + let merged = merge_column_lineage(inferred, annotated); + assert_eq!(merged.len(), 2); + // Annotation entry kept first and authoritative. + assert_eq!(merged[0].column, "total"); + assert_eq!(merged[0].inputs[0].from_column, "grand_total"); + // Inferred `qty` survives (no annotation for it); inferred `total` dropped. + assert_eq!(merged[1].column, "qty"); + } + + #[test] + fn column_lineage_malformed_lines_dropped_fail_safe() { + // No arrow, a multi-token output column, and a line whose every ref is + // malformed are all dropped entirely. + let out = parse_pipeline_annotations(concat!( + "// column no_arrow datatable://prod/x.y\n", // missing `<-` + "// column a b <- datatable://prod/x.y\n", // output not a single ident + "// column total <- bad_no_dot\n", // no valid ref + "// column\n", // bare keyword + )); + assert!(out.column_lineage.is_empty()); + } + + #[test] + fn count_malformed_data_tests_counts_only_broken_header_lines() { + let n = count_malformed_data_tests(concat!( + "-- data_test not_null id\n", // valid + "-- data_test unique id\n", // valid + "-- data_test accepted_values status paid\n", // malformed: missing `=` + "-- data_test relationships cust ducklake\n", // malformed: no `->` + "-- data_test\n", // malformed: bare keyword + "-- data_test f/tests/custom\n", // valid: custom script path + "SELECT 1 -- data_test not_a_test\n", // body line: not counted + )); + assert_eq!(n, 3); + // A clean header has zero. + assert_eq!( + count_malformed_data_tests("-- data_test unique id\nSELECT 1"), + 0 + ); + } } diff --git a/backend/parsers/windmill-parser/src/duckdb_builtins.rs b/backend/parsers/windmill-parser/src/duckdb_builtins.rs new file mode 100644 index 0000000000..d324fa4c30 --- /dev/null +++ b/backend/parsers/windmill-parser/src/duckdb_builtins.rs @@ -0,0 +1,936 @@ +// DuckDB built-in function names (scalar, aggregate, table, macro), used to +// reject workspace macros that would silently shadow a built-in (DuckDB +// allows the shadowing without error — verified on 1.5.4). Only +// identifier-shaped names are listed: workspace macro names are validated +// to `[a-z_][a-z0-9_]*` before this check, so operator names can't collide. +// +// Regenerate on DuckDB upgrades (sort in codepoint order — binary search): +// python3 -c "import duckdb,re; print('\\n'.join(sorted(set(r[0] for r in +// duckdb.connect().execute(\"SELECT DISTINCT lower(function_name) FROM +// duckdb_functions()\").fetchall() if re.fullmatch(r'[a-z_][a-z0-9_]*', r[0])))))" + +pub fn is_duckdb_builtin(name: &str) -> bool { + DUCKDB_BUILTIN_FUNCTIONS + .binary_search(&name.to_ascii_lowercase().as_str()) + .is_ok() +} + +const DUCKDB_BUILTIN_FUNCTIONS: &[&str] = &[ + "__internal_compress_integral_ubigint", + "__internal_compress_integral_uinteger", + "__internal_compress_integral_usmallint", + "__internal_compress_integral_utinyint", + "__internal_compress_string_hugeint", + "__internal_compress_string_ubigint", + "__internal_compress_string_uhugeint", + "__internal_compress_string_uinteger", + "__internal_compress_string_usmallint", + "__internal_compress_string_utinyint", + "__internal_decompress_integral_bigint", + "__internal_decompress_integral_hugeint", + "__internal_decompress_integral_integer", + "__internal_decompress_integral_smallint", + "__internal_decompress_integral_ubigint", + "__internal_decompress_integral_uhugeint", + "__internal_decompress_integral_uinteger", + "__internal_decompress_integral_usmallint", + "__internal_decompress_string", + "abs", + "acos", + "acosh", + "add", + "add_parquet_key", + "age", + "aggregate", + "ago", + "alias", + "all_profiling_output", + "any_value", + "apply", + "approx_count_distinct", + "approx_quantile", + "approx_top_k", + "arbitrary", + "arg_max", + "arg_max_null", + "arg_max_nulls_last", + "arg_min", + "arg_min_null", + "arg_min_nulls_last", + "argmax", + "argmin", + "array_agg", + "array_aggr", + "array_aggregate", + "array_append", + "array_apply", + "array_cat", + "array_concat", + "array_contains", + "array_cosine_distance", + "array_cosine_similarity", + "array_cross_product", + "array_distance", + "array_distinct", + "array_dot_product", + "array_extract", + "array_filter", + "array_grade_up", + "array_has", + "array_has_all", + "array_has_any", + "array_indexof", + "array_inner_product", + "array_intersect", + "array_length", + "array_negative_dot_product", + "array_negative_inner_product", + "array_pop_back", + "array_pop_front", + "array_position", + "array_prepend", + "array_push_back", + "array_push_front", + "array_reduce", + "array_resize", + "array_reverse", + "array_reverse_sort", + "array_select", + "array_slice", + "array_sort", + "array_to_json", + "array_to_string", + "array_to_string_comma_default", + "array_transform", + "array_unique", + "array_value", + "array_where", + "array_zip", + "arrow_scan", + "arrow_scan_dumb", + "ascii", + "asin", + "asinh", + "atan", + "atan2", + "atanh", + "avg", + "bar", + "base64", + "bin", + "bit_and", + "bit_count", + "bit_length", + "bit_or", + "bit_position", + "bit_xor", + "bitstring", + "bitstring_agg", + "bool_and", + "bool_or", + "can_cast_implicitly", + "cardinality", + "cast_to_type", + "cbrt", + "ceil", + "ceiling", + "century", + "char_length", + "character_length", + "checkpoint", + "chr", + "col_description", + "collations", + "combine", + "concat", + "concat_ws", + "constant_or_null", + "contains", + "copy_database", + "corr", + "cos", + "cosh", + "cot", + "count", + "count_if", + "count_star", + "countif", + "covar_pop", + "covar_samp", + "create_sort_key", + "cume_dist", + "current_catalog", + "current_connection_id", + "current_database", + "current_date", + "current_localtime", + "current_localtimestamp", + "current_query", + "current_query_id", + "current_role", + "current_schema", + "current_schemas", + "current_setting", + "current_transaction_id", + "current_user", + "currval", + "damerau_levenshtein", + "database_list", + "database_size", + "date_add", + "date_diff", + "date_part", + "date_sub", + "date_trunc", + "datediff", + "datepart", + "datesub", + "datetrunc", + "day", + "dayname", + "dayofmonth", + "dayofweek", + "dayofyear", + "days_in_month", + "decade", + "decode", + "degrees", + "dense_rank", + "disable_checkpoint_on_shutdown", + "disable_logging", + "disable_object_cache", + "disable_optimizer", + "disable_print_progress_bar", + "disable_profile", + "disable_profiling", + "disable_progress_bar", + "disable_verification", + "disable_verify_external", + "disable_verify_fetch_row", + "disable_verify_parallelism", + "disable_verify_serializer", + "divide", + "duckdb_approx_database_count", + "duckdb_columns", + "duckdb_connection_count", + "duckdb_constraints", + "duckdb_coordinate_systems", + "duckdb_databases", + "duckdb_dependencies", + "duckdb_extensions", + "duckdb_external_file_cache", + "duckdb_functions", + "duckdb_indexes", + "duckdb_keywords", + "duckdb_log_contexts", + "duckdb_logs", + "duckdb_logs_parsed", + "duckdb_memory", + "duckdb_optimizers", + "duckdb_prepared_statements", + "duckdb_profiling_settings", + "duckdb_schemas", + "duckdb_secret_types", + "duckdb_secrets", + "duckdb_sequences", + "duckdb_settings", + "duckdb_table_sample", + "duckdb_tables", + "duckdb_temporary_files", + "duckdb_types", + "duckdb_variables", + "duckdb_views", + "editdist3", + "element_at", + "enable_checkpoint_on_shutdown", + "enable_logging", + "enable_object_cache", + "enable_optimizer", + "enable_print_progress_bar", + "enable_profile", + "enable_profiling", + "enable_progress_bar", + "enable_verification", + "encode", + "ends_with", + "entropy", + "enum_code", + "enum_first", + "enum_last", + "enum_range", + "enum_range_boundary", + "epoch", + "epoch_ms", + "epoch_ns", + "epoch_us", + "equi_width_bins", + "era", + "error", + "even", + "exp", + "extension_versions", + "factorial", + "favg", + "fdiv", + "fill", + "filter", + "finalize", + "first", + "first_value", + "flatten", + "floor", + "fmod", + "force_checkpoint", + "format", + "format_bytes", + "format_pg_type", + "format_type", + "formatreadabledecimalsize", + "formatreadablesize", + "from_base64", + "from_binary", + "from_hex", + "from_json", + "from_json_strict", + "fsum", + "functions", + "gamma", + "gcd", + "gen_random_uuid", + "generate_series", + "generate_subscripts", + "geomean", + "geometric_mean", + "get_bit", + "get_block_size", + "get_current_time", + "get_current_timestamp", + "get_type", + "getvariable", + "glob", + "grade_up", + "greatest", + "greatest_common_divisor", + "group_concat", + "hamming", + "has_any_column_privilege", + "has_column_privilege", + "has_database_privilege", + "has_foreign_data_wrapper_privilege", + "has_function_privilege", + "has_language_privilege", + "has_schema_privilege", + "has_sequence_privilege", + "has_server_privilege", + "has_table_privilege", + "has_tablespace_privilege", + "hash", + "hex", + "histogram", + "histogram_exact", + "histogram_values", + "hour", + "icu_calendar_names", + "icu_collate_af", + "icu_collate_am", + "icu_collate_ar", + "icu_collate_ar_sa", + "icu_collate_as", + "icu_collate_az", + "icu_collate_be", + "icu_collate_bg", + "icu_collate_bn", + "icu_collate_bo", + "icu_collate_br", + "icu_collate_bs", + "icu_collate_ca", + "icu_collate_ceb", + "icu_collate_chr", + "icu_collate_cs", + "icu_collate_cy", + "icu_collate_da", + "icu_collate_de", + "icu_collate_de_at", + "icu_collate_dsb", + "icu_collate_dz", + "icu_collate_ee", + "icu_collate_el", + "icu_collate_en", + "icu_collate_en_us", + "icu_collate_eo", + "icu_collate_es", + "icu_collate_et", + "icu_collate_fa", + "icu_collate_fa_af", + "icu_collate_ff", + "icu_collate_fi", + "icu_collate_fil", + "icu_collate_fo", + "icu_collate_fr", + "icu_collate_fr_ca", + "icu_collate_fy", + "icu_collate_ga", + "icu_collate_gl", + "icu_collate_gu", + "icu_collate_ha", + "icu_collate_haw", + "icu_collate_he", + "icu_collate_he_il", + "icu_collate_hi", + "icu_collate_hr", + "icu_collate_hsb", + "icu_collate_hu", + "icu_collate_hy", + "icu_collate_id", + "icu_collate_id_id", + "icu_collate_ig", + "icu_collate_is", + "icu_collate_it", + "icu_collate_ja", + "icu_collate_ka", + "icu_collate_kk", + "icu_collate_kl", + "icu_collate_km", + "icu_collate_kn", + "icu_collate_ko", + "icu_collate_kok", + "icu_collate_ku", + "icu_collate_ky", + "icu_collate_lb", + "icu_collate_lij", + "icu_collate_lkt", + "icu_collate_ln", + "icu_collate_lo", + "icu_collate_lt", + "icu_collate_lv", + "icu_collate_mk", + "icu_collate_ml", + "icu_collate_mn", + "icu_collate_mr", + "icu_collate_ms", + "icu_collate_mt", + "icu_collate_my", + "icu_collate_nb", + "icu_collate_nb_no", + "icu_collate_ne", + "icu_collate_nl", + "icu_collate_nn", + "icu_collate_noaccent", + "icu_collate_nso", + "icu_collate_om", + "icu_collate_or", + "icu_collate_pa", + "icu_collate_pa_in", + "icu_collate_pl", + "icu_collate_ps", + "icu_collate_pt", + "icu_collate_ro", + "icu_collate_ru", + "icu_collate_sa", + "icu_collate_se", + "icu_collate_si", + "icu_collate_sk", + "icu_collate_sl", + "icu_collate_smn", + "icu_collate_sq", + "icu_collate_sr", + "icu_collate_sr_ba", + "icu_collate_sr_me", + "icu_collate_sr_rs", + "icu_collate_st", + "icu_collate_sv", + "icu_collate_sw", + "icu_collate_ta", + "icu_collate_te", + "icu_collate_th", + "icu_collate_tk", + "icu_collate_tn", + "icu_collate_to", + "icu_collate_tr", + "icu_collate_ug", + "icu_collate_uk", + "icu_collate_ur", + "icu_collate_uz", + "icu_collate_vi", + "icu_collate_wae", + "icu_collate_wo", + "icu_collate_xh", + "icu_collate_yi", + "icu_collate_yo", + "icu_collate_yue", + "icu_collate_yue_cn", + "icu_collate_zh", + "icu_collate_zh_cn", + "icu_collate_zh_hk", + "icu_collate_zh_mo", + "icu_collate_zh_sg", + "icu_collate_zh_tw", + "icu_collate_zu", + "icu_sort_key", + "ilike_escape", + "import_database", + "in_search_path", + "inet_client_addr", + "inet_client_port", + "inet_server_addr", + "inet_server_port", + "instr", + "is_histogram_other_bin", + "isfinite", + "isinf", + "isnan", + "isodow", + "isoyear", + "jaccard", + "jaro_similarity", + "jaro_winkler_similarity", + "json", + "json_array", + "json_array_length", + "json_contains", + "json_deserialize_sql", + "json_each", + "json_execute_serialized_sql", + "json_exists", + "json_extract", + "json_extract_path", + "json_extract_path_text", + "json_extract_string", + "json_group_array", + "json_group_object", + "json_group_structure", + "json_keys", + "json_merge_patch", + "json_object", + "json_pretty", + "json_quote", + "json_serialize_plan", + "json_serialize_sql", + "json_structure", + "json_transform", + "json_transform_strict", + "json_tree", + "json_type", + "json_valid", + "json_value", + "julian", + "kahan_sum", + "kurtosis", + "kurtosis_pop", + "lag", + "last", + "last_day", + "last_value", + "lcase", + "lcm", + "lead", + "least", + "least_common_multiple", + "left", + "left_grapheme", + "len", + "length", + "length_grapheme", + "levenshtein", + "lgamma", + "like_escape", + "list", + "list_aggr", + "list_aggregate", + "list_any_value", + "list_append", + "list_apply", + "list_approx_count_distinct", + "list_avg", + "list_bit_and", + "list_bit_or", + "list_bit_xor", + "list_bool_and", + "list_bool_or", + "list_cat", + "list_concat", + "list_contains", + "list_cosine_distance", + "list_cosine_similarity", + "list_count", + "list_distance", + "list_distinct", + "list_dot_product", + "list_element", + "list_entropy", + "list_extract", + "list_filter", + "list_first", + "list_grade_up", + "list_has", + "list_has_all", + "list_has_any", + "list_histogram", + "list_indexof", + "list_inner_product", + "list_intersect", + "list_kurtosis", + "list_kurtosis_pop", + "list_last", + "list_mad", + "list_max", + "list_median", + "list_min", + "list_mode", + "list_negative_dot_product", + "list_negative_inner_product", + "list_pack", + "list_position", + "list_prepend", + "list_product", + "list_reduce", + "list_resize", + "list_reverse", + "list_reverse_sort", + "list_select", + "list_sem", + "list_skewness", + "list_slice", + "list_sort", + "list_stddev_pop", + "list_stddev_samp", + "list_string_agg", + "list_sum", + "list_transform", + "list_unique", + "list_value", + "list_var_pop", + "list_var_samp", + "list_where", + "list_zip", + "listagg", + "ln", + "log", + "log10", + "log2", + "lower", + "lpad", + "ltrim", + "mad", + "make_date", + "make_time", + "make_timestamp", + "make_timestamp_ms", + "make_timestamp_ns", + "make_timestamptz", + "make_type", + "map", + "map_concat", + "map_contains", + "map_contains_entry", + "map_contains_value", + "map_entries", + "map_extract", + "map_extract_value", + "map_from_entries", + "map_keys", + "map_to_pg_oid", + "map_values", + "max", + "max_by", + "md5", + "md5_number", + "md5_number_lower", + "md5_number_upper", + "mean", + "median", + "metadata_info", + "microsecond", + "millennium", + "millisecond", + "min", + "min_by", + "minute", + "mismatches", + "mod", + "mode", + "month", + "monthname", + "multiply", + "nanosecond", + "nextafter", + "nextval", + "nfc_normalize", + "normalized_interval", + "not_ilike_escape", + "not_like_escape", + "now", + "nth_value", + "ntile", + "nullif", + "obj_description", + "octet_length", + "ord", + "pandas_scan", + "parquet_bloom_probe", + "parquet_file_metadata", + "parquet_full_metadata", + "parquet_kv_metadata", + "parquet_metadata", + "parquet_scan", + "parquet_schema", + "parse_dirname", + "parse_dirpath", + "parse_duckdb_log_message", + "parse_filename", + "parse_formatted_bytes", + "parse_path", + "percent_rank", + "pg_collation_is_visible", + "pg_conf_load_time", + "pg_conversion_is_visible", + "pg_function_is_visible", + "pg_get_constraintdef", + "pg_get_expr", + "pg_get_viewdef", + "pg_has_role", + "pg_is_other_temp_schema", + "pg_my_temp_schema", + "pg_opclass_is_visible", + "pg_operator_is_visible", + "pg_opfamily_is_visible", + "pg_postmaster_start_time", + "pg_size_pretty", + "pg_sleep", + "pg_table_is_visible", + "pg_timezone_names", + "pg_ts_config_is_visible", + "pg_ts_dict_is_visible", + "pg_ts_parser_is_visible", + "pg_ts_template_is_visible", + "pg_type_is_visible", + "pg_typeof", + "pi", + "platform", + "position", + "pow", + "power", + "pragma_collations", + "pragma_database_size", + "pragma_metadata_info", + "pragma_platform", + "pragma_show", + "pragma_storage_info", + "pragma_table_info", + "pragma_user_agent", + "pragma_version", + "prefix", + "printf", + "product", + "python_map_function", + "quantile", + "quantile_cont", + "quantile_disc", + "quarter", + "query", + "query_table", + "radians", + "random", + "range", + "rank", + "rank_dense", + "read_blob", + "read_csv", + "read_csv_auto", + "read_duckdb", + "read_json", + "read_json_auto", + "read_json_objects", + "read_json_objects_auto", + "read_ndjson", + "read_ndjson_auto", + "read_ndjson_objects", + "read_parquet", + "read_text", + "reduce", + "regexp_escape", + "regexp_extract", + "regexp_extract_all", + "regexp_full_match", + "regexp_matches", + "regexp_replace", + "regexp_split_to_array", + "regexp_split_to_table", + "regr_avgx", + "regr_avgy", + "regr_count", + "regr_intercept", + "regr_r2", + "regr_slope", + "regr_sxx", + "regr_sxy", + "regr_syy", + "remap_struct", + "repeat", + "repeat_row", + "replace", + "replace_type", + "reservoir_quantile", + "reverse", + "right", + "right_grapheme", + "round", + "round_even", + "roundbankers", + "row", + "row_number", + "row_to_json", + "rpad", + "rtrim", + "second", + "sem", + "seq_scan", + "session_user", + "set_bit", + "setseed", + "sha1", + "sha256", + "shobj_description", + "show", + "show_databases", + "show_tables", + "show_tables_expanded", + "sign", + "signbit", + "sin", + "sinh", + "skewness", + "sleep_ms", + "sniff_csv", + "split", + "split_part", + "sqrt", + "st_asbinary", + "st_astext", + "st_aswkb", + "st_aswkt", + "st_crs", + "st_geomfromwkb", + "st_intersects_extent", + "st_setcrs", + "starts_with", + "stats", + "stddev", + "stddev_pop", + "stddev_samp", + "storage_info", + "str_split", + "str_split_regex", + "strftime", + "string_agg", + "string_split", + "string_split_regex", + "string_to_array", + "strip_accents", + "strlen", + "strpos", + "strptime", + "struct_concat", + "struct_contains", + "struct_extract", + "struct_extract_at", + "struct_has", + "struct_indexof", + "struct_insert", + "struct_keys", + "struct_pack", + "struct_position", + "struct_update", + "struct_values", + "substr", + "substring", + "substring_grapheme", + "subtract", + "suffix", + "sum", + "sum_no_overflow", + "sumkahan", + "summary", + "switch", + "table_info", + "tan", + "tanh", + "test_all_types", + "test_vector_types", + "time_bucket", + "timetz_byte_comparable", + "timezone", + "timezone_hour", + "timezone_minute", + "to_base", + "to_base64", + "to_binary", + "to_centuries", + "to_days", + "to_decades", + "to_hex", + "to_hours", + "to_json", + "to_microseconds", + "to_millennia", + "to_milliseconds", + "to_minutes", + "to_months", + "to_quarters", + "to_seconds", + "to_timestamp", + "to_weeks", + "to_years", + "today", + "transaction_timestamp", + "translate", + "trim", + "trunc", + "truncate_duckdb_logs", + "try_strptime", + "txid_current", + "typeof", + "ucase", + "unbin", + "unhex", + "unicode", + "union_extract", + "union_tag", + "union_value", + "unnest", + "unpivot_list", + "upper", + "url_decode", + "url_encode", + "user", + "user_agent", + "uuid", + "uuid_extract_timestamp", + "uuid_extract_version", + "uuidv4", + "uuidv7", + "var_pop", + "var_samp", + "variance", + "variant_bytes_to_variant", + "variant_extract", + "variant_normalize", + "variant_to_parquet_variant", + "variant_typeof", + "vector_type", + "verify_external", + "verify_fetch_row", + "verify_parallelism", + "verify_serializer", + "version", + "wavg", + "week", + "weekday", + "weekofyear", + "weighted_avg", + "which_secret", + "write_log", + "xor", + "year", + "yearweek", +]; diff --git a/backend/parsers/windmill-parser/src/duckdb_macros.rs b/backend/parsers/windmill-parser/src/duckdb_macros.rs new file mode 100644 index 0000000000..20438d23d1 --- /dev/null +++ b/backend/parsers/windmill-parser/src/duckdb_macros.rs @@ -0,0 +1,591 @@ +//! Workspace DuckDB macro libraries (`// macros` annotation). +//! +//! A macro-library script's body is `CREATE [OR REPLACE] [TEMP] MACRO` +//! statements plus plain setup (ATTACH/INSTALL/LOAD/SET/PRAGMA). At deploy the +//! macros are parsed into the `macro_definition` registry; at job time the +//! worker injects the (transitively) called ones into consumer scripts as +//! `CREATE OR REPLACE TEMP MACRO` blocks. Everything is stored and re-emitted +//! **verbatim** (params, body) — no AST round-trip — so any expression DuckDB +//! accepts survives unchanged. +//! +//! DuckDB bind-checks macro bodies at CREATE time (macro→macro and table +//! references alike), so injected definitions must be emitted in dependency +//! order — hence `topo_order_macros` — and after the consumer's ATTACHes. + +use std::collections::{BTreeMap, BTreeSet, HashSet}; + +use crate::sql_materialize::{classify_block, split_statements, BlockClass}; + +#[derive(Debug, Clone, PartialEq)] +pub struct ParsedMacro { + /// Lowercased bare identifier (DuckDB identifiers are case-insensitive + /// unquoted; qualified / quoted names are rejected at parse). + pub name: String, + /// Verbatim text inside the parameter parens (may be empty). + pub params: String, + /// Verbatim text after `AS [TABLE]`, without the trailing `;`. + pub body: String, + pub is_table: bool, +} + +#[derive(Debug, Clone, PartialEq)] +pub enum LibStatement { + Macro(ParsedMacro), + /// A non-macro statement allowed in a library: setup-class only + /// (ATTACH/INSTALL/LOAD/SET/PRAGMA/USE/CREATE TEMP …). Re-emitted verbatim + /// ahead of the macro definitions when the lib is injected via `// use`. + Setup(String), +} + +/// Statement text for injecting one macro into a consumer job. Always +/// TEMP (session-scoped — no catalog writes) and OR REPLACE (idempotent). +pub fn macro_create_statement(name: &str, params: &str, is_table: bool, body: &str) -> String { + format!( + "CREATE OR REPLACE TEMP MACRO {}({}) AS {}{};", + name, + params, + if is_table { "TABLE " } else { "" }, + body + ) +} + +fn is_ident(s: &str) -> bool { + let mut chars = s.chars(); + matches!(chars.next(), Some(c) if c.is_ascii_alphabetic() || c == '_') + && chars.all(|c| c.is_ascii_alphanumeric() || c == '_') +} + +// Case-insensitive whole-word prefix strip (whitespace-bounded), returning the +// remainder with leading whitespace trimmed. `get` (not slicing) so a +// multi-byte char straddling the boundary yields None instead of panicking. +fn strip_kw<'a>(s: &'a str, kw: &str) -> Option<&'a str> { + let prefix = s.get(..kw.len())?; + if prefix.eq_ignore_ascii_case(kw) { + let after = &s[kw.len()..]; + if after.is_empty() || after.starts_with(|c: char| c.is_whitespace()) { + return Some(after.trim_start()); + } + } + None +} + +/// Parse one (comment-free, `;`-less) statement as a CREATE MACRO. Returns +/// `Ok(None)` when the statement is not macro-shaped at all (caller decides +/// whether it is acceptable setup), `Err` when it is macro-shaped but invalid. +fn parse_create_macro(stmt: &str) -> Result, String> { + let Some(mut rest) = strip_kw(stmt.trim(), "create") else { + return Ok(None); + }; + if let Some(r) = strip_kw(rest, "or") { + rest = strip_kw(r, "replace").ok_or("expected REPLACE after CREATE OR")?; + } + if let Some(r) = strip_kw(rest, "temp").or_else(|| strip_kw(rest, "temporary")) { + rest = r; + } + // `FUNCTION` is DuckDB's alias for `MACRO`. + let Some(rest) = strip_kw(rest, "macro").or_else(|| strip_kw(rest, "function")) else { + return Ok(None); + }; + + let name_end = rest + .find(|c: char| c.is_whitespace() || c == '(') + .unwrap_or(rest.len()); + let raw_name = &rest[..name_end]; + if raw_name.is_empty() { + return Err("CREATE MACRO: missing macro name".to_string()); + } + if raw_name.contains('.') || raw_name.contains('"') || !is_ident(raw_name) { + return Err(format!( + "macro name `{}` must be a plain unqualified identifier ([A-Za-z_][A-Za-z0-9_]*) in v1", + raw_name + )); + } + let name = raw_name.to_ascii_lowercase(); + + let rest = rest[name_end..].trim_start(); + if !rest.starts_with('(') { + return Err(format!( + "macro `{}`: expected a parenthesized parameter list after the name", + name + )); + } + // Balanced-paren scan for the verbatim param list. The statement comes + // from `split_statements` so comments are gone, but strings may contain + // parens (e.g. a default value) — skip quoted spans. + let bytes = rest.as_bytes(); + let mut depth = 0usize; + let mut i = 0usize; + let mut close = None; + while i < bytes.len() { + match bytes[i] { + b'(' => depth += 1, + b')' => { + depth -= 1; + if depth == 0 { + close = Some(i); + break; + } + } + q @ (b'\'' | b'"') => { + i += 1; + while i < bytes.len() && bytes[i] != q { + i += 1; + } + } + _ => {} + } + i += 1; + } + let Some(close) = close else { + return Err(format!( + "macro `{}`: unbalanced parameter parentheses", + name + )); + }; + let params = rest[1..close].trim().to_string(); + + let after_params = rest[close + 1..].trim_start(); + let Some(mut body) = strip_kw(after_params, "as") else { + return Err(format!( + "macro `{}`: expected AS after the parameter list", + name + )); + }; + let is_table = match strip_kw(body, "table") { + Some(r) => { + body = r; + true + } + None => false, + }; + let body = body.trim().trim_end_matches(';').trim_end().to_string(); + if body.is_empty() { + return Err(format!("macro `{}`: empty body", name)); + } + Ok(Some(ParsedMacro { name, params, body, is_table })) +} + +fn stmt_head(stmt: &str) -> String { + stmt.split_whitespace() + .take(4) + .collect::>() + .join(" ") +} + +/// Managed ATTACH forms (`ducklake://…`, `datatable://…`, resource URIs) are +/// rewritten by the worker's transform pass — which `// use`-injected lib +/// setup bypasses — so a library may only use plain, self-contained ATTACHes. +pub fn is_managed_attach(stmt: &str) -> bool { + let s = stmt.trim(); + if strip_kw(s, "attach").is_none() { + return false; + } + let lower = s.to_ascii_lowercase(); + [ + "'ducklake:", + "'datatable:", + "'windmill:", + "'$res:", + "\"ducklake:", + "\"datatable:", + "\"windmill:", + "\"$res:", + ] + .iter() + .any(|p| lower.contains(p)) +} + +/// Parse a `// macros` library body. Statements are either macro definitions +/// or setup; anything else is an error (user-facing message). +pub fn parse_macro_library(sql: &str) -> Result, String> { + let mut out = Vec::new(); + for stmt in split_statements(sql) { + match parse_create_macro(&stmt)? { + Some(m) => out.push(LibStatement::Macro(m)), + None => match classify_block(&stmt) { + BlockClass::Setup => { + if is_managed_attach(&stmt) { + return Err(format!( + "a `// macros` library cannot use managed ATTACH forms \ + (ducklake://, datatable://, resource URIs) in v1 — its setup is \ + injected verbatim into consumers, bypassing the ATTACH rewrite: `{}`", + stmt_head(&stmt) + )); + } + out.push(LibStatement::Setup(stmt)) + } + _ => { + return Err(format!( + "a `// macros` library may only contain CREATE [OR REPLACE] MACRO \ + statements plus setup (ATTACH/INSTALL/LOAD/SET/PRAGMA); found: `{}`", + stmt_head(&stmt) + )) + } + }, + } + } + Ok(out) +} + +/// Whether the statement is macro-definition-shaped (`CREATE [OR REPLACE] +/// [TEMP] MACRO|FUNCTION …`), regardless of whether the rest of the header +/// parses. Used by the worker's injection splice to keep injected blocks +/// *after* a script's own leading definitions (an injected body may only call +/// a local macro once the local CREATE has run — DuckDB binds at CREATE). +pub fn is_macro_definition(stmt: &str) -> bool { + let Some(mut rest) = strip_kw(stmt.trim(), "create") else { + return false; + }; + if let Some(r) = strip_kw(rest, "or") { + let Some(r2) = strip_kw(r, "replace") else { + return false; + }; + rest = r2; + } + if let Some(r) = strip_kw(rest, "temp").or_else(|| strip_kw(rest, "temporary")) { + rest = r; + } + strip_kw(rest, "macro") + .or_else(|| strip_kw(rest, "function")) + .is_some() +} + +/// Parse a single macro-definition statement (`CREATE [OR REPLACE] [TEMP] +/// MACRO …`). `None` when the statement isn't macro-shaped or its header is +/// malformed — callers using definitions as placement anchors skip those +/// (they fail at execution regardless). +pub fn parse_macro_definition(stmt: &str) -> Option { + parse_create_macro(stmt).ok().flatten() +} + +/// Names of macros the given statements define themselves (`CREATE [OR +/// REPLACE] [TEMP] MACRO …`). A consumer's own definition is authoritative +/// over a same-named workspace macro — the worker subtracts these before +/// planning injection, so deploying a library can never silently replace a +/// script's local macro. Malformed macro-shaped statements are skipped (they +/// fail at execution regardless). +pub fn locally_defined_macro_names(statements: &[String]) -> HashSet { + statements + .iter() + .filter_map(|s| parse_create_macro(s).ok().flatten().map(|m| m.name)) + .collect() +} + +/// Names from `names` that `sql` calls: an identifier token immediately +/// followed by `(` (after optional whitespace), not `.`-qualified. Scans the +/// comment-stripped statements; string-literal contents are skipped. Matching +/// is deliberately lexical — over-matching only injects an unused TEMP macro. +pub fn detect_macro_calls(sql: &str, names: &HashSet) -> HashSet { + let mut found = HashSet::new(); + if names.is_empty() { + return found; + } + for stmt in split_statements(sql) { + let bytes = stmt.as_bytes(); + let n = bytes.len(); + let mut i = 0usize; + let mut prev: Option = None; + while i < n { + let c = bytes[i]; + // skip quoted spans ('' escape for single quotes is handled by + // the fact that the reopened quote just starts another skip) + if c == b'\'' || c == b'"' { + let q = c; + i += 1; + while i < n && bytes[i] != q { + i += 1; + } + i += 1; + prev = Some(q); + continue; + } + let is_ident_start = c.is_ascii_alphabetic() || c == b'_'; + let prev_blocks = + matches!(prev, Some(p) if p == b'.' || p.is_ascii_alphanumeric() || p == b'_'); + if is_ident_start && !prev_blocks { + let start = i; + while i < n && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { + i += 1; + } + let word = stmt[start..i].to_ascii_lowercase(); + let mut j = i; + while j < n && (bytes[j] as char).is_whitespace() { + j += 1; + } + if j < n && bytes[j] == b'(' && names.contains(&word) { + found.insert(word); + } + prev = Some(bytes[i - 1]); + continue; + } + prev = Some(c); + i += 1; + } + } + found +} + +/// Order `selected` macro names so every macro comes after the macros its +/// body calls (Kahn's algorithm, name-sorted ties for determinism). `defs` +/// maps each selected name to its body. Errors on a dependency cycle (only +/// reachable via cross-library deploy interleaving — DuckDB itself could +/// never have bound a cycle). +pub fn topo_order_macros( + selected: &HashSet, + defs: &BTreeMap, +) -> Result, String> { + let all: HashSet = selected.clone(); + // deps[m] = selected macros m's body calls; rev[d] = macros depending on d + let mut deps: BTreeMap> = BTreeMap::new(); + let mut rev: BTreeMap> = BTreeMap::new(); + for name in selected { + let body = defs + .get(name) + .ok_or_else(|| format!("macro `{}` has no definition", name))?; + let mut called = detect_macro_calls(body, &all); + called.remove(name); // ignore self-recursion (DuckDB rejects it at CREATE anyway) + for d in &called { + rev.entry(d.clone()).or_default().insert(name.clone()); + } + deps.insert(name.clone(), called.into_iter().collect()); + } + let mut ready: BTreeSet = deps + .iter() + .filter(|(_, d)| d.is_empty()) + .map(|(n, _)| n.clone()) + .collect(); + let mut out = Vec::with_capacity(selected.len()); + while let Some(name) = ready.iter().next().cloned() { + ready.remove(&name); + out.push(name.clone()); + if let Some(dependents) = rev.get(&name) { + for dep in dependents.clone() { + let d = deps.get_mut(&dep).unwrap(); + d.remove(&name); + if d.is_empty() { + ready.insert(dep); + } + } + } + deps.remove(&name); + } + if !deps.is_empty() { + let cycle: Vec = deps.keys().cloned().collect(); + return Err(format!( + "macro dependency cycle involving: {}", + cycle.join(", ") + )); + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn names(list: &[&str]) -> HashSet { + list.iter().map(|s| s.to_string()).collect() + } + + #[test] + fn parses_scalar_macro() { + let lib = + parse_macro_library("CREATE MACRO surrogate_key(a, b) AS md5(concat_ws('||', a, b));") + .unwrap(); + assert_eq!( + lib, + vec![LibStatement::Macro(ParsedMacro { + name: "surrogate_key".into(), + params: "a, b".into(), + body: "md5(concat_ws('||', a, b))".into(), + is_table: false, + })] + ); + } + + #[test] + fn parses_table_macro_or_replace_temp_and_function_alias() { + let lib = parse_macro_library( + "CREATE OR REPLACE TEMP MACRO top_n(t_max) AS TABLE SELECT * FROM t WHERE x <= t_max;\n\ + create function dbl(a) as a * 2;", + ) + .unwrap(); + match &lib[0] { + LibStatement::Macro(m) => { + assert_eq!(m.name, "top_n"); + assert!(m.is_table); + assert_eq!(m.body, "SELECT * FROM t WHERE x <= t_max"); + } + other => panic!("expected macro, got {:?}", other), + } + match &lib[1] { + LibStatement::Macro(m) => { + assert_eq!(m.name, "dbl"); + assert!(!m.is_table); + } + other => panic!("expected macro, got {:?}", other), + } + } + + #[test] + fn parses_default_params_and_nested_parens() { + let lib = parse_macro_library( + "CREATE MACRO safe_div(a, b, fallback := (0)) AS CASE WHEN b = 0 THEN fallback ELSE a / b END;", + ) + .unwrap(); + match &lib[0] { + LibStatement::Macro(m) => { + assert_eq!(m.params, "a, b, fallback := (0)"); + assert!(m.body.starts_with("CASE WHEN")); + } + other => panic!("expected macro, got {:?}", other), + } + } + + #[test] + fn params_with_string_containing_paren() { + let lib = parse_macro_library("CREATE MACRO f(sep := '(') AS concat(sep, 'x');").unwrap(); + match &lib[0] { + LibStatement::Macro(m) => assert_eq!(m.params, "sep := '('"), + other => panic!("expected macro, got {:?}", other), + } + } + + #[test] + fn setup_statements_allowed_and_kept_in_order() { + let lib = + parse_macro_library("-- a comment\nATTACH 'x.duckdb' AS ext;\nCREATE MACRO m() AS 1;") + .unwrap(); + assert_eq!(lib.len(), 2); + assert!(matches!(&lib[0], LibStatement::Setup(s) if s.starts_with("ATTACH"))); + assert!(matches!(&lib[1], LibStatement::Macro(_))); + } + + #[test] + fn rejects_non_setup_statements() { + let err = parse_macro_library("CREATE MACRO m() AS 1; SELECT 1;").unwrap_err(); + assert!(err.contains("may only contain"), "{err}"); + let err = parse_macro_library("CREATE TABLE t(x int);").unwrap_err(); + assert!(err.contains("may only contain"), "{err}"); + } + + #[test] + fn rejects_managed_attach_setup() { + let err = + parse_macro_library("ATTACH 'ducklake://analytics' AS lake;\nCREATE MACRO m() AS 1;") + .unwrap_err(); + assert!(err.contains("managed ATTACH"), "{err}"); + } + + #[test] + fn rejects_qualified_and_quoted_names() { + assert!(parse_macro_library("CREATE MACRO lake.m(a) AS a;") + .unwrap_err() + .contains("unqualified")); + assert!(parse_macro_library("CREATE MACRO \"weird name\"(a) AS a;").is_err()); + } + + #[test] + fn rejects_missing_params_or_body() { + assert!(parse_macro_library("CREATE MACRO m AS 1;").is_err()); + assert!(parse_macro_library("CREATE MACRO m(a);").is_err()); + assert!(parse_macro_library("CREATE MACRO m(a) AS ;").is_err()); + } + + #[test] + fn detect_basic_and_word_boundaries() { + let ns = names(&["dbl", "avg_x"]); + let found = detect_macro_calls("SELECT dbl(1), my_dbl(2), avg_x (3) FROM t", &ns); + assert!(found.contains("dbl")); + assert!(found.contains("avg_x")); // whitespace before paren ok + assert_eq!(found.len(), 2); + } + + #[test] + fn detect_skips_qualified_strings_and_comments() { + let ns = names(&["dbl"]); + assert!(detect_macro_calls("SELECT lake.dbl(1)", &ns).is_empty()); + assert!(detect_macro_calls("SELECT 'dbl(1)'", &ns).is_empty()); + assert!(detect_macro_calls("-- dbl(1)\nSELECT 1", &ns).is_empty()); + assert!(detect_macro_calls("SELECT dbl FROM t", &ns).is_empty()); // no call parens + } + + #[test] + fn detect_case_insensitive_and_table_macro_position() { + let ns = names(&["top_n"]); + assert!(!detect_macro_calls("SELECT * FROM TOP_N(3)", &ns).is_empty()); + } + + #[test] + fn topo_chain_and_diamond() { + let mut defs = BTreeMap::new(); + defs.insert("a".to_string(), "b(1) + c(2)".to_string()); + defs.insert("b".to_string(), "d(1)".to_string()); + defs.insert("c".to_string(), "d(2)".to_string()); + defs.insert("d".to_string(), "1".to_string()); + let sel = names(&["a", "b", "c", "d"]); + let order = topo_order_macros(&sel, &defs).unwrap(); + let pos = |n: &str| order.iter().position(|x| x == n).unwrap(); + assert!(pos("d") < pos("b")); + assert!(pos("d") < pos("c")); + assert!(pos("b") < pos("a")); + assert!(pos("c") < pos("a")); + } + + #[test] + fn topo_cycle_errors() { + let mut defs = BTreeMap::new(); + defs.insert("a".to_string(), "b(1)".to_string()); + defs.insert("b".to_string(), "a(1)".to_string()); + let err = topo_order_macros(&names(&["a", "b"]), &defs).unwrap_err(); + assert!(err.contains("cycle"), "{err}"); + } + + #[test] + fn non_ascii_body_survives_verbatim() { + // Regression: the statement splitter must not Latin-1-mojibake + // multi-byte text — macro bodies are persisted and re-executed. + let lib = parse_macro_library("CREATE MACRO greet(a) AS a || ' café ☕';").unwrap(); + match &lib[0] { + LibStatement::Macro(m) => assert_eq!(m.body, "a || ' café ☕'"), + other => panic!("expected macro, got {:?}", other), + } + } + + #[test] + fn non_ascii_garbage_errors_instead_of_panicking() { + // Regression: keyword matching must not byte-slice across a char + // boundary (panicked on inputs like this before). + assert!(parse_macro_library("abcé foo;").is_err()); + assert!(parse_macro_library("créate macro m(a) AS a;").is_err()); + } + + #[test] + fn locally_defined_names_extracted() { + let blocks = vec![ + "ATTACH 'x' AS a;".to_string(), + "CREATE TEMP MACRO dbl(a) AS a * 2;".to_string(), + "SELECT dbl(2);".to_string(), + ]; + let local = locally_defined_macro_names(&blocks); + assert!(local.contains("dbl")); + assert_eq!(local.len(), 1); + } + + #[test] + fn create_statement_roundtrip() { + assert_eq!( + macro_create_statement("m", "a, b := 1", true, "SELECT a + b"), + "CREATE OR REPLACE TEMP MACRO m(a, b := 1) AS TABLE SELECT a + b;" + ); + } + + #[test] + fn builtin_lookup() { + use crate::duckdb_builtins::is_duckdb_builtin; + assert!(is_duckdb_builtin("concat")); + assert!(is_duckdb_builtin("CONCAT")); + assert!(is_duckdb_builtin("read_csv")); + assert!(!is_duckdb_builtin("surrogate_key")); + } +} diff --git a/backend/parsers/windmill-parser/src/lib.rs b/backend/parsers/windmill-parser/src/lib.rs index 5a7e90bf7e..4270edcc79 100644 --- a/backend/parsers/windmill-parser/src/lib.rs +++ b/backend/parsers/windmill-parser/src/lib.rs @@ -13,6 +13,9 @@ use serde::Serialize; use serde_json::Value; pub mod asset_parser; +pub mod duckdb_builtins; +pub mod duckdb_macros; +pub mod sql_materialize; /// S3 output format for SQL queries (moved here to avoid pulling sqlx into WASM via windmill-types) #[derive(Clone, Copy, Debug)] diff --git a/backend/parsers/windmill-parser/src/sql_materialize.rs b/backend/parsers/windmill-parser/src/sql_materialize.rs new file mode 100644 index 0000000000..0e6e6656eb --- /dev/null +++ b/backend/parsers/windmill-parser/src/sql_materialize.rs @@ -0,0 +1,2618 @@ +//! Eligibility classifier + materialization SQL codegen for managed `// materialize`. +//! +//! Managed `// materialize` (the default) promises the script is "setup +//! statements, then one trailing SELECT" — Windmill generates the write DDL +//! around that SELECT (the `// materialize manual` escape hatch opts out and +//! writes its own DDL). This module is the single source of truth for *which +//! block is that SELECT* and *what DDL gets generated*, so save-time validation +//! (deploy path) and run-time codegen (DuckDB executor) can never disagree. +//! +//! Everything here is pure and string-level: no SQL is executed, no type +//! inference is done. The classifier is leading-keyword based and deliberately +//! conservative — anything it can't positively recognize as a read-only output +//! or a known-safe setup statement is rejected, so a script is only accepted +//! for managed mode when its shape is unambiguous. + +/// One top-level statement's role in a wrap-mode script. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BlockClass { + /// Read-only relation the wrap writes from: `SELECT` / `WITH …SELECT` / + /// `FROM` (DuckDB from-first) / `VALUES` / `TABLE x` / `(UN)PIVOT`. + Output, + /// Known-safe preamble: `ATTACH` / `INSTALL` / `LOAD` / `SET` / `PRAGMA` / + /// `USE` / `CREATE TEMP …`. Runs verbatim before the generated write. + Setup, + /// Anything that writes or whose effect we can't vouch for: non-temp + /// `CREATE` / `INSERT` / `UPDATE` / `DELETE` / `MERGE` / `DROP` / `COPY` / + /// `ALTER` / `TRUNCATE`, or an unrecognized leading keyword. Disqualifies + /// managed mode (the user should use `// materialize manual`). + Disallowed, +} + +/// A script accepted for wrapping: zero+ setup blocks then one terminal SELECT. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WrapPlan { + /// Setup statements in source order, verbatim, **without** trailing `;`. + pub setup: Vec, + /// The single terminal output statement, verbatim, **without** trailing `;`. + pub output: String, +} + +/// Why a script is not eligible for managed `// materialize`. Carries enough to +/// render the targeted save-time messages. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum WrapError { + /// No statements at all (empty / comments only). + Empty, + /// No terminal SELECT — nothing to wrap. + NoOutput, + /// More than one top-level SELECT. `count` is how many were found. + MultipleOutputs { count: usize }, + /// A SELECT exists but isn't the last statement (something runs after it). + OutputNotLast, + /// A write/unknown statement appears among the setup blocks. `snippet` is a + /// short prefix of the offending statement for the error message. + DisallowedBlock { snippet: String }, +} + +impl WrapError { + /// Human-facing, actionable message (matches the spec's rejection text). + pub fn message(&self) -> String { + let base = + "managed `// materialize` requires the script to be setup statements then a single trailing SELECT"; + let manual = "use `// materialize manual` to write the DDL yourself"; + match self { + WrapError::Empty => format!("{base}: the script is empty."), + WrapError::NoOutput => format!("{base}: found no SELECT — {manual}."), + WrapError::MultipleOutputs { count } => format!( + "{base}: found {count} SELECT statements; combine them with a CTE, or {manual}." + ), + WrapError::OutputNotLast => format!( + "{base}: found statements after the SELECT — move them above it, or {manual}." + ), + WrapError::DisallowedBlock { snippet } => { + format!("{base}: `{snippet}` writes or is unrecognized — {manual}.") + } + } + } +} + +/// Split SQL into top-level, `;`-separated statements, skipping line comments +/// (`-- …`), block comments (`/* … */`), single-quoted strings (`'…'` with +/// `''` escape) and double-quoted identifiers (`"…"`). Semicolons inside any of +/// those are not separators. Returns each statement trimmed, comments stripped, +/// empties dropped. Self-contained so the parser crate stays dependency-free; +/// it must stay behaviourally aligned with the executor's block splitter (both +/// route wrap through `classify_wrap`, so the split they see is this one). +pub fn split_statements(sql: &str) -> Vec { + let mut out = Vec::new(); + let mut cur = String::new(); + // Char-wise, not byte-wise: the emitted strings are re-executed + // (materialize codegen) and persisted (macro registry), so a Latin-1 + // `bytes[i] as char` decode would corrupt any multi-byte text inside + // statements. All delimiters are ASCII — split positions are unaffected. + let chars: Vec = sql.chars().collect(); + let mut i = 0; + let n = chars.len(); + while i < n { + let c = chars[i]; + // line comment — `--` (SQL) or `//`. The `//` form is not SQL, but it + // is how Windmill pipeline annotations (`// materialize`, `// pipeline`, + // …) are written, and they sit above the SQL in the same script; strip + // them so they don't pollute the first statement block's classification + // or the generated setup SQL. + if (c == '-' && i + 1 < n && chars[i + 1] == '-') + || (c == '/' && i + 1 < n && chars[i + 1] == '/') + { + while i < n && chars[i] != '\n' { + i += 1; + } + continue; + } + // block comment + if c == '/' && i + 1 < n && chars[i + 1] == '*' { + i += 2; + while i + 1 < n && !(chars[i] == '*' && chars[i + 1] == '/') { + i += 1; + } + i += 2; + continue; + } + // single-quoted string + if c == '\'' { + cur.push(c); + i += 1; + while i < n { + cur.push(chars[i]); + if chars[i] == '\'' { + // doubled '' is an escaped quote, stay in string + if i + 1 < n && chars[i + 1] == '\'' { + cur.push('\''); + i += 2; + continue; + } + i += 1; + break; + } + i += 1; + } + continue; + } + // double-quoted identifier + if c == '"' { + cur.push(c); + i += 1; + while i < n { + cur.push(chars[i]); + if chars[i] == '"' { + i += 1; + break; + } + i += 1; + } + continue; + } + if c == ';' { + let t = cur.trim(); + if !t.is_empty() { + out.push(t.to_string()); + } + cur.clear(); + i += 1; + continue; + } + cur.push(c); + i += 1; + } + let t = cur.trim(); + if !t.is_empty() { + out.push(t.to_string()); + } + out +} + +/// Lowercased top-level keyword tokens of a single statement (parens collapsed +/// away: tokens *inside* balanced `(...)` are skipped, so a CTE body's verbs +/// don't leak up). Strings/identifiers are already gone from the split, but we +/// re-guard quotes defensively. Used to disambiguate `WITH …` and `CREATE …`. +fn top_level_keywords(stmt: &str) -> Vec { + let mut toks = Vec::new(); + let mut cur = String::new(); + let mut depth: i32 = 0; + let bytes = stmt.as_bytes(); + let mut i = 0; + let n = bytes.len(); + let flush = |cur: &mut String, toks: &mut Vec| { + if !cur.is_empty() { + toks.push(cur.to_lowercase()); + cur.clear(); + } + }; + while i < n { + let c = bytes[i] as char; + if c == '\'' || c == '"' { + let q = bytes[i]; + i += 1; + while i < n && bytes[i] != q { + i += 1; + } + i += 1; + continue; + } + if c == '(' { + flush(&mut cur, &mut toks); + depth += 1; + i += 1; + continue; + } + if c == ')' { + if depth > 0 { + depth -= 1; + } + i += 1; + continue; + } + if depth > 0 { + i += 1; + continue; + } + if c.is_alphanumeric() || c == '_' { + cur.push(c); + } else { + flush(&mut cur, &mut toks); + } + i += 1; + } + flush(&mut cur, &mut toks); + toks +} + +const OUTPUT_KW: &[&str] = &["select", "from", "values", "table", "pivot", "unpivot"]; +const SETUP_KW: &[&str] = &["attach", "install", "load", "set", "pragma", "use"]; +const WRITE_VERBS: &[&str] = &["insert", "update", "delete", "merge"]; + +/// Classify a single statement by its leading keyword (with `WITH`/`CREATE` +/// disambiguation). See [`BlockClass`]. +pub fn classify_block(stmt: &str) -> BlockClass { + let kws = top_level_keywords(stmt); + let Some(first) = kws.first().map(String::as_str) else { + return BlockClass::Disallowed; + }; + + // CREATE TEMP … is setup (staging); any other CREATE is a write. + if first == "create" { + let temp = kws + .iter() + .skip(1) + .take(3) + .any(|k| k == "temp" || k == "temporary"); + return if temp { + BlockClass::Setup + } else { + BlockClass::Disallowed + }; + } + + // WITH … : the main statement's verb decides. CTE bodies are parenthesized, + // so their verbs are not in `kws`; the first top-level write verb or SELECT + // after the CTE list is the real one. + if first == "with" { + for k in kws.iter().skip(1) { + if k == "select" { + return BlockClass::Output; + } + if WRITE_VERBS.contains(&k.as_str()) { + return BlockClass::Disallowed; + } + } + // `WITH x AS (...) SELECT` where SELECT got collapsed is impossible + // (SELECT here is top-level), so a WITH with no top-level verb is a + // malformed/unknown statement — reject conservatively. + return BlockClass::Disallowed; + } + + if OUTPUT_KW.contains(&first) { + return BlockClass::Output; + } + if SETUP_KW.contains(&first) { + return BlockClass::Setup; + } + BlockClass::Disallowed +} + +/// Validate a script for managed `// materialize` and, on success, return the +/// setup/output split. Enforces the four conditions from the spec: +/// 1. exactly one Output block, 2. it is last, 3. all preceding blocks are +/// Setup, 4. nothing after it. +pub fn classify_wrap(sql: &str) -> Result { + let stmts = split_statements(sql); + if stmts.is_empty() { + return Err(WrapError::Empty); + } + let classes: Vec = stmts.iter().map(|s| classify_block(s)).collect(); + + let output_idxs: Vec = classes + .iter() + .enumerate() + .filter(|(_, c)| **c == BlockClass::Output) + .map(|(i, _)| i) + .collect(); + + match output_idxs.len() { + 0 => return Err(WrapError::NoOutput), + 1 => {} + count => return Err(WrapError::MultipleOutputs { count }), + } + let out_idx = output_idxs[0]; + if out_idx != stmts.len() - 1 { + return Err(WrapError::OutputNotLast); + } + // Everything before the output must be Setup (no Disallowed preamble). + for (i, c) in classes.iter().enumerate().take(out_idx) { + if *c != BlockClass::Setup { + return Err(WrapError::DisallowedBlock { snippet: snippet(&stmts[i]) }); + } + } + Ok(WrapPlan { setup: stmts[..out_idx].to_vec(), output: stmts[out_idx].clone() }) +} + +fn snippet(stmt: &str) -> String { + let one_line: String = stmt.split_whitespace().collect::>().join(" "); + if one_line.chars().count() > 40 { + let truncated: String = one_line.chars().take(40).collect(); + format!("{truncated}…") + } else { + one_line + } +} + +// --------------------------------------------------------------------------- +// Codegen +// --------------------------------------------------------------------------- + +/// How a (partition of a) materialized table is reconciled on each run. +/// Derived at deploy from the annotation: `key=history` (or the `scd2` +/// alias) → `Scd2`, else `append` → `Append`, else `unique_key` → `Merge`, else +/// `Replace`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum MaterializeStrategy { + /// DELETE the current partition, then INSERT — partition becomes exactly + /// what the SELECT returned. Full-refresh of the slice. + Replace, + /// Upsert within the slice on `unique_key` (delete-by-key + insert); rows + /// absent from the SELECT are left in place. This is SCD type 1: a changed + /// row overwrites the prior value, keeping no history. + Merge { unique_key: String }, + /// INSERT only — immutable event-log semantics. + Append, + /// Slowly Changing Dimension type 2: the SELECT is the *current* snapshot + /// (one row per `key`); a change to any tracked column closes the prior + /// version (`valid_to`/`is_current=false`) and opens a new one, so the full + /// history is preserved. `track` empty ⇒ every non-key column is tracked. + /// `close_deleted` (opt-in `deletes=close`) also closes the current version + /// of a key that disappears from the snapshot (dbt's `hard_deletes=close`); + /// default (false) leaves absent keys current (soft delete). + /// Unpartitioned only (the worker rejects `// partitioned` + scd2). + Scd2 { key: String, track: Vec, close_deleted: bool }, +} + +/// SCD2 metadata columns appended to the managed history table. Fixed names so +/// the generated diff/close/open SQL and any `// data_test` on them agree. +const SCD2_VALID_FROM: &str = "valid_from"; +const SCD2_VALID_TO: &str = "valid_to"; +const SCD2_IS_CURRENT: &str = "is_current"; +/// Connection-local temp table holding the keys whose version must be rotated +/// this run (changed + new). Captured before the write so the close and the +/// open see the same set. `_wm_` prefix so it never collides with user tables. +const SCD2_CHANGED_KEYS: &str = "_wm_scd2_changed"; +/// Connection-local temp table holding the keys that disappeared from the +/// snapshot this run (present-and-current in the table, absent from the SELECT). +/// Only used when `close_deleted` (`deletes=close`) is set. +const SCD2_DELETED_KEYS: &str = "_wm_scd2_deleted"; + +/// Inputs to materialization codegen, all resolved at run time by the worker. +/// Pure: produces SQL text; executes nothing. +#[derive(Debug, Clone)] +pub struct MaterializeCodegen<'a> { + /// Fully-qualified target, e.g. `_wm_target.orders_daily`. Always qualified + /// so a user `USE …;` in setup can't redirect the write. + pub target_qualified: &'a str, + /// The user's output SELECT (verbatim, no trailing `;`) — embedded as a + /// subquery so its own shape is irrelevant to the generated wrapper. + pub select_sql: &'a str, + /// Physical partition column added to the managed table. + pub partition_col: &'a str, + /// SQL expression for the current partition value — a literal like + /// `'2026-06-19'` or a bind placeholder. The caller is responsible for + /// safe quoting/binding. + pub partition_value_sql: &'a str, + /// Whether `// partitioned` applies. When false the table is unpartitioned + /// and the partition column / `SET PARTITIONED BY` are omitted. + pub partitioned: bool, + pub strategy: MaterializeStrategy, + /// Write-time guardrail for a drifted SELECT vs the fixed table schema. + /// Only the persist-and-mutate strategies (partitioned replace, merge, + /// append) act on it: `Fail` emits an in-txn guard that raises on drift, + /// `Sync` writes BY NAME and expects the executor to inject `ALTER TABLE` + /// DDL at the [`SYNC_ALTER_SENTINEL`] slot, `Warn`/`Ignore` write + /// positionally (drift surfaced by the summary in `Warn`, silent in + /// `Ignore`). See [`MaterializeCodegen::is_persist_and_mutate`]. + pub on_schema_change: OnSchemaChange, +} + +/// The exact statement the `sync` codegen emits right after `BEGIN +/// TRANSACTION;` as the injection slot for `ALTER TABLE … ADD/DROP COLUMN` +/// DDL. The executor computes the drift with a pre-pass probe and replaces this +/// literal in the assembled query text (with the DDL, or removes it when there +/// is no drift). Classified `Write` so the EE write-audit-publish reassembly +/// keeps it inside the transaction with the mutations; a plain no-op SELECT so +/// that if it is somehow left un-replaced the run still succeeds unchanged. +pub const SYNC_ALTER_SENTINEL: &str = "SELECT '__wm_sync_alter_sentinel__' AS _wm_sync;"; + +impl<'a> MaterializeCodegen<'a> { + /// Whether this (strategy, partitioned) uses the positional persist-and- + /// mutate write whose table schema is fixed at first CREATE — the only case + /// the `on_schema_change` write-time guardrail applies to. Whole-table + /// replace (`CREATE OR REPLACE`) and scd2 self-heal / are name-mapped, so + /// they are excluded. + pub fn is_persist_and_mutate(&self) -> bool { + match self.strategy { + MaterializeStrategy::Scd2 { .. } => false, + MaterializeStrategy::Replace => self.partitioned, + MaterializeStrategy::Append | MaterializeStrategy::Merge { .. } => true, + } + } + + /// The ordered statements that perform the materialization, to be run after + /// the setup blocks and inside the caller's execution. The first-run + /// bootstrap is idempotent (`IF NOT EXISTS`), so this is safe to run every + /// time. The DELETE/INSERT body is wrapped in one transaction so a partial + /// failure leaves the prior snapshot intact. Every strategy reduces to + /// DELETE+INSERT (no `MERGE INTO`) — see the `Merge` arm for why. + pub fn statements(&self) -> Vec { + let t = self.target_qualified; + let sel = self.select_sql; + let pcol = self.partition_col; + let pval = self.partition_value_sql; + let mut out = Vec::new(); + + // SCD2 has a shape unlike the DELETE/INSERT strategies (diff → close old + // → open new) and does not support partitioning (rejected at the worker), + // so it is generated up front by its own helper. + if let MaterializeStrategy::Scd2 { key, track, close_deleted } = &self.strategy { + return self.scd2_statements(key, track, *close_deleted); + } + + // Whole-table replace: rebuild the table to match the SELECT's *current* + // schema each run with one atomic `CREATE OR REPLACE` (which DuckLake + // still snapshots). This is the only path that survives a changed SELECT + // or a pre-existing table with a different schema — the persist-and- + // mutate paths below fix the schema at first create. + if !self.partitioned && matches!(self.strategy, MaterializeStrategy::Replace) { + out.push(format!( + "CREATE OR REPLACE TABLE {t} AS SELECT * FROM ({sel});" + )); + return out; + } + + // Persist-and-mutate (partitioned, or merge/append): bootstrap the table + // if absent, then write into it. The schema is fixed at first create — + // a later SELECT-schema change needs a manual rebuild (schema evolution + // is a follow-up). + if self.partitioned { + out.push(format!( + "CREATE TABLE IF NOT EXISTS {t} AS \ + SELECT *, CAST(NULL AS VARCHAR) AS {pcol} FROM ({sel}) WHERE false;" + )); + out.push(format!("ALTER TABLE {t} SET PARTITIONED BY ({pcol});")); + } else { + out.push(format!( + "CREATE TABLE IF NOT EXISTS {t} AS SELECT * FROM ({sel}) WHERE false;" + )); + } + out.push("BEGIN TRANSACTION;".to_string()); + // Write-time schema guardrail, emitted right after BEGIN so it runs + // before any mutation (a failing guard aborts before touching data; the + // sync ALTERs run before the INSERT so the sets match). Reached only on + // the persist-and-mutate path here (whole-table replace and scd2 return + // above), so no extra strategy gate is needed. + match self.on_schema_change { + OnSchemaChange::Fail => out.push(schema_drift_guard_sql(sel, t, pcol)), + OnSchemaChange::Sync => out.push(SYNC_ALTER_SENTINEL.to_string()), + OnSchemaChange::Warn | OnSchemaChange::Ignore => {} + } + // The rows to write, with the partition column appended when partitioned. + let source = if self.partitioned { + format!("SELECT *, {pval} AS {pcol} FROM ({sel})") + } else { + format!("SELECT * FROM ({sel})") + }; + // `sync` maps columns by name (positional would cross-wire: an ALTERed + // ADD COLUMN appends at the end, so a positional INSERT of the SELECT + // would fill it from the wrong source column). + let by_name = if self.on_schema_change == OnSchemaChange::Sync { + " BY NAME" + } else { + "" + }; + match &self.strategy { + MaterializeStrategy::Replace => { + // Only reached when partitioned (whole-table replace returned above). + out.push(format!("DELETE FROM {t} WHERE {pcol} = {pval};")); + out.push(format!("INSERT INTO {t}{by_name} {source};")); + } + MaterializeStrategy::Append => { + out.push(format!("INSERT INTO {t}{by_name} {source};")); + } + MaterializeStrategy::Merge { unique_key } => { + // Upsert within the slice via delete-by-key + insert (dbt's + // `delete+insert`): rows whose key is in the incoming SELECT are + // replaced, others are left in place. This deliberately avoids + // `MERGE INTO` — DuckLake's MERGE fails writing the first rows of + // a fresh partition (HTTP 404 on the new parquet), and a failed + // write leaves the table needing a DROP. DELETE+INSERT is the + // same write shape as `replace`, which is reliable. The DELETE is + // scoped to the current partition when partitioned so it stays + // slice-local (a key present in another partition is untouched). + // + // Guard first: the DELETE+INSERT does not dedup the source, so + // two incoming rows sharing a key would both persist under it. + // Raise instead of silently double-writing (see the helper). + out.push(duplicate_source_key_guard_sql(sel, unique_key)); + let scope = if self.partitioned { + format!("{pcol} = {pval} AND ") + } else { + String::new() + }; + out.push(format!( + "DELETE FROM {t} WHERE {scope}{unique_key} IN (SELECT {unique_key} FROM ({sel}));" + )); + out.push(format!("INSERT INTO {t}{by_name} {source};")); + } + // Handled by the early return above (scd2 has no partitioned form). + MaterializeStrategy::Scd2 { .. } => unreachable!("scd2 handled before this match"), + } + out.push("COMMIT;".to_string()); + out + } + + /// SCD2 codegen: the incoming SELECT is the *current desired snapshot* (one + /// row per `key`); we diff it against the live current rows, close the prior + /// version of every changed/new key, and open a fresh one — so history is + /// kept. `track` empty ⇒ every non-key column is tracked for change + /// detection. + /// + /// Shape (all one transaction for the mutation, mirroring the other + /// strategies so a partial failure leaves the prior snapshot intact): + /// 1. bootstrap the table (business columns + `valid_from/valid_to/ + /// is_current`), idempotent; + /// 2. capture changed+new keys into a connection-local temp table *before* + /// the write — the close below flips `is_current`, so recomputing the + /// diff after it would see a different set; + /// 3. close the prior open version of those keys (`UPDATE` — not `MERGE`: + /// DuckLake's MERGE is the unreliable path, plain UPDATE works); + /// 3b. when `close_deleted`, also capture the keys that vanished from the + /// snapshot and close their current version (no reopen) — dbt's + /// `hard_deletes=close`; + /// 4. open a new current version from the snapshot; + /// 5. create the `_current` convenience view once (`IF NOT EXISTS`), + /// inside the same transaction so it doesn't advance the DuckLake snapshot + /// past the data write the summary records (and so an unchanged rerun, + /// whose UPDATE/INSERT touch no rows, stays a true no-op). + /// + /// Close/open match keys with `IS NOT DISTINCT FROM` (via a correlated + /// `EXISTS`), not `key IN (…)`: SQL `IN` never matches `NULL`, so a `NULL` + /// natural key would be flagged as changed yet silently skipped by both the + /// close and the open, dropping the row. Null-safe matching materializes it + /// instead (a `NULL` key is still ill-formed for a dimension — guard it with + /// `// data_test not_null ` — but it must not vanish). + /// + /// Without `close_deleted`, keys present in the table but absent from the + /// SELECT are left current (soft delete — dbt's `hard_deletes=ignore` default; + /// with `close_deleted` they are closed instead — see step 3b). The effective + /// timestamp is `now()`, which DuckDB fixes to + /// the transaction start, so `valid_from`/`valid_to` are consistent within a + /// run without a nondeterministic per-statement clock. + /// + /// Reserved columns: `valid_from`/`valid_to`/`is_current` are appended to the + /// user's SELECT with these fixed names (kept clean so consumers write + /// `WHERE is_current` / `ASOF JOIN … >= valid_from`). A SELECT that already + /// projects one of them is a v1 constraint violation — the bootstrap then + /// produces a duplicate output column and the run fails at execution + /// (documented; not statically checkable here since the SELECT's columns + /// aren't known at codegen time). + fn scd2_statements(&self, key: &str, track: &[String], close_deleted: bool) -> Vec { + let t = self.target_qualified; + let sel = self.select_sql; + let k = quote_ident(key); + let vf = SCD2_VALID_FROM; + let vt = SCD2_VALID_TO; + let ic = SCD2_IS_CURRENT; + let changed = SCD2_CHANGED_KEYS; + let deleted = SCD2_DELETED_KEYS; + // Transaction-stable effective timestamp (see doc above). Cast to plain + // TIMESTAMP so it matches the bootstrapped column type (now() is TZ-aware). + let ts = "CAST(now() AS TIMESTAMP)"; + + // Projection compared to detect change. Empty `track` ⇒ all business + // columns via `* EXCLUDE ()` on the table side (which carries + // the extra metadata columns) and `*` on the snapshot side. An explicit + // `track` ⇒ key + those columns on both sides. `EXCEPT` treats NULLs as + // equal, so an unchanged NULL is not read as a change. + let (src_proj, tgt_proj) = if track.is_empty() { + ( + format!("SELECT * FROM ({sel})"), + format!("SELECT * EXCLUDE ({vf}, {vt}, {ic}) FROM {t} WHERE {ic}"), + ) + } else { + let cols = std::iter::once(key) + .chain(track.iter().map(String::as_str)) + .map(quote_ident) + .collect::>() + .join(", "); + ( + format!("SELECT {cols} FROM ({sel})"), + format!("SELECT {cols} FROM {t} WHERE {ic}"), + ) + }; + + let mut out = vec![ + format!( + "CREATE TABLE IF NOT EXISTS {t} AS SELECT *, \ + CAST(NULL AS TIMESTAMP) AS {vf}, \ + CAST(NULL AS TIMESTAMP) AS {vt}, \ + CAST(NULL AS BOOLEAN) AS {ic} FROM ({sel}) WHERE false;" + ), + format!( + "CREATE OR REPLACE TEMP TABLE {changed} AS \ + SELECT {k} FROM ({src_proj} EXCEPT {tgt_proj});" + ), + ]; + // Hard-delete-close (`deletes=close`): the keys that vanished from the + // snapshot — present-and-current in the table, absent from the SELECT. + // Captured before the close (like `changed`) and disjoint from it (a + // key is either in the snapshot or not), so the two closes never overlap. + if close_deleted { + out.push(format!( + "CREATE OR REPLACE TEMP TABLE {deleted} AS \ + SELECT {k} FROM (SELECT {k} FROM {t} WHERE {ic} EXCEPT SELECT {k} FROM ({sel}));" + )); + } + out.push("BEGIN TRANSACTION;".to_string()); + out.push(format!( + "UPDATE {t} SET {vt} = {ts}, {ic} = false \ + WHERE {ic} AND EXISTS (SELECT 1 FROM {changed} \ + WHERE {changed}.{k} IS NOT DISTINCT FROM {t}.{k});" + )); + // Close vanished keys — no matching INSERT below, so they close without + // reopening. A key that later reappears isn't in `WHERE is_current`, so the + // `changed` diff treats it as new and opens a fresh version (a validity gap + // between the delete and the reactivation — correct SCD2). + if close_deleted { + out.push(format!( + "UPDATE {t} SET {vt} = {ts}, {ic} = false \ + WHERE {ic} AND EXISTS (SELECT 1 FROM {deleted} \ + WHERE {deleted}.{k} IS NOT DISTINCT FROM {t}.{k});" + )); + } + out.push(format!( + "INSERT INTO {t} SELECT s.*, {ts} AS {vf}, CAST(NULL AS TIMESTAMP) AS {vt}, \ + true AS {ic} FROM ({sel}) s WHERE EXISTS (SELECT 1 FROM {changed} c \ + WHERE c.{k} IS NOT DISTINCT FROM s.{k});" + )); + out.push( + // Consumer convenience: a `_current` view (the live slice) so the + // common "just the latest version" read needs no `WHERE is_current`, + // and downstream scripts can `// on` / read it directly. For the + // effective-dated payoff, consumers `ASOF JOIN ON fact.key = + // dim. AND fact.ts >= dim.valid_from`. + // + // `CREATE VIEW IF NOT EXISTS` (not `OR REPLACE`), created inside the + // write transaction, on purpose: the view definition never changes + // (`SELECT * WHERE is_current` always reflects live data), and a + // catalog write advances the DuckLake snapshot — so `OR REPLACE` on + // every run would (a) advance the snapshot on an otherwise no-op + // unchanged run and (b) make the summary's `max(snapshot_id)` record + // the view DDL instead of the data write. `IF NOT EXISTS` creates it + // once (folded into the first data-write snapshot) and is a true no-op + // afterwards. The `_current` name is reserved: if a real table by + // that name already exists, `IF NOT EXISTS` skips silently (no view, + // no error) — documented as a reserved suffix. + format!("CREATE VIEW IF NOT EXISTS {t}_current AS SELECT * FROM {t} WHERE {ic};"), + ); + out.push("COMMIT;".to_string()); + out + } +} + +// --------------------------------------------------------------------------- +// on_schema_change drift detection (write-time guardrail) +// --------------------------------------------------------------------------- +// +// Drift is computed entirely in SQL against the live DuckDB session: the +// SELECT's output columns come from `DESCRIBE`, the table's from `DESCRIBE` of +// the target (the managed `_wm_partition` column is excluded so it is compared +// as the producer's logical output, matching schema capture). `added` = SELECT +// columns absent from the table, `removed` = table columns absent from the +// SELECT. When the table was just created this run (first materialize) the two +// DESCRIBEs agree, so both lists are empty and no guard fires. + +/// A `DESCRIBE`-derived set of column names of `rel_sql` (any SELECT-able +/// relation, already parenthesized/qualified by the caller), optionally +/// dropping the managed partition column. +fn describe_col_names(rel_sql: &str, exclude_col: Option<&str>) -> String { + let filter = match exclude_col { + Some(c) => format!(" WHERE column_name <> {}", quote_lit(c)), + None => String::new(), + }; + format!("SELECT column_name FROM (DESCRIBE SELECT * FROM {rel_sql}){filter}") +} + +/// Scalar subqueries `(added, removed)` — the list of column names in the +/// SELECT but not the table, and vice versa. Each is a `list(...)` over an +/// `EXCEPT`; empty ⇒ `list()` yields an empty list (`len` 0). The partition +/// column is excluded on the table side only. +/// +/// Set difference, not ordered: this catches the add/remove/rename that a +/// positional INSERT misaligns on, but by design NOT a pure reorder of +/// same-named columns (identical sets ⇒ empty added/removed). See the +/// `OnSchemaChange` doc in asset_parser.rs — reorder-safety is `sync`'s job +/// (INSERT BY NAME); the set difference is deliberately kept over an ordered +/// comparison so the `fail` guard cannot false-positive on a correctly-aligned +/// write. +fn drift_lists(sel_sql: &str, target_qualified: &str, partition_col: &str) -> (String, String) { + let sel = describe_col_names(&format!("({sel_sql})"), None); + let tbl = describe_col_names(target_qualified, Some(partition_col)); + let added = format!("(SELECT list(column_name) FROM (({sel}) EXCEPT ({tbl})))"); + let removed = format!("(SELECT list(column_name) FROM (({tbl}) EXCEPT ({sel})))"); + (added, removed) +} + +/// The `fail`-mode guard: a single statement that raises via DuckDB `error(...)` +/// when the SELECT's columns diverge from the table's, naming the added/removed +/// columns and the target. Both `CASE` branches are cast to VARCHAR so the +/// planner cannot constant-fold the `error(...)` away, and the condition depends +/// on the runtime drift subqueries so it is never folded to a constant. +fn schema_drift_guard_sql(sel_sql: &str, target_qualified: &str, partition_col: &str) -> String { + let (added, removed) = drift_lists(sel_sql, target_qualified, partition_col); + // `target_qualified` is safe in table-reference position (quoted identifiers) + // but here it lands inside a SQL string literal, so single-quotes must be + // doubled (same as `quote_lit`). + let tq = target_qualified.replace('\'', "''"); + format!( + "SELECT CASE WHEN coalesce(len(_wm_added), 0) + coalesce(len(_wm_removed), 0) > 0 \ + THEN CAST(error('managed materialize: on_schema_change=fail blocked a schema-drifted \ + write to {tq} — added column(s): [' || coalesce(array_to_string(_wm_added, ', '), '') || \ + '], removed column(s): [' || coalesce(array_to_string(_wm_removed, ', '), '') || \ + ']. The table schema is fixed at first create; set on_schema_change=sync to auto-migrate, \ + or align the SELECT with the table.') AS VARCHAR) ELSE 'ok' END \ + FROM (SELECT {added} AS _wm_added, {removed} AS _wm_removed);", + tq = tq, + ) +} + +/// In-transaction guard for the keyed `merge` strategy: raises via DuckDB +/// `error(...)` when the source SELECT holds more than one row for the same +/// non-NULL `unique_key`. A keyed merge is delete-by-key + insert-all (it does +/// NOT deduplicate the source), so duplicate source keys would land every +/// duplicate row under one key — the exact silent double-write this guards +/// against. Erroring keeps the semantics explicit: the author must deduplicate +/// in the SELECT (or use `append`). NULL keys are excluded to match the delete's +/// `key IN (...)` scope, which never matches NULL. `unique_key` is embedded raw +/// in identifier position (matching the merge's own DELETE/IN) and doubled-quote +/// escaped where it lands inside the error string literal. +fn duplicate_source_key_guard_sql(sel_sql: &str, unique_key: &str) -> String { + let key_lit = unique_key.replace('\'', "''"); + format!( + "SELECT CASE WHEN _wm_dup_keys > 0 THEN CAST(error('managed materialize: keyed merge on \ + `{key_lit}` blocked — the source has ' || _wm_dup_keys || ' key value(s) with more than \ + one row. A keyed merge keeps one row per key and does not deduplicate; deduplicate in the \ + SELECT (e.g. QUALIFY row_number() OVER (PARTITION BY {key_lit} ORDER BY …) = 1) or use \ + `append`.') AS VARCHAR) ELSE 'ok' END FROM (SELECT count(*) AS _wm_dup_keys FROM (SELECT \ + {unique_key} FROM ({sel_sql}) WHERE {unique_key} IS NOT NULL GROUP BY {unique_key} HAVING \ + count(*) > 1));" + ) +} + +/// The `warn`-mode summary column: a `schema_drift` struct `{added, removed}` +/// when the SELECT drifted from the table, else NULL. Appended to the +/// materialize summary row so the executor can log it and fold it into the job +/// result without an extra round-trip. +fn schema_drift_summary_field( + sel_sql: &str, + target_qualified: &str, + partition_col: &str, +) -> String { + let (added, removed) = drift_lists(sel_sql, target_qualified, partition_col); + format!( + "(SELECT CASE WHEN coalesce(len(_wm_added), 0) + coalesce(len(_wm_removed), 0) > 0 \ + THEN {{'added': _wm_added, 'removed': _wm_removed}} END \ + FROM (SELECT {added} AS _wm_added, {removed} AS _wm_removed)) AS schema_drift" + ) +} + +/// The read that captures the DuckLake snapshot id produced by the write, for +/// the given attach alias (e.g. `_wm_target`). The worker runs this last and +/// records the result into `materialized_partition`. +pub fn snapshot_capture_sql(alias: &str) -> String { + format!("SELECT max(snapshot_id) AS snapshot_id FROM ducklake_snapshots('{alias}');") +} + +/// Reserved attach alias for the materialization target, fully-qualified in all +/// generated SQL so a user `USE …;` in the setup blocks can't redirect the +/// write. The worker resolves the real `ATTACH 'ducklake:…' AS _wm_target (…)` +/// from the target ducklake's config and passes it in as `target_attach`. +pub const TARGET_ALIAS: &str = "_wm_target"; + +/// Structural role of one statement in a [`MaterializePlan`]. The public build +/// executes the plan verbatim, so the kinds are pure metadata there; they exist +/// so a downstream assembler (`pipeline_advanced::finalize_materialize_query`) +/// can reason about the plan without parsing SQL. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum MaterializeStmtKind { + /// Pre-write statement: user setup, the target ATTACH, referenced-asset + /// ATTACHes. + Setup, + /// Write work against the target: bootstrap DDL, SCD2 temp-table captures, + /// the mutation itself, the `_current` view. + Write, + /// The `BEGIN TRANSACTION;` marker emitted by the strategy codegen. + TxnBegin, + /// The `COMMIT;` marker emitted by the strategy codegen. + TxnCommit, + /// The trailing one-row summary read (asset / rows / snapshot_id / + /// data_tests breakdown). + Summary, +} + +/// One planned statement: its structural role and the SQL text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct MaterializeStmt { + pub kind: MaterializeStmtKind, + pub sql: String, +} + +/// The full ordered materialization plan [`build_wrap_blocks`] produces: +/// statements in execution order plus the compiled data-test checks (also +/// embedded in the summary statement's breakdown). Assembled into the final +/// statement list by `pipeline_advanced::finalize_materialize_query`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct MaterializePlan { + pub stmts: Vec, + pub checks: Vec, +} + +/// Assemble the full ordered statement list the DuckDB executor runs for a +/// managed `// materialize` script. This is the single entry point the worker +/// calls; it composes the already-tested pieces (classifier split → target +/// ATTACH → strategy codegen → snapshot capture) so their ordering lives in one +/// tested place rather than inline in the executor. +/// +/// `target_attach` is the real `ATTACH 'ducklake:…' AS _wm_target (…);` string +/// the worker built from config (it depends on resolved credentials, so it +/// can't be generated here). `target_table` is the table within that catalog +/// (e.g. `orders_daily`), referenced as `_wm_target.
`. `asset_path` is +/// the full `/
` for the result summary. The trailing statement is +/// a one-row summary read (asset / rows / snapshot_id) that is both the job's +/// result (a useful preview) and what the worker records. +/// +/// Returns a [`MaterializePlan`] — the statements plus their structural role +/// and the compiled data-test checks — rather than raw SQL: the executor hands +/// the plan to `windmill_common::pipeline_advanced::finalize_materialize_query` +/// (which this crate cannot depend on), whose public-build implementation +/// assembles the statements verbatim. Everything this function produces runs +/// as-is on the public build; the plan's structure is metadata about it, not a +/// second mode. +pub fn build_wrap_blocks( + plan: &WrapPlan, + target_attach: &str, + target_table: &str, + asset_path: &str, + partition_col: &str, + partition_value_sql: &str, + partitioned: bool, + strategy: MaterializeStrategy, + on_schema_change: OnSchemaChange, + tests: &[DataTestResolved], +) -> Result { + let target_qualified = format!("{TARGET_ALIAS}.{target_table}"); + let scd2 = matches!(strategy, MaterializeStrategy::Scd2 { .. }); + let ctx = DataTestCtx { + target_qualified: &target_qualified, + asset_path, + partition_col, + partition_value_sql, + partitioned, + scd2, + }; + let test_sql = build_data_test_checks(tests, &ctx)?; + let cg = MaterializeCodegen { + target_qualified: &target_qualified, + select_sql: &plan.output, + partition_col, + partition_value_sql, + partitioned, + strategy, + on_schema_change, + }; + // `warn` folds the post-write drift into the summary row (executor logs it + + // returns it) — only for the positional persist-and-mutate path, and only in + // `warn`: `fail`/`sync` guard the write itself, `ignore` is silent. + let drift_summary_select = + if on_schema_change == OnSchemaChange::Warn && cg.is_persist_and_mutate() { + Some(plan.output.as_str()) + } else { + None + }; + let mut stmts: Vec = Vec::new(); + let setup = |sql: String| MaterializeStmt { kind: MaterializeStmtKind::Setup, sql }; + // Setup blocks come from the splitter with their `;` stripped — re-terminate + // each so that when the executor re-joins and re-splits the assembled query, + // adjacent statements (e.g. the user ATTACH and the synthetic target ATTACH) + // don't merge into one malformed statement. + stmts.extend(plan.setup.iter().map(|s| setup(terminate(s)))); + stmts.push(setup(target_attach.to_string())); + // Referenced-asset ATTACHes (relationships tests) — read-only, before the + // write and the summary that probes them. + stmts.extend(test_sql.attaches.into_iter().map(setup)); + // Classify the codegen statements by matching the exact transaction-marker + // literals this module emits (`BEGIN TRANSACTION;` / `COMMIT;`); everything + // else the codegen produces is write work. + stmts.extend(cg.statements().into_iter().map(|sql| { + let kind = match sql.as_str() { + "BEGIN TRANSACTION;" => MaterializeStmtKind::TxnBegin, + "COMMIT;" => MaterializeStmtKind::TxnCommit, + _ => MaterializeStmtKind::Write, + }; + MaterializeStmt { kind, sql } + })); + // The summary read carries the per-test breakdown (when any tests apply). + stmts.push(MaterializeStmt { + kind: MaterializeStmtKind::Summary, + sql: materialize_result_sql( + &target_qualified, + asset_path, + partition_col, + partition_value_sql, + partitioned, + &test_sql.checks, + drift_summary_select, + ), + }); + Ok(MaterializePlan { stmts, checks: test_sql.checks }) +} + +/// The trailing one-row summary the materialize run returns: the asset it +/// produced, the row count of the materialized slice (the partition when +/// partitioned, else the whole table), and the DuckLake snapshot it created. +/// This is both a useful preview result and the row the worker records. +pub fn materialize_result_sql( + target_qualified: &str, + asset_path: &str, + partition_col: &str, + partition_value_sql: &str, + partitioned: bool, + checks: &[DataTestCheck], + // `on_schema_change=warn` on a persist-and-mutate strategy: the SELECT to + // diff against the (post-write) table for the `schema_drift` summary column. + // `None` ⇒ no drift column (every other mode / strategy). + drift_summary_select: Option<&str>, +) -> String { + let (count_expr, partition_sel) = if partitioned { + // Row count is the slice this run wrote (the partition); `partition` + // lets the UI label the count and scope the preview to it. + ( + format!( + "(SELECT count(*) FROM {target_qualified} WHERE {partition_col} = {partition_value_sql})" + ), + format!("{partition_value_sql} AS partition, "), + ) + } else { + ( + format!("(SELECT count(*) FROM {target_qualified})"), + String::new(), + ) + }; + // Capture the materialized output schema (gap #2a) in the same summary row — + // no extra round-trip. `DESCRIBE SELECT * FROM ` yields one row per + // column (`column_name`, `column_type`); fold them into a list-of-struct the + // worker reads back and persists as asset metadata. The write just + // committed, so the latest snapshot (no `AT (VERSION)` needed) is exactly the + // slice recorded in `snapshot_id`. + // + // Two correctness details: + // - `_wm_ord` (a `row_number()` over the DESCRIBE) is captured so the + // list-of-struct is ordered *explicitly* (`list(... ORDER BY _wm_ord)`). + // DESCRIBE returns columns in physical order; without the explicit ORDER + // the `list()` aggregate could reorder them and spuriously bump the schema + // version on a re-materialize. + // - For a `// partitioned` asset the physical table carries the synthetic + // `_wm_partition` column; it must be filtered out so the recorded schema is + // the producer's logical output, not Windmill's storage detail (this is the + // grain #2b contract enforcement reads back). + let partition_filter = if partitioned { + format!( + " WHERE column_name <> '{}'", + partition_col.replace('\'', "''") + ) + } else { + String::new() + }; + let schema_capture = format!( + "(SELECT list({{'name': column_name, 'type': column_type}} ORDER BY _wm_ord) \ + FROM (SELECT column_name, column_type, row_number() OVER () AS _wm_ord \ + FROM (DESCRIBE SELECT * FROM {target_qualified}){partition_filter})) AS output_schema" + ); + // `on_schema_change=warn`: fold the drift `{added, removed}` (or NULL) into + // the same row so the executor logs it + returns it with no extra probe. + let drift_col = match drift_summary_select { + Some(sel) => format!( + ", {}", + schema_drift_summary_field(sel, target_qualified, partition_col) + ), + None => String::new(), + }; + let base_cols = format!( + "'ducklake://{asset_path}' AS materialized, \ + {partition_sel}{count_expr} AS rows, \ + (SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id, \ + {schema_capture}{drift_col}" + ); + if checks.is_empty() { + return format!("SELECT {base_cols};"); + } + // Per-test breakdown. Each check's one-row probe `(v, s)` becomes a CTE + // (`_wm_t0`, `_wm_t1`, …); `_wm_tr` cross-joins them (all one-row, so the + // join stays one row) and the `data_tests` list-of-struct references the + // flattened columns — DuckDB rejects scalar subqueries *inside* a + // struct/list literal, hence the CTE lift. Names are single-quote-escaped. + // The result row carries the whole breakdown so the worker runs every + // test (no abort-on-first) and decides pass/fail itself. + let probe_ctes = checks + .iter() + .enumerate() + .map(|(i, c)| format!("_wm_t{i} AS ({})", c.probe)) + .collect::>() + .join(", "); + let tr_cols = checks + .iter() + .enumerate() + .map(|(i, _)| format!("_wm_t{i}.v AS c{i}, _wm_t{i}.s AS s{i}")) + .collect::>() + .join(", "); + let tr_from = checks + .iter() + .enumerate() + .map(|(i, _)| format!("_wm_t{i}")) + .collect::>() + .join(", "); + let list_items = checks + .iter() + .enumerate() + .map(|(i, c)| { + let name = c.name.replace('\'', "''"); + format!("{{'test': '{name}', 'violating': c{i}, 'sample': s{i}}}") + }) + .collect::>() + .join(", "); + format!( + "WITH {probe_ctes}, _wm_tr AS (SELECT {tr_cols} FROM {tr_from}) \ + SELECT {base_cols}, [{list_items}] AS data_tests FROM _wm_tr;" + ) +} + +// Ensure a statement ends with a single `;`. +/// Convenience macro injected as the first setup statement of a partitioned +/// materialize: `wm_partition(ts)` renders a timestamp with the SAME identity +/// format the resolver used for `{partition}` (from +/// [`PartitionSpec::time_strftime_format`]). It lets a partitioned SELECT +/// filter to the active slice with a single grain-agnostic line — +/// `WHERE wm_partition() = {partition}` — so users never hand-write a +/// `strftime` format that can drift, nor reach for `= TIMESTAMP {partition}` +/// (which only parses for daily). `None` for `dynamic` (no wall-clock bucket). +/// +/// Timezone-agnostic by construction: it formats `ts` as given, matching the +/// prior documented `strftime(ts, fmt)` idiom. When a non-UTC `tz=` is set the +/// caller is responsible for expressing `ts` in that zone (same caveat the raw +/// idiom carried), so this doesn't silently reinterpret a column's instant. +pub fn wm_partition_macro(spec: &crate::asset_parser::PartitionSpec) -> Option { + let fmt = spec.time_strftime_format()?; + // fmt is a trusted per-grain constant or the author's `format=`; escape + // single quotes defensively so the emitted literal can't break out. + Some(format!( + "CREATE OR REPLACE TEMP MACRO wm_partition(ts) AS strftime(ts, '{}')", + fmt.replace('\'', "''") + )) +} + +fn terminate(stmt: &str) -> String { + let t = stmt.trim_end(); + if t.ends_with(';') { + t.to_string() + } else { + format!("{t};") + } +} + +// --------------------------------------------------------------------------- +// Data tests (`// data_test`) +// --------------------------------------------------------------------------- +// +// A data test is the FIRST extensible annotation: the parser yields a +// `DataTest` from a known vocabulary, and this module turns each into a +// *check* — a `(name, probe)` pair whose probe counts and samples the +// violating rows — that runs against the freshly-materialized target after +// the write commits. The materialize summary query embeds every check's +// outcome in one `data_tests` column, so all tests run in a single pass (no +// abort-on-first) and the worker, not the SQL, decides pass/fail and reports +// the full per-test breakdown. +// +// The pattern is deliberately open: a verifier is just `(name, violating-rows +// query)` handed to `push_check`. Built-ins differ only in their rows query; +// the `Custom` escape hatch supplies its own (a user SELECT returning the +// violating rows). A sibling annotation family (column-lineage) can emit its +// own checks through the same `push_check` shape rather than bolting on a +// parallel mechanism. See `docs/ducklake-materialization.md`. + +use crate::asset_parser::{AssetKind, DataTest, OnSchemaChange}; + +/// Target context a data-test probe runs against — the materialized table and +/// the partition slice (when partitioned, tests are scoped to the slice just +/// written, so a rerun/backfill is independent of other partitions' data). +#[derive(Debug, Clone)] +pub struct DataTestCtx<'a> { + /// Fully-qualified materialized target, e.g. `_wm_target.orders`. + pub target_qualified: &'a str, + /// `/
` of the target, for human-readable probe messages. + pub asset_path: &'a str, + /// Physical partition column on the managed table. + pub partition_col: &'a str, + /// SQL literal/expression for the current partition value (already escaped). + pub partition_value_sql: &'a str, + /// Whether the target is partitioned (scopes probes to the slice). + pub partitioned: bool, + /// Whether the target is an SCD2 history table. Built-in probes then assert + /// the *current snapshot* (`is_current` rows): the history legitimately + /// repeats the natural key across closed versions, so an unscoped + /// `unique()` would fail on the second change of any key. + pub scd2: bool, +} + +/// A data test resolved enough to generate SQL. Built-ins carry only their +/// parsed `DataTest`; `Custom` additionally carries the fetched script body +/// (the parser crate can't fetch it — the worker does and passes it in). +#[derive(Debug, Clone)] +pub enum DataTestResolved { + BuiltIn(DataTest), + Custom { path: String, body: String }, +} + +/// One compiled data-test check: a human-readable `name` and a one-row probe +/// query yielding `(v, s)` — the violating-row count (0 = pass) and a bounded +/// `to_json` sample of the violating rows as a VARCHAR (NULL when there are +/// none, or when the serialized sample exceeds the size cap). The materialize +/// summary query embeds every check's outcome so the worker gets the whole +/// breakdown in one result — all tests run (no abort-on-first) and the +/// worker, not the SQL, decides pass/fail. The sample is decoration only: +/// enforcement reads `v`, never `s`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DataTestCheck { + pub name: String, + /// One-row subquery `SELECT … AS v, … AS s FROM (…)` counting and + /// sampling the check's violating rows in a single scan. + pub probe: String, +} + +/// Row cap on a data-test sample. Bounded so the sample stays a debugging aid +/// (the full count is still reported); no ORDER BY on the violating rows, so +/// which rows land in the sample is nondeterministic. +const SAMPLE_MAX_ROWS: usize = 20; +/// Byte cap on one serialized sample. Oversized samples are dropped entirely +/// (NULL), never truncated — truncated JSON would fail parsing downstream +/// after paying the bytes anyway. +const SAMPLE_MAX_BYTES: usize = 51_200; + +/// The SQL a set of data tests compiles to: referenced-asset `ATTACH` +/// statements (resolved by the executor's ATTACH-transform pass) and the +/// per-test checks, both in declaration order. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct DataTestChecks { + pub attaches: Vec, + pub checks: Vec, +} + +/// Alias prefix for a relationships test's referenced asset, attached +/// read-only alongside the target. `_wm_ref_` so it never collides with the +/// user's aliases or the reserved `_wm_target`. +const REF_ALIAS_PREFIX: &str = "_wm_ref_"; + +// Double-quote a SQL identifier, escaping embedded quotes — so an arbitrary +// column name from an annotation can't break out of the identifier. +fn quote_ident(id: &str) -> String { + format!("\"{}\"", id.replace('"', "\"\"")) +} + +// Quote a possibly schema-qualified table reference (`schema.table`) by quoting +// each dotted segment independently: `main.dim_products` → `"main"."dim_products"`. +// Quoting the whole thing would make DuckDB read it as one table name containing +// a literal dot, querying the wrong table. +fn quote_qualified(name: &str) -> String { + name.split('.') + .map(quote_ident) + .collect::>() + .join(".") +} + +// Single-quote a SQL string literal, escaping embedded quotes. +fn quote_lit(s: &str) -> String { + format!("'{}'", s.replace('\'', "''")) +} + +// The `WHERE`/`AND` fragment scoping a probe to the current partition and/or +// the SCD2 current snapshot, or empty when neither applies. `prefix` is +// `WHERE ` or `AND ` per call site; further conditions chain with `AND`. +// (`partitioned` and `scd2` are mutually exclusive today — the combo is +// rejected at codegen — but the chaining keeps this correct if that changes.) +fn partition_scope(ctx: &DataTestCtx, prefix: &str, table_alias: Option<&str>) -> String { + let qualify = |col: String| match table_alias { + Some(a) => format!("{a}.{col}"), + None => col, + }; + let mut conds: Vec = Vec::new(); + if ctx.partitioned { + conds.push(format!( + "{} = {}", + qualify(quote_ident(ctx.partition_col)), + ctx.partition_value_sql + )); + } + if ctx.scd2 { + conds.push(qualify(SCD2_IS_CURRENT.to_string())); + } + if conds.is_empty() { + return String::new(); + } + format!("{prefix}{}", conds.join(" AND ")) +} + +// Record one check: its display `name` plus `rows_query`, the SELECT of its +// violating rows. The probe counts and samples those rows in one scan: +// `_wm_v` (the subquery alias) referenced as a column is the whole row as a +// struct; `to_json(...)::VARCHAR` keeps the sample a JSON *string* through +// the FFI — expanded rows would be visible to the executor's key-recursive +// `extract_i64(result, "rows"/"snapshot_id")` scans, which a user column of +// the same name could corrupt. `list()` over zero rows and an over-cap +// sample both degrade to NULL (`s` is optional by contract). +fn push_check(out: &mut DataTestChecks, name: String, rows_query: String) { + // `strlen` counts bytes (unlike `length`, characters) — the cap bounds + // payload size on the wire, so bytes are the right unit. + let probe = format!( + "SELECT v, CASE WHEN strlen(s_raw) <= {SAMPLE_MAX_BYTES} THEN s_raw END AS s \ + FROM (SELECT count(*) AS v, \ + to_json(list(_wm_v ORDER BY _wm_rn) FILTER (WHERE _wm_rn <= {SAMPLE_MAX_ROWS}))::VARCHAR AS s_raw \ + FROM (SELECT _wm_v, row_number() OVER () AS _wm_rn FROM ({rows_query}) _wm_v))" + ); + out.checks.push(DataTestCheck { name, probe }); +} + +// `SELECT *` for a sample rows-query, excluding the synthetic physical +// partition column on partitioned targets — it's Windmill's storage detail, +// not part of the producer's logical output (same rule as schema capture). +// `qualifier` scopes the star when the query aliases the target (`_wm_src`). +fn sample_star(ctx: &DataTestCtx, qualifier: Option<&str>) -> String { + let star = match qualifier { + Some(q) => format!("{q}.*"), + None => "*".to_string(), + }; + if ctx.partitioned { + format!("{star} EXCLUDE ({})", quote_ident(ctx.partition_col)) + } else { + star + } +} + +// Self-teaching tail appended to every malformed-custom-test error. It states +// the two rules that aren't documented or scaffolded anywhere else — the body +// is a single SELECT, and it reads the freshly-materialized target through the +// internal `_wm_target.
` alias — and doubles that alias into a copyable +// one-line example. `target_qualified` is already `_wm_target.
`. +fn custom_test_hint(target_qualified: &str) -> String { + format!( + "Write a single SELECT against `{target_qualified}` returning the offending rows, e.g. \ + `SELECT * FROM {target_qualified} WHERE ` — an empty result means the test \ + passes." + ) +} + +// Whether a custom-test statement reads the materialized target through the +// reserved `_wm_target` alias (the only handle the runtime attaches it under). +// SQL identifiers are case-insensitive, so match case-insensitively; +// `split_statements` has already stripped comments, so a match here is a real +// reference, not one buried in a comment. `TARGET_ALIAS` is lowercase. +fn references_target(stmt: &str) -> bool { + stmt.to_lowercase().contains(TARGET_ALIAS) +} + +/// Compile resolved data tests into ATTACH statements + per-test checks for +/// `ctx`'s target. Pure: returns SQL text, executes nothing. Errors carry an +/// actionable message (e.g. a relationships target that isn't an attachable +/// table). +pub fn build_data_test_checks( + tests: &[DataTestResolved], + ctx: &DataTestCtx, +) -> Result { + let t = ctx.target_qualified; + let mut out = DataTestChecks::default(); + // Dedup ref attaches by (kind, name): a database can't be attached twice, + // so multiple relationships into the same db share one alias. + let mut ref_aliases: Vec<(AssetKind, String, String)> = Vec::new(); + + for resolved in tests { + match resolved { + DataTestResolved::BuiltIn(DataTest::Unique { column }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + // The rows are the GROUP BY result — one `{value, count}` per + // duplicated key — so the count (number of duplicated values, + // not of rows) and the sample share one grain and can't + // contradict each other in the UI. + let q = format!( + "SELECT {c} AS \"value\", count(*) AS \"count\" FROM {t} \ + WHERE {c} IS NOT NULL{scope} GROUP BY {c} HAVING count(*) > 1" + ); + push_check(&mut out, format!("unique({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::NotNull { column }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let star = sample_star(ctx, None); + let q = format!("SELECT {star} FROM {t} WHERE {c} IS NULL{scope}"); + push_check(&mut out, format!("not_null({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::AcceptedValues { column, values }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let star = sample_star(ctx, None); + let list = values + .iter() + .map(|v| quote_lit(v)) + .collect::>() + .join(", "); + let q = format!( + "SELECT {star} FROM {t} WHERE {c} IS NOT NULL AND {c} NOT IN ({list}){scope}" + ); + push_check(&mut out, format!("accepted_values({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::Relationships { + column, + to_kind, + to_path, + to_column, + }) => { + let (ref_name, ref_table) = to_path.split_once('/').ok_or_else(|| { + format!("data_test relationships: target `{to_path}` must be `/
`") + })?; + if ref_table.is_empty() { + return Err(format!( + "data_test relationships: target `{to_path}` has no table" + )); + } + let scheme = match to_kind { + AssetKind::Ducklake => "ducklake", + AssetKind::DataTable => "datatable", + other => { + return Err(format!( + "data_test relationships: target kind {other:?} is not an attachable \ + table (use ducklake:// or datatable://)" + )) + } + }; + // The materialize target's ducklake is already attached as + // `_wm_target`; a reference into that same lake must reuse it + // rather than ATTACH the same database again under a fresh alias + // (DuckDB forbids attaching one database twice). `asset_path` is + // the target's `/
`, so its lake is the part before + // the first `/`. + let target_lake = ctx.asset_path.split('/').next().unwrap_or(""); + let alias = if *to_kind == AssetKind::Ducklake && ref_name == target_lake { + TARGET_ALIAS.to_string() + } else { + // Reuse an existing alias for the same (kind, name), else mint one. + match ref_aliases + .iter() + .find(|(k, n, _)| k == to_kind && n == ref_name) + { + Some((_, _, a)) => a.clone(), + None => { + let a = format!("{REF_ALIAS_PREFIX}{}", ref_aliases.len()); + // Escape the name — it is interpolated into a + // single-quoted DuckDB literal (defense-in-depth: the + // name is deploy-time annotation content, but the parser + // places no character restriction on asset paths). + let esc_name = ref_name.replace('\'', "''"); + out.attaches + .push(format!("ATTACH '{scheme}://{esc_name}' AS {a};")); + ref_aliases.push((*to_kind, ref_name.to_string(), a.clone())); + a + } + } + }; + let c = quote_ident(column); + let rc = quote_ident(to_column); + // `ref_table` may be schema-qualified (`schema.table`); quote each + // segment so the dot stays a schema separator, not a literal. + let rt = quote_qualified(ref_table); + let scope = partition_scope(ctx, " AND ", Some("_wm_src")); + let star = sample_star(ctx, Some("_wm_src")); + let q = format!( + "SELECT {star} FROM {t} _wm_src \ + WHERE _wm_src.{c} IS NOT NULL{scope} \ + AND NOT EXISTS (SELECT 1 FROM {alias}.{rt} _wm_ref \ + WHERE _wm_ref.{rc} = _wm_src.{c})" + ); + push_check( + &mut out, + format!("relationships({column} -> {to_path}.{to_column})"), + q, + ); + } + // A parsed Custom must be resolved (body fetched) before codegen. + DataTestResolved::BuiltIn(DataTest::Custom { path }) => { + return Err(format!( + "data_test custom `{path}`: body not resolved before codegen (internal)" + )); + } + DataTestResolved::Custom { path, body } => { + // dbt singular-test convention: the body is a *single* SELECT + // (or CTE) returning the violating rows, reading the + // freshly-materialized target through the internal `_wm_target` + // schema. It is embedded as a subquery (`FROM ()`), so a + // multi-statement or non-SELECT body would produce invalid SQL. + // Neither rule is documented or scaffolded elsewhere, so the + // errors are self-teaching: they name the exact violation and + // append a correct one-line example. It runs in the target's + // connection (can read `_wm_target` + the user's attaches); + // partition substitution is already applied by the worker. + let hint = custom_test_hint(t); + let stmts = split_statements(body); + if stmts.is_empty() { + return Err(format!( + "data_test custom `{path}`: empty test body. {hint}" + )); + } + if stmts.len() > 1 { + return Err(format!( + "data_test custom `{path}`: a custom data test must be a single SELECT, \ + but found {} statements. {hint}", + stmts.len() + )); + } + let stmt = &stmts[0]; + if classify_block(stmt) != BlockClass::Output { + return Err(format!( + "data_test custom `{path}`: a custom data test must be a single SELECT, \ + not a write or DDL statement. {hint}" + )); + } + if !references_target(stmt) { + return Err(format!( + "data_test custom `{path}`: the test never reads the freshly-materialized \ + target — reference it through the internal `{TARGET_ALIAS}` schema (as \ + `{t}`), not the table name on its own. {hint}" + )); + } + push_check(&mut out, format!("custom({path})"), stmt.to_string()); + } + } + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn ok(sql: &str) -> WrapPlan { + classify_wrap(sql).expect("expected wrap-eligible") + } + fn err(sql: &str) -> WrapError { + classify_wrap(sql).expect_err("expected wrap-ineligible") + } + + use crate::asset_parser::{PartitionKind, PartitionSpec}; + + fn pspec(kind: PartitionKind, format: Option<&str>) -> PartitionSpec { + PartitionSpec { kind, tz: None, format: format.map(String::from), start: None } + } + + #[test] + fn wm_partition_macro_uses_grain_identity_format() { + // Every time grain's macro must strftime with the exact format the + // resolver stamps the identity with — otherwise the equality filter + // silently returns no rows. This is the whole point of the shared source. + let cases = [ + (PartitionKind::Daily, "%Y-%m-%d"), + (PartitionKind::Hourly, "%Y-%m-%dT%H"), + (PartitionKind::Weekly, "%G-W%V"), + (PartitionKind::Monthly, "%Y-%m"), + ]; + for (kind, fmt) in cases { + assert_eq!( + wm_partition_macro(&pspec(kind.clone(), None)).as_deref(), + Some( + format!( + "CREATE OR REPLACE TEMP MACRO wm_partition(ts) AS strftime(ts, '{fmt}')" + ) + .as_str() + ), + "wrong macro format for {kind:?}" + ); + } + } + + #[test] + fn wm_partition_macro_honors_format_override_and_skips_dynamic() { + // Explicit `format=` wins over the grain default. + assert!( + wm_partition_macro(&pspec(PartitionKind::Hourly, Some("%Y/%m/%d %H"))) + .unwrap() + .contains("strftime(ts, '%Y/%m/%d %H')") + ); + // Dynamic has no wall-clock bucket → no macro (user filters on their key). + assert_eq!( + wm_partition_macro(&pspec( + PartitionKind::Dynamic { key: "$.tenant".into() }, + None + )), + None + ); + } + + #[test] + fn split_respects_strings_comments_idents() { + let sql = "SET x=1; -- a; comment\nSELECT ';' AS a, \"weird;col\" /* ; */ FROM t;"; + let s = split_statements(sql); + assert_eq!(s.len(), 2); + assert_eq!(s[0], "SET x=1"); + assert!(s[1].starts_with("SELECT")); + assert!(s[1].contains("\"weird;col\"")); + } + + #[test] + fn split_handles_escaped_quote() { + let s = split_statements("SELECT 'it''s; fine' AS a;"); + assert_eq!(s.len(), 1); + assert!(s[0].contains("it''s; fine")); + } + + #[test] + fn pipeline_annotations_are_stripped() { + // The real shape: `//` annotation lines above the SQL must not pollute + // the first block's classification (regression — they were being read + // as a leading `pipeline` keyword and rejected). + let p = ok("// pipeline\n// materialize ducklake://main/t\n// partitioned daily\nATTACH 'ducklake://main' AS dl;\nSELECT 1 AS id"); + assert_eq!(p.setup.len(), 1); + // The annotation lines are gone — the setup block starts at the real + // SQL (the `//` inside `ducklake://main` is legitimately retained). + assert!(p.setup[0].starts_with("ATTACH")); + assert!(p.output.starts_with("SELECT")); + } + + #[test] + fn bare_select_is_eligible() { + let p = ok("SELECT a, b FROM t WHERE c = '{partition}'"); + assert!(p.setup.is_empty()); + assert!(p.output.starts_with("SELECT")); + } + + #[test] + fn setup_then_select_is_eligible() { + let p = ok( + "ATTACH 'ducklake://main' AS dl;\n SET memory_limit='4GB';\n SELECT * FROM dl.orders", + ); + assert_eq!(p.setup.len(), 2); + assert!(p.output.starts_with("SELECT")); + } + + #[test] + fn create_temp_staging_is_setup() { + let p = ok("CREATE TEMP TABLE s AS SELECT 1; SELECT * FROM s"); + assert_eq!(p.setup.len(), 1); + assert_eq!( + classify_block("CREATE TEMP TABLE s AS SELECT 1"), + BlockClass::Setup + ); + assert_eq!( + classify_block("CREATE OR REPLACE TEMPORARY VIEW v AS SELECT 1"), + BlockClass::Setup + ); + } + + #[test] + fn with_cte_select_is_output_write_is_disallowed() { + assert_eq!( + classify_block("WITH x AS (SELECT 1) SELECT * FROM x"), + BlockClass::Output + ); + // CTE whose main statement inserts is a write, even though it starts WITH. + assert_eq!( + classify_block("WITH x AS (SELECT 1) INSERT INTO t SELECT * FROM x"), + BlockClass::Disallowed + ); + } + + #[test] + fn from_first_and_values_are_output() { + assert_eq!(classify_block("FROM t SELECT a"), BlockClass::Output); + assert_eq!(classify_block("VALUES (1),(2)"), BlockClass::Output); + assert_eq!(classify_block("TABLE t"), BlockClass::Output); + } + + #[test] + fn trailing_write_rejected() { + assert_eq!( + err("SELECT * FROM t; INSERT INTO u VALUES (1)"), + WrapError::OutputNotLast + ); + } + + #[test] + fn write_in_preamble_rejected() { + match err("INSERT INTO t VALUES (1); SELECT * FROM t") { + WrapError::DisallowedBlock { snippet } => assert!(snippet.starts_with("INSERT")), + e => panic!("wrong error: {e:?}"), + } + } + + #[test] + fn multiple_selects_rejected() { + assert_eq!( + err("SELECT 1; SELECT 2"), + WrapError::MultipleOutputs { count: 2 } + ); + } + + #[test] + fn no_select_and_empty_rejected() { + assert_eq!(err("CREATE TABLE t (a INT)"), WrapError::NoOutput); + assert_eq!(err(" -- just a comment\n"), WrapError::Empty); + } + + #[test] + fn use_cannot_redirect_is_classified_setup() { + // `USE` is allowed setup; generated SQL is fully qualified regardless. + assert_eq!(classify_block("USE dl"), BlockClass::Setup); + } + + #[test] + fn codegen_replace_partitioned() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.orders_daily", + select_sql: "SELECT a FROM dl.orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + strategy: MaterializeStrategy::Replace, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + assert!(st[0].contains("CREATE TABLE IF NOT EXISTS _wm_target.orders_daily")); + assert!(st[0].contains("CAST(NULL AS VARCHAR) AS _wm_partition")); + assert!(st.iter().any( + |s| s == "ALTER TABLE _wm_target.orders_daily SET PARTITIONED BY (_wm_partition);" + )); + assert!(st.iter().any(|s| s.starts_with( + "DELETE FROM _wm_target.orders_daily WHERE _wm_partition = '2026-06-19'" + ))); + assert!(st.iter().any(|s| s.contains( + "INSERT INTO _wm_target.orders_daily SELECT *, '2026-06-19' AS _wm_partition" + ))); + assert_eq!(st.first().map(|_| &st[st.len() - 1]).unwrap(), "COMMIT;"); + } + + #[test] + fn codegen_merge_is_delete_by_key_plus_insert() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.orders_daily", + select_sql: "SELECT order_id, amount FROM dl.orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + strategy: MaterializeStrategy::Merge { unique_key: "order_id".to_string() }, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + // upsert = delete-by-key (partition-scoped) + insert — NO `MERGE INTO` + // (DuckLake's MERGE fails on fresh partitions). + assert!(!st.iter().any(|s| s.contains("MERGE INTO"))); + let del = st + .iter() + .find(|s| s.starts_with("DELETE FROM")) + .expect("delete stmt"); + assert!(del.contains( + "WHERE _wm_partition = '2026-06-19' AND order_id IN (SELECT order_id FROM (SELECT order_id, amount FROM dl.orders))" + )); + assert!(st + .iter() + .any(|s| s.starts_with("INSERT INTO _wm_target.orders_daily SELECT *, '2026-06-19'"))); + } + + #[test] + fn codegen_merge_emits_duplicate_source_key_guard() { + // A keyed merge does not dedup its source, so codegen must emit a guard + // that fails the write when the SELECT has >1 row per key — otherwise + // duplicate source keys silently double-write. + let cg = MaterializeCodegen { + target_qualified: "_wm_target.orders_daily", + select_sql: "SELECT order_id, amount FROM dl.orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: false, + strategy: MaterializeStrategy::Merge { unique_key: "order_id".to_string() }, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + let guard = st + .iter() + .find(|s| s.contains("_wm_dup_keys")) + .expect("duplicate-key guard stmt"); + // raises via error(), counts non-NULL keys appearing more than once + assert!(guard.contains("error('managed materialize: keyed merge on `order_id` blocked")); + assert!(guard.contains("GROUP BY order_id HAVING count(*) > 1")); + assert!(guard.contains("WHERE order_id IS NOT NULL")); + // must run before the mutations so a violation aborts before any write + let guard_pos = st.iter().position(|s| s.contains("_wm_dup_keys")).unwrap(); + let del_pos = st + .iter() + .position(|s| s.starts_with("DELETE FROM")) + .unwrap(); + let ins_pos = st + .iter() + .position(|s| s.starts_with("INSERT INTO")) + .unwrap(); + assert!(guard_pos < del_pos && guard_pos < ins_pos); + } + + #[test] + fn codegen_append_inserts_only() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.events", + select_sql: "SELECT * FROM dl.raw", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + strategy: MaterializeStrategy::Append, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + assert!(st + .iter() + .any(|s| s.starts_with("INSERT INTO _wm_target.events"))); + assert!(!st.iter().any(|s| s.starts_with("DELETE"))); + assert!(!st.iter().any(|s| s.starts_with("MERGE"))); + } + + #[test] + fn codegen_whole_table_replace_is_create_or_replace() { + // Unpartitioned replace must use CREATE OR REPLACE so a changed SELECT + // schema (or a pre-existing table with a different schema) doesn't break + // — and nothing else (no bootstrap / DELETE / INSERT / txn). + let cg = MaterializeCodegen { + target_qualified: "_wm_target.customer_dim", + select_sql: "SELECT a, b, c FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy: MaterializeStrategy::Replace, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + assert_eq!( + st, + vec![ + "CREATE OR REPLACE TABLE _wm_target.customer_dim AS SELECT * FROM (SELECT a, b, c FROM dl.src);" + .to_string() + ] + ); + } + + #[test] + fn codegen_scd2_default_track_closes_old_opens_new() { + // Empty `track` ⇒ diff on all business columns via `* EXCLUDE (scd cols)`. + let cg = MaterializeCodegen { + target_qualified: "_wm_target.dim_scd2", + select_sql: "SELECT id, name FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy: MaterializeStrategy::Scd2 { + key: "id".to_string(), + track: vec![], + close_deleted: false, + }, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + // bootstrap adds the three SCD metadata columns + assert!(st[0].starts_with("CREATE TABLE IF NOT EXISTS _wm_target.dim_scd2 AS SELECT *,")); + assert!(st[0].contains("AS valid_from")); + assert!(st[0].contains("AS valid_to")); + assert!(st[0].contains("AS is_current")); + // changed-key set captured before the transaction, all cols compared + assert!( + st[1].contains("CREATE OR REPLACE TEMP TABLE _wm_scd2_changed AS SELECT \"id\" FROM") + ); + assert!(st[1].contains("SELECT * FROM (SELECT id, name FROM dl.src) EXCEPT")); + assert!(st[1].contains("SELECT * EXCLUDE (valid_from, valid_to, is_current) FROM _wm_target.dim_scd2 WHERE is_current")); + assert_eq!(st[2], "BEGIN TRANSACTION;"); + // close: UPDATE (not MERGE) the prior open version of changed keys, with + // null-safe key matching (IS NOT DISTINCT FROM, not IN — IN drops NULLs) + assert!(st[3].starts_with("UPDATE _wm_target.dim_scd2 SET valid_to = CAST(now() AS TIMESTAMP), is_current = false")); + assert!(st[3].contains( + "WHERE is_current AND EXISTS (SELECT 1 FROM _wm_scd2_changed \ + WHERE _wm_scd2_changed.\"id\" IS NOT DISTINCT FROM _wm_target.dim_scd2.\"id\");" + )); + // open: INSERT the new current version, null-safe key matching + assert!(st[4].starts_with( + "INSERT INTO _wm_target.dim_scd2 SELECT s.*, CAST(now() AS TIMESTAMP) AS valid_from" + )); + assert!(st[4].contains( + "true AS is_current FROM (SELECT id, name FROM dl.src) s WHERE EXISTS \ + (SELECT 1 FROM _wm_scd2_changed c WHERE c.\"id\" IS NOT DISTINCT FROM s.\"id\");" + )); + // consumer-convenience `_current` view: `IF NOT EXISTS` (created once, + // no-op on unchanged reruns) and INSIDE the txn (folded into the write snapshot) + assert_eq!( + st[5], + "CREATE VIEW IF NOT EXISTS _wm_target.dim_scd2_current AS SELECT * FROM _wm_target.dim_scd2 WHERE is_current;" + ); + assert_eq!(st[6], "COMMIT;"); + // no fragile constructs: no MERGE INTO, and no NULL-dropping `IN (SELECT` + assert!(!st.iter().any(|s| s.contains("MERGE INTO"))); + assert!(!st.iter().any(|s| s.contains("IN (SELECT"))); + // soft-delete default: no deleted-key set, no second close + assert!(!st.iter().any(|s| s.contains("_wm_scd2_deleted"))); + } + + #[test] + fn codegen_scd2_explicit_track_projects_key_and_tracked_cols() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.dim", + select_sql: "SELECT id, name, addr FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy: MaterializeStrategy::Scd2 { + key: "id".to_string(), + track: vec!["name".to_string()], + close_deleted: false, + }, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + // only key + tracked cols are compared (addr changes don't rotate a version) + assert!(st[1].contains("SELECT \"id\", \"name\" FROM (SELECT id, name, addr FROM dl.src) EXCEPT SELECT \"id\", \"name\" FROM _wm_target.dim WHERE is_current")); + } + + #[test] + fn codegen_scd2_close_deleted_adds_deleted_set_and_second_close() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.dim", + select_sql: "SELECT id, name FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy: MaterializeStrategy::Scd2 { + key: "id".to_string(), + track: vec![], + close_deleted: true, + }, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + // the deleted-key set: current keys absent from the snapshot, captured + // before the transaction (like `changed`) + assert!(st.iter().any(|s| s.contains( + "CREATE OR REPLACE TEMP TABLE _wm_scd2_deleted AS SELECT \"id\" FROM \ + (SELECT \"id\" FROM _wm_target.dim WHERE is_current EXCEPT SELECT \"id\" FROM (SELECT id, name FROM dl.src));" + ))); + // a second close UPDATE against the deleted set (null-safe), and NO INSERT + // that reopens deleted keys (the only INSERT filters on `_wm_scd2_changed`) + assert!(st + .iter() + .any(|s| s.starts_with("UPDATE _wm_target.dim SET valid_to") + && s.contains( + "EXISTS (SELECT 1 FROM _wm_scd2_deleted \ + WHERE _wm_scd2_deleted.\"id\" IS NOT DISTINCT FROM _wm_target.dim.\"id\");" + ))); + assert_eq!( + st.iter().filter(|s| s.starts_with("INSERT INTO")).count(), + 1 + ); + assert!(st + .iter() + .find(|s| s.starts_with("INSERT INTO")) + .unwrap() + .contains("_wm_scd2_changed")); + // the deleted close is inside the transaction (between BEGIN and COMMIT) + let begin = st.iter().position(|s| s == "BEGIN TRANSACTION;").unwrap(); + let commit = st.iter().position(|s| s == "COMMIT;").unwrap(); + let del_close = st + .iter() + .position(|s| s.starts_with("UPDATE") && s.contains("_wm_scd2_deleted")) + .unwrap(); + assert!(begin < del_close && del_close < commit); + } + + fn persist_cg(strategy: MaterializeStrategy, osc: OnSchemaChange) -> Vec { + MaterializeCodegen { + target_qualified: "_wm_target.t", + select_sql: "SELECT a, c FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy, + on_schema_change: osc, + } + .statements() + } + + #[test] + fn codegen_fail_emits_drift_guard_inside_txn_before_write() { + let st = persist_cg( + MaterializeStrategy::Merge { unique_key: "a".into() }, + OnSchemaChange::Fail, + ); + let begin = st.iter().position(|s| s == "BEGIN TRANSACTION;").unwrap(); + let guard = st + .iter() + .position(|s| s.contains("error(") && s.contains("on_schema_change=fail")) + .expect("fail emits a guard"); + let del = st + .iter() + .position(|s| s.starts_with("DELETE FROM")) + .unwrap(); + let insert = st + .iter() + .position(|s| s.starts_with("INSERT INTO")) + .unwrap(); + let commit = st.iter().position(|s| s == "COMMIT;").unwrap(); + // guard runs right after BEGIN and before any mutation, inside the txn + assert!(begin < guard && guard < del && del < insert && insert < commit); + // both CASE branches are VARCHAR so the planner can't fold error() away + assert!(st[guard].contains("CAST(error(")); + assert!(st[guard].contains("ELSE 'ok' END")); + // fail is positional (no BY NAME) and emits no sync sentinel + assert!(st[insert].starts_with("INSERT INTO _wm_target.t SELECT")); + assert!(!st.iter().any(|s| s == SYNC_ALTER_SENTINEL)); + } + + #[test] + fn fail_guard_drift_is_name_set_based_not_ordered() { + // Documents a deliberate boundary: the fail guard fires on the column + // SET difference (EXCEPT over column_name), so a pure reorder of + // same-named columns is NOT caught here — that is `sync`'s job (BY NAME). + // Keeping this name-set (not an ordered comparison) is what prevents the + // guard from false-positive-aborting a correctly-aligned write. If this + // ever moves to an ordered comparison, update the OnSchemaChange doc. + let guard = + schema_drift_guard_sql("SELECT b, a FROM dl.src", "_wm_target.t", "_wm_partition"); + assert!(guard.contains("column_name")); + assert!(guard.contains("EXCEPT")); + // No positional/ordinal comparison in the guard condition. + assert!(!guard.to_lowercase().contains("ordinal")); + assert!(!guard.to_lowercase().contains("row_number")); + } + + #[test] + fn codegen_sync_uses_by_name_and_sentinel() { + let st = persist_cg(MaterializeStrategy::Append, OnSchemaChange::Sync); + let begin = st.iter().position(|s| s == "BEGIN TRANSACTION;").unwrap(); + let sentinel = st.iter().position(|s| s == SYNC_ALTER_SENTINEL).unwrap(); + let insert = st + .iter() + .position(|s| s.starts_with("INSERT INTO")) + .unwrap(); + // the ALTER-injection slot is right after BEGIN, before the write + assert!(begin < sentinel && sentinel < insert); + // name-mapped insert (a positional insert would cross-wire ALTERed cols) + assert!(st[insert].starts_with("INSERT INTO _wm_target.t BY NAME SELECT")); + assert!(!st.iter().any(|s| s.contains("error("))); + } + + #[test] + fn codegen_ignore_and_warn_write_positionally_no_guard() { + for osc in [OnSchemaChange::Ignore, OnSchemaChange::Warn] { + let st = persist_cg(MaterializeStrategy::Append, osc); + assert!(!st.iter().any(|s| s.contains("error(")), "{osc:?}"); + assert!(!st.iter().any(|s| s == SYNC_ALTER_SENTINEL), "{osc:?}"); + let insert = st.iter().find(|s| s.starts_with("INSERT INTO")).unwrap(); + assert!( + insert.starts_with("INSERT INTO _wm_target.t SELECT"), + "{osc:?}" + ); + } + } + + #[test] + fn codegen_whole_table_replace_and_scd2_ignore_guardrail() { + // Whole-table replace (unpartitioned) and scd2 are not persist-and-mutate: + // fail/sync must not add a guard/sentinel/BY NAME there. + let repl = MaterializeCodegen { + target_qualified: "_wm_target.t", + select_sql: "SELECT a FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy: MaterializeStrategy::Replace, + on_schema_change: OnSchemaChange::Fail, + }; + assert!(!repl.is_persist_and_mutate()); + let st = repl.statements(); + assert!(!st + .iter() + .any(|s| s.contains("error(") || s == SYNC_ALTER_SENTINEL)); + + let scd2 = MaterializeCodegen { + strategy: MaterializeStrategy::Scd2 { + key: "a".into(), + track: vec![], + close_deleted: false, + }, + on_schema_change: OnSchemaChange::Sync, + ..repl + }; + assert!(!scd2.is_persist_and_mutate()); + let st = scd2.statements(); + assert!(!st.iter().any(|s| s == SYNC_ALTER_SENTINEL)); + assert!(!st.iter().any(|s| s.contains(" BY NAME "))); + } + + #[test] + fn summary_schema_drift_field_only_for_warn_persist_and_mutate() { + // warn + persist-and-mutate ⇒ summary carries the drift column + let warn = plan_for_osc(MaterializeStrategy::Append, false, OnSchemaChange::Warn); + assert!(warn.stmts.last().unwrap().sql.contains("AS schema_drift")); + // ignore / fail / sync ⇒ no summary drift column (they guard the write) + for osc in [ + OnSchemaChange::Ignore, + OnSchemaChange::Fail, + OnSchemaChange::Sync, + ] { + let p = plan_for_osc(MaterializeStrategy::Append, false, osc); + assert!( + !p.stmts.last().unwrap().sql.contains("schema_drift"), + "{osc:?} must not emit the summary drift column" + ); + } + // whole-table replace + warn ⇒ not persist-and-mutate ⇒ no drift column + let repl = plan_for_osc(MaterializeStrategy::Replace, false, OnSchemaChange::Warn); + assert!(!repl.stmts.last().unwrap().sql.contains("schema_drift")); + } + + #[test] + fn fail_guard_is_write_kind_between_txn_markers() { + use MaterializeStmtKind::*; + let plan = plan_for_osc( + MaterializeStrategy::Merge { unique_key: "a".into() }, + false, + OnSchemaChange::Fail, + ); + let begin = kidx(&plan, |s| s.kind == TxnBegin); + let commit = kidx(&plan, |s| s.kind == TxnCommit); + let guard = kidx(&plan, |s| { + s.sql.contains("error(") && s.sql.contains("on_schema_change=fail") + }); + assert_eq!(plan.stmts[guard].kind, Write); + assert!(begin < guard && guard < commit); + } + + #[test] + fn snapshot_capture_targets_alias() { + assert_eq!( + snapshot_capture_sql("_wm_target"), + "SELECT max(snapshot_id) AS snapshot_id FROM ducklake_snapshots('_wm_target');" + ); + } + + #[test] + fn build_wrap_blocks_orders_setup_attach_codegen_snapshot() { + let plan = ok("ATTACH 'ducklake://main' AS dl;\n SELECT a FROM dl.orders WHERE d = '{p}'"); + let blocks: Vec = build_wrap_blocks( + &plan, + "ATTACH 'ducklake:postgres:…' AS _wm_target (DATA_PATH 's3://b/p');", + "orders_daily", + "main/orders_daily", + "_wm_partition", + "'2026-06-19'", + true, + MaterializeStrategy::Replace, + OnSchemaChange::Warn, + &[], + ) + .unwrap() + .stmts + .into_iter() + .map(|s| s.sql) + .collect(); + // setup block first, then the target ATTACH, then codegen, then result. + assert!(blocks[0].starts_with("ATTACH 'ducklake://main' AS dl")); + // every setup block must be `;`-terminated so re-splitting can't merge it + // with the synthetic target ATTACH that follows. + assert!(blocks[0].ends_with(';')); + assert_eq!( + blocks[1], + "ATTACH 'ducklake:postgres:…' AS _wm_target (DATA_PATH 's3://b/p');" + ); + assert!(blocks.iter().any(|b| b.contains("_wm_target.orders_daily"))); + assert!(blocks.iter().any(|b| b.starts_with( + "DELETE FROM _wm_target.orders_daily WHERE _wm_partition = '2026-06-19'" + ))); + // the trailing block is the one-row summary (asset / rows / snapshot_id), + // partition-scoped for the row count + let last = blocks.last().unwrap(); + assert!(last.contains("'ducklake://main/orders_daily' AS materialized")); + assert!(last.contains("'2026-06-19' AS partition")); + assert!(last.contains("WHERE _wm_partition = '2026-06-19') AS rows")); + assert!(last.contains("ducklake_snapshots('_wm_target')")); + } + + // -- materialize plan structure ------------------------------------------ + + fn plan_for(strategy: MaterializeStrategy, partitioned: bool) -> MaterializePlan { + plan_for_osc(strategy, partitioned, OnSchemaChange::Warn) + } + + fn plan_for_osc( + strategy: MaterializeStrategy, + partitioned: bool, + on_schema_change: OnSchemaChange, + ) -> MaterializePlan { + let plan = ok("SELECT a, b FROM src"); + build_wrap_blocks( + &plan, + "ATTACH 'ducklake:…' AS _wm_target;", + "orders", + "main/orders", + "_wm_partition", + "'2026-06-19'", + partitioned, + strategy, + on_schema_change, + &[ + DataTestResolved::BuiltIn(DataTest::NotNull { column: "a".into() }), + DataTestResolved::BuiltIn(DataTest::Unique { column: "b".into() }), + ], + ) + .unwrap() + } + + fn kidx(plan: &MaterializePlan, pred: impl Fn(&MaterializeStmt) -> bool) -> usize { + plan.stmts + .iter() + .position(|s| pred(s)) + .expect("stmt present") + } + + #[test] + fn plan_tags_structure_and_carries_checks() { + use MaterializeStmtKind::*; + let plan = plan_for(MaterializeStrategy::Replace, true); + // leading statements are Setup, ending with the target ATTACH + assert!(plan.stmts[0].kind == Setup); + assert!(plan + .stmts + .iter() + .take_while(|s| s.kind == Setup) + .any(|s| s.sql.contains("_wm_target"))); + // txn markers are tagged, everything between them is Write + let begin = kidx(&plan, |s| s.kind == TxnBegin); + let commit = kidx(&plan, |s| s.kind == TxnCommit); + assert!(begin < commit); + assert!(plan.stmts[begin + 1..commit] + .iter() + .all(|s| s.kind == Write)); + // bootstrap DDL is Write work (it targets the table, not the session) + let bootstrap = kidx(&plan, |s| s.sql.starts_with("CREATE TABLE IF NOT EXISTS")); + assert_eq!(plan.stmts[bootstrap].kind, Write); + // summary is last and carries the breakdown; checks ride along + let last = plan.stmts.last().unwrap(); + assert_eq!(last.kind, Summary); + assert!(last.sql.contains("AS data_tests")); + assert_eq!(plan.checks.len(), 2); + assert!(plan.checks[0].name.contains("not_null(a)")); + } + + #[test] + fn plan_whole_table_replace_has_no_txn_markers() { + use MaterializeStmtKind::*; + let plan = plan_for(MaterializeStrategy::Replace, false); + assert!(!plan.stmts.iter().any(|s| s.kind == TxnBegin)); + assert!(!plan.stmts.iter().any(|s| s.kind == TxnCommit)); + assert_eq!( + plan.stmts.iter().filter(|s| s.kind == Write).count(), + 1, + "single atomic CREATE OR REPLACE" + ); + } + + #[test] + fn plan_scd2_captures_are_write_kind() { + use MaterializeStmtKind::*; + let plan = plan_for( + MaterializeStrategy::Scd2 { key: "a".into(), track: vec![], close_deleted: false }, + false, + ); + let capture = kidx(&plan, |s| s.sql.contains("TEMP TABLE _wm_scd2_changed")); + assert_eq!(plan.stmts[capture].kind, Write); + // no test declared ⇒ empty checks + let plain = ok("SELECT a FROM src"); + let no_tests = build_wrap_blocks( + &plain, + "ATTACH 'ducklake:…' AS _wm_target;", + "orders", + "main/orders", + "_wm_partition", + "''", + false, + MaterializeStrategy::Append, + OnSchemaChange::Warn, + &[], + ) + .unwrap(); + assert!(no_tests.checks.is_empty()); + } + + // -- data tests --------------------------------------------------------- + + fn ctx_partitioned() -> DataTestCtx<'static> { + DataTestCtx { + target_qualified: "_wm_target.orders", + asset_path: "analytics/orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + scd2: false, + } + } + fn ctx_unpartitioned() -> DataTestCtx<'static> { + DataTestCtx { partitioned: false, ..ctx_partitioned() } + } + fn ctx_scd2() -> DataTestCtx<'static> { + DataTestCtx { partitioned: false, scd2: true, ..ctx_partitioned() } + } + + #[test] + fn data_test_unique_and_not_null_partition_scoped() { + let tests = vec![ + DataTestResolved::BuiltIn(DataTest::Unique { column: "order_id".into() }), + DataTestResolved::BuiltIn(DataTest::NotNull { column: "user_id".into() }), + ]; + let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap(); + assert!(sql.attaches.is_empty()); + assert_eq!(sql.checks.len(), 2); + // short, asset-free names (the asset is shown once by the breakdown). + assert_eq!(sql.checks[0].name, "unique(order_id)"); + assert_eq!(sql.checks[1].name, "not_null(user_id)"); + // each probe counts and samples the violating rows in one scan, with + // the size guard on the serialized sample. + for c in &sql.checks { + assert!(c + .probe + .starts_with("SELECT v, CASE WHEN strlen(s_raw) <= 51200 THEN s_raw END AS s")); + assert!(c.probe.contains("count(*) AS v")); + assert!(c.probe.contains("FILTER (WHERE _wm_rn <= 20)")); + } + // unique: groups non-null keys within the slice, having count>1; the + // sample is `{value, count}` pairs at the same grain as the count. + assert!(sql.checks[0] + .probe + .contains("GROUP BY \"order_id\" HAVING count(*) > 1")); + assert!(sql.checks[0] + .probe + .contains("SELECT \"order_id\" AS \"value\", count(*) AS \"count\"")); + assert!(sql.checks[0] + .probe + .contains("\"order_id\" IS NOT NULL AND \"_wm_partition\" = '2026-06-19'")); + // not_null: null rows in the slice; the sample excludes the synthetic + // partition column (storage detail, not producer output). + assert!(sql.checks[1] + .probe + .contains("WHERE \"user_id\" IS NULL AND \"_wm_partition\" = '2026-06-19'")); + assert!(sql.checks[1] + .probe + .contains("SELECT * EXCLUDE (\"_wm_partition\") FROM")); + } + + #[test] + fn data_test_unpartitioned_has_no_partition_scope() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::NotNull { + column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0].probe.contains("WHERE \"id\" IS NULL")); + assert!(!sql.checks[0].probe.contains("_wm_partition")); + assert!(!sql.checks[0].probe.contains("EXCLUDE")); + } + + #[test] + fn data_test_scd2_scopes_builtins_to_current_rows() { + // On an SCD2 history table the natural key repeats across closed + // versions, so built-in probes must assert the current snapshot only. + let tests = vec![ + DataTestResolved::BuiltIn(DataTest::Unique { column: "customer_id".into() }), + DataTestResolved::BuiltIn(DataTest::NotNull { column: "tier".into() }), + DataTestResolved::BuiltIn(DataTest::AcceptedValues { + column: "region".into(), + values: vec!["emea".into()], + }), + ]; + let sql = build_data_test_checks(&tests, &ctx_scd2()).unwrap(); + assert!(sql.checks[0] + .probe + .contains("WHERE \"customer_id\" IS NOT NULL AND is_current")); + assert!(sql.checks[1] + .probe + .contains("WHERE \"tier\" IS NULL AND is_current")); + assert!(sql.checks[2].probe.contains("AND is_current")); + // no partition scope leaks in (scd2 is unpartitioned in v1) + assert!(!sql.checks[0].probe.contains("_wm_partition")); + } + + #[test] + fn data_test_accepted_values_escapes_literals() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::AcceptedValues { + column: "status".into(), + values: vec!["paid".into(), "o'brien".into()], + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0].probe.contains("NOT IN ('paid', 'o''brien')")); + assert!(sql.checks[0].probe.contains("\"status\" IS NOT NULL")); + } + + #[test] + fn data_test_relationships_attaches_ref_and_dedups() { + let tests = vec![ + DataTestResolved::BuiltIn(DataTest::Relationships { + column: "user_id".into(), + to_kind: AssetKind::DataTable, + to_path: "prod/users".into(), + to_column: "id".into(), + }), + // second relationship into the SAME db reuses the alias (no 2nd attach) + DataTestResolved::BuiltIn(DataTest::Relationships { + column: "buyer_id".into(), + to_kind: AssetKind::DataTable, + to_path: "prod/buyers".into(), + to_column: "id".into(), + }), + ]; + let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap(); + assert_eq!(sql.attaches.len(), 1, "same db attached once"); + assert_eq!(sql.attaches[0], "ATTACH 'datatable://prod' AS _wm_ref_0;"); + assert!(sql.checks[0] + .probe + .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"users\"")); + assert!(sql.checks[1] + .probe + .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"buyers\"")); + assert!(sql.checks[0] + .probe + .contains("_wm_src.\"_wm_partition\" = '2026-06-19'")); + // sample rows come from the aliased target and drop the synthetic + // partition column. + assert!(sql.checks[0] + .probe + .contains("SELECT _wm_src.* EXCLUDE (\"_wm_partition\") FROM")); + assert_eq!( + sql.checks[0].name, + "relationships(user_id -> prod/users.id)" + ); + } + + #[test] + fn data_test_relationships_escapes_ref_name_in_attach() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "k".into(), + to_kind: AssetKind::DataTable, + to_path: "ev'il/users".into(), + to_column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + // single quote in the name is doubled so it can't break out of the literal + assert_eq!(sql.attaches[0], "ATTACH 'datatable://ev''il' AS _wm_ref_0;"); + } + + #[test] + fn data_test_relationships_same_lake_reuses_target() { + // A relationship into the SAME ducklake as the materialize target + // (asset_path = "analytics/orders") must NOT re-ATTACH it — _wm_target + // already holds that catalog; reuse it. + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "user_id".into(), + to_kind: AssetKind::Ducklake, + to_path: "analytics/users".into(), + to_column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!( + sql.attaches.is_empty(), + "same-lake ref must not ATTACH again" + ); + assert!(sql.checks[0] + .probe + .contains("NOT EXISTS (SELECT 1 FROM _wm_target.\"users\"")); + } + + #[test] + fn data_test_relationships_schema_qualified_target() { + // `/.
` — the schema-qualified table must quote each + // segment so the dot stays a separator, not part of one identifier. + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "sku".into(), + to_kind: AssetKind::Ducklake, + to_path: "warehouse/main.dim_products".into(), + to_column: "sku".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert_eq!( + sql.attaches[0], + "ATTACH 'ducklake://warehouse' AS _wm_ref_0;" + ); + assert!( + sql.checks[0] + .probe + .contains("FROM _wm_ref_0.\"main\".\"dim_products\""), + "schema-qualified target should be quoted per segment: {}", + sql.checks[0].probe + ); + } + + #[test] + fn data_test_relationships_rejects_non_attachable_kind() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "k".into(), + to_kind: AssetKind::S3Object, + to_path: "bucket/file".into(), + to_column: "c".into(), + })]; + assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err()); + } + + #[test] + fn data_test_custom_wraps_body() { + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SELECT * FROM _wm_target.orders WHERE amount < 0;".into(), + }]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + // trailing ; stripped, body embedded as the probe's rows query + assert!(sql.checks[0] + .probe + .contains("FROM (SELECT * FROM _wm_target.orders WHERE amount < 0) _wm_v")); + assert!(sql.checks[0].probe.contains("count(*) AS v")); + assert_eq!(sql.checks[0].name, "custom(f/tests/amount)"); + } + + #[test] + fn data_test_custom_rejects_multi_statement_body() { + // The body is embedded as a subquery, so a setup-then-SELECT body would + // produce invalid SQL — reject it up front with a self-teaching error + // that names the violation and shows the correct single-SELECT shape. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SET threads = 1; SELECT * FROM _wm_target.orders WHERE amount < 0".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("single SELECT"), "unexpected error: {err}"); + assert!( + err.contains("found 2 statements"), + "unexpected error: {err}" + ); + // the copyable example points at the internal target alias. + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); + } + + #[test] + fn data_test_custom_rejects_non_select_body() { + // A write/DDL body can't be embedded as `FROM ()`; the error must + // say so and teach the single-SELECT convention. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "DELETE FROM _wm_target.orders WHERE amount < 0".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("single SELECT"), "unexpected error: {err}"); + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); + } + + #[test] + fn data_test_custom_rejects_wrong_target_alias() { + // Referencing the target by its bare table name (not `_wm_target.
`) + // is the most common custom-test mistake — the runtime only attaches the + // freshly-materialized target under `_wm_target`, so the query would fail + // at runtime. Catch it at codegen with a self-teaching error. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SELECT * FROM orders WHERE amount < 0".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("_wm_target"), "unexpected error: {err}"); + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); + } + + #[test] + fn data_test_custom_accepts_from_first_and_uppercased_alias() { + // DuckDB's FROM-first syntax is a valid Output, and the alias match is + // case-insensitive (SQL identifiers are), so this passes. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "FROM _WM_TARGET.orders WHERE amount < 0".into(), + }]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0] + .probe + .contains("FROM (FROM _WM_TARGET.orders WHERE amount < 0) _wm_v")); + } + + #[test] + fn data_test_custom_empty_body_teaches_shape() { + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: " \n-- just a comment\n".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("empty test body"), "unexpected error: {err}"); + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); + } + + #[test] + fn data_test_unresolved_custom_is_internal_error() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Custom { + path: "f/x".into(), + })]; + assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err()); + } + + #[test] + fn materialize_result_sql_embeds_data_tests_breakdown() { + let checks = vec![ + DataTestCheck { name: "unique(order_id)".into(), probe: "SELECT v, s FROM q0".into() }, + DataTestCheck { name: "custom(f/t)".into(), probe: "SELECT v, s FROM q1".into() }, + ]; + let sql = materialize_result_sql( + "_wm_target.orders", + "analytics/orders", + "_wm_partition", + "'2026-06-19'", + false, + &checks, + None, + ); + // each probe runs once as a one-row CTE; _wm_tr cross-joins them and + // the list-of-struct references the flattened count/sample columns. + assert!(sql.starts_with( + "WITH _wm_t0 AS (SELECT v, s FROM q0), _wm_t1 AS (SELECT v, s FROM q1), \ + _wm_tr AS (SELECT _wm_t0.v AS c0, _wm_t0.s AS s0, _wm_t1.v AS c1, _wm_t1.s AS s1 \ + FROM _wm_t0, _wm_t1)" + )); + assert!(sql.contains("[{'test': 'unique(order_id)', 'violating': c0, 'sample': s0}, ")); + assert!( + sql.contains("{'test': 'custom(f/t)', 'violating': c1, 'sample': s1}] AS data_tests") + ); + assert!(sql.contains("FROM _wm_tr;")); + // no tests -> plain summary, no CTE / data_tests column. + let plain = materialize_result_sql( + "_wm_target.orders", + "analytics/orders", + "_wm_partition", + "'x'", + false, + &[], + None, + ); + assert!(plain.starts_with("SELECT 'ducklake://analytics/orders' AS materialized")); + assert!(!plain.contains("data_tests")); + // Schema capture (gap #2a) is in every summary, tests or not. Unpartitioned + // → explicit ordering, no partition-column filter. + for s in [&sql, &plain] { + assert!(s.contains( + "(SELECT list({'name': column_name, 'type': column_type} ORDER BY _wm_ord) \ + FROM (SELECT column_name, column_type, row_number() OVER () AS _wm_ord \ + FROM (DESCRIBE SELECT * FROM _wm_target.orders))) AS output_schema" + )); + assert!(!s.contains("WHERE column_name <>")); + } + } + + #[test] + fn materialize_result_sql_schema_excludes_partition_column() { + // Partitioned → the synthetic `_wm_partition` column is filtered out so + // the captured schema is the producer's logical output only. + let sql = materialize_result_sql( + "_wm_target.orders_daily", + "analytics/orders_daily", + "_wm_partition", + "'2026-06-19'", + true, + &[], + None, + ); + assert!(sql.contains( + "FROM (DESCRIBE SELECT * FROM _wm_target.orders_daily) \ + WHERE column_name <> '_wm_partition')) AS output_schema" + )); + } +} diff --git a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json index 158246022c..226f32914c 100644 --- a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json +++ b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json @@ -48,7 +48,11 @@ "expected": { "in_pipeline": true, "asset_triggers": [], - "native_triggers": ["kafka", "schedule", "data_upload"], + "native_triggers": [ + "kafka", + "schedule", + "data_upload" + ], "partition": null, "freshness": null, "tag": null, @@ -154,7 +158,10 @@ "partition": null, "freshness": null, "tag": null, - "retry": { "count": 3, "delay": "5s" } + "retry": { + "count": 3, + "delay": "5s" + } } }, { @@ -167,7 +174,10 @@ "partition": null, "freshness": null, "tag": null, - "retry": { "count": 2, "delay": null } + "retry": { + "count": 2, + "delay": null + } } }, { @@ -201,7 +211,9 @@ "code": "// pipeline\n// on s3://bucket/daily/{partition}/data.parquet\nexport function main() {}", "expected": { "in_pipeline": true, - "asset_triggers": ["s3object:bucket/daily/{partition}/data.parquet"], + "asset_triggers": [ + "s3object:bucket/daily/{partition}/data.parquet" + ], "native_triggers": [], "partition": null, "freshness": null, @@ -214,7 +226,9 @@ "code": " -- pipeline\n\t-- on datatable://main/x\nSELECT 1;", "expected": { "in_pipeline": true, - "asset_triggers": ["datatable:main/x"], + "asset_triggers": [ + "datatable:main/x" + ], "native_triggers": [], "partition": null, "freshness": null, @@ -234,5 +248,630 @@ "tag": null, "retry": null } + }, + { + "name": "materialize managed (default) with merge key", + "code": "// pipeline\n// materialize ducklake://analytics/orders_daily key=order_id\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders_daily", + "unique_key": "order_id" + } + } + }, + { + "name": "materialize scd2 via history flag with key, track and deletes=close", + "code": "// pipeline\n// materialize ducklake://analytics/dim_customer key=id history track=name,tier deletes=close\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/dim_customer", + "unique_key": "id", + "scd2": true, + "track": [ + "name", + "tier" + ], + "close_deleted": true + } + } + }, + { + "name": "materialize scd2 keyword alias with key only (track all non-key cols)", + "code": "// materialize scd2 ducklake://analytics/dim key=id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/dim", + "unique_key": "id", + "scd2": true + } + } + }, + { + "name": "materialize on_schema_change=ignore opt", + "code": "// pipeline\n// materialize ducklake://analytics/orders on_schema_change=ignore\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders", + "on_schema_change": "ignore" + } + } + }, + { + "name": "materialize on_schema_change unknown value keeps warn default (fail-safe)", + "code": "// materialize ducklake://analytics/orders key=id on_schema_change=bogus\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders", + "unique_key": "id", + "on_schema_change": "warn" + } + } + }, + { + "name": "materialize key without history is plain merge (SCD1, not scd2)", + "code": "// materialize ducklake://analytics/dim key=id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/dim", + "unique_key": "id" + } + } + }, + { + "name": "materialize manual escape hatch, first value wins", + "code": "// materialize manual ducklake://analytics/orders_daily\n// materialize ducklake://other/x\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders_daily", + "manual": true + } + } + }, + { + "name": "materialize default-syntax shorthand with append", + "code": "// materialize ducklake append\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "main", + "append": true + } + } + }, + { + "name": "materialize manual with no target is dropped", + "code": "// materialize manual\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "data_test built-ins accumulate in order", + "code": "-- pipeline\n-- materialize ducklake://analytics/orders key=order_id\n-- data_test unique order_id\n-- data_test not_null user_id\n-- data_test accepted_values status = paid,pending,refunded\n-- data_test relationships user_id -> datatable://prod/users.id\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders", + "unique_key": "order_id" + }, + "data_tests": [ + { + "type": "unique", + "column": "order_id" + }, + { + "type": "not_null", + "column": "user_id" + }, + { + "type": "accepted_values", + "column": "status", + "values": [ + "paid", + "pending", + "refunded" + ] + }, + { + "type": "relationships", + "column": "user_id", + "to_kind": "datatable", + "to_path": "prod/users", + "to_column": "id" + } + ] + } + }, + { + "name": "data_test accepted_values strips quotes and spacing", + "code": "# data_test accepted_values kind = \"a b\", 'c' ,d\nprint(1)", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { + "type": "accepted_values", + "column": "kind", + "values": [ + "a b", + "c", + "d" + ] + } + ] + } + }, + { + "name": "data_test custom escape hatch is a script path", + "code": "// data_test f/tests/orders_amount_sane\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { + "type": "custom", + "path": "f/tests/orders_amount_sane" + } + ] + } + }, + { + "name": "data_test relationships with ducklake shorthand target", + "code": "// data_test relationships sku -> ducklake://warehouse/dim_products.sku\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { + "type": "relationships", + "column": "sku", + "to_kind": "ducklake", + "to_path": "warehouse/dim_products", + "to_column": "sku" + } + ] + } + }, + { + "name": "malformed data_test lines are dropped fail-safe", + "code": "// data_test uniq order_id\n// data_test accepted_values s =\n// data_test relationships a -> b\n// data_test unique\n// data_test\n// data_test unique id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { + "type": "unique", + "column": "id" + } + ] + } + }, + { + "name": "ci test annotation is not a data test", + "code": "// test: f/foo/bar\n// data_test unique id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { + "type": "unique", + "column": "id" + } + ] + } + }, + { + "name": "column lineage maps output columns to upstream sources", + "code": "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n// column user_name <- datatable://prod/users.name\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "order_total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "tax" + } + ] + }, + { + "column": "user_name", + "inputs": [ + { + "from_kind": "datatable", + "from_path": "prod/users", + "from_column": "name" + } + ] + } + ] + } + }, + { + "name": "column lineage keeps duplicate input refs (dedup is a view concern)", + "code": "// column total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.amount\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + } + ] + } + ] + } + }, + { + "name": "column lineage keeps schema-qualified table, drops malformed refs", + "code": "// column sku <- ducklake://warehouse/main.dim_products.sku, bad_no_dot\n// column no_arrow datatable://prod/x.y\n// column total <- bad_no_dot\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "sku", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/main.dim_products", + "from_column": "sku" + } + ] + } + ] + } + }, + { + "name": "bare macros marker", + "code": "// macros\nCREATE MACRO dbl(a) AS a * 2;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "macros": true + } + }, + { + "name": "macros with trailing prose is not a marker", + "code": "// macros are defined below\nCREATE MACRO dbl(a) AS a * 2;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "macros marker with sql comment prefix", + "code": "-- macros\n-- pipeline\nCREATE MACRO dbl(a) AS a * 2;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "macros": true + } + }, + { + "name": "use accumulates in order and dedups", + "code": "// use f/lib/stats\n// use f/lib/dates\n// use f/lib/stats\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "use_libs": [ + "f/lib/stats", + "f/lib/dates" + ] + } + }, + { + "name": "use requires a slashed single token", + "code": "// use this script to compute stuff\n// use standalone\n// use f/lib/ok extra\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "use stops at first code line", + "code": "// pipeline\nSELECT 1;\n-- use f/lib/late\n", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "s3 triple-slash default-storage trigger canonicalizes to bare key", + "code": "// pipeline\n// on s3:///exports/x\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": [ + "s3object:exports/x" + ], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "s3 quad-slash trigger strips all leading slashes to the bare key", + "code": "// pipeline\n// on s3:////x\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": [ + "s3object:x" + ], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "mute suppresses a single ducklake read edge", + "code": "// pipeline\n// mute ducklake://main.orders\nselect 1", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "mute": [ + "ducklake:main.orders" + ] + } + }, + { + "name": "mute all opts out of all auto-derivation", + "code": "// pipeline\n// mute all\nselect 1", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "mute_all": true + } + }, + { + "name": "mute accumulates in order and dedups", + "code": "// pipeline\n// mute ducklake://main.a\n// mute s3://raw/b\n// mute ducklake://main.a\nselect 1", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "mute": [ + "ducklake:main.a", + "s3object:raw/b" + ] + } + }, + { + "name": "mute of a native trigger kind is dropped (only assets are muteable)", + "code": "// pipeline\n// mute kafka\nselect 1", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "mute prose without an asset ref never false-positives", + "code": "// pipeline\n// muted for now\nselect 1", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "mute all coexists with explicit on edges", + "code": "// pipeline\n// mute all\n// on ducklake://main.orders\nselect 1", + "expected": { + "in_pipeline": true, + "asset_triggers": [ + "ducklake:main.orders" + ], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "mute_all": true + } + }, + { + "name": "on asset ref strips trailing key=value opts", + "code": "// pipeline\n// on ducklake://main.orders debounce=60s\nselect 1", + "expected": { + "in_pipeline": true, + "asset_triggers": [ + "ducklake:main.orders" + ], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } } ] diff --git a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs index 3efc9f0c9d..fff818edb2 100644 --- a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs +++ b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs @@ -34,6 +34,57 @@ struct Expected { freshness: Option, tag: Option, retry: Option, + // Default-on-absent so the pre-existing fixtures (which omit it) keep + // deserializing; only fixtures exercising materialization set it. + #[serde(default)] + materialize: Option, + // Snake_case `DataTest` serde shape (e.g. {"type":"unique","column":"x"}), + // compared against `serde_json::to_value(got.data_tests)`. Absent === []. + #[serde(default)] + data_tests: Vec, + // Snake_case `ColumnLineage` serde shape (e.g. {"column":"x","inputs": + // [{"from_kind":"datatable","from_path":"p","from_column":"c"}]}), compared + // against `to_value(got.column_lineage)`. Absent === []. + #[serde(default)] + column_lineage: Vec, + // `// macros` marker (strict, alone on the line). Absent === false. + #[serde(default)] + macros: bool, + // `// use ` accumulation, declaration order, deduped. Absent === []. + #[serde(default)] + use_libs: Vec, + // `// mute ` accumulation as `kind:path`, declaration order, deduped. + // Absent === []. + #[serde(default)] + mute: Vec, + // `// mute all` marker. Absent === false. + #[serde(default)] + mute_all: bool, +} + +#[derive(Deserialize)] +struct ExpectedMaterialize { + target_kind: String, + target_path: String, + #[serde(default)] + manual: bool, + #[serde(default)] + append: bool, + #[serde(default)] + unique_key: Option, + #[serde(default)] + scd2: bool, + #[serde(default)] + track: Vec, + #[serde(default)] + close_deleted: bool, + // "warn" | "ignore"; absent === "warn" (the default). + #[serde(default = "default_on_schema_change")] + on_schema_change: String, +} + +fn default_on_schema_change() -> String { + "warn".to_string() } #[derive(Deserialize)] @@ -153,5 +204,72 @@ fn pipeline_annotation_fixtures_match() { want.is_some() ), } + + match (&got.materialize, &f.expected.materialize) { + (None, None) => {} + (Some(m), Some(e)) => { + assert_eq!( + kind_str(m.target_kind), + e.target_kind, + "{ctx}: materialize kind" + ); + assert_eq!(m.target_path, e.target_path, "{ctx}: materialize path"); + assert_eq!(m.manual, e.manual, "{ctx}: materialize manual"); + assert_eq!(m.append, e.append, "{ctx}: materialize append"); + assert_eq!(m.unique_key, e.unique_key, "{ctx}: materialize key"); + assert_eq!(m.scd2, e.scd2, "{ctx}: materialize scd2"); + assert_eq!(m.track, e.track, "{ctx}: materialize track"); + assert_eq!( + m.close_deleted, e.close_deleted, + "{ctx}: materialize close_deleted" + ); + let osc = match m.on_schema_change { + windmill_parser::asset_parser::OnSchemaChange::Warn => "warn", + windmill_parser::asset_parser::OnSchemaChange::Ignore => "ignore", + windmill_parser::asset_parser::OnSchemaChange::Fail => "fail", + windmill_parser::asset_parser::OnSchemaChange::Sync => "sync", + }; + assert_eq!( + osc, e.on_schema_change, + "{ctx}: materialize on_schema_change" + ); + } + (got, want) => panic!( + "{ctx}: materialize mismatch — got {:?}, want present={}", + got, + want.is_some() + ), + } + + let got_tests = serde_json::to_value(&got.data_tests).expect("data_tests serialize"); + assert_eq!( + got_tests, + serde_json::Value::Array(f.expected.data_tests.clone()), + "{ctx}: data tests" + ); + + let got_lineage = + serde_json::to_value(&got.column_lineage).expect("column_lineage serialize"); + assert_eq!( + got_lineage, + serde_json::Value::Array(f.expected.column_lineage.clone()), + "{ctx}: column lineage" + ); + + assert_eq!(got.macros, f.expected.macros, "{ctx}: macros"); + assert_eq!(got.use_libs, f.expected.use_libs, "{ctx}: use_libs"); + + let mute: Vec = got + .mute + .iter() + .filter_map(|t| match t { + TriggerSpec::Asset { asset_kind, path, .. } => { + Some(format!("{}:{}", kind_str(*asset_kind), path)) + } + _ => None, + }) + .collect(); + assert_eq!(mute, f.expected.mute, "{ctx}: mute"); + assert_eq!(got.mute_all, f.expected.mute_all, "{ctx}: mute_all"); } } diff --git a/backend/src/db_connect.rs b/backend/src/db_connect.rs index deda032058..d4dee19f82 100644 --- a/backend/src/db_connect.rs +++ b/backend/src/db_connect.rs @@ -3,9 +3,11 @@ use windmill_common::{ get_database_url, DatabaseUrl, }; -pub const DEFAULT_MAX_CONNECTIONS_SERVER: u32 = 50; -pub const DEFAULT_MAX_CONNECTIONS_WORKER: u32 = 5; -pub const DEFAULT_MAX_CONNECTIONS_INDEXER: u32 = 5; +// Single source of truth in windmill_common so the DB-health sizing guidance +// (windmill-api/src/db_health.rs) and the actual pool sizing here can't drift. +pub use windmill_common::{ + DEFAULT_MAX_CONNECTIONS_INDEXER, DEFAULT_MAX_CONNECTIONS_SERVER, DEFAULT_MAX_CONNECTIONS_WORKER, +}; #[cfg(feature = "operator")] pub const DEFAULT_MAX_CONNECTIONS_OPERATOR: u32 = 2; diff --git a/backend/src/main.rs b/backend/src/main.rs index 57cc02598a..4115e2a12b 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -578,6 +578,7 @@ fn print_help() { println!(" JSON_FMT = false Output logs in JSON instead of logfmt"); println!(" METRICS_ADDR = None (EE only) Prometheus metrics addr at /metrics; set \"true\" to use :8001"); println!(" SUPERADMIN_SECRET = None Virtual superadmin token (server)"); + println!(" NO_AUTH = false Bypass all auth; every request acts as the admin@windmill.dev superadmin (only behind a trusted gateway; ignored when CLOUD_HOSTED)"); println!(" LICENSE_KEY = None (EE only) Enterprise license key (workers require valid key)"); println!(" RUN_UPDATE_CA_CERTIFICATE_AT_START = false Run system CA update at startup"); println!(" RUN_UPDATE_CA_CERTIFICATE_PATH = /usr/sbin/update-ca-certificates Path to CA update tool"); @@ -641,6 +642,15 @@ async fn windmill_main() -> anyhow::Result<()> { println!("Running in MCP mode"); } + if *windmill_common::worker::NO_AUTH { + println!("############################################################"); + println!("# NO_AUTH mode is ENABLED: authentication is fully #"); + println!("# bypassed and every request is treated as the #"); + println!("# admin@windmill.dev superadmin. Only run this behind a #"); + println!("# trusted authenticating gateway on a private network. #"); + println!("############################################################"); + } + #[cfg(all(not(target_env = "msvc"), feature = "jemalloc"))] println!("jemalloc enabled"); @@ -1446,19 +1456,45 @@ Windmill Community Edition {GIT_VERSION} } else { None }; - monitor_db( - &conn, - &base_internal_url, - server_mode, - worker_mode, - false, - tx.clone(), - Some(MonitorIteration { - rd_shift, - iter: monitor_iteration, - }), + // Hard cap on a single monitor pass. monitor_db runs all its + // periodic tasks under one join!, so a single task stuck on a + // non-DB await (statement_timeout only bounds DB statements) + // would otherwise freeze the whole loop indefinitely — silently + // stopping critical maintenance like audit-partition creation. + // Larger than statement_timeout (5min) so a slow-but-progressing + // statement is never killed prematurely. + const MONITOR_DB_TIMEOUT: Duration = Duration::from_secs(600); + let monitor_timed_out = tokio::time::timeout( + MONITOR_DB_TIMEOUT, + monitor_db( + &conn, + &base_internal_url, + server_mode, + worker_mode, + false, + tx.clone(), + Some(MonitorIteration { + rd_shift, + iter: monitor_iteration, + }), + ), ) - .await; + .await + .is_err(); + if monitor_timed_out { + windmill_common::utils::report_critical_error( + format!( + "monitor task did not finish within {}s and was aborted; \ + a background maintenance task is likely stuck. \ + Continuing to the next iteration.", + MONITOR_DB_TIMEOUT.as_secs() + ), + db.clone(), + None, + None, + ) + .await; + } monitor_iteration += 1; if let Some(handle) = warn_handle { handle.abort(); @@ -1653,6 +1689,13 @@ async fn process_notify_event( ); windmill_queue::asset_dispatch::ASSET_PRODUCER_WRITES_CACHE.remove(payload); } + "notify_macro_registry_change" => { + tracing::debug!( + "Macro registry change for workspace {}, invalidating macro registry cache", + payload + ); + windmill_common::assets::MACRO_REGISTRY_CACHE.remove(payload); + } "notify_workspace_key_change" => { tracing::info!( "Workspace key change detected, invalidating workspace key cache: {}", @@ -1682,6 +1725,10 @@ async fn process_notify_event( match *source_type { "script" => { windmill_common::DEPLOYED_SCRIPT_HASH_CACHE.remove(&key); + // Bundle-cache key resolution for imported scripts; evicted + // together with the content-side caches below so key and + // inlined content flip to the new version in the same window. + windmill_common::IMPORTED_SCRIPT_HASH_CACHE.remove(&key); // Evict the relative-import latest-hash cache so a redeployed // imported script flips the content cache to its new version // across all replicas within a poll interval (see #6769). Keyed diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index be54edf468..60b4024c0a 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -75,6 +75,7 @@ use windmill_common::{ WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING, }, indexer::load_indexer_config, + jobs::delete_jobs, jwt::JWT_SECRET, oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH, server::load_smtp_config, @@ -177,6 +178,14 @@ lazy_static::lazy_static! { .and_then(|x| x.parse::().ok()) .unwrap_or(false); + // Ops kill switch for the pipeline freshness watchdog (a background + // pusher — being able to stop it without a redeploy matters more than + // for read-only monitors). + pub static ref DISABLE_FRESHNESS_WATCHDOG: bool = std::env::var("DISABLE_FRESHNESS_WATCHDOG") + .ok() + .and_then(|x| x.parse::().ok()) + .unwrap_or(false); + pub static ref WORKERS_NAMES: Arc>> = Arc::new(RwLock::new(Vec::new())); static ref QUEUE_COUNT_TAGS: Arc>> = Arc::new(RwLock::new(Vec::new())); @@ -1275,6 +1284,19 @@ pub async fn delete_expired_items(db: &DB) -> () { tracing::error!("Error deleting autoscaling event on CE: {:?}", e); } + // native_retry_attempt has no FK to v2_job (kept off the hot bulk delete). + // Retention sweeps markers alongside the jobs it deletes, but direct job + // deletions (workspace/job/schedule clearing) leave markers orphaned — reap + // any whose job is gone. The table is sparse, so this anti-join is cheap. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt nra WHERE NOT EXISTS (SELECT 1 FROM v2_job WHERE id = nra.job_id)" + ) + .execute(db) + .await + { + tracing::error!("Error reaping orphaned native retry markers: {:?}", e); + } + if let Err(e) = windmill_queue::cascade::reap_stale_join_slots(db).await { tracing::error!("Error reaping stale join_pending_inputs slots: {:?}", e); } @@ -1324,6 +1346,9 @@ pub async fn delete_expired_items(db: &DB) -> () { let cleanup_start = Instant::now(); let mut total_deleted = 0u64; let mut batch_num = 0i32; + // Watermark carried across batches so each one resumes after the rows the previous batch + // already processed instead of re-scanning the (potentially undeletable) oldest prefix. + let mut completed_at_floor: Option> = None; // Process batches until no more expired jobs or max batches reached loop { @@ -1336,14 +1361,17 @@ pub async fn delete_expired_items(db: &DB) -> () { } // Each batch runs in its own transaction to avoid long-running locks - let batch_result = delete_expired_jobs_batch(db, job_retention_secs, batch_size).await; + let batch_result = + delete_expired_jobs_batch(db, job_retention_secs, batch_size, completed_at_floor) + .await; match batch_result { - Ok(deleted_count) => { + Ok((deleted_count, max_completed_at)) => { if deleted_count == 0 { // No more expired jobs to delete break; } + completed_at_floor = max_completed_at.or(completed_at_floor); total_deleted += deleted_count as u64; batch_num += 1; } @@ -1510,12 +1538,20 @@ pub async fn check_expiring_tokens(db: &DB) { /// Delete a batch of expired jobs with LIMIT and SKIP LOCKED for high-scale environments. /// Uses a single transaction per batch to minimize lock duration. -/// Returns the number of jobs deleted in this batch. +/// +/// `completed_at_floor` is the watermark from the previous batch in the same cleanup run (the +/// max `completed_at` it deleted); pass `None` for the first batch. It is re-applied as +/// `completed_at >= floor` so the scan resumes past the rows already processed instead of +/// re-walking them (see the inline comment on the DELETE for why this matters). +/// +/// Returns `(jobs deleted in this batch, max completed_at deleted)`. The caller feeds the +/// returned watermark back in as `completed_at_floor` for the next batch. async fn delete_expired_jobs_batch( db: &DB, job_retention_secs: i64, batch_size: i64, -) -> error::Result { + completed_at_floor: Option>, +) -> error::Result<(usize, Option>)> { let mut tx = db.begin().await?; // Fetch active ROOT job IDs that started before the retention period. We only care about @@ -1531,26 +1567,70 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; - // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas - // ORDER BY completed_at ensures we delete oldest jobs first - let deleted_jobs: Vec = sqlx::query_scalar!( - "DELETE FROM v2_job_completed - WHERE id IN ( - SELECT jc.id FROM v2_job_completed jc - LEFT JOIN v2_job j ON j.id = jc.id - WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3) - ORDER BY jc.completed_at ASC - LIMIT $2 - FOR UPDATE OF jc SKIP LOCKED - ) - RETURNING id", - job_retention_secs, - batch_size, - &active_root_job_ids - ) - .fetch_all(&mut *tx) - .await?; + // `completed_at_floor` is a watermark carried across batches within a cleanup run: it is the + // max(completed_at) deleted by the previous batch. Re-applying it as `completed_at >= floor` + // lets each batch resume after the rows the previous batch already processed instead of + // re-scanning them. This matters when the oldest rows are undeletable (children of a + // still-active root flow): without the floor the `ORDER BY completed_at ASC` scan walks that + // same protected prefix on every batch, turning a cleanup run quadratic in prefix size. + // Floor only ever skips rows the current run already deleted, was protecting, or skip-locked — + // all correctly deferred to the next run, identical to the unbounded scan's semantics. + // + // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas; ORDER BY completed_at + // deletes oldest jobs first. + let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() { + // Common case: no old root flow is still running, so nothing is protected and the + // v2_job join (a PK lookup per candidate) is pure overhead — skip it entirely. + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT id FROM v2_job_completed + WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($3::timestamptz IS NULL OR completed_at >= $3) + ORDER BY completed_at ASC + LIMIT $2 + FOR UPDATE SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + } else { + // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`: + // the subquery form lets the planner build a one-time hashed SubPlan and apply it as a + // filter on the ordered index scan, giving O(1) membership per candidate instead of a + // per-row linear array scan (which degrades sharply when many root jobs are active). The + // `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT jc.id FROM v2_job_completed jc + LEFT JOIN v2_job j ON j.id = jc.id + WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($4::timestamptz IS NULL OR jc.completed_at >= $4) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) + ORDER BY jc.completed_at ASC + LIMIT $2 + FOR UPDATE OF jc SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + &active_root_job_ids, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + }; let deleted_count = deleted_jobs.len(); @@ -1589,10 +1669,21 @@ async fn delete_expired_jobs_batch( Err(e) => tracing::error!("Error deleting job logs: {:?}", e), } - if let Err(e) = sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", &deleted_jobs) - .execute(&mut *tx) - .await + // Native retry markers have no FK (to keep this bulk delete cheap) — sweep + // them with their jobs here too (the periodic retention path), same as the + // other side tables. The table is created by a startup migration, so it + // always exists by the time cleanup runs. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + &deleted_jobs + ) + .execute(&mut *tx) + .await { + tracing::error!("Error deleting native retry markers: {:?}", e); + } + + if let Err(e) = delete_jobs(&mut *tx, &deleted_jobs).await { tracing::error!("Error deleting job: {:?}", e); } @@ -1610,7 +1701,7 @@ async fn delete_expired_jobs_batch( tx.commit().await?; - Ok(deleted_count) + Ok((deleted_count, max_completed_at)) } async fn delete_log_files_from_disk_and_store( @@ -1638,11 +1729,30 @@ async fn delete_log_files_from_disk_and_store( .collect(); let stream = futures::stream::iter(s3_paths).boxed(); let mut result = os.delete_stream(stream); + let mut deleted = 0u64; + let mut not_found = 0u64; + let mut failed = 0u64; while let Some(r) = result.next().await { - if let Err(e) = r { - tracing::error!("Failed to delete from object store: {e}"); + match r { + Ok(_) => deleted += 1, + // Deleting a non-existent object is a successful no-op. S3's + // DeleteObjects ignores missing keys, but GCS returns 404 per + // delete, surfacing as NotFound — count it separately rather + // than logging it as an error. + Err(windmill_object_store::object_store_reexports::ObjectStoreError::NotFound { .. }) => { + not_found += 1; + } + Err(e) => { + failed += 1; + tracing::error!("Failed to delete from object store: {e}"); + } } } + if deleted + not_found + failed > 0 { + tracing::info!( + "object store log cleanup: {deleted} deleted, {not_found} already absent (404), {failed} failed" + ); + } } } } @@ -2614,6 +2724,9 @@ pub async fn monitor_db( if let Err(e) = cleanup_debounce_orphaned_keys(&db).await { tracing::error!("Error cleaning up debounce keys: {:?}", e); } + if let Err(e) = cleanup_consumed_debounce_batches(&db).await { + tracing::error!("Error cleaning up consumed debounce batches: {:?}", e); + } } } }; @@ -2889,6 +3002,28 @@ pub async fn monitor_db( } }; + // run every ~60s (2 iterations * 30s). Enterprise feature: the active + // `// freshness` backstop lives in windmill-queue's `freshness_watchdog` + // (`private`); OSS gets a no-op stub. Runtime-gated on an Enterprise + // license like the audit export above. Safe on concurrent servers — the + // watchdog claims per-script state rows atomically before pushing. + let pipeline_freshness_watchdog_f = async { + if server_mode + && !*DISABLE_FRESHNESS_WATCHDOG + && iteration.is_some() + && iteration.as_ref().unwrap().should_run(2) + { + if let Some(db) = conn.as_sql() { + if matches!( + windmill_common::ee_oss::get_license_plan().await, + windmill_common::ee_oss::LicensePlan::Enterprise + ) { + windmill_queue::freshness_watchdog::tick(db).await; + } + } + } + }; + join!( expired_items_f, zombie_jobs_f, @@ -2916,6 +3051,7 @@ pub async fn monitor_db( manage_audit_partitions_f, export_audit_logs_to_object_store_f, cleanup_scheduled_job_deletions_f, + pipeline_freshness_watchdog_f, ); } @@ -4316,6 +4452,40 @@ RETURNING key,job_id Ok(()) } +/// GC for claim-based debounce batches: once a batch row has been consumed (its +/// args accumulated into some survivor's run), it only lingers to let a later-pulled +/// survivor of the same batch tell "already consumed" from "never batched". A generous +/// grace period (>> any debounce window) makes that decision safe; after it, the rows +/// are dead weight. A re-pulled survivor whose row was GC'd correctly falls back to its +/// own (already-accumulated, persisted) args, so the grace period is not correctness- +/// critical. +async fn cleanup_consumed_debounce_batches(db: &DB) -> error::Result<()> { + // Only reclaim a consumed row once its job has LEFT the queue. A consumed sibling + // (its contribution already accumulated by another survivor) can sit queued well + // past any time-based grace under a concurrency limit / worker backlog; removing its + // row while still queued would make its eventual pull treat it as never-batched and + // re-run its item (a duplicate). Keeping the row until the job is no longer queued + // guarantees that pull still sees "already consumed" and runs empty. The age floor + // is just a safety margin on top. + let deleted = sqlx::query_scalar!( + "WITH del AS ( + DELETE FROM v2_job_debounce_batch + WHERE consumed_at IS NOT NULL + AND consumed_at < now() - interval '10 minutes' + AND id NOT IN (SELECT id FROM v2_job_queue) + RETURNING 1 + ) SELECT count(*) FROM del" + ) + .fetch_one(db) + .await? + .unwrap_or(0); + + if deleted > 0 { + tracing::info!("Cleaned up {deleted} consumed debounce batch rows"); + } + Ok(()) +} + async fn cleanup_debounce_keys_for_completed_jobs(db: &DB) -> error::Result<()> { // If min version doesn't support runnable settings, clean up debounce keys for completed jobs if !windmill_common::min_version::MIN_VERSION_SUPPORTS_RUNNABLE_SETTINGS_V0 @@ -4349,39 +4519,90 @@ RETURNING key,job_id Ok(()) } -async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { - let result = sqlx::query_scalar!( - "DELETE FROM job_perms -WHERE job_id NOT IN (SELECT id FROM v2_job_queue) -RETURNING job_id" - ) - .fetch_all(db) - .await?; +// Per-statement cap keeps each delete short and lock-light; the per-cycle batch +// cap bounds total work per monitor iteration so monitor_db stays responsive. +// A large backlog drains across several iterations rather than one long delete. +// +// These sweeps anti-join the whole table to find orphans, so their cost tracks the heap's +// physical size. job_perms / job_result_stream_v2 are high-churn (one row per job, deleted +// here), so their bloat — not the query shape — is what makes the sweep slow. These sweeps run +// every monitor cycle, but the bulk vacuuming_tables() runs only ~hourly, so dead tuples pile +// up between bulk vacuums; each sweep VACUUMs its own table right after deleting (see below) to +// keep the heap near the live working set. The outer `ctid IN (SELECT ... LIMIT)` is +// deliberate: a `job_id IN (...)` rewrite adds a second scan/probe for the delete and +// benchmarks slower, so don't "simplify" it. +const ORPHAN_CLEANUP_BATCH_SIZE: u64 = 100_000; +const ORPHAN_CLEANUP_MAX_BATCHES: usize = 10; - if !result.is_empty() { - tracing::info!("Cleaned up {} orphaned job_perms rows", result.len()); +// Reclaim the dead tuples a sweep just created so the next sweep's anti-join scans a lean heap +// instead of a bloated one. Plain VACUUM (not FULL) only takes SHARE UPDATE EXCLUSIVE, so +// concurrent reads/writes (every job create touches job_perms) keep running, and the visibility +// map lets it skip unchanged pages so repeated runs are cheap. SKIP_LOCKED means HA replicas +// don't pile up: one vacuums, the rest skip rather than queue behind it. +async fn vacuum_after_sweep(db: &DB, table: &str) { + if let Err(e) = sqlx::query(&format!("VACUUM (SKIP_LOCKED) {table}")) + .execute(db) + .await + { + tracing::warn!("Error vacuuming {table} after orphan cleanup: {e:?}"); + } +} + +async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_perms + WHERE ctid IN ( + SELECT jp.ctid FROM job_perms jp + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id) + LIMIT 100000 + )" + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } + + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_perms rows"); + vacuum_after_sweep(db, "job_perms").await; } Ok(()) } async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> { - let result = sqlx::query!( - "DELETE FROM job_result_stream_v2 - WHERE job_id NOT IN (SELECT id FROM v2_job_queue) - AND job_id NOT IN ( - SELECT id FROM v2_job_completed - WHERE completed_at > NOW() - INTERVAL '60 seconds' - ) - RETURNING job_id", - ) - .fetch_all(db) - .await?; + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_result_stream_v2 + WHERE ctid IN ( + SELECT jrs.ctid FROM job_result_stream_v2 jrs + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id) + AND NOT EXISTS ( + SELECT 1 FROM v2_job_completed c + WHERE c.id = jrs.job_id + AND c.completed_at > NOW() - INTERVAL '60 seconds' + ) + LIMIT 100000 + )", + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } - if result.len() > 0 { - tracing::info!( - "Cleaned up {} orphaned job_result_stream_v2 rows", - result.len() - ); + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_result_stream_v2 rows"); + vacuum_after_sweep(db, "job_result_stream_v2").await; } Ok(()) } @@ -4417,11 +4638,19 @@ async fn audit_log_retention_days() -> i64 { } } +/// Number of days ahead (including today) for which an audit partition must +/// always exist. A missing partition in this window means audit inserts fail +/// once that date is reached — and because some callers (notably login) write +/// the audit row in the same transaction as their own work, that failure +/// poisons the whole transaction, so a missing partition is a hard outage, not +/// just a dropped audit row. +const AUDIT_PARTITION_LOOKAHEAD_DAYS: i64 = 3; + async fn manage_audit_partitions(db: &DB, retention_days: i64) { let today = chrono::Utc::now().date_naive(); - // Create partitions for today and the next 3 days - for days_ahead in 0..=3i64 { + // Create partitions for today and the next few days + for days_ahead in 0..=AUDIT_PARTITION_LOOKAHEAD_DAYS { let date = today + chrono::Duration::days(days_ahead); let next_date = date + chrono::Duration::days(1); let partition_name = format!("audit_{}", date.format("%Y%m%d")); @@ -4437,9 +4666,6 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { } } - // Drop expired partitions - let cutoff_date = today - chrono::Duration::days(retention_days); - let partitions = sqlx::query_scalar::<_, String>( "SELECT c.relname::text \ FROM pg_inherits i \ @@ -4449,28 +4675,62 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { .fetch_all(db) .await; - match partitions { - Ok(partitions) => { - for partition_name in partitions { - if let Some(date_str) = partition_name.strip_prefix("audit_") { - if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { - if date < cutoff_date { - let quoted_name = - format!("\"{}\"", partition_name.replace('"', "\"\"")); - let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); - match sqlx::query(&sql).execute(db).await { - Ok(_) => tracing::info!( - "Dropped expired audit partition {partition_name}" - ), - Err(e) => tracing::error!( - "Error dropping audit partition {partition_name}: {e:?}" - ), - } + let partitions = match partitions { + Ok(partitions) => partitions, + Err(e) => { + tracing::error!("Error listing audit partitions: {e:?}"); + return; + } + }; + + // Verify the lookahead window is actually covered. If a create above failed + // (or this loop has not run for several days), alert loudly instead of + // letting it surface days later as failed audit inserts and broken logins. + let existing: std::collections::HashSet<&str> = partitions.iter().map(|s| s.as_str()).collect(); + let missing: Vec = (0..=AUDIT_PARTITION_LOOKAHEAD_DAYS) + .map(|days_ahead| { + format!( + "audit_{}", + (today + chrono::Duration::days(days_ahead)).format("%Y%m%d") + ) + }) + .filter(|name| !existing.contains(name.as_str())) + .collect(); + if !missing.is_empty() { + report_critical_error( + format!( + "Audit log partitions missing after maintenance run: {}. \ + Audit inserts will fail once these dates are reached, which also \ + breaks logins (the login audit row shares the login transaction). \ + Check for earlier 'Error creating audit partition' logs and verify \ + the audit-partition maintenance loop is still running.", + missing.join(", ") + ), + db.clone(), + None, + None, + ) + .await; + } + + // Drop expired partitions + let cutoff_date = today - chrono::Duration::days(retention_days); + for partition_name in &partitions { + if let Some(date_str) = partition_name.strip_prefix("audit_") { + if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { + if date < cutoff_date { + let quoted_name = format!("\"{}\"", partition_name.replace('"', "\"\"")); + let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); + match sqlx::query(&sql).execute(db).await { + Ok(_) => { + tracing::info!("Dropped expired audit partition {partition_name}") } + Err(e) => tracing::error!( + "Error dropping audit partition {partition_name}: {e:?}" + ), } } } } - Err(e) => tracing::error!("Error listing audit partitions: {e:?}"), } } diff --git a/backend/summarized_schema.txt b/backend/summarized_schema.txt index 3c88f16d0c..ccd46a9310 100644 --- a/backend/summarized_schema.txt +++ b/backend/summarized_schema.txt @@ -117,6 +117,10 @@ kafka_pending_commits: id(bigint), workspace_id(char), kafka_trigger_path(char), FK: (workspace_id, kafka_trigger_path) -> kafka_trigger(workspace_id, path) kafka_trigger: path(char), kafka_resource_path(char), topics(char), group_id(char), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), server_id(char), last_server_ping(ts), error(text), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode), filters(jsonb[]), auto_commit(bool), labels(text[]) log_file: hostname(char), log_ts(ts), ok_lines(bigint), err_lines(bigint), mode(log_mode), worker_group(char), file_path(char), json_fmt(bool) +macro_definition: workspace_id(char), name(char), provider_path(char), params(text), body(text), is_table_macro(bool), created_at(ts) + FK: (workspace_id) -> workspace(id) +macro_usage: workspace_id(char), consumer_path(char), macro_name(char) + FK: (workspace_id) -> workspace(id) magic_link: email(char), token(char), expiration(ts) mcp_oauth_client: mcp_server_url(text), client_id(text), client_secret(text), client_secret_expires_at(ts), token_endpoint(text), created_at(ts) mcp_oauth_refresh_token: id(bigint), refresh_token(char), access_token_hash(char), client_id(char), user_email(char), workspace_id(char), scopes(text[]), token_family(uuid), created_at(ts), expires_at(ts), used_at(ts), revoked(bool) @@ -184,7 +188,7 @@ windmill_migrations: name(text), created_at(ts) worker_group_job_stats: hour(bigint), worker_group(text), script_lang(char), workspace_id(char), job_count(int), total_duration_ms(bigint) FK: (workspace_id) -> workspace(id) worker_ping: worker(char), worker_instance(char), ping_at(ts), started_at(ts), ip(char), jobs_executed(int), custom_tags(text[]), worker_group(char), dedicated_worker(char), wm_version(char), current_job_id(uuid), current_job_workspace_id(char), vcpus(bigint), memory(bigint), occupancy_rate(float), memory_usage(bigint), wm_memory_usage(bigint), occupancy_rate_15s(float), occupancy_rate_5m(float), occupancy_rate_30m(float), job_isolation(text), dedicated_workers(text[]) -workspace: id(char), name(char), owner(char), deleted(bool), premium(bool), parent_workspace_id(char) +workspace: id(char), name(char), owner(char), deleted(bool), premium(bool), parent_workspace_id(char), is_dev_workspace(bool) FK: (parent_workspace_id) -> workspace(id) workspace_dependencies: id(bigint), name(char), content(text), language(script_lang), description(text), archived(bool), workspace_id(char), created_at(ts) workspace_diff: source_workspace_id(char), fork_workspace_id(char), path(char), kind(char), ahead(int), behind(int), has_changes(bool), exists_in_source(bool), exists_in_fork(bool) diff --git a/backend/tests/asset_trigger_dispatch.rs b/backend/tests/asset_trigger_dispatch.rs index e79937f702..7184d4810d 100644 --- a/backend/tests/asset_trigger_dispatch.rs +++ b/backend/tests/asset_trigger_dispatch.rs @@ -54,6 +54,14 @@ async fn seed_script( .bind(language) .execute(db) .await?; + // These tests use #[sqlx::test] isolated DBs that share one workspace id and + // reuse script paths, while the same path is seeded with different content + // (hence different hashes) across tests. The process-global deployed-script + // caches are keyed by (workspace, path)/(workspace, hash), so a concurrent + // test resolves a path to a hash that lives in another test's DB and the + // dispatch 404s. Disable them so every resolution reads the test's own DB. + windmill_common::DEPLOYED_SCRIPT_CACHE_DISABLED + .store(true, std::sync::atomic::Ordering::Relaxed); Ok(h) } @@ -392,14 +400,44 @@ async fn end_to_end_asset_dispatch(db: Pool) -> anyhow::Result<()> { ); } - // producer with parent_job (flow step) is ineligible + // flow step (carries flow_step_id) is ineligible clear_dispatched(&db).await?; let id = seed_producer_job(&db, json!({})).await?; let mut mini = make_mini(id, PRODUCER); - mini.parent_job = Some(Uuid::new_v4()); + mini.flow_step_id = Some("a".to_string()); let r = dispatch_asset_triggers(&db, &mini).await; assert_eq!(r.dispatched.len(), 0, "flow step producer ineligible"); + // A native retry attempt has a native_retry_attempt marker — it stays eligible, + // so a subscriber that recovers on retry still cascades. + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({})).await?; + sqlx::query("INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, 1)") + .bind(id) + .execute(&db) + .await?; + let mut mini = make_mini(id, PRODUCER); + mini.parent_job = Some(Uuid::new_v4()); + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!( + r.dispatched.len(), + 2, + "native retry attempt (marked) still dispatches" + ); + + // A parented Script child WITHOUT the marker — a schedule handler or a WAC + // inline child (which re-runs the same runnable) — must NOT cascade. + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({})).await?; + let mut mini = make_mini(id, PRODUCER); + mini.parent_job = Some(Uuid::new_v4()); // no marker => not a retry + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!( + r.dispatched.len(), + 0, + "parented child without the marker (handler/WAC inline) does not cascade" + ); + // producer with kind=Flow is ineligible let id = seed_producer_job(&db, json!({})).await?; let mut mini = make_mini(id, PRODUCER); @@ -640,13 +678,13 @@ async fn debounce_setting_applied_to_dispatched_subscriber( Ok(()) } -/// `// retry []` opts the subscriber into the flow-runtime retry -/// path. Implementation-wise, the dispatcher wraps the script in a one-step -/// flow (`JobPayload::SingleStepFlow`) so the existing flow retry machinery -/// handles re-runs. Subscribers without retry continue to be pushed as -/// `JobKind::Script` (no wrapping). +/// `// retry []` opts the subscriber into native retry: the +/// dispatcher pushes a real `JobKind::Script` (not a one-step flow) carrying +/// the policy in its `runnable_settings_handle`, so a failed subscriber re-runs +/// natively and stays eligible to trigger its own downstream. Subscribers +/// without retry are pushed as a plain `Script` with no settings handle. #[sqlx::test(fixtures("base"))] -async fn retry_setting_wraps_dispatched_subscriber_as_flow( +async fn retry_setting_dispatches_subscriber_as_native_script( db: Pool, ) -> anyhow::Result<()> { initialize_tracing().await; @@ -654,8 +692,8 @@ async fn retry_setting_wraps_dispatched_subscriber_as_flow( seed_script(&db, SUB_S3, "echo retrying", "bash").await?; seed_script(&db, SUB_RES, "echo plain", "bash").await?; seed_asset_write(&db, PRODUCER, "s3object", "f/blob").await?; - // Retry policy on the s3 edge; res edge stays vanilla so we also assert - // the "no retry" path keeps the cheaper ScriptHash push. + // Retry policy on the s3 edge; res edge stays vanilla so we also assert the + // "no retry" path carries no settings handle. seed_subscription_with_retry(&db, SUB_S3, "s3://f/blob", 3, 5).await?; seed_subscription(&db, SUB_RES, "script", "s3://f/blob").await?; @@ -663,37 +701,46 @@ async fn retry_setting_wraps_dispatched_subscriber_as_flow( let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; assert_eq!(r.dispatched.len(), 2, "both subscribers dispatched"); - let kinds: Vec<(String, String)> = sqlx::query!( - r#"SELECT runnable_path AS "runnable_path!", kind::text AS "kind!" - FROM v2_job - WHERE workspace_id = $1 AND trigger_kind = 'asset' - ORDER BY runnable_path"#, + let rows: Vec<(String, String, Option)> = sqlx::query!( + r#"SELECT j.runnable_path AS "runnable_path!", j.kind::text AS "kind!", + q.runnable_settings_handle + FROM v2_job j JOIN v2_job_queue q ON q.id = j.id + WHERE j.workspace_id = $1 AND j.trigger_kind = 'asset' + ORDER BY j.runnable_path"#, WS, ) .fetch_all(&db) .await? .into_iter() - .map(|r| (r.runnable_path, r.kind)) + .map(|r| (r.runnable_path, r.kind, r.runnable_settings_handle)) .collect(); - let s3_kind = kinds + let s3 = rows .iter() - .find(|(p, _)| p == SUB_S3) - .map(|(_, k)| k.as_str()) - .unwrap_or(""); - let res_kind = kinds + .find(|(p, _, _)| p == SUB_S3) + .expect("s3 dispatched"); + let res = rows .iter() - .find(|(p, _)| p == SUB_RES) - .map(|(_, k)| k.as_str()) - .unwrap_or(""); + .find(|(p, _, _)| p == SUB_RES) + .expect("res dispatched"); + // Native retry: a real Script (not a SingleStepFlow), with the policy in the + // runnable_settings_handle. assert_eq!( - s3_kind, "singlestepflow", - "retry-bearing subscriber wraps as SingleStepFlow so flow-runtime retry kicks in" + s3.1, "script", + "retry subscriber dispatched as a native Script" + ); + assert!( + s3.2.is_some(), + "retry subscriber carries a runnable_settings_handle (the retry policy)" ); assert_eq!( - res_kind, "script", - "no-retry subscriber stays as the cheaper ScriptHash push" + res.1, "script", + "no-retry subscriber stays a plain ScriptHash push" + ); + assert!( + res.2.is_none(), + "no-retry subscriber carries no settings handle" ); Ok(()) @@ -997,3 +1044,132 @@ async fn reaper_clears_only_stale_join_slots(db: Pool) -> anyhow::Resu Ok(()) } + +/// Seed one `materialized_partition` row (the state a managed `// materialize` +/// write records) so dispatch has a snapshot to look up. +async fn seed_materialization( + db: &Pool, + kind: &str, + asset_path: &str, + partition: &str, + status: &str, + snapshot_id: Option, +) -> anyhow::Result<()> { + sqlx::query( + r#"INSERT INTO materialized_partition + (workspace_id, asset_kind, asset_path, partition, status, snapshot_id) + VALUES ($1, $2::asset_kind, $3, $4, $5::materialization_status, $6)"#, + ) + .bind(WS) + .bind(kind) + .bind(asset_path) + .bind(partition) + .bind(status) + .bind(snapshot_id) + .execute(db) + .await?; + Ok(()) +} + +/// Dispatch records, on the consumer's `trigger` arg, the latest captured +/// materialization snapshot of each of its direct upstream assets +/// (`upstream_snapshots`) — the forensic "what did this run see" record. +/// Covered here: +/// - latest = highest `snapshot_id` with status `materialized` (a stale +/// partition and a failed/no-snapshot row are both passed over), +/// - whole-table (sentinel '') upstreams omit `partition`, +/// - upstreams with no captured snapshot produce no entry, +/// - a consumer with no materialized upstream at all gets no +/// `upstream_snapshots` key. +#[sqlx::test(fixtures("base"))] +async fn upstream_snapshots_recorded_on_dispatch(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + seed_script(&db, SUB_S3, "echo lake consumer", "bash").await?; + seed_script(&db, SUB_RES, "echo raw consumer", "bash").await?; + seed_asset_write(&db, PRODUCER, "ducklake", "analytics/orders").await?; + seed_asset_write(&db, PRODUCER, "s3object", "f/raw").await?; + + // SUB_S3's direct upstream set: the firing ducklake asset, a second + // materialized ducklake dimension (written by some other producer), and a + // plain s3 object that is never materialized. + seed_subscription(&db, SUB_S3, "script", "ducklake://analytics/orders").await?; + seed_subscription(&db, SUB_S3, "script", "ducklake://analytics/customers").await?; + seed_subscription(&db, SUB_S3, "script", "s3://f/raw").await?; + // SUB_RES subscribes only to the raw (non-materialized) asset. + seed_subscription(&db, SUB_RES, "script", "s3://f/raw").await?; + + // orders: an older partition, the latest one, and a failed slice with no + // snapshot — only 2026-06-19 @ 42 must be recorded. + seed_materialization( + &db, + "ducklake", + "analytics/orders", + "2026-06-18", + "materialized", + Some(41), + ) + .await?; + seed_materialization( + &db, + "ducklake", + "analytics/orders", + "2026-06-19", + "materialized", + Some(42), + ) + .await?; + seed_materialization( + &db, + "ducklake", + "analytics/orders", + "2026-06-20", + "failed", + None, + ) + .await?; + // customers: unpartitioned (sentinel '') → entry without `partition`. + seed_materialization( + &db, + "ducklake", + "analytics/customers", + "", + "materialized", + Some(7), + ) + .await?; + + let id = seed_producer_job(&db, json!({})).await?; + let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; + assert_eq!( + r.dispatched.len(), + 3, + "SUB_S3 fired for both written assets, SUB_RES for the raw one" + ); + + let expected_snaps = json!([ + { "asset": "ducklake://analytics/customers", "snapshot_id": 7 }, + { "asset": "ducklake://analytics/orders", "snapshot_id": 42, "partition": "2026-06-19" }, + ]); + for (path, _, args) in fetch_dispatched(&db).await? { + let trigger = args + .as_ref() + .and_then(|a| a.get("trigger")) + .cloned() + .expect("dispatched job carries a trigger arg"); + match path.as_str() { + SUB_S3 => assert_eq!( + trigger.get("upstream_snapshots"), + Some(&expected_snaps), + "latest materialized snapshot per upstream, sorted by ref, raw asset absent" + ), + SUB_RES => assert!( + trigger.get("upstream_snapshots").is_none(), + "no materialized upstream → no upstream_snapshots key" + ), + other => panic!("unexpected dispatched path {other}"), + } + } + + Ok(()) +} diff --git a/backend/tests/batch_rerun.rs b/backend/tests/batch_rerun.rs index b750379834..b1708465f9 100644 --- a/backend/tests/batch_rerun.rs +++ b/backend/tests/batch_rerun.rs @@ -71,6 +71,7 @@ fn ssf_script_payload(hash: Option, retry: Option) -> JobPayl path: SCRIPT_PATH.to_string(), hash, flow_version: None, + language: None, args: Default::default(), retry, error_handler_path: None, @@ -92,6 +93,7 @@ fn ssf_flow_payload() -> JobPayload { path: FLOW_PATH.to_string(), hash: None, flow_version: Some(FLOW_VERSION), + language: None, args: Default::default(), retry: None, error_handler_path: None, diff --git a/backend/tests/bun_jobs.rs b/backend/tests/bun_jobs.rs index d51e51f58b..fc766253bc 100644 --- a/backend/tests/bun_jobs.rs +++ b/backend/tests/bun_jobs.rs @@ -1,3 +1,4 @@ +use futures::StreamExt; use sqlx::postgres::Postgres; use sqlx::Pool; use uuid::Uuid; @@ -818,6 +819,126 @@ export function main() { Ok(()) } +// ============================================================================ +// Bundle cache invalidation on transitive relative-import change +// ============================================================================ + +async fn insert_deployed_bun_script(db: &Pool, path: &str, hash: i64, content: &str) { + // What gen_bun_lockfile stores for a script with no npm dependencies; a + // bare '' lock fails split_lockfile when the script is run directly. + const EMPTY_BUN_LOCK: &str = "{\n \"dependencies\": {}\n}\n//bun.lock\n"; + // Runtime query to avoid touching the sqlx offline cache. + sqlx::query( + "INSERT INTO script (workspace_id, created_by, content, schema, summary, description, path, hash, language, lock) + VALUES ('test-workspace', 'test-user', $1, '{\"$schema\":\"https://json-schema.org/draft/2020-12/schema\",\"properties\":{},\"required\":[],\"type\":\"object\"}', '', '', $2, $3, 'bun', $4)", + ) + .bind(content) + .bind(path) + .bind(hash) + .bind(EMPTY_BUN_LOCK) + .execute(db) + .await + .unwrap(); +} + +fn run_main_script_job(hash: i64) -> RunJob { + RunJob::from(JobPayload::ScriptHash { + path: "f/stale_bundle/main_script".to_string(), + hash: windmill_common::scripts::ScriptHash(hash), + cache_ttl: None, + cache_ignore_s3_path: None, + dedicated_worker: None, + language: ScriptLang::Bun, + priority: None, + apply_preprocessor: false, + concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default(), + debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(), + labels: None, + }) +} + +/// Editing a script that a runnable imports only TRANSITIVELY (main -> mid -> +/// leaf) must invalidate the runnable's cached bundle: the leaf's code is +/// inlined in the bundle, so the cache key has to cover the whole closure, not +/// just direct imports. +#[sqlx::test(fixtures("base"))] +async fn test_bun_transitive_import_change_rebundles(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Hashes/paths unique across this test binary: the script/hash caches are + // process-global while parallel tests each run in their own DB. + const LEAF_V1: i64 = 41230001; + const MID: i64 = 41230002; + const MAIN: i64 = 41230003; + const LEAF_V2: i64 = 41230004; + + insert_deployed_bun_script( + &db, + "f/stale_bundle/leaf", + LEAF_V1, + r#"export function leafValue() { return "V1_FROM_LEAF"; }"#, + ) + .await; + insert_deployed_bun_script( + &db, + "f/stale_bundle/mid", + MID, + r#"import { leafValue } from "./leaf"; +export function midValue() { return `M(${leafValue()})`; }"#, + ) + .await; + insert_deployed_bun_script( + &db, + "f/stale_bundle/main_script", + MAIN, + r#"import { midValue } from "./mid"; +export function main() { return midValue(); }"#, + ) + .await; + + let mut completed = listen_for_completed_jobs(&db).await; + let db2 = db.clone(); + in_test_worker( + &db, + async move { + let job = run_main_script_job(MAIN).push(&db2).await; + completed.next().await; + let result = completed_job(job, &db2).await.json_result().unwrap(); + assert_eq!(result, serde_json::json!("M(V1_FROM_LEAF)")); + + // Deploy a new version of ONLY the leaf; main_script and mid keep + // their hash, content, and lock byte-identical. + insert_deployed_bun_script( + &db2, + "f/stale_bundle/leaf", + LEAF_V2, + r#"export function leafValue() { return "V2_FROM_LEAF"; }"#, + ) + .await; + + // Tests don't run the notify_event poll loop, so replay what its + // `notify_runnable_version_change` handler (main.rs) does on deploy: + // evict the leaf's latest-hash cache entries. + windmill_common::IMPORTED_SCRIPT_HASH_CACHE.remove(&( + "test-workspace".to_string(), + "f/stale_bundle/leaf".to_string(), + )); + windmill_api_scripts::scripts::RAW_SCRIPT_LATEST_HASH_CACHE + .remove(&format!("test-workspace:f/stale_bundle/leaf")); + + let job = run_main_script_job(MAIN).push(&db2).await; + completed.next().await; + let result = completed_job(job, &db2).await.json_result().unwrap(); + assert_eq!(result, serde_json::json!("M(V2_FROM_LEAF)")); + }, + port, + ) + .await; + Ok(()) +} + #[sqlx::test(fixtures("base", "bun_edge_cases"))] async fn test_bun_shared_imports_both_styles(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; diff --git a/backend/tests/debounce_e2e.rs b/backend/tests/debounce_e2e.rs new file mode 100644 index 0000000000..9a2351f135 --- /dev/null +++ b/backend/tests/debounce_e2e.rs @@ -0,0 +1,112 @@ +// End-to-end debounce test: drives the FULL real path — real `push()` (EE +// `maybe_debounce` collapsing the batch), real `pull()`, real +// `maybe_apply_debouncing` (claim + accumulate), and a real worker executing the +// surviving flow — then asserts the executed result contains every accumulated item. +// Runs on --features deno_core,enterprise,private (debounce is EE/compile-gated). +#[cfg(all(feature = "deno_core", feature = "enterprise", feature = "private"))] +mod debounce_e2e { + use serde_json::json; + use sqlx::{Pool, Postgres}; + use uuid::Uuid; + use windmill_common::flows::FlowValue; + use windmill_common::jobs::JobPayload; + use windmill_common::worker::Connection; + use windmill_test_utils::*; + + async fn initialize_tracing() { + use std::sync::Once; + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let _ = windmill_common::tracing_init::initialize_tracing( + "test", + &windmill_common::utils::Mode::Standalone, + "test", + ); + }); + } + + /// A one-step flow that debounces on `items` and returns `flow_input.items`, + /// so the flow's result is exactly the accumulated batch. + fn debounce_flow() -> FlowValue { + serde_json::from_value(json!({ + "modules": [{ + "id": "a", + "value": { + "type": "rawscript", + "language": "deno", + "content": "export async function main(items: any[]) { return items }", + "input_transforms": { + "items": { "type": "javascript", "expr": "flow_input.items" } + } + } + }], + "debounce_delay_s": 1, + "debounce_key": "e2e_debounce_key", + "debounce_args_to_accumulate": ["items"] + })) + .expect("valid flow value") + } + + /// Fire three same-key messages; only the survivor runs, and it must execute once + /// with ALL accumulated items (none dropped, none duplicated). + #[sqlx::test(fixtures("base"))] + async fn debounce_accumulation_runs_once_with_all_items( + db: Pool, + ) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Push 3 debounced flow jobs (same key) — each collapses the previous; the last + // is the survivor. scheduled_for is ~1s out, so all 3 land before any worker pull. + let mut survivor = Uuid::nil(); + let mut superseded = Vec::new(); + for n in [1i64, 2, 3] { + if survivor != Uuid::nil() { + superseded.push(survivor); + } + survivor = RunJob::from(JobPayload::RawFlow { + value: debounce_flow(), + path: None, + restarted_from: None, + }) + .arg("items", json!([n])) + .push(&db) + .await; + } + + // Run a real worker until the survivor completes. + let listener = listen_for_completed_jobs(&db).await; + in_test_worker(Connection::Sql(db.clone()), listener.find(&survivor), port).await; + + let cj = completed_job(survivor, &db).await; + assert!(cj.success, "survivor flow must succeed"); + let mut result: Vec = + serde_json::from_value(cj.json_result().expect("result present")) + .expect("result is an array of numbers"); + result.sort(); + assert_eq!( + result, + vec![1, 2, 3], + "survivor must execute exactly once with ALL accumulated items" + ); + + // The two superseded messages must have been debounced (skipped), not run. + // (use a lightweight status query — skipped jobs have NULL started_at, which the + // full CompletedJob row decoder rejects) + for s in superseded { + let skipped: Option = + sqlx::query_scalar("SELECT status = 'skipped' FROM v2_job_completed WHERE id = $1") + .bind(s) + .fetch_optional(&db) + .await?; + assert_eq!( + skipped, + Some(true), + "superseded message {s} must be debounced (skipped), not executed" + ); + } + + Ok(()) + } +} diff --git a/backend/tests/fixtures/jobs_read_auth.sql b/backend/tests/fixtures/jobs_read_auth.sql index e6b28fba0c..456ac8c801 100644 --- a/backend/tests/fixtures/jobs_read_auth.sql +++ b/backend/tests/fixtures/jobs_read_auth.sql @@ -19,6 +19,45 @@ INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, sc ARRAY['jobs:read', 'if_jobs:filter_tags:deno'] ); +-- App embed token for the admin viewer (test-user). Mirrors a minted sandboxed +-- low-code app token: carries the `app_embed` sentinel plus the embed scope set. +-- Used to assert the token is confined to jobs the viewer LAUNCHED, not every job +-- the (admin) viewer could otherwise read. +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, scopes) VALUES ( + encode(sha256('EMBED_APP_TOKEN'::bytea), 'hex'), 'EMBED_APP_', 'EMBED_APP_TOKEN', + 'test@windmill.dev', 'app embed token', false, + ARRAY['apps:run', 'jobs:read', 'app_embed', 'resources:run', 'users:read', 'folders:read'] +); + +-- A completed app-component job LAUNCHED BY the admin viewer (created_by = +-- test-user), running as the app owner. The embed token must keep reading its own +-- launched job (the `created_by == viewer` fast path). +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner, args +) VALUES ( + '12121212-1212-1212-1212-121212121212', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false, + '{"own": "arg"}' +); +INSERT INTO public.v2_job_completed (id, workspace_id, duration_ms, status, result) VALUES + ('12121212-1212-1212-1212-121212121212', 'test-workspace', 1000, 'success'::job_status, + '{"own": "EMBED_OWN_RESULT"}'); + +-- A QUEUED job launched by the admin embed viewer (created_by = test-user). The +-- embed token may cancel its own launched job; it must NOT cancel another user's. +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner +) VALUES ( + '13131313-1313-1313-1313-131313131313', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false +); +INSERT INTO public.v2_job_queue (id, workspace_id, scheduled_for, running, tag) VALUES + ('13131313-1313-1313-1313-131313131313', 'test-workspace', '2023-01-01 00:00:00', false, 'deno'); + -- RUNNING job: queued (no completed row) and owned by test-user-2. Used to check -- that `completed/get_result_maybe?get_started=true` authorizes before disclosing -- running-state to a non-reader. diff --git a/backend/tests/freshness_watchdog.rs b/backend/tests/freshness_watchdog.rs new file mode 100644 index 0000000000..17851d0a1e --- /dev/null +++ b/backend/tests/freshness_watchdog.rs @@ -0,0 +1,294 @@ +//! End-to-end tests for the pipeline freshness watchdog (Enterprise). +//! +//! `windmill_queue::freshness_watchdog::tick` is called directly against +//! seeded `script` / `v2_job(_completed)` rows — no worker or API server is +//! needed, since the watchdog's job ends at the push (the pushed job sitting +//! in `v2_job_queue` is itself part of the assertions). Covers: staleness on +//! never-ran and aged-out members, the fresh short-circuit + state reset, +//! the in-flight suppression, the backoff claim, and the skip rules +//! (partitioned, malformed window, non-pipeline scripts). + +#![cfg(feature = "private")] + +use sqlx::{Pool, Postgres}; +use windmill_queue::freshness_watchdog::tick; +use windmill_test_utils::initialize_tracing; + +const WS: &str = "test-workspace"; +const PATH: &str = "u/test-user/freshness_producer"; + +/// Seed a deployed pipeline-member script. Mirrors the deploy path's output: +/// `auto_kind = 'pipeline'`, empty (non-NULL) lock so run-by-path resolution +/// treats it as deployed, hash derived from path+content for uniqueness. +async fn seed_pipeline_script( + db: &Pool, + path: &str, + content: &str, +) -> anyhow::Result<()> { + let mut h = 0i64; + for b in path.bytes().chain(content.bytes()) { + h = h.wrapping_mul(31).wrapping_add(b as i64); + } + sqlx::query( + r#"INSERT INTO script (workspace_id, hash, path, summary, description, content, + created_by, language, tag, lock, auto_kind) + VALUES ($1, $2, $3, '', '', $4, 'test-user', 'bash'::script_lang, 'bash', '', 'pipeline') + ON CONFLICT DO NOTHING"#, + ) + .bind(WS) + .bind(h) + .bind(path) + .bind(content) + .execute(db) + .await?; + // Process-global deployed-script caches are keyed by (workspace, path) / + // (workspace, hash) and would leak between #[sqlx::test] isolated DBs + // that reuse both — resolve everything from this test's own DB. + windmill_common::DEPLOYED_SCRIPT_CACHE_DISABLED + .store(true, std::sync::atomic::Ordering::Relaxed); + Ok(()) +} + +/// Seed a completed root run of `path` that finished `age_s` seconds ago. +async fn seed_completed_run( + db: &Pool, + path: &str, + age_s: i64, + success: bool, +) -> anyhow::Result<()> { + let id = uuid::Uuid::new_v4(); + sqlx::query( + r#"INSERT INTO v2_job (id, workspace_id, runnable_path, kind, created_at, + created_by, permissioned_as, permissioned_as_email, tag) + VALUES ($1, $2, $3, 'script'::job_kind, + now() - ($4::bigint::text || ' seconds')::interval, + 'test-user', 'u/test-user', 'test@windmill.dev', 'bash')"#, + ) + .bind(id) + .bind(WS) + .bind(path) + .bind(age_s) + .execute(db) + .await?; + sqlx::query( + r#"INSERT INTO v2_job_completed (id, workspace_id, duration_ms, status, started_at, completed_at) + VALUES ($1, $2, 0, CASE WHEN $3 THEN 'success'::job_status ELSE 'failure'::job_status END, + now() - ($4::bigint::text || ' seconds')::interval, + now() - ($4::bigint::text || ' seconds')::interval)"#, + ) + .bind(id) + .bind(WS) + .bind(success) + .bind(age_s) + .execute(db) + .await?; + Ok(()) +} + +/// Jobs the watchdog pushed: (path, created_by, args) rows attributed to +/// `trigger_kind = 'freshness'`. +async fn fetch_pushed( + db: &Pool, +) -> anyhow::Result)>> { + let rows = sqlx::query!( + r#"SELECT runnable_path AS "runnable_path!", created_by AS "created_by!", + args AS "args: sqlx::types::Json" + FROM v2_job + WHERE workspace_id = $1 AND trigger_kind = 'freshness' + ORDER BY created_at"#, + WS, + ) + .fetch_all(db) + .await?; + Ok(rows + .into_iter() + .map(|r| (r.runnable_path, r.created_by, r.args.map(|a| a.0))) + .collect()) +} + +async fn state_row(db: &Pool, path: &str) -> anyhow::Result> { + Ok(sqlx::query_scalar!( + "SELECT attempts FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2", + WS, + path, + ) + .fetch_optional(db) + .await?) +} + +#[sqlx::test(fixtures("base"))] +async fn never_ran_member_is_pushed_once(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 30s\necho hi\n").await?; + + tick(&db).await; + + let pushed = fetch_pushed(&db).await?; + assert_eq!(pushed.len(), 1, "one watchdog push expected"); + let (path, created_by, args) = &pushed[0]; + assert_eq!(path, PATH); + assert_eq!(created_by, &format!("freshness-{PATH}")); + let args = args.as_ref().expect("args recorded"); + assert_eq!( + args.get("_wmill_skip_asset_dispatch"), + Some(&serde_json::json!(true)), + "watchdog runs must not re-fire the cascade" + ); + assert_eq!( + args.pointer("/trigger/kind"), + Some(&serde_json::json!("freshness")) + ); + assert_eq!(state_row(&db, PATH).await?, Some(1), "claim row recorded"); + + // Second tick: the pushed job is queued-and-due, so the in-flight guard + // suppresses a duplicate regardless of backoff. + tick(&db).await; + assert_eq!( + fetch_pushed(&db).await?.len(), + 1, + "no duplicate while queued" + ); + + // Simulate the queued job vanishing without a completion: the backoff + // claim (next_attempt_at in the future) now carries the suppression. + sqlx::query!("DELETE FROM v2_job_queue WHERE workspace_id = $1", WS) + .execute(&db) + .await?; + tick(&db).await; + assert_eq!(fetch_pushed(&db).await?.len(), 1, "backoff holds the retry"); + + // Force the backoff window open: the watchdog retries and escalates. + sqlx::query!( + "UPDATE pipeline_freshness_state SET next_attempt_at = now() - interval '1 second' + WHERE workspace_id = $1 AND script_path = $2", + WS, + PATH, + ) + .execute(&db) + .await?; + tick(&db).await; + assert_eq!(fetch_pushed(&db).await?.len(), 2, "due retry pushed"); + assert_eq!(state_row(&db, PATH).await?, Some(2), "attempts escalated"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn fresh_member_is_skipped_and_state_reset(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 1h\necho hi\n").await?; + seed_completed_run(&db, PATH, 10, true).await?; + // Leftover backoff row from an earlier staleness episode. + sqlx::query!( + "INSERT INTO pipeline_freshness_state (workspace_id, script_path, attempts) VALUES ($1, $2, 3)", + WS, + PATH, + ) + .execute(&db) + .await?; + + tick(&db).await; + + assert!( + fetch_pushed(&db).await?.is_empty(), + "fresh member not pushed" + ); + assert_eq!(state_row(&db, PATH).await?, None, "backoff reset on fresh"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn aged_out_member_is_pushed(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 1h\necho hi\n").await?; + // Old success outside the window + a recent failure: still stale. + seed_completed_run(&db, PATH, 7200, true).await?; + seed_completed_run(&db, PATH, 60, false).await?; + + tick(&db).await; + + assert_eq!(fetch_pushed(&db).await?.len(), 1, "aged-out member pushed"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn partitioned_malformed_and_plain_members_are_skipped( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + // Partitioned: freshness means partition-gap detection, out of scope. + seed_pipeline_script( + &db, + "u/test-user/partitioned", + "# pipeline\n# partitioned daily\n# freshness 1h\necho hi\n", + ) + .await?; + // Malformed window: fails safe to unwatched. + seed_pipeline_script( + &db, + "u/test-user/malformed", + "# pipeline\n# freshness soonish\necho hi\n", + ) + .await?; + // Freshness only in prose (parser must reject; ILIKE prefilter passes). + seed_pipeline_script( + &db, + "u/test-user/prose", + "# pipeline\n# ensure freshness of data below\necho hi\n", + ) + .await?; + + tick(&db).await; + + assert!(fetch_pushed(&db).await?.is_empty(), "no member is watched"); + let rows = sqlx::query_scalar!( + r#"SELECT COUNT(*) AS "count!" FROM pipeline_freshness_state WHERE workspace_id = $1"#, + WS, + ) + .fetch_one(&db) + .await?; + assert_eq!(rows, 0, "no state rows for unwatched members"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn archived_workspace_is_not_resurrected(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + // Workspace archival stops all execution but keeps script rows for + // unarchival — the watchdog must not keep pushing runs there. + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 30s\necho hi\n").await?; + sqlx::query!("UPDATE workspace SET deleted = true WHERE id = $1", WS) + .execute(&db) + .await?; + + tick(&db).await; + + assert!( + fetch_pushed(&db).await?.is_empty(), + "no pushes into an archived workspace" + ); + assert_eq!( + state_row(&db, PATH).await?, + None, + "no state bookkeeping either" + ); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn state_of_unwatched_member_is_cleaned_up(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + // A stale-bookkeeping row whose script no longer declares freshness + // (e.g. annotation removed and redeployed) must not survive the sweep. + sqlx::query!( + "INSERT INTO pipeline_freshness_state (workspace_id, script_path) VALUES ($1, $2)", + WS, + "u/test-user/gone", + ) + .execute(&db) + .await?; + + tick(&db).await; + + assert_eq!(state_row(&db, "u/test-user/gone").await?, None); + Ok(()) +} diff --git a/backend/tests/jobs_read_auth.rs b/backend/tests/jobs_read_auth.rs index f19daf0f92..1eab0bffee 100644 --- a/backend/tests/jobs_read_auth.rs +++ b/backend/tests/jobs_read_auth.rs @@ -38,6 +38,10 @@ const TOP_SECRET_FLOW: &str = "ffffffff-ffff-ffff-ffff-ffffffffffff"; const DEEP_LEAF_JOB: &str = "88888888-8888-8888-8888-888888888888"; // A queued/running job (no completed row) owned by test-user-2. const RUNNING_JOB: &str = "77777777-7777-7777-7777-777777777777"; +// An app-component job launched BY the admin embed viewer (created_by test-user). +const EMBED_OWN_JOB: &str = "12121212-1212-1212-1212-121212121212"; +// A QUEUED job launched by the embed viewer (created_by test-user) — cancelable by it. +const EMBED_OWN_QUEUED: &str = "13131313-1313-1313-1313-131313131313"; // Secrets that must never leak to an unauthorized viewer. const RESULT_SECRET: &str = "RESULT_SECRET"; @@ -59,6 +63,19 @@ async fn get(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCod (status, body) } +async fn post(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCode, String) { + let mut req = client() + .post(format!("{base}/{path}")) + .json(&serde_json::json!({})); + if let Some(token) = token { + req = req.header("Authorization", format!("Bearer {token}")); + } + let resp = req.send().await.expect("request"); + let status = resp.status(); + let body = resp.text().await.expect("body"); + (status, body) +} + #[sqlx::test(fixtures("base", "jobs_read_auth"))] async fn test_single_job_read_authorization(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; @@ -88,6 +105,10 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul format!("get_completed_logs_tail/{VICTIM}"), ), ("get_flow_all_logs", format!("get_flow_all_logs/{VICTIM}")), + ( + "get_flow_all_logs_structured", + format!("get_flow_all_logs_structured/{VICTIM}"), + ), ( "completed/get_timing", format!("completed/get_timing/{VICTIM}"), @@ -281,6 +302,75 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul "top flow in an unreadable folder must stay denied (got {status}): {body}" ); + // ---- APP EMBED TOKEN: confined to jobs the viewer LAUNCHED, not everything + // the (admin) viewer can otherwise read. The token carries the `app_embed` + // sentinel; an admin's normal token reads VICTIM (asserted above), but the + // embed token must stop at the `created_by == viewer` grant so user-authored + // app JS can't reuse it to read unrelated jobs by UUID. + // Its own launched component job (created_by == viewer) still reads. + let (status, body) = get( + &base, + &format!("completed/get_result/{EMBED_OWN_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must read a job it launched (got {status}): {body}" + ); + assert!( + body.contains("EMBED_OWN_RESULT"), + "embed token should get its own launched job result: {body}" + ); + // The VICTIM job — created by another user but readable by this admin viewer's + // normal token (asserted above) — is denied to the embed token across result / + // logs / live update. NotFound (not 403) so the untrusted app can't even probe + // existence, and no secret leaks. + for path in [ + format!("completed/get_result/{VICTIM}"), + format!("get_logs/{VICTIM}"), + format!("getupdate/{VICTIM}?only_result=true"), + ] { + let (status, body) = get(&base, &path, Some("EMBED_APP_TOKEN")).await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not read a job it did not launch ({path}, got {status}): {body}" + ); + for secret in [RESULT_SECRET, ARGS_SECRET, LOGS_SECRET] { + assert!( + !body.contains(secret), + "embed token response for {path} leaked `{secret}`: {body}" + ); + } + } + + // ---- APP EMBED TOKEN: cancellation confined to the app's own jobs. The token + // may cancel a job it launched (created_by == viewer), but `cancel_job_api` + // denies (NotFound) a job created by someone else, even though cancel + // otherwise has no per-job ownership check. + let (status, body) = post( + &base, + &format!("queue/cancel/{EMBED_OWN_QUEUED}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must cancel a job it launched (got {status}): {body}" + ); + let (status, body) = post( + &base, + &format!("queue/cancel/{RUNNING_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not cancel another user's job (got {status}): {body}" + ); + // ---- UNAUTHENTICATED, unchanged: an anonymous-created job is readable // without a token (public trigger / public app result polling). let (status, body) = get(&base, &format!("completed/get_result/{ANON_JOB}"), None).await; diff --git a/backend/tests/preserve_on_behalf_of.rs b/backend/tests/preserve_on_behalf_of.rs index 7ed5fc7697..d808348937 100644 --- a/backend/tests/preserve_on_behalf_of.rs +++ b/backend/tests/preserve_on_behalf_of.rs @@ -2704,7 +2704,9 @@ async fn test_schedule_permissions_workspace_admin(db: Pool) -> anyhow Ok(()) } -/// Superadmin NOT in workspace creates a schedule — uses email as permissioned_as +/// Superadmin NOT in workspace creates a schedule — uses their instance-derived +/// username (`password.username`) as permissioned_as, not the raw email. The +/// email is still stored directly on the schedule for downstream resolution. #[sqlx::test(fixtures("preserve_on_behalf_of"))] async fn test_schedule_permissions_superadmin_not_in_workspace( db: Pool, @@ -2758,16 +2760,16 @@ async fn test_schedule_permissions_superadmin_not_in_workspace( .fetch_one(&db) .await?; - // Superadmin not in workspace: username_to_permissioned_as uses the email directly - // since the authed username for a superadmin not in workspace IS the email + // Superadmin not in workspace: the authed username is now their instance-derived + // username (`password.username` = 'superadmin-external'), so permissioned_as is + // `u/` rather than the raw email. The email is still stored directly. assert_eq!( schedule.email, "superadmin-external@windmill.dev", "schedule email should be superadmin email" ); assert_eq!( - schedule.permissioned_as, - schedule.email.clone(), - "permissioned_as should match email for superadmin not in workspace" + schedule.permissioned_as, "u/superadmin-external", + "permissioned_as should use the instance-derived username, not the email" ); // Update by the same superadmin diff --git a/backend/tests/script_rename_clears_assets.rs b/backend/tests/script_rename_clears_assets.rs new file mode 100644 index 0000000000..315deaf152 --- /dev/null +++ b/backend/tests/script_rename_clears_assets.rs @@ -0,0 +1,106 @@ +//! Regression test: renaming a script clears the OLD path's static asset +//! usage rows. +//! +//! A script deploy persists its producer/consumer asset lineage as `asset` +//! rows keyed by `usage_path = + + +
+ + +"##; + TEMPLATE.replace("__SECRET__", secret) +} + // async fn get_app_version( // authed: ApiAuthed, // Extension(user_db): Extension, @@ -937,6 +1110,17 @@ async fn get_public_app_by_secret( let mut app = not_found_if_none(app_o, "App", id.to_string())?; + // Confine the app embed token (the only credential handed to untrusted app JS, + // carrying the viewer's identity + `apps:read:`) to the app the secret + // resolves to: without this, app JS could reuse the viewer's identity to read any + // app it can see by secret via the RLS check below. Scoped to embed tokens only — + // other callers (anonymous, cookie, plain external JWT) keep their existing access. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:read:{}", app.path))?; + } + } + let policy = serde_json::from_str::(app.policy.0.get()).map_err(to_anyhow)?; if !matches!(policy.execution_mode, ExecutionMode::Anonymous) { @@ -971,6 +1155,300 @@ async fn get_public_app_by_secret( Ok(Json(app)) } +/// Scopes granted to a short-lived "app embed token". This is the token the +/// app-embedder page hands the (opaque-origin) app iframe at startup so the app +/// never receives the viewer's session cookie. Instead of restricting which +/// routes a *domain* may hit, we restrict which routes the *token* may hit, so +/// that even a malicious or compromised app document can only reach the +/// endpoints an app legitimately needs. The `app_embed` sentinel turns each of +/// these into a strict route allowlist (`app_embed_route_denied`): +/// - `jobs:read` → by-id job poll/cancel only; enumeration, counts, exports, +/// and `job_signature`/`resume_urls` are denied, and by-id +/// reads are confined to the app's own runs. +/// - `app_embed` → sentinel tagging this as an app embed token (grants nothing). +/// - `resources:run` → resource metadata only (pickers, type schemas), never values. +/// - `users:read` → `users/whoami` only. +/// - `folders:read` → `folders/listnames` only. +/// Plus two path-scoped scopes minted per app (see `mint_app_embed_token`): +/// - `apps:read:` → the app's own definition (`apps/get/p/`); no +/// `apps:write`, so management routes are unreachable. +/// - `apps:run:` → run THIS app's components (`execute_component`, which +/// re-checks the path); `apps_u/*` public-serving routes. +pub const APP_EMBED_SCOPES: [&str; 5] = [ + "jobs:read", + windmill_api_auth::scopes::APP_EMBED_SENTINEL, + "resources:run", + "users:read", + "folders:read", +]; + +/// How long an app embed token stays valid. The embedder re-mints on demand +/// (e.g. after a `401` from the iframe) so this can stay short. +const APP_EMBED_TOKEN_VALIDITY_HOURS: i64 = 12; + +#[derive(Serialize)] +pub struct EmbedTokenResponse { + /// Narrowly-scoped token for the iframe. `None` for fully anonymous access + /// (the iframe then calls the public endpoints anonymously). + pub token: Option, + pub expiration: Option>, + /// WIN-2006: raw apps render single-iframe (the bundle is already isolated in + /// its own opaque iframe), so the viewer skips the opaque-viewer indirection + /// and the embed token entirely — it loads the app with the page credential. + #[serde(default)] + pub raw_app: bool, + /// WIN-2006: publisher opted this app into sandbox isolation. When false the + /// viewer runs the app same-origin with its full session (the default, + /// pre-isolation behavior). + #[serde(default)] + pub sandbox: bool, + /// WIN-2006: the resolved app path. The embedder uses it (together with + /// `workspace_id`) to scope the app's backing `localStorage` per app (so + /// sandboxed apps don't share one store). Not a new disclosure — the viewer + /// already receives `path` when it loads the app (e.g. `get_public_app_by_secret`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub app_path: Option, + /// WIN-2006: the resolved workspace. Pairs with `app_path` for the per-app + /// `localStorage` key so two apps at the same path in different workspaces don't + /// share a store. For custom-path apps the viewer can't derive this itself. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub workspace_id: Option, +} + +/// Mint a short-lived, narrowly-scoped embed token for `app_path` when a caller +/// is authenticated. When `opt_authed` is `None` (anonymous access to an +/// anonymous app) no token is minted and the iframe relies on the public +/// endpoints. +/// +/// The CALLER MUST verify the viewer's access to `app_path` before calling: this +/// mints a token on behalf of `opt_authed` unconditionally (DB access remains +/// gated by the viewer's own RLS, but the token's existence is not access-checked +/// here). All current call sites (`get_app_embed_token`, +/// `get_app_embed_token_for_path`, and the EE custom-path variant) do this. +/// +/// Scope confinement IS enforced here: the minted scopes must be within the +/// caller's own (`ensure_scopes_within_caller`), so a scope-restricted bearer +/// token cannot bootstrap a broader-scoped embed token. For the normal caller — +/// an unscoped browser session — this is a no-op and the mint is purely +/// narrowing. +pub async fn mint_app_embed_token( + db: &DB, + w_id: &str, + app_path: &str, + opt_authed: Option<&ApiAuthed>, +) -> Result { + let token_and_exp = if let Some(authed) = opt_authed { + // An app embed token represents untrusted app JS in the sandboxed iframe; it + // must never reach this mint path to renew itself. The 12h expiry is the + // blast-radius cap on a leaked embed token, and `ensure_scopes_within_caller` + // below would pass a same-scoped renewal (the requested scopes equal the + // caller's own), making the credential indefinitely self-renewable. Refresh + // minting is the trusted embedder session/JWT's job. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotAuthorized( + "App embed tokens cannot mint or renew embed tokens".to_string(), + )); + } + let expiration = + chrono::Utc::now() + chrono::Duration::hours(APP_EMBED_TOKEN_VALIDITY_HOURS); + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Path-scoped read so the app can fetch its OWN definition (apps/get/p, + // which the in-workspace sandboxed viewer uses) — but no other app's. The + // public viewer fetches via apps_u/public_app and doesn't rely on this. + scopes.push(format!("apps:read:{app_path}")); + // Path-scoped run (NOT unqualified `apps:run`) so the token can only execute + // THIS app's components: `execute_component` re-checks `apps:run:` for + // the requested app, so the token can't drive another app's runnables. + scopes.push(format!("apps:run:{app_path}")); + // A scope-restricted caller token must not bootstrap a broader-scoped + // embed token (`create_token_internal` deliberately does not check this + // itself). No-op for unscoped sessions — the normal embed flow. + ensure_scopes_within_caller(authed, Some(&scopes))?; + let token_config = NewToken::new( + Some(format!("embed_app:{app_path}")), + Some(expiration), + None, + Some(scopes), + Some(w_id.to_string()), + // Never let an embed token gain write capability the caller's own + // session lacks. + Some(authed.read_only), + ); + let mut tx = db.begin().await?; + let token = create_token_internal(&mut *tx, db, authed, token_config).await?; + tx.commit().await?; + Some((token, expiration)) + } else { + None + }; + + Ok(EmbedTokenResponse { + token: token_and_exp.as_ref().map(|(t, _)| t.clone()), + expiration: token_and_exp.map(|(_, e)| e), + raw_app: false, + sandbox: false, + app_path: Some(app_path.to_string()), + workspace_id: Some(w_id.to_string()), + }) +} + +/// Issue an embed token for a public app addressed by its (secret) share id. +/// Mirrors the access check in [`get_public_app_by_secret`]: anonymous apps are +/// reachable without auth, otherwise the caller must be logged in and have read +/// access to the app. +async fn get_app_embed_token( + OptAuthed(opt_authed): OptAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, secret)): Path<(String, String)>, +) -> JsonResult { + let id = get_id_from_secret(&db, &w_id, secret, None).await?; + + let app = sqlx::query!( + "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.id = $1 AND a.workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + let app = not_found_if_none(app, "App", id.to_string())?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient field-level read instead of a strict `Policy` parse: a legacy app + // whose stored policy predates newer required fields must still resolve to + // its (unsandboxed) render here rather than erroring out of the viewer. + let policy = parse_embed_policy(&policy_str)?; + + let authed_for_token = if policy.anonymous_execution { + // Anonymous app: still mint a scoped token if the viewer happens to be + // logged in (so the app sees their identity), otherwise stay anonymous. + opt_authed + } else { + let authed = opt_authed.ok_or_else(|| { + Error::NotAuthorized( + "App visibility does not allow public access and you are not logged in".to_string(), + ) + })?; + let mut tx = user_db.begin(&authed).await?; + let is_visible = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM app WHERE id = $1 AND workspace_id = $2)", + id, + &w_id + ) + .fetch_one(&mut *tx) + .await?; + tx.commit().await?; + if !is_visible.unwrap_or(false) { + return Err(Error::NotAuthorized( + "App visibility does not allow public access and you are logged in but you have no read-access to that app".to_string(), + )); + } + Some(authed) + }; + + // The token is only consumed by the sandboxed low-code render. Raw apps + // render single-iframe with the page credential (WIN-2006 Variant A), and + // unsandboxed apps render same-origin with the viewer's own session — minting + // for those would write a useless token row per view and, worse, could fail + // the whole render for a scope-restricted caller (`ensure_scopes_within_caller`) + // even though no token is needed. The access check above still gates + // visibility in every case. + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, &app.path, authed_for_token.as_ref()).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(app.path); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + +/// Minimal, lenient view of an app policy for the embed-token endpoints +/// (WIN-2006). Reads only the fields the sandbox decision needs, via +/// `serde_json::Value`, so a legacy policy that no longer satisfies the strict +/// [`Policy`] struct (e.g. `triggerables_v2` entries predating now-required +/// fields) still renders instead of failing the viewer with "Not found". +/// A missing/unknown `execution_mode` is treated as NOT anonymous — the +/// strictest access interpretation. +pub struct EmbedPolicyView { + pub anonymous_execution: bool, + pub sandbox: bool, +} + +pub fn parse_embed_policy(policy_str: &str) -> Result { + let v: serde_json::Value = serde_json::from_str(policy_str).map_err(to_anyhow)?; + Ok(EmbedPolicyView { + anonymous_execution: v.get("execution_mode").and_then(|m| m.as_str()) == Some("anonymous"), + sandbox: v.get("sandbox").and_then(|b| b.as_bool()).unwrap_or(false), + }) +} + +/// Authenticated, path-based embed token for the in-workspace app viewer +/// (WIN-2006). Mirrors [`get_app_embed_token`] but keyed by app path and gated by +/// the caller's read access (RLS), so the logged-in `/apps/get` viewer can render +/// the app sandboxed — isolated from the member's full session — using the same +/// scoped token. Raw apps get no token (single-iframe with the page credential). +async fn get_app_embed_token_for_path( + authed: ApiAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, path)): Path<(String, StripPath)>, +) -> JsonResult { + let path = path.to_path(); + check_scopes(&authed, || format!("apps:read:{}", path))?; + // RLS: the caller must have read access to this app, otherwise it's not found. + let mut tx = user_db.begin(&authed).await?; + let app = sqlx::query!( + "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.path = $1 AND a.workspace_id = $2", + path, + &w_id + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + let app = not_found_if_none(app, "App", path)?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient parse + mint only for the sandboxed low-code render — see + // [`get_app_embed_token`] for the rationale (identical here). + let policy = parse_embed_policy(&policy_str)?; + + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, path, Some(&authed)).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(path.to_string()); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + async fn get_id_from_secret( db: &DB, w_id: &str, @@ -1245,8 +1723,6 @@ async fn create_app_raw<'a>( ) .await?; - check_scopes(&authed, || format!("apps:write:{}", path))?; - webhook.send_message( w_id.clone(), WebhookMessage::CreateApp { workspace: w_id, path: path.clone() }, @@ -1290,7 +1766,6 @@ async fn create_app( )); } let path = app.path.clone(); - check_scopes(&authed, || format!("apps:write:{}", &path))?; if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, @@ -1304,6 +1779,7 @@ async fn create_app( return Err(Error::PermissionDenied(msg)); } + // scope is enforced inside create_app_internal, before any persistence. let (new_tx, _path, _id) = create_app_internal(authed, db, user_db, &w_id, false, app).await?; new_tx.commit().await?; @@ -1342,6 +1818,50 @@ fn custom_path_conflict_error( } } +/// App values live in a `json` column, which — unlike `jsonb` — accepts the +/// `\u0000` escape. Any later `json`→`jsonb` conversion (a workspace fork's +/// `clone_apps`, search indexing, …) then aborts with "unsupported Unicode +/// escape sequence". Strip genuine NULs so the value is jsonb-safe before it +/// lands in the DB; the usual source is a binary file such as `.DS_Store` +/// accidentally bundled into a raw app's file map. A real NUL is unstorable +/// either way, and frontend code that needs the character writes it as the +/// source escape `\u0000`, which JSON-encodes to `\\u0000` (an escaped +/// backslash — the even-parity case below) and is left untouched. +/// +/// Returns `Cow::Borrowed` (no allocation) when the value is already clean. +fn strip_null_chars(raw: &str) -> Cow<'_, str> { + let bytes = raw.as_bytes(); + let mut out: Option = None; + let mut copied_to = 0; + let mut search_from = 0; + // A genuine NUL is `\u0000`: a `u0000` introduced by an *odd* run of + // backslashes. An even run (`\\u0000`) is an escaped backslash then the + // literal text "u0000" (common in minified JS regexes) and is preserved. + while let Some(rel) = raw[search_from..].find("u0000") { + let at = search_from + rel; + let mut backslashes = 0; + let mut j = at; + while j > 0 && bytes[j - 1] == b'\\' { + backslashes += 1; + j -= 1; + } + if backslashes % 2 == 1 { + // Drop the escaping backslash + `u0000` — the 6 chars in [at-1, at+5). + let out = out.get_or_insert_with(String::new); + out.push_str(&raw[copied_to..at - 1]); + copied_to = at + 5; + } + search_from = at + 5; + } + match out { + Some(mut out) => { + out.push_str(&raw[copied_to..]); + Cow::Owned(out) + } + None => Cow::Borrowed(raw), + } +} + async fn create_app_internal<'a>( authed: ApiAuthed, db: sqlx::Pool, @@ -1350,6 +1870,10 @@ async fn create_app_internal<'a>( raw_app: bool, mut app: CreateApp, ) -> Result<(sqlx::Transaction<'a, sqlx::Postgres>, String, i64)> { + // Enforce scope before any persistence: the raw-app create path commits + // inside process_app_multipart!, so checking after this call would leave a + // denied app committed in the DB. + check_scopes(&authed, || format!("apps:write:{}", &app.path))?; if *CLOUD_HOSTED { let nb_apps = sqlx::query_scalar!("SELECT COUNT(*) FROM app WHERE workspace_id = $1", &w_id) @@ -1367,7 +1891,9 @@ async fn create_app_internal<'a>( )); } } - let mut tx = user_db.clone().begin(&authed).await?; + // Resolve the on-behalf-of defaults on the (non-RLS) pool *before* opening + // the RLS transaction below: doing these lookups mid-transaction would hold + // a second simultaneous connection while `tx` is still checked out. let should_preserve = app.preserve_on_behalf_of.unwrap_or(false) && windmill_common::can_preserve_on_behalf_of(&authed) && app.policy.on_behalf_of.is_some(); @@ -1393,6 +1919,8 @@ async fn create_app_internal<'a>( app.policy.on_behalf_of_email = Some(authed.email.clone()); } } + + let mut tx = user_db.clone().begin(&authed).await?; let path = app.path.clone(); if &app.path == "" { return Err(Error::BadRequest("App path cannot be empty".to_string())); @@ -1477,13 +2005,18 @@ async fn create_app_internal<'a>( ) .fetch_one(&mut *tx) .await?; + // `.get()` keeps the raw text (and thus key order); strip any NUL so the + // `json`→`jsonb` conversion downstream (fork, indexing) can't choke on it. + let value = strip_null_chars(app.value.0.get()); + if matches!(value, Cow::Owned(_)) { + tracing::warn!(path = %app.path, "stripped NUL character(s) from app value on create"); + } let v_id = sqlx::query_scalar!( "INSERT INTO app_version (app_id, value, created_by, raw_app) VALUES ($1, $2::text::json, $3, $4) RETURNING id", id, - //to preserve key orders - serde_json::to_string(&app.value).unwrap(), + value.as_ref(), authed.username, raw_app ) @@ -1891,6 +2424,13 @@ async fn update_app_internal<'a>( ns: EditApp, ) -> Result<(sqlx::Transaction<'a, sqlx::Postgres>, String, i64)> { use sql_builder::prelude::*; + + // A rename moves the app to ns.path, so the destination must also be within + // the token's write scope, not just the source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("apps:write:{}", npath))?; + } + let mut tx = user_db.clone().begin(&authed).await?; let mut preserved_on_behalf_of: Option = None; @@ -2050,13 +2590,18 @@ async fn update_app_internal<'a>( .fetch_one(&mut *tx) .await?; + // `.get()` keeps the raw text (and thus key order); strip any NUL so the + // `json`→`jsonb` conversion downstream (fork, indexing) can't choke on it. + let value = strip_null_chars(nvalue.0.get()); + if matches!(value, Cow::Owned(_)) { + tracing::warn!(path = %npath, "stripped NUL character(s) from app value on update"); + } let v_id = sqlx::query_scalar!( "INSERT INTO app_version (app_id, value, created_by, raw_app) VALUES ($1, $2::text::json, $3, $4) RETURNING id", app_id, - //to preserve key orders - serde_json::to_string(&nvalue).unwrap(), + value.as_ref(), authed.username, raw_app ) @@ -2276,6 +2821,17 @@ async fn execute_component( Path((w_id, path)): Path<(String, StripPath)>, Json(mut payload): Json, ) -> Result { + let path = path.to_path(); + // Authorize FIRST, before touching the payload: confine the app embed token (the + // only credential handed to untrusted app JS, carrying `apps:run:`) to + // the app it was minted for. The route layer can't path-check the apps domain, so + // enforce it here. Scoped to embed tokens only — other callers (anonymous, cookie, + // plain external JWT) keep their existing access; the run is still policy-gated. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path))?; + } + } // Only honor temp_script_refs for the inline-script preview path: // preview/editor mode (force_viewer_static_fields set, == `is_preview`), // raw_code present, and no deployed app_script id — i.e. `wmill app dev`. @@ -2298,7 +2854,6 @@ async fn execute_component( _ => {} }; - let path = path.to_path(); let (arc_policy, policy): (Arc, Policy); let policy_triggerables_default = Default::default(); // Preview mode means the request was issued from the editor; the editing @@ -2762,6 +3317,15 @@ async fn upload_s3_file_from_app( Query(query): Query, request: axum::extract::Request, ) -> JsonResult { + // Confine an app embed token (untrusted app JS) to uploading for its OWN app. + // The route is reachable with `apps:run` (RUN_PATH_ACTIONS), so without this a + // token minted for app A could drive app B's upload policy. Mirrors + // execute_component / download_s3_file; other callers are unaffected. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path.to_path()))?; + } + } let policy = if let Some(file_key_regex) = query.force_viewer_file_key_regex { // `force_viewer_*` lets the caller supply a synthetic upload policy that // bypasses the deployed app's file_key_regex / resource restrictions. @@ -2796,6 +3360,7 @@ async fn upload_s3_file_from_app( .unwrap_or_default(), }]), allowed_s3_keys: None, + sandbox: None, }) } else { let policy_o = sqlx::query_scalar!( @@ -3038,7 +3603,49 @@ async fn upload_s3_file_from_app( ]) .into(); - let _put_result = upload_file_from_req(s3_client, &file_key, request, options).await?; + // Only workspace storage is quota-metered; a custom-resource upload lands in + // the user's own bucket and is neither capped nor counted. An overwrite of an + // existing key only spends the difference over its current size. + let _is_workspace_storage = query.s3_resource_path.is_none(); + #[cfg(all(feature = "parquet", not(feature = "enterprise")))] + if _is_workspace_storage { + reject_reserved_volume_key(&file_key)?; + } + #[cfg(all(feature = "parquet", not(feature = "enterprise")))] + let (max_size, _existing_size) = if _is_workspace_storage { + let content_length = request + .headers() + .get(http::header::CONTENT_LENGTH) + .and_then(|h| h.to_str().ok()) + .and_then(|s| s.parse::().ok()); + let budget = ce_upload_budget(&db, &w_id, &s3_client, &file_key, content_length).await?; + (Some(budget.max_size), budget.existing_size) + } else { + (None, 0) + }; + #[cfg(any(not(feature = "parquet"), feature = "enterprise"))] + let max_size: Option = None; + + match upload_file_from_req(s3_client, &file_key, request, options, max_size).await { + Ok((_, _size)) => + { + #[cfg(all(feature = "parquet", not(feature = "enterprise")))] + if _is_workspace_storage { + bump_storage_usage( + &db, + &w_id, + windmill_object_store::DEFAULT_STORAGE, + _size as i64 - _existing_size, + ) + .await; + } + } + Err(e) => { + #[cfg(all(feature = "parquet", not(feature = "enterprise")))] + spawn_storage_usage_recount_floored(&db, &w_id); + return Err(e); + } + } let delete_token = jwt::encode_with_internal_secret(S3DeleteTokenClaims { file_key: file_key.clone(), @@ -3155,6 +3762,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: Some(force_allowed_s3_keys), + sandbox: None, } } else { // TODO: improve db query to not return uneeded fields @@ -3177,6 +3785,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: None, + sandbox: None, }) }; @@ -3220,34 +3829,46 @@ async fn check_if_allowed_to_access_s3_file_from_app( return Err(Error::InternalErr( "Internal error: signature validation is not supported in open source mode".to_string(), )); - } else if opt_authed.is_some() { + } else if opt_authed.as_ref().is_some_and(|authed| { + !windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) + }) { + // A normal logged-in caller (editor / full session) may fetch any file they + // can reach. An app embed token also carries an identity but represents + // untrusted app JS, so it falls through to the allowlist below instead of + // this bypass — otherwise the app could read arbitrary S3 keys the + // viewer/on-behalf identity can see, beyond its own declared keys/outputs. Ok(()) } else { - let allowed = policy - .allowed_s3_keys + // Anonymous viewer, or an app embed token: confine to the app's declared S3 + // keys, or files produced by THIS app's own component runs. The producing + // identity is the embed viewer for a token, else `anonymous`. + let creator = opt_authed .as_ref() - .unwrap() - .iter() - .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) - || { - sqlx::query_scalar!( - r#"SELECT EXISTS ( + .map(|authed| authed.username.clone()) + .unwrap_or_else(|| "anonymous".to_string()); + let allowed = policy.allowed_s3_keys.as_ref().is_some_and(|keys| { + keys.iter() + .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) + }) || { + sqlx::query_scalar!( + r#"SELECT EXISTS ( SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id) WHERE j.workspace_id = $2 AND (j.kind = 'appscript' OR j.kind = 'preview') - AND j.created_by = 'anonymous' + AND j.created_by = $4 AND c.started_at > now() - interval '3 hours' AND j.runnable_path LIKE $3 || '/%' AND c.result @> ('{"s3":"' || $1 || '"}')::jsonb )"#, - file_query.s3, - w_id, - path, - ) - .fetch_one(db) - .await? - .unwrap_or(false) - }; + file_query.s3, + w_id, + path, + creator, + ) + .fetch_one(db) + .await? + .unwrap_or(false) + }; if !allowed { Err(Error::BadRequest("File restricted".to_string())) @@ -3286,6 +3907,15 @@ async fn download_s3_file_from_app( let path = path.to_path(); + // Authorize the app path first: a scoped caller (notably an app embed token, + // which carries `apps:read:`) may only download files for the app it + // was minted for — otherwise it could read another app's S3 files via that app's + // on-behalf policy. Unscoped sessions / anonymous callers pass through (the + // latter still gated by the policy allowlist in `check_if_allowed_...`). + if let Some(authed) = opt_authed.as_ref() { + check_scopes(authed, || format!("apps:read:{}", path))?; + } + let force_viewer_allowed_s3_keys = if let Some(force_viewer_allowed_s3_keys) = query.force_viewer_allowed_s3_keys.clone() { @@ -3562,3 +4192,312 @@ async fn build_args( job_id, )) } + +#[cfg(test)] +mod embed_token_tests { + use super::APP_EMBED_SCOPES; + use windmill_api_auth::scopes::check_scopes_for_route; + + /// The embed token must reach exactly the endpoints an app needs and nothing + /// else. This locks the allow/deny matrix that confines a malicious or + /// compromised app to app-only routes (WIN-2006). + #[test] + fn embed_scopes_allow_app_routes_and_deny_the_rest() { + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Mirror mint_app_embed_token: the per-app path-scoped read + run. + scopes.push("apps:read:u/admin/app".to_string()); + scopes.push("apps:run:u/admin/app".to_string()); + let scopes = Some(scopes.as_slice()); + + // Allowed: the routes a running app legitimately calls. + let allowed = [ + // Own definition + the public app-serving / execution endpoints. + ("/api/w/test/apps/get/p/u/admin/app", "GET"), + ("/api/w/test/apps_u/public_app/secret", "GET"), + ("/api/w/test/apps_u/get_data/v/secret.js", "GET"), + ("/api/w/test/apps_u/public_resource/f/app_themes/t", "GET"), + ("/api/w/test/apps_u/execute_component/u/admin/app", "POST"), + // S3 file upload from the app's S3 File Input component: a `run` action + // (RUN_PATH_ACTIONS) so the embed token reaches it; the handler re-checks + // `apps:run:` to confine it to this app, like execute_component. + ("/api/w/test/apps_u/upload_s3_file/u/admin/app", "POST"), + // By-id job poll routes (the JobLoader surface) stay allowed. + ("/api/w/test/jobs_u/get/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate_sse/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_result/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_timing/some-uuid", "GET"), + // By-id cancel (POST): permitted at the route layer; the handler confines + // it to the app's own jobs (created_by == viewer). + ("/api/w/test/jobs_u/queue/cancel/some-uuid", "POST"), + ("/api/w/test/users/whoami", "GET"), + // Resource METADATA only (picker list + type schemas) — never values. + ("/api/w/test/resources/list", "GET"), + ("/api/w/test/resources/exists/u/admin/r", "GET"), + ("/api/w/test/resources/type/list", "GET"), + ("/api/w/test/folders/listnames", "GET"), + ]; + for (path, method) in allowed { + assert!( + check_scopes_for_route(scopes, path, method).is_ok(), + "embed token should allow {method} {path}" + ); + } + + // Denied: anything outside what an app needs, including app management + // (apps:write is intentionally withheld), resource VALUE reads (which can + // hold credentials), and other workspace domains. + let denied = [ + ("/api/w/test/apps/update/u/admin/app", "POST"), + ("/api/w/test/apps/delete/u/admin/app", "DELETE"), + // Workspace app inventory must NOT be reachable (Apps domain is + // default-denied for the embed sentinel; only own-def + apps_u/* allowed). + ("/api/w/test/apps/exists/u/admin/app", "GET"), + ("/api/w/test/apps/custom_path_exists/foo", "GET"), + ( + "/api/w/test/apps/list_paths_from_workspace_runnable/script/u/admin/x", + "GET", + ), + ("/api/w/test/apps/list", "GET"), + // The embed-token MINT endpoints are public app routes (`apps_u/`) but + // create credentials — denied so a captured embed token can't renew + // itself indefinitely past the 12h expiry (refresh is the embedder's job). + ("/api/w/test/apps_u/embed_token/secret", "GET"), + ("/api/w/test/apps_u/embed_token_by_custom_path/foo", "GET"), + ("/api/w/test/scripts/list", "GET"), + ("/api/w/test/variables/list", "GET"), + ("/api/w/test/resources/update/u/admin/r", "POST"), + // Resource value reads must NOT be reachable with the embed token. + ("/api/w/test/resources/get/u/admin/r", "GET"), + ("/api/w/test/resources/get_value/u/admin/r", "GET"), + ( + "/api/w/test/resources/get_value_interpolated/u/admin/r", + "GET", + ), + ("/api/w/test/resources/list_search", "GET"), + // Workspace-wide job enumeration/export must NOT be reachable — an app + // reads only jobs it launched, by id (blocked via the app_embed sentinel). + ("/api/w/test/jobs/list", "GET"), + ("/api/w/test/jobs/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/completed/list", "GET"), + ("/api/w/test/jobs/completed/export", "GET"), + ("/api/w/test/jobs/queue/list", "GET"), + ("/api/w/test/jobs/queue/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/queue/export", "GET"), + // Job counts (workspace-wide aggregates) and the capability-minting + // routes (signed resume/approval URLs) are NOT by-id polling — denied. + ("/api/w/test/jobs/completed/count", "GET"), + ("/api/w/test/jobs/completed/count_jobs", "GET"), + ("/api/w/test/jobs/queue/count", "GET"), + ("/api/w/test/jobs/job_signature/some-uuid/some-rid", "GET"), + ("/api/w/test/jobs/resume_urls/some-uuid/some-rid", "GET"), + // get_root_job_id has no access check in its handler and the app never + // calls it — denied so the token can't probe foreign jobs' flow lineage. + ("/api/w/test/jobs_u/get_root_job_id/some-uuid", "GET"), + // `users:read`/`folders:read` exist only for whoami/listnames — every + // other route in those domains is denied via the app_embed sentinel + // (the whole /users and /folders routers are CORS-enabled for the iframe). + ("/api/w/test/users/list", "GET"), + ("/api/w/test/users/list_usage", "GET"), + ("/api/w/test/users/username_to_email/admin", "GET"), + ("/api/w/test/folders/list", "GET"), + ("/api/w/test/folders/get/myfolder", "GET"), + ("/api/w/test/folders/getusage/myfolder", "GET"), + ]; + for (path, method) in denied { + assert!( + check_scopes_for_route(scopes, path, method).is_err(), + "embed token should deny {method} {path}" + ); + } + } + + /// `apps:run` satisfies read at the route layer, so `apps/list` / `apps/list_search` + /// pass the route check — that's why those handlers ALSO call + /// `check_scopes(apps:read)`, which uses `ScopeDefinition::includes` (where run + /// does NOT include read). Lock that: no embed scope, including the + /// dynamically-minted path-scoped read, satisfies a domain-level `apps:read`, so + /// the token cannot list all apps' definitions (their full `value`/code). + #[test] + fn embed_scopes_cannot_satisfy_domain_app_read() { + use windmill_api_auth::scopes::ScopeDefinition; + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // mint_app_embed_token also grants read scoped to the single app path: + scopes.push("apps:read:u/admin/app".to_string()); + let required = ScopeDefinition::from_scope_string("apps:read").unwrap(); + for s in &scopes { + // The `app_embed` sentinel intentionally doesn't parse as a domain:action + // scope (it grants nothing; it only drives the job-enumeration deny). + let Ok(def) = ScopeDefinition::from_scope_string(s) else { + continue; + }; + assert!( + !def.includes(&required), + "embed scope {s} must not satisfy domain-level apps:read (would leak apps/list[_search])" + ); + } + // Sanity: a genuine domain-level apps:read token does satisfy it. + assert!(ScopeDefinition::from_scope_string("apps:read") + .unwrap() + .includes(&required)); + } + + /// The token carries path-scoped `apps:run:` and `apps:read:` + /// (NOT unqualified `apps:run`). Every handler that resolves an app and acts on + /// its behalf re-checks the requested path via `ScopeDefinition::includes`, so the + /// token is confined to its OWN app: + /// - `apps:run:` — `execute_component`. + /// - `apps:read:` — `get_app` (apps/get/p), `get_public_app_by_secret`, + /// the EE custom-path `get_public_app_by_custom_path`, and + /// `download_s3_file_from_app`. + /// This blocks cross-app execution, definition reads (by secret / custom path), + /// and S3 file reads through another app's on-behalf policy. + #[test] + fn embed_run_scope_is_path_scoped_to_its_app() { + use windmill_api_auth::scopes::ScopeDefinition; + // The mint must not grant unqualified run (which would include any path). + assert!( + !APP_EMBED_SCOPES.contains(&"apps:run"), + "embed scopes must not include unqualified apps:run" + ); + for action in ["run", "read"] { + let own = + ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")).unwrap(); + assert!( + own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")) + .unwrap() + ), + "apps:{action} must grant its own app" + ); + assert!( + !own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/other")) + .unwrap() + ), + "apps:{action} must NOT grant another app (cross-app)" + ); + } + } + + /// `mint_app_embed_token` guards its `create_token_internal` call with + /// `ensure_scopes_within_caller`, so a scope-restricted bearer token cannot + /// bootstrap the broader embed-scope set. Lock that boundary on the exact + /// scope vec the mint builds: rejected for a path-scoped caller, no-op for + /// the unscoped browser session that is the normal embed flow. + #[test] + fn embed_token_mint_is_scope_bounded() { + use windmill_api_auth::{ensure_scopes_within_caller, ApiAuthed}; + + // Same scope set mint_app_embed_token assembles for an app. + let mut minted: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + minted.push("apps:read:u/admin/app".to_string()); + + // A caller restricted to a single app read must not widen to the full + // embed set (apps:run, jobs:read, resources:read, ...). + let restricted = ApiAuthed { + scopes: Some(vec!["apps:read:u/admin/app".to_string()]), + ..Default::default() + }; + assert!( + ensure_scopes_within_caller(&restricted, Some(&minted)).is_err(), + "a path-scoped caller must not mint the broader embed-scope set" + ); + + // An unscoped session (the normal embed flow) passes — the mint only + // narrows. + let unscoped = ApiAuthed { scopes: None, ..Default::default() }; + assert!(ensure_scopes_within_caller(&unscoped, Some(&minted)).is_ok()); + } + + /// The embed-token endpoints must keep working for legacy apps whose stored + /// policy no longer satisfies the strict `Policy` struct (pre-dating + /// now-required fields): `parse_embed_policy` reads only the sandbox-decision + /// fields, leniently, and treats a missing/unknown `execution_mode` as NOT + /// anonymous (the strictest access interpretation). + #[test] + fn embed_policy_parse_is_lenient() { + use super::parse_embed_policy; + + // Quirky legacy policy: triggerables_v2 entry missing required fields, + // no execution_mode at all — must still parse, and absent `sandbox` + // resolves to the unsandboxed default. + let p = parse_embed_policy(r#"{"triggerables_v2": {"x": {}}}"#).unwrap(); + assert!(!p.sandbox); + assert!( + !p.anonymous_execution, + "missing execution_mode must not grant anonymous access" + ); + + // Normal policies map field-for-field. + let p = parse_embed_policy(r#"{"execution_mode": "anonymous", "sandbox": true}"#).unwrap(); + assert!(p.anonymous_execution); + assert!(p.sandbox); + + // Unknown execution_mode value: lenient parse, but not anonymous. + let p = parse_embed_policy(r#"{"execution_mode": "weird"}"#).unwrap(); + assert!(!p.anonymous_execution); + + // Invalid JSON still errors. + assert!(parse_embed_policy("not json").is_err()); + } +} + +#[cfg(test)] +mod strip_null_chars_tests { + use super::strip_null_chars; + use std::borrow::Cow; + + // Build `{"k":"u0000"}` without writing the escape literally + // (a real NUL can't live in Rust source). Odd n => the trailing `u0000` is a + // genuine NUL escape; even n => an escaped backslash then the text "u0000". + fn doc(backslashes: usize) -> String { + format!(r#"{{"k":"{}u0000"}}"#, "\\".repeat(backslashes)) + } + + #[test] + fn strips_genuine_null_escape() { + // 1 backslash: the NUL escape is dropped, the string value becomes "". + assert_eq!(strip_null_chars(&doc(1)).as_ref(), r#"{"k":""}"#); + // 3 backslashes: escaped backslash + NUL -> keep the escaped backslash. + let three = doc(3); + let out = strip_null_chars(&three); + assert_eq!(out.as_ref(), r#"{"k":"\\"}"#); + // Result is now valid, NUL-free JSON (i.e. jsonb-safe). + let v: serde_json::Value = serde_json::from_str(out.as_ref()).unwrap(); + assert!(!v["k"].as_str().unwrap().as_bytes().contains(&0u8)); + } + + #[test] + fn preserves_escaped_backslash_then_literal_u0000() { + // Even runs are the literal text "u0000" (e.g. a minified JS regex char + // class) and must be returned untouched, with no allocation. + for n in [2usize, 4] { + let s = doc(n); + let out = strip_null_chars(&s); + assert_eq!(out.as_ref(), s.as_str()); + assert!(matches!(out, Cow::Borrowed(_)), "n={n} should be borrowed"); + } + } + + #[test] + fn preserves_clean_values() { + // Plain value, and the bare token "u0000" with no preceding backslash. + for s in [r#"{"files":{"/index.tsx":"hello"}}"#, r#"{"k":"u0000"}"#] { + let out = strip_null_chars(s); + assert_eq!(out.as_ref(), s); + assert!(matches!(out, Cow::Borrowed(_))); + } + // The escape for a literal backslash char (`u005c`) then text "u0000": + // the only "u0000" match is preceded by `c` (0 backslashes) -> no NUL. + let s = format!(r#"{{"k":"{}u005cu0000"}}"#, "\\"); + assert!(matches!(strip_null_chars(&s), Cow::Borrowed(_))); + } + + #[test] + fn strips_multiple_and_preserves_surrounding() { + // Mirrors the .DS_Store case: several NULs interleaved with real text. + let s = format!(r#"{{"a":"x{b}u0000{b}u0000y","b":"ok"}}"#, b = "\\"); + assert_eq!(strip_null_chars(&s).as_ref(), r#"{"a":"xy","b":"ok"}"#); + } +} diff --git a/backend/windmill-api/src/args.rs b/backend/windmill-api/src/args.rs index 3793e94b02..dc8b36fbaf 100644 --- a/backend/windmill-api/src/args.rs +++ b/backend/windmill-api/src/args.rs @@ -90,6 +90,10 @@ impl RawWebhookArgs { db: &DB, w_id: &str, ) -> Result>, Error> { + #[cfg(not(feature = "enterprise"))] + use crate::job_helpers_oss::{ + bump_storage_usage, ce_storage_quota_remaining, spawn_storage_usage_recount_floored, + }; use crate::job_helpers_oss::{ get_random_file_name, get_workspace_s3_resource, upload_file_internal, }; @@ -139,8 +143,38 @@ impl RawWebhookArgs { .into_stream() .map_err(|err| std::io::Error::new(std::io::ErrorKind::Other, err)); - upload_file_internal(s3_client.clone(), &file_key, bytes_stream, options) - .await?; + // file_key is always freshly random here, so this never + // overwrites an existing object; the full size is the delta. + #[cfg(not(feature = "enterprise"))] + let max_size = Some(ce_storage_quota_remaining(db, w_id, None).await? as usize); + #[cfg(feature = "enterprise")] + let max_size: Option = None; + + match upload_file_internal( + s3_client.clone(), + &file_key, + bytes_stream, + options, + max_size, + ) + .await + { + Ok((_, _size)) => { + #[cfg(not(feature = "enterprise"))] + bump_storage_usage( + db, + w_id, + windmill_object_store::DEFAULT_STORAGE, + _size as i64, + ) + .await; + } + Err(e) => { + #[cfg(not(feature = "enterprise"))] + spawn_storage_usage_recount_floored(db, w_id); + return Err(e); + } + } files.entry(name).or_insert(vec![]).push(serde_json::json!({ "s3": &file_key diff --git a/backend/windmill-api/src/db.rs b/backend/windmill-api/src/db.rs index 7b8d468da8..8b9489263b 100644 --- a/backend/windmill-api/src/db.rs +++ b/backend/windmill-api/src/db.rs @@ -84,12 +84,26 @@ lazy_static::lazy_static! { (20260228000000, include_str!( "../../migrations/20260228000000_v2_job_completed_failure_index.up.sql" ).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")), + (20260610151334, include_str!( + "../../migrations/20260610151334_folder_labels.up.sql" + ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()), + (20260614075900, include_str!( + "../../migrations/20260614075900_dedup_folder_labels.up.sql" + ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()), ].into_iter().collect(); } pub struct CustomMigrator { inner: PoolConnection, } +impl CustomMigrator { + /// The connection the migrator already holds (with the migration advisory lock). + /// Migration housekeeping runs on it instead of re-acquiring: a second connection + /// while this one is held deadlocks a single-connection backend (e.g. embedded pglite). + pub fn connection(&mut self) -> &mut PgConnection { + &mut *self.inner + } +} impl Migrate for CustomMigrator { fn ensure_migrations_table( &mut self, @@ -266,7 +280,7 @@ pub async fn migrate( version=20250131115248 OR version=20250902085503 OR version=20250201145630 OR version=20250201145631 OR version=20250201145632 OR version=20251006143821" ) - .execute(db) + .execute(custom_migrator.connection()) .await { tracing::info!("Could not remove sqlx migrations: {err:#}"); @@ -288,6 +302,12 @@ pub async fn migrate( // idempotent, so re-applying on an already-migrated DB is a no-op. 20260423050000, 20260523055641, + // Reworked to stop reading pg_authid (via pg_has_role) from an elevated + // context, which managed providers (e.g. Cloud SQL) forbid — the original + // aborted startup. The new file is idempotent (CREATE OR REPLACE + an + // epoch-guarded UPDATE that no-ops once anchored), so re-applying on an + // already-migrated DB is safe. + 20260626132251, ]; for m in migrator.migrations.iter() { if m.migration_type.is_down_migration() { @@ -298,7 +318,7 @@ pub async fn migrate( sqlx::query("DELETE FROM _sqlx_migrations WHERE version = $1 AND checksum != $2") .bind(m.version) .bind(&*m.checksum) - .execute(db) + .execute(custom_migrator.connection()) .await { tracing::info!("Could not clean up stale migration {}: {err:#}", m.version); @@ -328,7 +348,7 @@ pub async fn migrate( } } - crate::live_migrations::custom_migrations(&mut custom_migrator, db).await?; + crate::live_migrations::custom_migrations(&mut custom_migrator).await?; Ok(None) } diff --git a/backend/windmill-api/src/db_health.rs b/backend/windmill-api/src/db_health.rs index 412c126dec..c487d29da9 100644 --- a/backend/windmill-api/src/db_health.rs +++ b/backend/windmill-api/src/db_health.rs @@ -96,8 +96,35 @@ pub struct ConnectionPoolInfo { pub pg_total_connections: i64, pub pg_active_connections: i64, pub pg_idle_connections: i64, + pub pg_superuser_reserved_connections: i64, pub status: HealthLevel, pub message: String, + /// Connection sizing guidance derived from the live Windmill fleet. + pub sizing: ConnectionSizingInfo, +} + +#[derive(Serialize)] +pub struct ConnectionSizingInfo { + /// Live DB-connected worker processes (distinct worker_instance pinged recently). + pub live_worker_instances: i64, + /// Live individual DB-connected workers across all instances. + pub live_workers: i64, + /// Live agent workers (HTTP-only, hold no postgres connections; excluded from the estimate). + pub live_agent_workers: i64, + /// Effective per-server pool ceiling: DATABASE_CONNECTIONS if set, else DEFAULT_MAX_CONNECTIONS_SERVER. + pub server_pool_size: i64, + /// Effective per-worker-instance pool ceiling (single-worker baseline; grows +1 per extra worker unless DATABASE_CONNECTIONS is set). + pub worker_pool_size: i64, + /// The DATABASE_CONNECTIONS override, if this server has one set (caps every process's pool). + pub database_connections_override: Option, + /// Estimated peak connections opened by all live worker instances. + pub estimated_worker_connections: i64, + /// Recommended max_connections floor (workers + one server + headroom). + pub recommended_max_connections: i64, + /// Per-additional-server increment to add to the recommendation. + pub per_server_increment: i64, + /// Human-readable sizing explanation. + pub message: String, } #[derive(Serialize)] @@ -379,6 +406,13 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result windmill_common::error::Result now() - interval '30 seconds'"#, + db_worker_pattern, + agent_worker_pattern, + ) + .fetch_one(db) + .await?; + + // Matches how db_connect.rs reads it: when DATABASE_CONNECTIONS is set it caps + // every process's pool (server, indexer, worker) regardless of worker count. + let database_connections_override = std::env::var("DATABASE_CONNECTIONS") + .ok() + .and_then(|n| n.parse::().ok()) + .filter(|n| *n > 0); + + let sizing = compute_connection_sizing( + fleet.live_workers, + fleet.live_instances, + fleet.live_agent_workers, + reserved, + database_connections_override, + ); + let pg_max = max_row; let pg_total = stats_row.total; let pg_active = stats_row.active; @@ -438,11 +507,105 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result, +) -> ConnectionSizingInfo { + // Never recommend below this floor: postgres defaults to 100 and headroom + // for growth/bursts/psql is cheap, so 200 is a safe baseline for any fleet. + const MIN_RECOMMENDED_MAX_CONNECTIONS: i64 = 200; + + let default_server_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_SERVER as i64; + let default_worker_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_WORKER as i64; + + let (server_pool, worker_pool_size, estimated_worker_connections) = + match database_connections_override { + // Override caps every process identically; per-instance pool is the override. + Some(n) => (n, n, n * live_instances), + None => ( + default_server_pool, + default_worker_pool, + (default_worker_pool - 1) * live_instances + live_workers, + ), + }; + + // Workers + one server, plus 20% headroom and the superuser reserve, so the + // recommendation leaves room for psql/monitoring sessions and bursts, then + // floored at MIN_RECOMMENDED_MAX_CONNECTIONS. + let base = estimated_worker_connections + server_pool; + let recommended = ((((base as f64) * 1.20).ceil() as i64) + reserved.max(3)) + .max(MIN_RECOMMENDED_MAX_CONNECTIONS); + + let pool_source = if database_connections_override.is_some() { + format!("DATABASE_CONNECTIONS={server_pool}") + } else { + "defaults, configurable via DATABASE_CONNECTIONS".to_string() + }; + + let message = if live_instances == 0 { + format!( + "No live workers detected. Each Windmill server and worker instance opens up to {server_pool} connections ({pool_source}). Size max_connections as (servers + worker instances) × {server_pool} + ~20% headroom, and at least {MIN_RECOMMENDED_MAX_CONNECTIONS}." + ) + } else { + let per_instance = if database_connections_override.is_some() { + format!("each instance up to {worker_pool_size}") + } else { + format!("each instance up to {worker_pool_size}, +1 per extra worker") + }; + let agent_note = if live_agent_workers > 0 { + format!( + " ({live_agent_workers} agent worker(s) excluded — they use HTTP, not postgres connections.)" + ) + } else { + String::new() + }; + format!( + "{live_workers} live worker(s) across {live_instances} instance(s) can open up to ~{estimated_worker_connections} connections ({per_instance}; {pool_source}). Each Windmill server adds up to {server_pool}. Recommended max_connections ≥ {recommended} for a single server; add {server_pool} per additional server.{agent_note}" + ) + }; + + ConnectionSizingInfo { + live_worker_instances: live_instances, + live_workers, + live_agent_workers, + server_pool_size: server_pool, + worker_pool_size, + database_connections_override, + estimated_worker_connections, + recommended_max_connections: recommended, + per_server_increment: server_pool, + message, + } +} + async fn fetch_table_maintenance( db: &DB, ) -> windmill_common::error::Result> { @@ -638,3 +801,91 @@ async fn fetch_datatables(db: &DB) -> windmill_common::error::Result 500 worker connections. + let s = compute_connection_sizing(20, 5, 0, 3, Some(100)); + assert_eq!(s.server_pool_size, 100); + assert_eq!(s.worker_pool_size, 100); + assert_eq!(s.database_connections_override, Some(100)); + assert_eq!(s.estimated_worker_connections, 500); + assert_eq!(s.per_server_increment, 100); + // ceil((500 + 100) * 1.20) + 3 = 720 + 3 = 723. + assert_eq!(s.recommended_max_connections, 723); + assert!(s.message.contains("DATABASE_CONNECTIONS=100")); + } + + #[test] + fn agent_workers_are_excluded_from_the_estimate() { + // 50 agent workers alongside 2 DB workers/2 instances: only the DB + // workers count toward connections; the agent count is reported. + let s = compute_connection_sizing(2, 2, 50, 3, None); + assert_eq!(s.live_agent_workers, 50); + // (5 - 1) * 2 + 2 = 10, agent workers contribute nothing. + assert_eq!(s.estimated_worker_connections, 10); + let without_agents = compute_connection_sizing(2, 2, 0, 3, None); + assert_eq!( + s.recommended_max_connections, + without_agents.recommended_max_connections + ); + assert!(s.message.contains("50 agent worker(s) excluded")); + } +} diff --git a/backend/windmill-api/src/docs/corpus.rs b/backend/windmill-api/src/docs/corpus.rs new file mode 100644 index 0000000000..a797879cfe --- /dev/null +++ b/backend/windmill-api/src/docs/corpus.rs @@ -0,0 +1,86 @@ +//! The vendored documentation snapshot, embedded into the binary and parsed once. +//! +//! `docs_snapshot/*.gz` are refreshed by `docs_snapshot/fetch.sh`. Embedding them +//! lets docs search work with no runtime egress (including air-gapped instances). + +use std::io::Read; +use std::sync::OnceLock; + +use flate2::read::GzDecoder; + +use super::search::{ + canonical_docs_page_url, canonical_search_url, parse_docs_full_text, parse_docs_index, + DocsFullPage, DocsIndexEntry, +}; + +const LLMS_FULL_GZ: &[u8] = + include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/docs_snapshot/llms-full.txt.gz")); +const LLMS_INDEX_GZ: &[u8] = + include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/docs_snapshot/llms.txt.gz")); + +pub struct DocsCorpus { + /// Every docs page, keyed by its `Source:` URL (from llms-full.txt). + pub pages: Vec, + /// The curated page index with one-line descriptions (from llms.txt). + pub index: Vec, +} + +impl DocsCorpus { + /// Finds the page whose `Source:` URL matches a model/CLI-supplied path or URL + /// after canonicalization (origin re-anchored, `.md` and ordering prefixes + /// stripped). + pub fn find_page(&self, path: &str) -> Option<&DocsFullPage> { + let key = canonical_search_url(&canonical_docs_page_url(path)); + self.pages.iter().find(|p| canonical_search_url(&p.url) == key) + } +} + +static CORPUS: OnceLock = OnceLock::new(); + +fn decompress(bytes: &[u8]) -> String { + let mut out = String::new(); + if let Err(e) = GzDecoder::new(bytes).read_to_string(&mut out) { + // The embedded snapshot is valid gzip text; a decode failure is a + // build-time packaging error, so failing closed to an empty corpus + // (docs search returns "no matches") is acceptable. + tracing::error!("failed to decompress embedded docs snapshot: {e}"); + return String::new(); + } + out +} + +/// Returns the parsed docs corpus, decompressing and parsing the embedded +/// snapshot once on first access. +pub fn corpus() -> &'static DocsCorpus { + CORPUS.get_or_init(|| { + let pages = parse_docs_full_text(&decompress(LLMS_FULL_GZ)); + let index = parse_docs_index(&decompress(LLMS_INDEX_GZ)); + DocsCorpus { pages, index } + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn embedded_corpus_parses_to_non_empty() { + let corpus = corpus(); + assert!(corpus.pages.len() > 50, "expected many pages, got {}", corpus.pages.len()); + assert!(corpus.index.len() > 50, "expected many index entries, got {}", corpus.index.len()); + assert!(corpus + .pages + .iter() + .all(|p| p.url.starts_with("https://www.windmill.dev/docs/") && !p.body.is_empty())); + } + + #[test] + fn find_page_matches_by_canonical_url() { + let corpus = corpus(); + // A page that is expected to exist in the published docs. + let by_path = corpus.find_page("/docs/core_concepts/worker_groups"); + let by_url = corpus.find_page("https://www.windmill.dev/docs/core_concepts/worker_groups.md"); + assert!(by_path.is_some()); + assert_eq!(by_path.map(|p| &p.url), by_url.map(|p| &p.url)); + } +} diff --git a/backend/windmill-api/src/docs/mod.rs b/backend/windmill-api/src/docs/mod.rs new file mode 100644 index 0000000000..cd903e593f --- /dev/null +++ b/backend/windmill-api/src/docs/mod.rs @@ -0,0 +1,124 @@ +//! Self-hosted documentation search. +//! +//! The backend embeds a vendored docs snapshot (see [`corpus`]) and exposes two +//! read-only endpoints over it, so docs search works with no runtime egress: +//! - `GET /api/docs/search?query=...` — full-text + index search +//! - `GET /api/docs/page?url=...§ion=...` — read one page (or a section) +//! +//! These back the AI chat `search_docs`/`read_docs_page` tools, the MCP +//! `searchDocs`/`readDocsPage` tools, and the `wmill docs` CLI. The routes are +//! nested behind the global authed service in `lib.rs`, so a valid token is +//! required but no workspace. + +mod corpus; +mod search; + +use axum::{extract::Query, routing::get, Json, Router}; +use serde::{Deserialize, Serialize}; +use windmill_common::error::JsonResult; + +use search::DocsSearchResult; + +pub fn global_service() -> Router { + Router::new() + .route("/search", get(search_docs)) + .route("/page", get(read_docs_page)) +} + +#[derive(Deserialize)] +struct SearchQuery { + query: String, +} + +#[derive(Serialize)] +struct SearchResponse { + /// Model-ready rendering of the results (the exact string the AI/MCP tool + /// returns). Built once here so every consumer is identical. + text: String, + /// Structured results for non-AI consumers (e.g. the CLI's pretty/`--json`). + results: Vec, +} + +#[derive(Deserialize)] +struct PageQuery { + /// A page's `Source` URL (as returned by the docs search tool); a bare `/docs/...` + /// path is also accepted and canonicalized before lookup. + url: String, + section: Option, +} + +#[derive(Serialize)] +struct PageResponse { + text: String, + source_url: String, +} + +async fn search_docs(Query(q): Query) -> JsonResult { + let query = q.query.trim().to_string(); + if query.is_empty() { + return Ok(Json(SearchResponse { + text: "No search query was provided. Provide a `query` of one or more keywords." + .to_string(), + results: Vec::new(), + })); + } + + // Lazy corpus init (gzip decompress + parse) and the per-query full-corpus scan + // are CPU-bound; keep them off the async runtime. + let (text, results) = tokio::task::spawn_blocking(move || { + let corpus = corpus::corpus(); + // Body grep first (concrete content hits), then index titles/descriptions to + // surface named features body grep misses; merge dedupes by canonical URL. + let body = search::search_docs_pages(&corpus.pages, &query, 5); + let index = search::search_docs_index(&corpus.index, &query, 4); + let results = search::merge_docs_search_results(body, index, search::SEARCH_MAX_PAGES); + let text = search::format_docs_search_results(&query, &results); + (text, results) + }) + .await + .map_err(|e| windmill_common::error::Error::InternalErr(format!("docs search task: {e}")))?; + + Ok(Json(SearchResponse { text, results })) +} + +async fn read_docs_page(Query(q): Query) -> JsonResult { + let url = q.url.trim(); + if url.is_empty() { + return Ok(Json(PageResponse { + text: "No documentation page URL was provided. Provide a `url` — e.g. a `Source` URL returned by the docs search tool.".to_string(), + source_url: String::new(), + })); + } + + let url = url.to_string(); + let section = q.section.filter(|s| !s.trim().is_empty()); + + // Corpus init + page sanitize/render are CPU-bound; keep them off the runtime. + let resp = tokio::task::spawn_blocking(move || { + let corpus = corpus::corpus(); + match corpus.find_page(&url) { + Some(page) => { + // Rewrite docusaurus source-file links to canonical published URLs + // so the model never echoes a broken `.mdx` path. + let sanitized = search::sanitize_docs_markdown_links(&page.body, &page.url); + let rendered = search::render_docs_page_result(&sanitized, section.as_deref()); + let text = format!( + "Source page — cite this URL when referencing this page: {}\n\n{}", + page.url, rendered + ); + PageResponse { text, source_url: page.url.clone() } + } + None => PageResponse { + text: format!( + "No documentation page found for \"{}\". Use the docs search tool to find the correct Source URL first.", + url + ), + source_url: search::canonical_docs_page_url(&url), + }, + } + }) + .await + .map_err(|e| windmill_common::error::Error::InternalErr(format!("docs page task: {e}")))?; + + Ok(Json(resp)) +} diff --git a/backend/windmill-api/src/docs/search.rs b/backend/windmill-api/src/docs/search.rs new file mode 100644 index 0000000000..733558ceb4 --- /dev/null +++ b/backend/windmill-api/src/docs/search.rs @@ -0,0 +1,729 @@ +//! Documentation search & page rendering. +//! +//! Direct port of the pure functions in the frontend's +//! `copilot/chat/docs/core.ts`, so the AI chat, the MCP `searchDocs`/`readDocsPage` +//! tools and the `wmill docs` CLI all return identical results from one place. +//! Operates over the vendored corpus parsed in [`super::corpus`]. + +use lazy_static::lazy_static; +use regex::Regex; +use serde::Serialize; +use std::collections::HashSet; + +pub const DOCS_ORIGIN: &str = "https://www.windmill.dev"; + +// Above this size, return an outline of the page's headings instead of the full +// content, prompting the model to request a specific section. +const FULL_PAGE_CHAR_LIMIT: usize = 20_000; + +// search result caps — keep the returned payload small (the whole point of search +// vs. dumping the index or full pages is token economy). +pub const SEARCH_MAX_PAGES: usize = 8; +const SEARCH_MAX_SNIPPETS_PER_PAGE: usize = 3; +const SEARCH_MAX_SNIPPET_CHARS: usize = 200; +// Each distinct query term triggers a full-corpus scan; cap it so a long, +// caller-controlled query can't multiply the scan cost without bound. Real +// queries are a handful of keywords, so this never truncates a useful search. +const MAX_QUERY_TERMS: usize = 24; + +// --------------------------------------------------------------------------- +// Corpus record types +// --------------------------------------------------------------------------- + +/// A single page extracted from llms-full.txt, keyed by its `Source:` URL. +pub struct DocsFullPage { + pub url: String, + pub title: String, + pub body: String, + /// `body` lowercased once at parse time, so the per-query full-corpus scan + /// doesn't re-allocate a lowercase copy of every page on each request. + pub body_lower: String, +} + +/// A line of the llms.txt index: title, URL and one-line description. +pub struct DocsIndexEntry { + pub title: String, + pub url: String, + pub description: String, + /// `title`/`description` lowercased once at parse time (same rationale as + /// `DocsFullPage::body_lower`). + pub title_lower: String, + pub description_lower: String, +} + +#[derive(Serialize, Clone)] +pub struct DocsSearchResult { + pub url: String, + pub title: String, + /// Higher = more relevant. Distinct query terms matched dominate raw occurrences. + pub score: i64, + pub snippets: Vec, +} + +// --------------------------------------------------------------------------- +// Corpus parsing +// --------------------------------------------------------------------------- + +lazy_static! { + // In llms-full.txt every page's `Source:` line is preceded by a category-header + // lead-in: `...page body...\n\n---\n\n## \n\nSource: `. Splitting + // on `Source:` lines leaves that lead-in on the *previous* page, so strip a + // trailing `---` + level-2-heading block to avoid mis-attributing the next + // page's category title to the previous page. + static ref TRAILING_LEAD_IN_RE: Regex = + Regex::new(r"\n+-{3,}[ \t]*\n+#{2}[ \t]+.*[ \t]*\n*$").unwrap(); + // A line in llms.txt: `- [Title](https://.../page.md): question-phrased description`. + static ref INDEX_ENTRY_RE: Regex = + Regex::new(r"^\s*-\s*\[([^\]]+)\]\(([^)\s]+)\)\s*:?\s*(.*)$").unwrap(); + static ref ORDERING_PREFIX_RE: Regex = Regex::new(r"^\d+[_-]").unwrap(); + // Markdown inline link `](target "optional title")`. + static ref MD_LINK_RE: Regex = Regex::new(r#"\]\(([^)\s]+?)(\s+"[^"]*")?\)"#).unwrap(); +} + +/// `^Source:\s*(\S+)\s*$` — returns the single non-whitespace URL token. +fn parse_source_line(line: &str) -> Option<&str> { + let rest = line.strip_prefix("Source:")?; + let trimmed = rest.trim(); + if trimmed.is_empty() || trimmed.contains(char::is_whitespace) { + return None; + } + Some(trimmed) +} + +/// Splits the llms-full.txt corpus into per-page records keyed by the `Source:` +/// URL. Content before the first `Source:` line (the corpus preamble) is dropped. +pub fn parse_docs_full_text(full_text: &str) -> Vec { + let mut pages = Vec::new(); + let mut url: Option = None; + let mut buffer: Vec<&str> = Vec::new(); + + for line in full_text.split('\n') { + if let Some(u) = parse_source_line(line) { + flush_page(&mut pages, &url, &buffer); + url = Some(u.to_string()); + buffer.clear(); + continue; + } + if url.is_some() { + buffer.push(line); + } + } + flush_page(&mut pages, &url, &buffer); + pages +} + +fn flush_page(pages: &mut Vec, url: &Option, buffer: &[&str]) { + let Some(url) = url else { + return; + }; + let joined = buffer.join("\n"); + let body = TRAILING_LEAD_IN_RE.replace(&joined, ""); + let body = body.trim(); + if !body.is_empty() { + let title = first_heading(body).unwrap_or_else(|| url.clone()); + let body = body.to_string(); + let body_lower = body.to_lowercase(); + pages.push(DocsFullPage { url: url.clone(), title, body, body_lower }); + } +} + +fn first_heading(body: &str) -> Option { + for line in body.split('\n') { + if let Some((_, title)) = parse_heading_line(line, 6) { + return Some(title); + } + } + None +} + +/// Parses the llms.txt index into per-page entries (title, URL, description). +pub fn parse_docs_index(index_text: &str) -> Vec { + let mut entries = Vec::new(); + for line in index_text.split('\n') { + if let Some(caps) = INDEX_ENTRY_RE.captures(line) { + let url = caps[2].trim(); + if !url.contains("/docs/") { + continue; + } + let title = caps[1].trim().to_string(); + let description = caps[3].trim().to_string(); + entries.push(DocsIndexEntry { + title_lower: title.to_lowercase(), + description_lower: description.to_lowercase(), + title, + url: url.to_string(), + description, + }); + } + } + entries +} + +// --------------------------------------------------------------------------- +// Headings, outline, section extraction +// --------------------------------------------------------------------------- + +pub struct DocsHeading { + pub level: usize, + pub title: String, + /// Byte offset of the start of the heading line within the document. + pub start_index: usize, +} + +/// `^(#{1,max_level})\s+(.*\S)\s*$` — heading level and trimmed title. +fn parse_heading_line(line: &str, max_level: usize) -> Option<(usize, String)> { + let hashes = line.bytes().take_while(|&b| b == b'#').count(); + if hashes < 1 || hashes > max_level { + return None; + } + let rest = &line[hashes..]; + // require at least one whitespace after the hashes (\s+) + if !rest.starts_with(|c: char| c.is_whitespace()) { + return None; + } + let title = rest.trim(); + if title.is_empty() { + return None; + } + Some((hashes, title.to_string())) +} + +fn match_fence(line: &str) -> Option { + let trimmed = line.trim_start(); + let first = trimmed.chars().next()?; + if first != '`' && first != '~' { + return None; + } + let count = trimmed.chars().take_while(|&c| c == first).count(); + if count >= 3 { + Some(std::iter::repeat(first).take(count).collect()) + } else { + None + } +} + +/// Parses the markdown headings (`#`–`####`) of a docs page, ignoring any +/// heading-like lines inside fenced code blocks (which are common in samples). +pub fn parse_docs_headings(content: &str) -> Vec { + let mut headings = Vec::new(); + let mut offset = 0usize; + let mut fence_marker: Option = None; + + for line in content.split('\n') { + if let Some(fence) = match_fence(line) { + match &fence_marker { + None => fence_marker = Some(fence), + Some(marker) if line.trim_start().starts_with(marker.as_str()) => { + fence_marker = None; + } + _ => {} + } + offset += line.len() + 1; + continue; + } + + if fence_marker.is_none() { + if let Some((level, title)) = parse_heading_line(line, 4) { + headings.push(DocsHeading { level, title, start_index: offset }); + } + } + offset += line.len() + 1; + } + headings +} + +fn section_end_index(content: &str, headings: &[DocsHeading], index: usize) -> usize { + let level = headings[index].level; + // A section ends at the next heading of the same or higher (shallower) level. + for h in &headings[index + 1..] { + if h.level <= level { + return h.start_index; + } + } + content.len() +} + +/// Builds a human-readable outline of a page's headings, with an approximate +/// size for each section. Used when a page is too large to return whole. +pub fn build_docs_outline(content: &str) -> String { + let headings = parse_docs_headings(content); + if headings.is_empty() { + return "(no markdown headings found on this page)".to_string(); + } + headings + .iter() + .enumerate() + .map(|(i, h)| { + let end = section_end_index(content, &headings, i); + let approx = end.saturating_sub(h.start_index); + let indent = " ".repeat(h.level.saturating_sub(1)); + format!("{}- {} (~{} chars)", indent, h.title, approx) + }) + .collect::>() + .join("\n") +} + +/// Normalizes a heading title for tolerant, case/punctuation-insensitive matching. +fn normalize_heading_title(title: &str) -> String { + // toLowerCase, replace /[^a-z0-9]+/g with ' ', trim + let mut out = String::new(); + let mut prev_space = false; + for ch in title.to_lowercase().chars() { + if ch.is_ascii_alphanumeric() { + out.push(ch); + prev_space = false; + } else if !prev_space { + out.push(' '); + prev_space = true; + } + } + out.trim().to_string() +} + +/// Extracts the content of the section whose heading matches `section` (from the +/// heading up to the next heading of the same/higher level). Case-insensitive and +/// tolerant of minor punctuation differences. `None` when no heading matches. +pub fn extract_docs_section(content: &str, section: &str) -> Option { + let headings = parse_docs_headings(content); + let target = normalize_heading_title(section); + if target.is_empty() { + return None; + } + let match_index = headings + .iter() + .position(|h| normalize_heading_title(&h.title) == target) + .or_else(|| { + // Fall back to a contains match so "Result streaming" matches "Result". + headings + .iter() + .position(|h| normalize_heading_title(&h.title).contains(&target)) + })?; + + let start = headings[match_index].start_index; + let end = section_end_index(content, &headings, match_index); + Some(content[start..end].trim().to_string()) +} + +/// Decides what to return for read_docs_page: a requested section, the full page, +/// or an outline asking the model to pick a section. +pub fn render_docs_page_result(content: &str, section: Option<&str>) -> String { + if let Some(section) = section { + if let Some(extracted) = extract_docs_section(content, section) { + return extracted; + } + return format!( + "No section matching \"{}\" was found on this page. Available sections:\n\n{}", + section, + build_docs_outline(content) + ); + } + + if content.len() <= FULL_PAGE_CHAR_LIMIT { + return content.to_string(); + } + + format!( + "This documentation page is large. Below is its list of sections with approximate sizes.\n\ + Call the docs page-reading tool again with the same `url` argument and a `section` set to one of these headings to read that section.\n\n{}", + build_docs_outline(content) + ) +} + +// --------------------------------------------------------------------------- +// Ranking +// --------------------------------------------------------------------------- + +/// Splits a query into distinct, lowercased, non-empty terms (insertion order), +/// capped at `MAX_QUERY_TERMS`. +fn tokenize_query(query: &str) -> Vec { + let mut seen = HashSet::new(); + let mut out = Vec::new(); + for term in query.to_lowercase().split_whitespace() { + if seen.insert(term.to_string()) { + out.push(term.to_string()); + if out.len() >= MAX_QUERY_TERMS { + break; + } + } + } + out +} + +fn count_occurrences(haystack: &str, needle: &str) -> usize { + if needle.is_empty() { + return 0; + } + haystack.matches(needle).count() +} + +struct Scored { + res: DocsSearchResult, + distinct_terms: usize, + order: usize, +} + +/// Prefer pages that cover every query term; sort by score desc then input order; +/// take the top `max_pages`. +fn finalize_pool(mut scored: Vec, term_count: usize, max_pages: usize) -> Vec { + let has_full = scored.iter().any(|s| s.distinct_terms == term_count); + if has_full { + scored.retain(|s| s.distinct_terms == term_count); + } + scored.sort_by(|a, b| b.res.score.cmp(&a.res.score).then(a.order.cmp(&b.order))); + scored.into_iter().take(max_pages).map(|s| s.res).collect() +} + +/// Ranks docs pages for a keyword query. Score is `distinctTermsMatched` +/// (dominant) then total occurrences. Each result carries up to +/// `SEARCH_MAX_SNIPPETS_PER_PAGE` of its most term-dense lines. +pub fn search_docs_pages(pages: &[DocsFullPage], query: &str, max_pages: usize) -> Vec { + let terms = tokenize_query(query); + if terms.is_empty() { + return Vec::new(); + } + + let mut scored = Vec::new(); + for (order, page) in pages.iter().enumerate() { + let mut distinct = 0usize; + let mut occurrences = 0usize; + for term in &terms { + let count = count_occurrences(&page.body_lower, term); + if count > 0 { + distinct += 1; + occurrences += count; + } + } + if distinct == 0 { + continue; + } + scored.push(Scored { + res: DocsSearchResult { + url: page.url.clone(), + title: page.title.clone(), + // distinctTerms dominates so a page matching all terms always + // outranks one matching fewer, regardless of raw occurrences. + score: distinct as i64 * 1_000_000 + occurrences as i64, + snippets: select_snippets( + &page.body, + &terms, + SEARCH_MAX_SNIPPETS_PER_PAGE, + SEARCH_MAX_SNIPPET_CHARS, + ), + }, + distinct_terms: distinct, + order, + }); + } + finalize_pool(scored, terms.len(), max_pages) +} + +/// Ranks index entries by matching query terms against each entry's title and +/// description (title matches weigh more). The description becomes the result's +/// single snippet. Recovers "named feature" discovery that full-text grep misses. +pub fn search_docs_index(entries: &[DocsIndexEntry], query: &str, max_pages: usize) -> Vec { + let terms = tokenize_query(query); + if terms.is_empty() { + return Vec::new(); + } + + let mut scored = Vec::new(); + for (order, entry) in entries.iter().enumerate() { + let mut distinct = 0usize; + let mut score = 0i64; + for term in &terms { + let in_title = entry.title_lower.contains(term.as_str()); + let in_desc = entry.description_lower.contains(term.as_str()); + if in_title || in_desc { + distinct += 1; + score += if in_title { 5 } else { 0 } + if in_desc { 1 } else { 0 }; + } + } + if distinct == 0 { + continue; + } + scored.push(Scored { + res: DocsSearchResult { + url: entry.url.clone(), + title: entry.title.clone(), + score: distinct as i64 * 1_000_000 + score, + snippets: if entry.description.is_empty() { + Vec::new() + } else { + vec![entry.description.clone()] + }, + }, + distinct_terms: distinct, + order, + }); + } + finalize_pool(scored, terms.len(), max_pages) +} + +/// Picks the most term-dense lines of a page body as snippets, in document order, +/// deduped, each trimmed to `max_chars` around the first matched term. +fn select_snippets(body: &str, terms: &[String], max_snippets: usize, max_chars: usize) -> Vec { + struct LineHit { + text: String, + distinct: usize, + order: usize, + } + let mut hits = Vec::new(); + for (order, line) in body.split('\n').enumerate() { + let lower = line.to_lowercase(); + let distinct = terms.iter().filter(|t| lower.contains(t.as_str())).count(); + if distinct == 0 { + continue; + } + let text = make_snippet(line, terms, max_chars); + if !text.is_empty() { + hits.push(LineHit { text, distinct, order }); + } + } + hits.sort_by(|a, b| b.distinct.cmp(&a.distinct).then(a.order.cmp(&b.order))); + + let mut seen = HashSet::new(); + let mut result = Vec::new(); + for hit in hits { + if !seen.insert(hit.text.clone()) { + continue; + } + result.push(hit.text); + if result.len() >= max_snippets { + break; + } + } + result +} + +/// Collapses a matched line to a single-line snippet of at most `max_chars`, +/// windowed around the first matched term (with ellipses) when the line is long. +/// Operates on `char`s so multibyte content can't split mid-codepoint. +fn make_snippet(line: &str, terms: &[String], max_chars: usize) -> String { + let collapsed = line.split_whitespace().collect::>().join(" "); + let chars: Vec = collapsed.chars().collect(); + if chars.len() <= max_chars { + return collapsed; + } + + let lower = collapsed.to_lowercase(); + let mut first_index: Option = None; + for term in terms { + if let Some(byte_idx) = lower.find(term.as_str()) { + let char_idx = lower[..byte_idx].chars().count(); + first_index = Some(first_index.map_or(char_idx, |f| f.min(char_idx))); + } + } + + let total = chars.len(); + match first_index { + None => { + let slice: String = chars[..max_chars].iter().collect(); + format!("{}…", slice.trim_end()) + } + Some(fi) => { + let start = fi.saturating_sub(max_chars / 3); + let end = (start + max_chars).min(total); + let prefix = if start > 0 { "…" } else { "" }; + let suffix = if end < total { "…" } else { "" }; + let slice: String = chars[start..end].iter().collect(); + format!("{}{}{}", prefix, slice.trim(), suffix) + } + } +} + +/// Strips the `.md` suffix and trailing slash so index/body URLs dedupe. +pub fn canonical_search_url(url: &str) -> String { + let stripped = strip_md_suffix(url); + stripped.strip_suffix('/').unwrap_or(stripped).to_string() +} + +fn strip_md_suffix(s: &str) -> &str { + if s.len() >= 3 && s[s.len() - 3..].eq_ignore_ascii_case(".md") { + &s[..s.len() - 3] + } else { + s + } +} + +/// Merges full-text (body) results with index-description results. Body matches +/// come first; index-only matches fill remaining slots — so a named feature +/// surfaced only by its index entry still appears even when body grep missed it. +pub fn merge_docs_search_results( + body_results: Vec, + index_results: Vec, + max_pages: usize, +) -> Vec { + let mut seen: HashSet = + body_results.iter().map(|r| canonical_search_url(&r.url)).collect(); + let mut merged = body_results; + for entry in index_results { + let key = canonical_search_url(&entry.url); + if seen.insert(key) { + merged.push(entry); + } + } + merged.truncate(max_pages); + merged +} + +/// Renders search results as the string returned to the model. +pub fn format_docs_search_results(query: &str, results: &[DocsSearchResult]) -> String { + if results.is_empty() { + return format!( + "No documentation pages matched \"{}\". Try fewer or more general keywords (a single distinctive term often works best).", + query + ); + } + + let blocks = results + .iter() + .map(|r| { + let mut lines = vec![format!("## {}", r.title), format!("Source: {}", r.url)]; + for snippet in &r.snippets { + lines.push(format!(" - {}", snippet)); + } + lines.join("\n") + }) + .collect::>() + .join("\n\n"); + + format!( + "Found {} documentation page(s) matching \"{}\", most relevant first:\n\n{}\n\n\ + Cite the exact \"Source\" URL when referencing a page. If these snippets are not enough, call the docs page-reading tool with a Source URL as its `url` argument to read the full page or a section.", + results.len(), + query, + blocks + ) +} + +// --------------------------------------------------------------------------- +// URL normalization & link sanitization +// --------------------------------------------------------------------------- + +/// Strips docusaurus numeric ordering prefixes (`13_`, `8-`) from each path +/// segment so it matches the published route. +fn strip_docs_path_prefixes(path: &str) -> String { + path.split('/') + .map(|seg| ORDERING_PREFIX_RE.replace(seg, "").into_owned()) + .collect::>() + .join("/") +} + +/// Normalizes a user/model-supplied docs reference to a fully-qualified `.md` URL +/// on the docs origin. Accepts a full URL, `/docs/...`, or `docs/...`. +pub fn normalize_docs_url(input: &str) -> String { + let mut value = input.trim().to_string(); + + if is_http_url(&value) { + // Strip the origin so we re-anchor to DOCS_ORIGIN and normalize the path. + if let Ok(parsed) = url::Url::parse(&value) { + value = parsed.path().to_string(); + } + } + + // Drop any query string or hash fragment. + value = value.split('#').next().unwrap().split('?').next().unwrap().to_string(); + + if !value.starts_with('/') { + value = format!("/{}", value); + } + // Strip a trailing slash (but keep the leading one). + if value.len() > 1 && value.ends_with('/') { + value.pop(); + } + + value = strip_docs_path_prefixes(&value); + if let Some(stripped) = value.strip_suffix(".mdx") { + value = format!("{}.md", stripped); + } + if !value.ends_with(".md") { + value = format!("{}.md", value); + } + + format!("{}{}", DOCS_ORIGIN, value) +} + +/// The canonical published URL a model should cite for a docs page (the `.md` +/// fetch URL without the suffix). +pub fn canonical_docs_page_url(path: &str) -> String { + strip_md_suffix(&normalize_docs_url(path)).to_string() +} + +fn is_http_url(s: &str) -> bool { + let lower = s.to_ascii_lowercase(); + lower.starts_with("http://") || lower.starts_with("https://") +} + +/// Rewrites relative/source-file doc links inside raw page markdown to canonical +/// published URLs, so the model never echoes a docusaurus source path into its +/// answer as a broken link. Non-doc links (external, images, anchors) and `../` +/// cross-directory links are left untouched. +pub fn sanitize_docs_markdown_links(content: &str, page_url: &str) -> String { + let base = url::Url::parse(page_url).ok(); + MD_LINK_RE + .replace_all(content, |caps: ®ex::Captures| { + let whole = caps.get(0).unwrap().as_str(); + let target = &caps[1]; + let title = caps.get(2).map(|m| m.as_str()).unwrap_or(""); + + // Only rewrite links to docusaurus source files (.md/.mdx); leave + // images, external URLs and bare anchors untouched. + if !is_md_target(target) { + return whole.to_string(); + } + // `../` cross-directory links are authored against the docusaurus + // source tree, whose depth differs from the published URL, so strict + // resolution is unreliable. Leave them for the canonical-URL header. + if has_parent_traversal(target) { + return whole.to_string(); + } + let Some(base) = &base else { + return whole.to_string(); + }; + let Ok(resolved) = base.join(target) else { + return whole.to_string(); + }; + if resolved.scheme() != "https" + || resolved.host_str() != Some("www.windmill.dev") + || !resolved.path().starts_with("/docs/") + { + return whole.to_string(); + } + let pathname = strip_docs_path_prefixes(resolved.path()); + let pathname = strip_md_or_mdx(&pathname); + let hash = resolved.fragment().map(|f| format!("#{}", f)).unwrap_or_default(); + format!("]({}{}{}{})", DOCS_ORIGIN, pathname, hash, title) + }) + .into_owned() +} + +/// `\.mdx?($|[#?])` — the target points at a markdown source file. +fn is_md_target(target: &str) -> bool { + for marker in [".md", ".mdx"] { + if let Some(idx) = target.to_ascii_lowercase().find(marker) { + let after = &target[idx + marker.len()..]; + // `.md` must not be a prefix of `.mdx` here: only accept when the + // extension is followed by end / `#` / `?`. + if after.is_empty() || after.starts_with('#') || after.starts_with('?') { + return true; + } + } + } + false +} + +/// `(^|/)\.\./` — the target contains a parent-directory traversal segment. +fn has_parent_traversal(target: &str) -> bool { + target == ".." || target.starts_with("../") || target.contains("/../") +} + +fn strip_md_or_mdx(path: &str) -> &str { + if let Some(stripped) = path.strip_suffix(".mdx") { + stripped + } else { + strip_md_suffix(path) + } +} + +#[cfg(test)] +mod tests; diff --git a/backend/windmill-api/src/docs/search/tests.rs b/backend/windmill-api/src/docs/search/tests.rs new file mode 100644 index 0000000000..84c422aa73 --- /dev/null +++ b/backend/windmill-api/src/docs/search/tests.rs @@ -0,0 +1,267 @@ +//! Parity tests ported from the frontend `copilot/chat/docs/core.test.ts`. + +use super::*; + +const SAMPLE: &str = "# Jobs\n\nIntro text about jobs.\n\n## Job kinds\n\nSome kinds.\n\n## Result\n\n### Result of jobs that failed\n\n```\n{ \"error\": \"boom\" }\n```\n\n### Result streaming\n\n#### Returning a stream directly\n\n```python\n# Returning a stream directly is a comment heading that must be ignored\ndef main():\n pass\n```\n\n## Retention policy\n\nFinal section.\n"; + +// Mirrors the llms-full.txt layout: a corpus preamble, then per-page blocks each +// introduced by a `---` + `## ` lead-in followed by a `Source:` line. +const SAMPLE_FULL: &str = "# Windmill\n\n> Preamble blurb that precedes the first Source line and must be ignored.\n\n## Browser automation\n\nSource: https://www.windmill.dev/docs/advanced/browser_automation\n\n# Browser automation\n\nBy default, a worker group named `reports` handles jobs with the `chromium` tag.\nThe chromium binary will be available on these workers at /usr/bin/chromium.\nYou can disable the sandbox by passing the --no-sandbox flag.\n\n---\n\n## Worker groups\n\nSource: https://www.windmill.dev/docs/core_concepts/worker_groups\n\n# Worker groups\n\nWorker groups let you assign tags to workers.\nSet the chromium tag on a worker so it can run browser jobs.\n\n---\n\n## Scheduling\n\nSource: https://www.windmill.dev/docs/core_concepts/scheduling\n\n# Scheduling\n\nUse cron expressions to schedule scripts and flows.\n"; + +#[test] +fn parses_headings_and_ignores_fenced_blocks() { + let titles: Vec = parse_docs_headings(SAMPLE) + .iter() + .map(|h| format!("{}:{}", h.level, h.title)) + .collect(); + assert_eq!( + titles, + vec![ + "1:Jobs", + "2:Job kinds", + "2:Result", + "3:Result of jobs that failed", + "3:Result streaming", + "4:Returning a stream directly", + "2:Retention policy", + ] + ); +} + +#[test] +fn heading_start_index_points_at_the_heading_line() { + for h in parse_docs_headings(SAMPLE) { + let at = &SAMPLE[h.start_index..]; + assert!(at.starts_with(&"#".repeat(h.level))); + assert!(at[h.level..].trim_start().starts_with(&h.title)); + } +} + +#[test] +fn handles_tilde_fences() { + let content = "# Title\n\n~~~\n# not a heading\n~~~\n\n## Real\n"; + let titles: Vec = parse_docs_headings(content).iter().map(|h| h.title.clone()).collect(); + assert_eq!(titles, vec!["Title", "Real"]); +} + +#[test] +fn extracts_section_up_to_next_same_or_higher_heading() { + let section = extract_docs_section(SAMPLE, "Result").unwrap(); + assert!(section.contains("## Result")); + assert!(section.contains("### Result of jobs that failed")); + assert!(section.contains("### Result streaming")); + assert!(!section.contains("## Retention policy")); +} + +#[test] +fn extract_section_is_case_and_punctuation_tolerant() { + let section = extract_docs_section(SAMPLE, "retention-policy!").unwrap(); + assert!(section.contains("## Retention policy")); + assert!(section.contains("Final section.")); +} + +#[test] +fn extract_section_returns_none_when_missing() { + assert!(extract_docs_section(SAMPLE, "Nonexistent section").is_none()); +} + +#[test] +fn build_outline_lists_headings_with_indent() { + let outline = build_docs_outline(SAMPLE); + assert!(outline.contains("- Jobs (~")); + assert!(outline.contains(" - Job kinds (~")); + assert!(outline.contains(" - Result of jobs that failed (~")); +} + +#[test] +fn build_outline_handles_no_headings() { + assert_eq!( + build_docs_outline("just some text\nwith no headings"), + "(no markdown headings found on this page)" + ); +} + +#[test] +fn render_page_returns_whole_small_page() { + assert_eq!(render_docs_page_result(SAMPLE, None), SAMPLE); +} + +#[test] +fn render_page_returns_outline_for_large_page() { + let large = format!("# Big\n\n{}\n\n## Tail\n\nmore", "x".repeat(25_000)); + let result = render_docs_page_result(&large, None); + assert!(result.contains("This documentation page is large")); + assert!(result.contains("same `url` argument")); + assert!(!result.contains("same path")); + assert!(!result.contains("read_docs_page")); + assert!(result.contains("- Big (~")); + assert!(result.contains("- Tail (~")); +} + +#[test] +fn render_page_returns_requested_section() { + let result = render_docs_page_result(SAMPLE, Some("Job kinds")); + assert!(result.contains("## Job kinds")); + assert!(result.contains("Some kinds.")); +} + +#[test] +fn render_page_missing_section_returns_outline_note() { + let result = render_docs_page_result(SAMPLE, Some("Does not exist")); + assert!(result.contains("No section matching \"Does not exist\" was found")); + assert!(result.contains("- Jobs (~")); +} + +#[test] +fn normalize_docs_url_cases() { + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("docs/core_concepts/jobs"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!( + normalize_docs_url("https://www.windmill.dev/docs/core_concepts/jobs#result?foo=bar"), + "https://www.windmill.dev/docs/core_concepts/jobs.md" + ); + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs.md"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs/"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("/docs/flows/13_flow_branches"), "https://www.windmill.dev/docs/flows/flow_branches.md"); + assert_eq!(normalize_docs_url("/docs/flows/13_flow_branches.mdx"), "https://www.windmill.dev/docs/flows/flow_branches.md"); +} + +#[test] +fn canonical_docs_page_url_cases() { + assert_eq!(canonical_docs_page_url("/docs/flows/flow_editor"), "https://www.windmill.dev/docs/flows/flow_editor"); + assert_eq!(canonical_docs_page_url("/docs/flows/14_retries.md"), "https://www.windmill.dev/docs/flows/retries"); +} + +#[test] +fn sanitize_markdown_links_cases() { + let page = "https://www.windmill.dev/docs/flows/flow_editor.md"; + assert_eq!( + sanitize_docs_markdown_links("See [retries](./14_retries.mdx) for more.", page), + "See [retries](https://www.windmill.dev/docs/flows/retries) for more." + ); + assert_eq!( + sanitize_docs_markdown_links("[handling](./8_error_handling.mdx)", page), + "[handling](https://www.windmill.dev/docs/flows/error_handling)" + ); + assert_eq!( + sanitize_docs_markdown_links("[branch all](./13_flow_branches.mdx#branch-all)", page), + "[branch all](https://www.windmill.dev/docs/flows/flow_branches#branch-all)" + ); + // images & external links untouched + let external = "![diagram](./assets/flow_example.png) and [site](https://example.com/page.md)"; + assert_eq!(sanitize_docs_markdown_links(external, page), external); + // bare anchor untouched + assert_eq!(sanitize_docs_markdown_links("[top](#introduction)", page), "[top](#introduction)"); + // ../ cross-directory links untouched + let parent = "[handling](../core_concepts/8_error_handling.mdx)"; + assert_eq!(sanitize_docs_markdown_links(parent, page), parent); + let parent2 = "[retries](../../flows/14_retries.md)"; + assert_eq!(sanitize_docs_markdown_links(parent2, page), parent2); +} + +#[test] +fn parse_full_text_splits_pages_and_drops_preamble() { + let pages = parse_docs_full_text(SAMPLE_FULL); + assert_eq!( + pages.iter().map(|p| p.url.clone()).collect::>(), + vec![ + "https://www.windmill.dev/docs/advanced/browser_automation", + "https://www.windmill.dev/docs/core_concepts/worker_groups", + "https://www.windmill.dev/docs/core_concepts/scheduling", + ] + ); + assert_eq!( + pages.iter().map(|p| p.title.clone()).collect::>(), + vec!["Browser automation", "Worker groups", "Scheduling"] + ); + // The next page's "## Worker groups" lead-in must not leak into this body. + let browser = pages.iter().find(|p| p.url.ends_with("/browser_automation")).unwrap(); + assert!(!browser.body.contains("Worker groups")); + assert!(!browser.body.contains("---")); +} + +#[test] +fn search_pages_ranks_more_occurrences_first() { + let pages = parse_docs_full_text(SAMPLE_FULL); + let results = search_docs_pages(&pages, "chromium", 5); + assert_eq!( + results.iter().map(|r| r.url.clone()).collect::>(), + vec![ + "https://www.windmill.dev/docs/advanced/browser_automation", + "https://www.windmill.dev/docs/core_concepts/worker_groups", + ] + ); + assert!(!results[0].snippets.is_empty()); + assert!(results[0].snippets.join("\n").contains("chromium")); +} + +#[test] +fn search_prefers_pages_covering_all_terms() { + let pages = parse_docs_full_text(SAMPLE_FULL); + // Only browser_automation contains "sandbox"; worker_groups has "chromium" but + // not "sandbox". A full-coverage page exists, so partial matches are dropped. + let results = search_docs_pages(&pages, "chromium sandbox", 5); + assert_eq!( + results.iter().map(|r| r.url.clone()).collect::>(), + vec!["https://www.windmill.dev/docs/advanced/browser_automation"] + ); +} + +#[test] +fn merge_dedupes_index_against_body_by_canonical_url() { + let body = vec![DocsSearchResult { + url: "https://www.windmill.dev/docs/a".to_string(), + title: "A".to_string(), + score: 10, + snippets: vec![], + }]; + let index = vec![ + DocsSearchResult { + url: "https://www.windmill.dev/docs/a.md".to_string(), + title: "A".to_string(), + score: 5, + snippets: vec![], + }, + DocsSearchResult { + url: "https://www.windmill.dev/docs/b.md".to_string(), + title: "B".to_string(), + score: 5, + snippets: vec![], + }, + ]; + let merged = merge_docs_search_results(body, index, SEARCH_MAX_PAGES); + assert_eq!( + merged.iter().map(|r| r.url.clone()).collect::>(), + vec!["https://www.windmill.dev/docs/a", "https://www.windmill.dev/docs/b.md"] + ); +} + +#[test] +fn empty_query_returns_no_results() { + let pages = parse_docs_full_text(SAMPLE_FULL); + assert!(search_docs_pages(&pages, " ", 5).is_empty()); +} + +#[test] +fn format_search_results_no_matches() { + let out = format_docs_search_results("zzz", &[]); + assert!(out.contains("No documentation pages matched \"zzz\"")); +} + +#[test] +fn format_search_results_uses_caller_neutral_followup_guidance() { + let out = format_docs_search_results( + "jobs", + &[DocsSearchResult { + url: "https://www.windmill.dev/docs/core_concepts/jobs".to_string(), + title: "Jobs".to_string(), + score: 1, + snippets: vec!["Jobs run scripts and flows.".to_string()], + }], + ); + + assert!(out.contains("Source: https://www.windmill.dev/docs/core_concepts/jobs")); + assert!(out.contains("Source URL as its `url` argument")); + assert!(!out.contains("read_docs_page")); + assert!(!out.contains("readDocsPage")); +} diff --git a/backend/windmill-api/src/drafts.rs b/backend/windmill-api/src/drafts.rs index 9d4976489b..72791de78f 100644 --- a/backend/windmill-api/src/drafts.rs +++ b/backend/windmill-api/src/drafts.rs @@ -18,6 +18,7 @@ use windmill_common::{ db::UserDB, error::{Error, Result}, user_drafts::{DraftUserRef, UserDraftItemKind, ENCRYPTED_DRAFT_PREFIX}, + users::resolve_username_to_email, variables::{build_crypt, encrypt}, }; @@ -171,11 +172,16 @@ fn list_drafts_query(all_users: bool) -> String { // draft author at this (path, kind), legacy NULL-email row surfaced as a // null username. Restricted to the shared full-page-editor kinds — drawer // kinds keep their drafts private, so we never reveal their authors. + // A superadmin authoring in a workspace they are not a member of has no `usr` + // row: fall back to their instance-derived username (`password.username`), or + // their email when derivation is disabled (`password.username` is NULL). This + // keeps the raw email out of the payload whenever a derived username exists. let draft_users = r#"CASE WHEN d.typ::text IN ('script', 'flow', 'app', 'raw_app') THEN ( - SELECT json_agg(json_build_object('username', COALESCE(u.username, CASE WHEN du.workspace_id = 'admins' THEN du.email END)) - ORDER BY COALESCE(u.username, CASE WHEN du.workspace_id = 'admins' THEN du.email END) NULLS LAST) + SELECT json_agg(json_build_object('username', COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN du.email END)) + ORDER BY COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN du.email END) NULLS LAST) FROM draft du LEFT JOIN usr u ON u.workspace_id = du.workspace_id AND u.email = du.email + LEFT JOIN password p ON p.email = du.email AND p.super_admin = true WHERE du.workspace_id = d.workspace_id AND du.path = d.path AND du.typ = d.typ ) ELSE NULL END"#; // Default lists the user's own drafts AND the legacy NULL-email rows; with @@ -578,20 +584,12 @@ async fn get_draft_for_user( // Username -> email, scoped to the workspace. None signals "fetch the // legacy NULL-email row" (distinct from a username with no draft, which - // 404s below). + // 404s below). Resolution falls back to the instance `password` table so a + // superadmin's draft (they are not a `usr` member of the workspace, and + // their username is their instance-derived one) still resolves. let owner_email: Option = if let Some(username) = &query.username { - let email = sqlx::query_scalar!( - r#"SELECT email FROM usr WHERE workspace_id = $1 AND username = $2"#, - &w_id, - username, - ) - .fetch_optional(&db) - .await?; - match email { + match resolve_username_to_email(&w_id, username, &db).await? { Some(e) => Some(e), - // The `admins` workspace has no `usr` rows (username IS the email - // there), so accept it as the owner email directly. - None if w_id == "admins" => Some(username.clone()), None => { return Err(Error::NotFound(format!( "no user with username {username} in workspace" @@ -805,7 +803,6 @@ async fn require_can_read_path( Err(Error::NotFound(format!("no draft visible at {path}"))) } - #[cfg(test)] mod tests { use super::strip_json_nul; @@ -841,7 +838,9 @@ mod tests { // "a" carries a real NUL; "b" carries the literal text backslash-u0000. // The value walk strips the former and leaves the latter intact — the // pathological case that needed a fallback in SQL is trivial in Rust. - let v = parsed(strip_json_nul(r#"{"a":"x\u0000y","b":"p\\u0000q"}"#.to_string())); + let v = parsed(strip_json_nul( + r#"{"a":"x\u0000y","b":"p\\u0000q"}"#.to_string(), + )); assert_eq!(v["a"], "xy"); assert_eq!(v["b"], "p\\u0000q"); } diff --git a/backend/windmill-api/src/health.rs b/backend/windmill-api/src/health.rs index fab9979c12..4dc8947de6 100644 --- a/backend/windmill-api/src/health.rs +++ b/backend/windmill-api/src/health.rs @@ -219,12 +219,16 @@ struct DatabaseCheckResult { async fn check_database_with_latency(db: &DB) -> DatabaseCheckResult { let start = std::time::Instant::now(); + // `pg_is_in_recovery()` is true on standbys/read-only replicas, so a primary + // returns true here. A read-only replica (e.g. after a failover where the + // primary became a secondary) reports unhealthy, letting liveness probes + // restart the pod instead of silently failing all writes. let healthy = tokio::time::timeout( HEALTH_CHECK_TIMEOUT, - sqlx::query_scalar!("SELECT 1").fetch_one(db), + sqlx::query_scalar!("SELECT NOT pg_is_in_recovery()").fetch_one(db), ) .await - .map(|r| r.is_ok()) + .map(|r| matches!(r, Ok(Some(true)))) .unwrap_or(false); let latency_ms = start.elapsed().as_millis() as i64; diff --git a/backend/windmill-api/src/inkeep_oss.rs b/backend/windmill-api/src/inkeep_oss.rs deleted file mode 100644 index a34ea91435..0000000000 --- a/backend/windmill-api/src/inkeep_oss.rs +++ /dev/null @@ -1,23 +0,0 @@ -#[cfg(feature = "private")] -#[allow(unused)] -pub use crate::inkeep_ee::*; - -#[cfg(not(feature = "private"))] -use axum::{routing::post, Router}; - -#[cfg(not(feature = "private"))] -use windmill_common::error::Error; - -#[cfg(not(feature = "private"))] -pub fn global_service() -> Router { - Router::new().route("/", post(inkeep_not_available)) -} - -#[cfg(not(feature = "private"))] -async fn inkeep_not_available() -> windmill_common::error::Result<()> { - Err(Error::Generic( - http::StatusCode::FORBIDDEN, - "Inkeep AI documentation assistant is only available in Windmill Enterprise Edition" - .to_string(), - )) -} diff --git a/backend/windmill-api/src/job_helpers_oss.rs b/backend/windmill-api/src/job_helpers_oss.rs index 23f20459c3..c6d8e397f2 100644 --- a/backend/windmill-api/src/job_helpers_oss.rs +++ b/backend/windmill-api/src/job_helpers_oss.rs @@ -87,7 +87,8 @@ pub async fn upload_file_from_req( _file_key: &str, _req: axum::extract::Request, _options: PutMultipartOpts, -) -> error::Result { + _max_size: Option, +) -> error::Result<(PutResult, usize)> { Err(error::Error::internal_err( "Not implemented in Windmill's Open Source repository".to_string(), )) @@ -99,12 +100,77 @@ pub async fn upload_file_internal( _file_key: &str, _stream: impl Stream> + Unpin, _options: PutMultipartOpts, -) -> error::Result<()> { + _max_size: Option, +) -> error::Result<(PutResult, usize)> { Err(error::Error::internal_err( "Not implemented in Windmill's Open Source repository".to_string(), )) } +// These stubs stand in for the CE quota helpers in a pure-OSS build; their only +// callers (apps.rs / args.rs uploads) are `not(enterprise)`, so gate them the +// same way — an enterprise-without-private build compiles neither. +#[cfg(all( + feature = "parquet", + not(feature = "private"), + not(feature = "enterprise") +))] +pub async fn ce_storage_quota_remaining( + _db: &DB, + _w_id: &str, + _exclude_upload_id: Option<&str>, +) -> error::Result { + Ok(i64::MAX) +} + +#[cfg(all( + feature = "parquet", + not(feature = "private"), + not(feature = "enterprise") +))] +pub fn reject_reserved_volume_key(_file_key: &str) -> error::Result<()> { + Ok(()) +} + +#[cfg(all( + feature = "parquet", + not(feature = "private"), + not(feature = "enterprise") +))] +pub struct CeUploadBudget { + pub max_size: usize, + pub existing_size: i64, +} + +#[cfg(all( + feature = "parquet", + not(feature = "private"), + not(feature = "enterprise") +))] +pub async fn ce_upload_budget( + _db: &DB, + _w_id: &str, + _s3_client: &Arc, + _file_key: &str, + _content_length: Option, +) -> error::Result { + Ok(CeUploadBudget { max_size: usize::MAX, existing_size: 0 }) +} + +#[cfg(all( + feature = "parquet", + not(feature = "private"), + not(feature = "enterprise") +))] +pub async fn bump_storage_usage(_db: &DB, _w_id: &str, _storage: &str, _delta: i64) {} + +#[cfg(all( + feature = "parquet", + not(feature = "private"), + not(feature = "enterprise") +))] +pub fn spawn_storage_usage_recount_floored(_db: &DB, _w_id: &str) {} + #[cfg(all(feature = "parquet", not(feature = "private")))] pub async fn download_s3_file_internal( _authed: OptJobAuthed, diff --git a/backend/windmill-api/src/jobs.rs b/backend/windmill-api/src/jobs.rs index c6d978e0d6..14cbb76c91 100644 --- a/backend/windmill-api/src/jobs.rs +++ b/backend/windmill-api/src/jobs.rs @@ -390,6 +390,10 @@ pub fn workspace_unauthed_service() -> Router { .route("/get/{id}", get(get_job)) .route("/get_logs/{id}", get(get_job_logs)) .route("/get_flow_all_logs/{id}", get(get_flow_all_logs)) + .route( + "/get_flow_all_logs_structured/{id}", + get(get_flow_all_logs_structured), + ) .route( "/get_completed_logs_tail/{id}", get(get_completed_job_logs_tail), @@ -513,6 +517,26 @@ async fn cancel_job_api( Path((w_id, id)): Path<(String, Uuid)>, Json(CancelJob { reason }): Json, ) -> error::Result { + // App embed tokens (the sandboxed app iframe) may cancel ONLY jobs they launched + // — their app's component runs, stamped created_by == viewer. cancel_job_api has + // no other per-job ownership check, so without this an embed token (which carries + // the viewer's identity) could cancel any job by id. NotFound (not 403) so the + // untrusted app can't probe job existence. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + let created_by = sqlx::query_scalar!( + "SELECT created_by FROM v2_job WHERE id = $1 AND workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + if created_by.as_deref() != Some(authed.username.as_str()) { + return Err(Error::NotFound(format!("Job {id} not found"))); + } + } + } + let tx = db.begin().await?; let audit_author: AuditAuthor = match opt_authed.as_ref() { @@ -1007,6 +1031,17 @@ async fn require_job_read_access( return Ok(()); } + // App embed tokens (the sandboxed app iframe) carry the viewer's identity so the + // app can read its own component runs — which are stamped `created_by == viewer` + // and so already returned above. They must NOT inherit the viewer's *broader* + // job access (share links, folder ACLs, admin RLS): user-authored app JS holds + // this token, and letting it reach any job merely visible to the viewer would + // expose unrelated runs' results/logs. Stop at the launched-by-viewer grant. + // NotFound (not PermissionDenied) so the untrusted app can't probe job existence. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotFound(format!("Job {job_id} not found"))); + } + // `username_override` is derived from the token *label* (`username_override_from_label`), // which is fully user-controlled with no uniqueness/ownership check (webhook-/http-/ // email-/ws- trigger tokens, `ephemeral-script-end-user-*`, and the generic `label-*` @@ -1388,6 +1423,7 @@ macro_rules! get_job_query { @impl "v2_job_completed", ($($opts)*), "v2_job_completed.duration_ms, v2_job_completed.completed_at, CASE WHEN status = 'success' OR status = 'skipped' THEN true ELSE false END as success, result_columns, deleted, status = 'skipped' as is_skipped, \ v2_job.labels, \ + EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry, \ CASE WHEN result is null or pg_column_size(result) < 90000 THEN result ELSE '\"WINDMILL_TOO_BIG\"'::jsonb END as result", "", ) @@ -1397,7 +1433,8 @@ macro_rules! get_job_query { @impl "v2_job_queue", ($($opts)*), "scheduled_for, running, ping as last_ping, suspend, suspend_until, same_worker, pre_run_error, visible_to_owner, \ flow_innermost_root_job AS root_job, flow_leaf_jobs AS leaf_jobs, concurrent_limit, concurrency_time_window_s, timeout, flow_step_id, cache_ttl, cache_ignore_s3_path, runnable_settings_handle, \ - script_entrypoint_override, v2_job.labels", + script_entrypoint_override, v2_job.labels, \ + EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry", "LEFT JOIN v2_job_runtime ON v2_job_runtime.id = v2_job_queue.id LEFT JOIN v2_job_status ON v2_job_status.id = v2_job_queue.id", ) }; @@ -2177,14 +2214,40 @@ async fn resolve_logs_to_string( logs.to_string() } -async fn get_flow_all_logs( - OptViewToken(view_token): OptViewToken, - OptAuthed(opt_authed): OptAuthed, +/// A single job in a flow's execution tree, with its resolved logs and a +/// human-readable label describing its position (iteration, branch, subflow…). +#[derive(Serialize)] +struct FlowLogEntry { + job_id: String, + /// Human-readable label, e.g. "Step a (iteration 2/3)" or "Flow". + label: String, + /// Job kind (script, flow, forloopflow, …). + kind: String, + /// The flow step id this job corresponds to, if any. + flow_step_id: Option, + /// Materialized step path (e.g. "a/b") used to locate the step in the flow. + step_path: Option, + /// Depth in the flow tree (0 for the root flow job). + depth: i32, + /// The parent module type (forloopflow, branchall, …), if any. + parent_module_type: Option, + /// 1-based index of this job among its siblings sharing the same step. + sibling_index: i32, + /// Total number of siblings sharing the same step. + sibling_count: i32, + /// Resolved logs for this job (pulled from disk/object store as needed). + logs: String, +} + +async fn collect_flow_log_entries( + view_token: Option, + opt_authed: Option, opt_tokened: OptTokened, - Extension(db): Extension, - Extension(user_db): Extension, - Path((w_id, id)): Path<(String, Uuid)>, -) -> error::Result { + db: &DB, + user_db: &UserDB, + w_id: &str, + id: Uuid, +) -> error::Result> { let tags = opt_authed .as_ref() .map(|authed| get_scope_tags(authed).map(|v| v.iter().map(|s| s.to_string()).collect_vec())) @@ -2197,17 +2260,17 @@ async fn get_flow_all_logs( w_id, tags.as_ref().map(|v| v.as_slice()) ) - .fetch_optional(&db) + .fetch_optional(db) .await?; let root_job = not_found_if_none(root_job, "Job", id.to_string())?; if let Some(authed) = opt_authed.as_ref() { require_job_read_access( - &db, - &user_db, + db, + user_db, authed, - &w_id, + w_id, &id, &root_job.created_by, view_token.as_deref(), @@ -2220,10 +2283,10 @@ async fn get_flow_all_logs( } log_job_view( - &db, + db, opt_authed.as_ref(), opt_tokened.token.as_deref(), - &w_id, + w_id, &id, ) .await?; @@ -2290,10 +2353,10 @@ async fn get_flow_all_logs( w_id, id, ) - .fetch_all(&db) + .fetch_all(db) .await?; - let mut all_logs = String::new(); + let mut entries = Vec::with_capacity(records.len()); for record in &records { let kind = record.kind.as_deref().unwrap_or(""); @@ -2356,18 +2419,89 @@ async fn get_flow_all_logs( }; let job_id = record.id.map(|u| u.to_string()).unwrap_or_default(); - all_logs.push_str(&format!("\n=== {} (Job: {}) ===\n", label, job_id)); let logs = record.logs.as_deref().unwrap_or(""); let resolved = resolve_logs_to_string(record.log_offset.unwrap_or(0), logs, &record.log_file_index) .await; - all_logs.push_str(&resolved); + + entries.push(FlowLogEntry { + job_id, + label, + kind: kind.to_string(), + flow_step_id: record.flow_step_id.clone(), + step_path: record.path_label.clone(), + depth, + parent_module_type: if parent_module_type.is_empty() { + None + } else { + Some(parent_module_type.to_string()) + }, + sibling_index, + sibling_count, + logs: resolved, + }); + } + + Ok(entries) +} + +async fn get_flow_all_logs( + OptViewToken(view_token): OptViewToken, + OptAuthed(opt_authed): OptAuthed, + opt_tokened: OptTokened, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, Uuid)>, +) -> error::Result { + let entries = collect_flow_log_entries( + view_token, + opt_authed, + opt_tokened, + &db, + &user_db, + &w_id, + id, + ) + .await?; + + let mut all_logs = String::new(); + for entry in &entries { + all_logs.push_str(&format!( + "\n=== {} (Job: {}) ===\n", + entry.label, entry.job_id + )); + all_logs.push_str(&entry.logs); all_logs.push('\n'); } Ok(content_plain(Body::from(all_logs))) } +/// Structured alternative to `get_flow_all_logs`: returns the same flow log +/// tree as a JSON array of entries (one per job) instead of a flat text blob, +/// so callers can render or process logs per-step without parsing delimiters. +async fn get_flow_all_logs_structured( + OptViewToken(view_token): OptViewToken, + OptAuthed(opt_authed): OptAuthed, + opt_tokened: OptTokened, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, Uuid)>, +) -> JsonResult> { + let entries = collect_flow_log_entries( + view_token, + opt_authed, + opt_tokened, + &db, + &user_db, + &w_id, + id, + ) + .await?; + + Ok(Json(entries)) +} + async fn get_args( OptViewToken(view_token): OptViewToken, OptAuthed(opt_authed): OptAuthed, @@ -4022,11 +4156,17 @@ fn conditionally_require_authed_user( } pub async fn create_job_signature( - _authed: ApiAuthed, + authed: ApiAuthed, Extension(db): Extension, Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>, Query(approver): Query, ) -> error::Result { + // The HMAC is treated as full authority by the resume endpoints, so minting + // it requires run scope on the suspended job's flow — not merely any + // jobs:run scope. No-op for unscoped tokens (incl. the in-flow substep token + // used by wmill.get_resume_urls()). + let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?; + check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?; let key = get_workspace_key(&w_id, &db).await?; create_signature(key, job_id, resume_id, approver.approver) } @@ -4109,11 +4249,17 @@ fn build_resume_url( } pub async fn get_resume_urls( - _authed: ApiAuthed, + authed: ApiAuthed, Extension(db): Extension, Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>, Query(approver): Query, ) -> error::JsonResult { + // These URLs embed a resume signature (full resume capability), so a scoped + // token must hold run scope on the suspended job's flow. No-op for unscoped + // tokens (incl. the in-flow substep token). Trusted internal callers use + // get_resume_urls_internal directly and are unaffected. + let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?; + check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?; get_resume_urls_internal( Extension(db), Path((w_id, job_id, resume_id)), @@ -4180,6 +4326,46 @@ pub async fn get_resume_urls_internal( Ok(Json(res)) } +/// Resolve the runnable path of the flow a (possibly step) job belongs to, used +/// to scope-check resume-signature minting against `jobs:run:flows:`. +/// Returns an empty string when the path can't be resolved (e.g. previews or an +/// unknown job); an empty path only matters for path-restricted tokens, which +/// would not be running such a flow. Never hard-fails, so it can't break resume +/// for unscoped tokens (the in-flow `get_resume_urls()` path). +async fn resume_target_flow_path(db: &DB, w_id: &str, job_id: Uuid) -> error::Result { + let job = sqlx::query!( + r#"SELECT kind::text as "kind!", parent_job, runnable_path + FROM v2_job WHERE id = $1 AND workspace_id = $2"#, + job_id, + w_id + ) + .fetch_optional(db) + .await?; + let Some(job) = job else { + return Ok(String::new()); + }; + // All flow kinds: the job itself is the flow whose path scopes the resume. + if matches!( + job.kind.as_str(), + "flow" | "flowpreview" | "flownode" | "singlestepflow" + ) { + return Ok(job.runnable_path.unwrap_or_default()); + } + // Otherwise it's a step; its parent is the flow. + if let Some(parent) = job.parent_job { + return Ok(sqlx::query_scalar!( + "SELECT runnable_path FROM v2_job WHERE id = $1 AND workspace_id = $2", + parent, + w_id + ) + .fetch_optional(db) + .await? + .flatten() + .unwrap_or_default()); + } + Ok(job.runnable_path.unwrap_or_default()) +} + /// Get the flow ID for a job. If the job is a flow, returns the job_id. /// If the job is a step in a flow, returns the parent flow ID. async fn get_flow_id_for_job(db: &DB, job_id: Uuid) -> error::Result { @@ -7416,26 +7602,32 @@ async fn run_dynamic_select( )); } + if !is_valid_entrypoint_name(&request.entrypoint_function) { + return Err(error::Error::BadRequest(format!( + "Invalid entrypoint_function {:?}: must match \ + ^[A-Za-z_][A-Za-z0-9_]*$ (letters, digits and underscores, \ + not starting with a digit)", + request.entrypoint_function + ))); + } + + // Deployed scripts keep their normal deployed-run path (a `script` job, resolved below by + // push_script_job_by_path_into_queue). Deployed flows and inline snippets have no deployed + // runnable, so they run their dyn-select code as a `preview`; the flow carries its path and + // worker tag so its option-fetching job lines up with the script run. let dynamic_input: DynamicInput; + let runnable_path: Option; + let mut tag: Option = None; match request.runnable_ref { DynamicSelectRunnableRef::Deployed { path, runnable_kind } => match runnable_kind { RunnableKind::Script => { - if !is_valid_entrypoint_name(&request.entrypoint_function) { - return Err(error::Error::BadRequest(format!( - "Invalid entrypoint_function {:?}: must match \ - ^[A-Za-z_][A-Za-z0-9_]*$ (letters, digits and underscores, \ - not starting with a digit)", - request.entrypoint_function - ))); - } let mut script_args = request.args.unwrap_or_default(); script_args.insert( - "_ENTRYPOINT_OVERRIDE".to_string(), + ENTRYPOINT_OVERRIDE.to_string(), serde_json::value::to_raw_value(&request.entrypoint_function)?, ); - - let push_args = PushArgsOwned { extra: None, args: script_args.clone() }; + let push_args = PushArgsOwned { extra: None, args: script_args }; let (uuid, _, _) = push_script_job_by_path_into_queue( authed.clone(), @@ -7445,7 +7637,7 @@ async fn run_dynamic_select( w_id.clone(), StripPath(path), run_query.clone(), - push_args.clone(), + push_args, None, ) .await?; @@ -7453,13 +7645,40 @@ async fn run_dynamic_select( return Ok((StatusCode::CREATED, uuid.to_string()).into_response()); } RunnableKind::Flow => { - // Runs the deployed flow's dynamic-select code. Enforce the same - // path-scoped check the script branch gets via - // push_script_job_by_path_into_queue, so a token not scoped to this - // flow cannot trigger its code through dynamic select. + // Runs the deployed flow's dynamic-select code. Path-scoped so a token not + // scoped to this flow cannot trigger its code through dynamic select. check_scopes(&authed, || format!("jobs:run:flows:{path}"))?; let mut conn = user_db.clone().begin(&authed).await?; + // Read the flow's tag under RLS. This runs on every request (including the + // cache hit below), so it doubles as the access check and routes the + // option-fetching preview to the flow's worker group, matching the script branch. + let Some(flow_tag) = sqlx::query_scalar!( + "SELECT tag FROM flow WHERE workspace_id = $1 AND path = $2", + &w_id, + &path + ) + .fetch_optional(&mut *conn) + .await? + else { + conn.commit().await?; + let exists = sqlx::query_scalar!( + "SELECT 1 FROM flow WHERE workspace_id = $1 AND path = $2 LIMIT 1", + &w_id, + &path + ) + .fetch_optional(&db) + .await? + .is_some(); + if exists { + return Err(error::Error::NotAuthorized(format!( + "You are not authorized to access this flow: {path}" + ))); + } + return Err(Error::NotFound(format!("Flow not found at path {path}"))); + }; + tag = flow_tag; + let dynamic_input_res = match DYNAMIC_INPUT_CACHE.get(&format!("{}:{}", w_id, path)) { Some(cached) => cached.as_ref().clone(), @@ -7502,21 +7721,34 @@ async fn run_dynamic_select( conn.commit().await?; dynamic_input = dynamic_input_res; + runnable_path = Some(path); } }, DynamicSelectRunnableRef::Inline { code, lang: language } => { // Inline dynamic select runs arbitrary, request-supplied code; require the broad // jobs:run scope so a narrowly-scoped token cannot escape its scope. The Deployed - // branches are path-scoped instead (scripts via push_script_job_by_path_into_queue, - // flows via the check_scopes above). + // branches are path-scoped instead. check_scopes(&authed, || format!("jobs:run"))?; dynamic_input = DynamicInput { x_windmill_dyn_select_code: code, x_windmill_dyn_select_lang: language.unwrap_or_default(), }; + runnable_path = None; } } + // Same tag-permission gate a normal run gets (run_flow / push_script_job_by_path_into_queue): + // a caller allowed to read the flow must still be allowed to use its worker tag. No-op for + // inline (tag is None); the script branch checked this inside its helper and returned above. + check_tag_available_for_workspace(&db, &w_id, &tag, &authed).await?; + + // Invoke the dyn-select entrypoint instead of `main`. + let mut args = request.args.unwrap_or_default(); + args.insert( + ENTRYPOINT_OVERRIDE.to_string(), + serde_json::value::to_raw_value(&request.entrypoint_function)?, + ); + let scheduled_for = run_query.get_scheduled_for(&db).await?; let tx = PushIsolationLevel::Isolated(user_db.clone(), authed.clone().into()); @@ -7527,7 +7759,7 @@ async fn run_dynamic_select( JobPayload::Code(RawCode { hash: None, content: dynamic_input.x_windmill_dyn_select_code, - path: None, + path: runnable_path, language: dynamic_input.x_windmill_dyn_select_lang, lock: None, cache_ttl: None, @@ -7536,9 +7768,11 @@ async fn run_dynamic_select( concurrency_settings: ConcurrencySettings::default().into(), debouncing_settings: DebouncingSettings::default(), modules: None, + // RawCode.tag is ignored by the queue path (`JobPayload::Code` destructures it as + // `tag: _`); the effective tag is the `tag` argument to `push` below. tag: None, }), - PushArgs::from(&request.args.unwrap_or_default()), + PushArgs::from(&args), authed.display_username(), &authed.email, username_to_permissioned_as(&authed.username), @@ -7553,7 +7787,8 @@ async fn run_dynamic_select( false, None, true, - None, + // Deployed flow → the flow's worker tag; inline → `None` (language default). + tag, run_query.timeout, None, None, @@ -9488,16 +9723,31 @@ mod approval_view_gate_tests { fn anonymous_cannot_view_when_auth_required() { // The regression: an unauthenticated holder of the approval token must see nothing. let c = Some(conds(true, vec![])); - assert!(!can_view(&None, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(!can_view( + &None, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); } #[test] fn anonymous_can_view_when_no_auth_required() { // Unchanged behaviour: token alone is sufficient when auth isn't required. let c = Some(conds(false, vec![])); - assert!(can_view(&None, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &None, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); // No approval conditions at all also allows token-only view. - assert!(can_view(&None, &None, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &None, + &None, + Some("f/team/flow"), + "trigger@example.com" + )); } #[test] @@ -9522,7 +9772,17 @@ mod approval_view_gate_tests { let member = Some(authed("carol", false, vec!["approvers".to_string()])); let outsider = Some(authed("dave", false, vec!["other".to_string()])); // Use a non-owned folder path so ownership doesn't short-circuit the check. - assert!(can_view(&member, &c, Some("f/team/flow"), "trigger@example.com")); - assert!(!can_view(&outsider, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &member, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); + assert!(!can_view( + &outsider, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); } } diff --git a/backend/windmill-api/src/lib.rs b/backend/windmill-api/src/lib.rs index c34a153619..73593c595a 100644 --- a/backend/windmill-api/src/lib.rs +++ b/backend/windmill-api/src/lib.rs @@ -69,6 +69,7 @@ mod ai; #[cfg(feature = "private")] mod ai_free_tier_ee; mod ai_free_tier_oss; +mod ai_skills; mod apps; pub mod args; mod audit; @@ -80,6 +81,7 @@ mod capture; mod concurrency_groups; mod db; mod db_health; +mod docs; mod drafts; #[cfg(feature = "private")] @@ -97,9 +99,6 @@ mod health; #[cfg(feature = "private")] pub mod indexer_ee; mod indexer_oss; -#[cfg(feature = "private")] -mod inkeep_ee; -mod inkeep_oss; mod integration; mod internal_db; mod live_migrations; @@ -548,7 +547,15 @@ pub async fn run_server( Router::new() // Reordered alphabetically .nest("/acls", granular_acls::workspaced_service()) - .nest("/apps", apps::workspaced_service(request_size_limit * 5)) + // CORS so the opaque-origin in-workspace app viewer (WIN-2006, + // sandboxed /apps/get) can read the app definition by path + // (apps/get/p, apps/embed_token/p) with a scoped embed token. + // Bearer-token-only (no cookies), consistent with the other + // workspaced services the iframe calls. + .nest( + "/apps", + apps::workspaced_service(request_size_limit * 5).layer(cors.clone()), + ) .nest("/assets", windmill_api_assets::workspaced_service()) .nest("/audit", audit::workspaced_service()) .nest("/capture", capture::workspaced_service()) @@ -568,7 +575,13 @@ pub async fn run_server( "/flow_conversations", windmill_api_flow_conversations::workspaced_service(), ) - .nest("/folders", folders::workspaced_service()) + // CORS so an opaque-origin app iframe (WIN-2006 embed, + // no separate domain) can read folders/listnames with a + // scoped embed token. Consistent with apps_u/jobs_u cors. + .nest( + "/folders", + folders::workspaced_service().layer(cors.clone()), + ) .nest("/folders_history", folder_history::workspaced_service()) .nest("/groups", groups::workspaced_service()) .nest("/groups_history", group_history::workspaced_service()) @@ -611,20 +624,30 @@ pub async fn run_server( Router::new() }) .nest("/ai", ai::workspaced_service()) + .nest("/ai_skills", ai_skills::workspaced_service()) .nest("/npm_proxy", windmill_api_npm_proxy::workspaced_service()) .nest( "/path_autocomplete", path_autocomplete::workspaced_service(), ) .nest("/raw_apps", raw_apps::workspaced_service()) - .nest("/resources", resources::workspaced_service()) + // CORS so the opaque-origin app iframe can read + // resources/list, resources/type/* with a scoped token. + .nest( + "/resources", + resources::workspaced_service().layer(cors.clone()), + ) .nest("/shared_ui", workspace_shared_ui::workspaced_service()) .nest("/schedules", windmill_api_schedule::workspaced_service()) .nest("/scripts", scripts::workspaced_service()) .nest("/trash", trash::workspaced_service()) .nest( "/users", - users::workspaced_service().layer(Extension(argon2.clone())), + // CORS so the opaque-origin app iframe can read + // users/whoami with a scoped embed token. + users::workspaced_service() + .layer(Extension(argon2.clone())) + .layer(cors.clone()), ) .nest("/variables", variables::workspaced_service()) .nest("/volumes", volumes_oss::workspaced_service()) @@ -665,7 +688,7 @@ pub async fn run_server( .nest("/schedules", windmill_api_schedule::global_service()) .nest("/embeddings", embeddings::global_service()) .nest("/ai", ai::global_service()) - .nest("/inkeep", inkeep_oss::global_service()) + .nest("/docs", docs::global_service()) .nest("/indexer", indexer_oss::management_service()) .nest("/mcp/w/{workspace_id}/list_tools", mcp_list_tools_service) .nest("/db_health", db_health::global_service()) @@ -730,7 +753,11 @@ pub async fn run_server( .nest("/apps_u", { #[cfg(feature = "enterprise")] { - apps_oss::global_unauthed_service() + // CORS so the opaque-origin app viewer (WIN-2006 embed, no + // separate domain) can load a custom-path public app via + // public_app_by_custom_path cross-origin. Consistent with + // the workspaced /w/{workspace_id}/apps_u mount below. + apps_oss::global_unauthed_service().layer(cors.clone()) } #[cfg(not(feature = "enterprise"))] diff --git a/backend/windmill-api/src/live_migrations.rs b/backend/windmill-api/src/live_migrations.rs index e7fa3eb13f..d5c759fc18 100644 --- a/backend/windmill-api/src/live_migrations.rs +++ b/backend/windmill-api/src/live_migrations.rs @@ -9,26 +9,26 @@ use sqlx::Postgres; use windmill_common::error::Error; -use crate::db::{CustomMigrator, DB}; +use crate::db::CustomMigrator; use sqlx::migrate::Migrate; +use sqlx::Acquire; use sqlx::Executor; -pub async fn custom_migrations(migrator: &mut CustomMigrator, db: &DB) -> Result<(), Error> { - if let Err(err) = fix_flow_versioning_migration(migrator, db).await { +pub async fn custom_migrations(migrator: &mut CustomMigrator) -> Result<(), Error> { + if let Err(err) = fix_flow_versioning_migration(migrator).await { tracing::error!("Could not apply flow versioning fix migration: {err:#}"); } Ok(()) } -async fn fix_flow_versioning_migration( - migrator: &mut CustomMigrator, - db: &DB, -) -> Result<(), Error> { +// Runs on the migrator's held connection (see CustomMigrator::connection): re-acquiring +// from the pool here would deadlock a single-connection backend. +async fn fix_flow_versioning_migration(migrator: &mut CustomMigrator) -> Result<(), Error> { let has_done_migration = sqlx::query_scalar!( "SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = 'fix_flow_versioning_2')", ) - .fetch_one(db) + .fetch_one(migrator.connection()) .await? .unwrap_or(false); @@ -44,14 +44,14 @@ async fn fix_flow_versioning_migration( let has_done_migration = sqlx::query_scalar!( "SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = 'fix_flow_versioning_2')", ) - .fetch_one(db) + .fetch_one(migrator.connection()) .await? .unwrap_or(false); if !has_done_migration { let query = include_str!("../../custom_migrations/fix_flow_versioning_2.sql"); tracing::info!("Applying fix_flow_versioning_2.sql"); - let mut tx: sqlx::Transaction<'_, Postgres> = db.begin().await?; + let mut tx: sqlx::Transaction<'_, Postgres> = migrator.connection().begin().await?; tx.execute(query).await?; tracing::info!("Applied fix_flow_versioning_2.sql"); sqlx::query!( diff --git a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs index 2e9b5babe7..d44484f9db 100644 --- a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs +++ b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs @@ -7,25 +7,53 @@ use windmill_mcp::server::EndpointTool; pub fn all_tools() -> Vec { vec![ EndpointTool { - name: Cow::Borrowed("queryDocumentation"), - description: Cow::Borrowed("query Windmill AI documentation assistant (EE only)"), + name: Cow::Borrowed("searchDocs"), + description: Cow::Borrowed("Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more."), instructions: Cow::Borrowed(""), - path: Cow::Borrowed("/inkeep"), - method: Cow::Borrowed("POST"), + path: Cow::Borrowed("/docs/search"), + method: Cow::Borrowed("GET"), path_params_schema: None, - query_params_schema: None, - body_schema: Some(serde_json::json!({ + query_params_schema: Some(serde_json::json!({ "type": "object", "properties": { "query": { "type": "string", - "description": "The documentation query to send to the AI assistant" + "description": "Keywords to search for in the documentation body, e.g. \"chromium worker tag\" or \"retry exponential backoff\". Fewer, more distinctive words match better." } }, "required": [ "query" ] })), + body_schema: None, + path_field_renames: None, + query_field_renames: None, + body_field_renames: None, + }, + EndpointTool { + name: Cow::Borrowed("readDocsPage"), + description: Cow::Borrowed("Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section."), + instructions: Cow::Borrowed(""), + path: Cow::Borrowed("/docs/page"), + method: Cow::Borrowed("GET"), + path_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted." + }, + "section": { + "type": "string", + "description": "Optional. A heading title from the page outline to read just that section instead of the full page." + } + }, + "required": [ + "url" + ] +})), + body_schema: None, path_field_renames: None, query_field_renames: None, body_field_renames: None, @@ -84,6 +112,9 @@ pub fn all_tools() -> Vec { "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -169,6 +200,9 @@ pub fn all_tools() -> Vec { "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the variable (body parameter)" @@ -210,6 +244,10 @@ pub fn all_tools() -> Vec { "include_encrypted": { "type": "boolean", "description": "ask to include the encrypted value if secret and decrypt secret is not true (default: false)\n" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -260,6 +298,10 @@ pub fn all_tools() -> Vec { "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft variables whose path has no\ndeployed variable. Synthesized rows carry `draft_only: true`\nso the home page can render a \"Draft\" badge.\n" } }, "required": [] @@ -309,6 +351,9 @@ pub fn all_tools() -> Vec { "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -383,6 +428,9 @@ pub fn all_tools() -> Vec { "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the resource (body parameter)" @@ -414,7 +462,16 @@ pub fn all_tools() -> Vec { "path" ] })), - query_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +})), body_schema: None, path_field_renames: None, query_field_renames: None, @@ -469,6 +526,10 @@ pub fn all_tools() -> Vec { "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft resources whose path has\nno deployed resource. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] @@ -719,6 +780,10 @@ Creates a new version of an existing script when called with the same path and t "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -847,6 +912,10 @@ Creates a new version of an existing script when called with the same path and t "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -1184,6 +1253,14 @@ Creates a new version of an existing script when called with the same path and t "language" ] } + }, + "temp_script_refs": { + "type": "object", + "nullable": true, + "description": "Map of relative-import script path -> temp storage hash so the preview job resolves those imports from not-yet-deployed local content instead of the deployed script", + "additionalProperties": { + "type": "string" + } } }, "required": [ @@ -1456,6 +1533,10 @@ Creates a new version of an existing script when called with the same path and t "type": "boolean", "description": "filter on successful jobs" }, + "status": { + "type": "string", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.. Possible values: success, failure, canceled, skipped" + }, "all_workspaces": { "type": "boolean", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)" @@ -1464,6 +1545,10 @@ Creates a new version of an existing script when called with the same path and t "type": "boolean", "description": "is not a scheduled job" }, + "excludes_entrypoint_override": { + "type": "boolean", + "description": "exclude jobs that were started with a `_ENTRYPOINT_OVERRIDE` arg (e.g. dynamic-select helper runs and preprocessor previews)" + }, "broad_filter": { "type": "string", "description": "broad search across multiple fields (case-insensitive substring match on path, tag, schedule path, trigger kind, label)" @@ -1502,6 +1587,10 @@ Creates a new version of an existing script when called with the same path and t }, "no_code": { "type": "boolean" + }, + "approval_token": { + "type": "string", + "description": "Approval token granting read access to the job when not logged in. The token must be the one issued for this job's flow (i.e. the flow id used when generating the approval URL)." } }, "required": [] @@ -1559,7 +1648,7 @@ You should get the schema of the script or flow before creating the schedule to "properties": { "path": { "type": "string", - "description": "The unique path identifier for this schedule" + "description": "The unique Windmill path for this schedule. Must be of the form `u//` or `f//`." }, "schedule": { "type": "string", @@ -2002,7 +2091,16 @@ You should get the schema of the script or flow before updating the schedule to "path" ] })), - query_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +})), body_schema: None, path_field_renames: None, query_field_renames: None, @@ -2061,6 +2159,10 @@ You should get the schema of the script or flow before updating the schedule to "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft schedules whose path has\nno deployed schedule. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] diff --git a/backend/windmill-api/src/mcp/oauth_server.rs b/backend/windmill-api/src/mcp/oauth_server.rs index 7cfabd2319..c5142c3031 100644 --- a/backend/windmill-api/src/mcp/oauth_server.rs +++ b/backend/windmill-api/src/mcp/oauth_server.rs @@ -18,6 +18,7 @@ use windmill_common::{ }; use crate::db::ApiAuthed; +use windmill_mcp::parse_mcp_scopes; /// Token expiration for MCP OAuth tokens (1 week in seconds) const MCP_OAUTH_TOKEN_EXPIRATION_SECS: u64 = 7 * 24 * 60 * 60; @@ -585,6 +586,8 @@ async fn handle_refresh_token_grant( Some(&new_access_token) }; let new_refresh_token = rd_string(32); + // Re-issues the already-approved (hence already-contained) scopes verbatim; + // containment is enforced once at approval time, so no re-check here. let scopes = token_row.scopes; // Create new access token (rejects archived workspaces inline) @@ -820,6 +823,40 @@ async fn oauth_approve_inner( .map(|s| s.to_string()) .collect(); + // The approver's own token bounds what it may grant: a scope-restricted MCP + // token must not approve a broader one (e.g. mcp:scripts:f/x -> mcp:all). An + // unrestricted approver (interactive session, scopes None) grants freely, + // which is the normal consent flow. This is the legitimate MCP-narrowing + // path, so it uses MCP-pattern containment rather than the byte-identical + // rule ensure_scopes_within_caller applies on the generic token endpoints. + let caller_restricted = authed + .scopes + .as_deref() + .is_some_and(|s| s.iter().any(|x| !x.starts_with("if_jobs:filter_tags:"))); + if caller_restricted { + // An empty grant would mint a token the auth layer treats as unscoped + // (full privileges), so a restricted approver must not produce one. + if scopes.is_empty() { + return Err(Error::NotAuthorized( + "A scope-restricted token cannot approve an empty scope grant".to_string(), + )); + } + if scopes.iter().any(|s| !s.starts_with("mcp:")) { + return Err(Error::NotAuthorized( + "A scope-restricted token can only approve MCP (mcp:*) scopes".to_string(), + )); + } + let caller_config = parse_mcp_scopes(authed.scopes.as_deref().unwrap_or(&[])) + .map_err(|e| Error::InternalErr(format!("Failed to parse caller MCP scopes: {e}")))?; + let requested_config = parse_mcp_scopes(&scopes) + .map_err(|e| Error::BadRequest(format!("Failed to parse requested MCP scopes: {e}")))?; + if !caller_config.contains(&requested_config) { + return Err(Error::NotAuthorized( + "Requested scopes exceed the approving token's own MCP scopes".to_string(), + )); + } + } + sqlx::query!( "INSERT INTO mcp_oauth_server_code (code, client_id, user_email, workspace_id, scopes, redirect_uri, code_challenge, code_challenge_method) diff --git a/backend/windmill-api/src/mcp/utils.rs b/backend/windmill-api/src/mcp/utils.rs index 4073758ba9..2184c1548f 100644 --- a/backend/windmill-api/src/mcp/utils.rs +++ b/backend/windmill-api/src/mcp/utils.rs @@ -382,12 +382,18 @@ pub fn build_query_string( .map(|value| { // Use the original name for the query parameter key let original_name = get_original_name(param_name, query_field_renames); - let value_str = value.to_string(); - let str_val = value_str.trim_matches('"'); + // For string values, use the raw content: to_string() would JSON-encode + // it, and stripping the outer quotes leaves inner quotes backslash-escaped + // (e.g. `{\"k\":\"v\"}`), which breaks downstream JSON parsing of params + // like `args`/`result`. Non-string values keep their JSON serialization. + let str_val = value + .as_str() + .map(|s| s.to_string()) + .unwrap_or_else(|| value.to_string()); format!( "{}={}", urlencoding::encode(&original_name), - urlencoding::encode(str_val) + urlencoding::encode(&str_val) ) }) }) @@ -455,9 +461,35 @@ pub async fn create_http_request( } }; + // Bound the minted JWT to exactly this proxied route so a scope-restricted + // MCP token can't be widened into a full-privilege blank check. The + // endpoint-name gate (in the MCP runner) already authorized *which* endpoint + // may be called; this constrains what the resulting request can do. Unscoped + // callers (cookie / full-privilege tokens) keep an unscoped JWT to preserve + // existing behavior. A scope-restricted caller whose route can't be resolved + // fails closed. + let caller_restricted = api_authed + .scopes + .as_deref() + .is_some_and(|s| s.iter().any(|x| !x.starts_with("if_jobs:filter_tags:"))); + let scopes = if caller_restricted { + let parsed = reqwest::Url::parse(url) + .map_err(|e| ErrorData::internal_error(format!("Invalid proxied URL: {}", e), None))?; + let scope = + windmill_api_auth::scopes::scope_for_route(method, parsed.path()).ok_or_else(|| { + ErrorData::internal_error( + "Could not derive route scope for proxied MCP endpoint".to_string(), + None, + ) + })?; + Some(vec![scope]) + } else { + None + }; + // Add authorization header let authed = Authed::from(api_authed.clone()); - let token = create_jwt_token(authed, workspace_id, 3600, None, None, None, None) + let token = create_jwt_token(authed, workspace_id, 3600, None, None, None, scopes) .await .map_err(|e| ErrorData::internal_error(e.to_string(), None))?; request_builder = request_builder.header("Authorization", format!("Bearer {}", token)); @@ -598,4 +630,55 @@ mod tests { .expect("legitimate path should substitute"); assert_eq!(result, "/w/dev/scripts/get/p/u/alice/my_script"); } + + fn single_query_schema(param: &str) -> Option { + Some(json!({ + "type": "object", + "properties": { param: { "type": "string" } } + })) + } + + #[test] + fn build_query_string_preserves_json_string_content() { + // A string param carrying JSON (e.g. the `args` filter on listJobs) must be + // emitted as its raw content so the backend can `serde_json::from_str` it. + let mut args = serde_json::Map::new(); + args.insert("args".to_string(), json!("{\"key\":\"val\"}")); + + let qs = build_query_string(&args, &single_query_schema("args"), &None); + + // No backslash escaping: %5C must not appear; the encoded braces/quotes are exact. + assert_eq!(qs, "?args=%7B%22key%22%3A%22val%22%7D"); + assert!( + !qs.contains("%5C"), + "must not contain backslash escapes: {qs}" + ); + } + + #[test] + fn build_query_string_keeps_non_string_serialization() { + let mut args = serde_json::Map::new(); + args.insert("per_page".to_string(), json!(42)); + assert_eq!( + build_query_string(&args, &single_query_schema("per_page"), &None), + "?per_page=42" + ); + + let mut args = serde_json::Map::new(); + args.insert("running".to_string(), json!(true)); + assert_eq!( + build_query_string(&args, &single_query_schema("running"), &None), + "?running=true" + ); + } + + #[test] + fn build_query_string_encodes_plain_string() { + let mut args = serde_json::Map::new(); + args.insert("path".to_string(), json!("u/alice/my script")); + assert_eq!( + build_query_string(&args, &single_query_schema("path"), &None), + "?path=u%2Falice%2Fmy%20script" + ); + } } diff --git a/backend/windmill-api/src/offboarding.rs b/backend/windmill-api/src/offboarding.rs index 3d5342e3a5..24b6d3f10d 100644 --- a/backend/windmill-api/src/offboarding.rs +++ b/backend/windmill-api/src/offboarding.rs @@ -1,13 +1,13 @@ use std::collections::HashMap; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use axum::{ extract::{Extension, Path}, Json, }; use serde::{Deserialize, Serialize}; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_api_users::users::delete_workspace_user_internal; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; @@ -483,11 +483,13 @@ pub(crate) async fn global_offboard_preview( pub(crate) async fn offboard_global_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Path(email): Path, Json(req): Json, ) -> Result> { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let workspaces = sqlx::query!( "SELECT workspace_id, username FROM usr WHERE email = $1", diff --git a/backend/windmill-api/src/secret_backend_ext.rs b/backend/windmill-api/src/secret_backend_ext.rs index f76ba0ba64..fd17b8271e 100644 --- a/backend/windmill-api/src/secret_backend_ext.rs +++ b/backend/windmill-api/src/secret_backend_ext.rs @@ -8,245 +8,24 @@ //! Secret backend extension for the API layer //! -//! This module provides helper functions for integrating the SecretBackend -//! trait with variable operations in the API. +//! Backend resolution and read helpers live in +//! `windmill_common::secret_backend`; this module keeps the API-specific bulk +//! rename helper used when renaming users. //! //! Note: HashiCorp Vault integration requires Enterprise Edition. //! The OSS version only supports the database backend. -#[cfg(all(feature = "private", feature = "enterprise"))] -use std::sync::Arc; - use windmill_common::{db::DB, error::Result}; -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::error::Error; - -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::secret_backend::{database::DatabaseBackend, SecretBackend}; - #[cfg(all(feature = "private", feature = "enterprise"))] use windmill_common::{ - global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, + error::Error, secret_backend::{ - AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, - AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, + get_secret_backend, is_aws_sm_stored_value, is_azure_kv_stored_value, + is_external_stored_value, is_vault_backend_configured, }, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use tokio::sync::RwLock; - -// Cached Vault backend to avoid recreating it for every request -// This enables connection pooling and avoids repeated setup overhead -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedVaultBackend { - backend: Arc, - settings: VaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached Azure Key Vault backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAzureKvBackend { - backend: Arc, - settings: AzureKeyVaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached AWS Secrets Manager backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAwsSmBackend { - backend: Arc, - settings: AwsSecretsManagerSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -/// Get the current secret backend based on global settings (EE only) -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_secret_backend(db: &DB) -> Result> { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - match config { - SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), - SecretBackendConfig::HashiCorpVault(settings) => { - get_or_create_vault_backend(db, settings).await - } - SecretBackendConfig::AzureKeyVault(settings) => { - get_or_create_azure_kv_backend(db, settings).await - } - SecretBackendConfig::AwsSecretsManager(settings) => { - get_or_create_aws_sm_backend(db, settings).await - } - } -} - -/// Get a cached Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_vault_backend( - _db: &DB, - settings: VaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = VAULT_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = VAULT_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = { - #[cfg(feature = "openidconnect")] - if settings.token.is_none() { - Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) - } else { - Arc::new(VaultBackend::new(settings.clone())) - } - - #[cfg(not(feature = "openidconnect"))] - Arc::new(VaultBackend::new(settings.clone())) - }; - - // Cache it - *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached Azure Key Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_azure_kv_backend( - _db: &DB, - settings: AzureKeyVaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = AZURE_KV_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = AZURE_KV_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); - - // Cache it - *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached AWS SM backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_aws_sm_backend( - _db: &DB, - settings: AwsSecretsManagerSettings, -) -> Result> { - { - let cache = AWS_SM_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - let mut cache = AWS_SM_BACKEND_CACHE.write().await; - - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - let backend: Arc = - Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); - - *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Check if an external secret backend is currently configured (EE only) -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn is_vault_backend_configured(db: &DB) -> Result { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - Ok(matches!( - config, - SecretBackendConfig::HashiCorpVault(_) - | SecretBackendConfig::AzureKeyVault(_) - | SecretBackendConfig::AwsSecretsManager(_) - )) -} - -/// Check if a value is stored in Vault (indicated by the $vault: prefix) -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_vault_stored_value(value: &str) -> bool { - value.starts_with("$vault:") -} - -/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_azure_kv_stored_value(value: &str) -> bool { - value.starts_with("$azure_kv:") -} - -/// Check if a value is stored in AWS Secrets Manager -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_aws_sm_stored_value(value: &str) -> bool { - value.starts_with("$aws_sm:") -} - -/// Check if a value is stored in any external secret backend -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_external_stored_value(value: &str) -> bool { - is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) -} - /// Bulk rename secrets in Vault when a path prefix changes (e.g., user rename) /// EE only feature. /// diff --git a/backend/windmill-api/src/token.rs b/backend/windmill-api/src/token.rs index ac4da6497e..7a4a4887f8 100644 --- a/backend/windmill-api/src/token.rs +++ b/backend/windmill-api/src/token.rs @@ -98,6 +98,7 @@ fn build_standard_scope_domains() -> Vec { ("configs", "Configs", "Configuration management", false), ("oauth", "OAuth", "OAuth management", false), ("ai", "AI", "AI feature management", false), + ("ai_skills", "AI Skills", "AI skill management", false), ( "agent_workers", "Agent Workers", @@ -193,6 +194,19 @@ lazy_static! { ], }]; + // Read-only: `/api/docs/*` exposes only GET routes, so there is no + // `docs:write`. Kept out of build_standard_scope_domains (which mints a + // read+write pair) for that reason. + groups.push(ScopeDomain { + name: "Documentation".to_string(), + description: Some("Read-only documentation search".to_string()), + scopes: vec![ScopeOption { + value: "docs:read".to_string(), + label: "Read".to_string(), + requires_resource_path: false, + }], + }); + groups.extend(build_standard_scope_domains()); groups.extend(build_trigger_scope_domains()); @@ -207,3 +221,21 @@ pub fn global_service() -> Router { async fn get_all_available_scopes() -> JsonResult> { Ok(Json(ALL_SCOPES.clone())) } + +#[cfg(test)] +mod tests { + use super::*; + + /// The token-scope picker is driven by this catalog, so a scope that is + /// enforced but absent here can't be granted through the supported UI. + #[test] + fn docs_read_scope_is_exposed_read_only() { + let values: Vec<&str> = ALL_SCOPES + .iter() + .flat_map(|d| d.scopes.iter()) + .map(|s| s.value.as_str()) + .collect(); + assert!(values.contains(&"docs:read"), "docs:read must be selectable"); + assert!(!values.contains(&"docs:write"), "docs has no write surface"); + } +} diff --git a/backend/windmill-api/src/triggers/http/handler.rs b/backend/windmill-api/src/triggers/http/handler.rs index 0fd7717582..0508586182 100644 --- a/backend/windmill-api/src/triggers/http/handler.rs +++ b/backend/windmill-api/src/triggers/http/handler.rs @@ -216,7 +216,7 @@ async fn get_http_route_trigger( let email = windmill_common::users::get_email_from_permissioned_as( &trigger.permissioned_as, &trigger.workspace_id, - &db, + db, ) .await?; let authed = windmill_api_auth::fetch_api_authed_from_permissioned_as( diff --git a/backend/windmill-api/src/users.rs b/backend/windmill-api/src/users.rs index 0da987e675..36b0d6c734 100644 --- a/backend/windmill-api/src/users.rs +++ b/backend/windmill-api/src/users.rs @@ -11,7 +11,7 @@ pub use windmill_api_users::users::*; use std::sync::Arc; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use argon2::Argon2; use axum::{ @@ -21,7 +21,7 @@ use axum::{ }; use hyper::StatusCode; use serde::Deserialize; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; use windmill_common::audit::AuditAuthor; @@ -71,11 +71,13 @@ pub fn make_unauthed_service() -> Router { async fn create_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Extension(webhook): Extension, Extension(argon2): Extension>>, Json(nu): Json, ) -> Result<(StatusCode, String)> { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::create_user(authed, db, webhook, argon2, nu).await } @@ -141,8 +143,10 @@ async fn set_password( Extension(db): Extension, Extension(argon2): Extension>>, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let email = authed.email.clone(); crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -152,9 +156,11 @@ async fn set_password_of_user( Extension(argon2): Extension>>, Path(email): Path, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -165,11 +171,13 @@ struct RenameUser { async fn rename_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(user_email): Path, Extension(db): Extension, Json(ru): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; diff --git a/backend/windmill-api/src/workspaces_export.rs b/backend/windmill-api/src/workspaces_export.rs index dae1be0614..839057b4b9 100644 --- a/backend/windmill-api/src/workspaces_export.rs +++ b/backend/windmill-api/src/workspaces_export.rs @@ -12,6 +12,8 @@ use crate::db::ApiAuthed; use crate::{apps::AppWithLastVersion, db::DB, folders::Folder}; +use windmill_api_auth::check_scopes; + #[cfg(any( feature = "http_trigger", feature = "websocket", @@ -582,6 +584,18 @@ pub(crate) async fn tarball_workspace( skip_resources ); + // The route is gated by workspaces:read, but exporting DECRYPTED secrets is a + // variable-read capability beyond workspace metadata. Require variables:read + // only on the plaintext-secret path: ordinary tarball pulls (structure and + // encrypted-only values) keep working with workspaces:read, and the workspace + // key itself stays admin-only (include_key). No-op for unscoped tokens. + if plain_secret.or(plain_secrets).unwrap_or(false) + && !skip_secrets.unwrap_or(false) + && !skip_variables.unwrap_or(false) + { + check_scopes(&authed, || "variables:read".to_string())?; + } + // Opt-in behavior for surfacing per-resource ACLs on flow/app rows. // Folder and group rows have always carried `extra_perms` in source and // continue to do so unconditionally (`KeepEvenEmpty`) so existing @@ -592,8 +606,37 @@ pub(crate) async fn tarball_workspace( ExtraPermsBehavior::Drop }; + // Resolve workspace dependencies on the pool *before* opening the RLS + // transaction: fetching them mid-transaction would hold a second + // simultaneous connection while `tx` is still checked out. + let workspace_dependencies = if include_workspace_dependencies.unwrap_or(false) + && require_admin(authed.is_admin, &authed.username).is_ok() + { + Some(WorkspaceDependencies::list(&w_id, &db).await?) + } else { + None + }; + let mut tx = user_db.begin(&authed).await?; + // Exporting decrypted secrets in bulk is the same capability as a per-item + // secret read, so record it for parity with variables.decrypt_secret. + if plain_secret.or(plain_secrets).unwrap_or(false) + && !skip_variables.unwrap_or(false) + && !skip_secrets.unwrap_or(false) + { + windmill_audit::audit_oss::audit_log( + &mut *tx, + &authed, + "variables.decrypt_secret", + windmill_audit::ActionKind::Execute, + &w_id, + Some("workspace_tarball_export"), + None, + ) + .await?; + } + // Source-of-truth for fork-ness: the workspace's parent_workspace_id column. // The wm-fork-* prefix is a creation-time naming convention that could in // principle drift (rename, manual SQL); the column is the contract that @@ -851,11 +894,8 @@ pub(crate) async fn tarball_workspace( } } - if include_workspace_dependencies.unwrap_or(false) - && require_admin(authed.is_admin, &authed.username).is_ok() - { + if let Some(workspace_dependencies) = workspace_dependencies { tracing::info!("Including workspace dependencies in tarball export"); - let workspace_dependencies = WorkspaceDependencies::list(&w_id, &db).await?; tracing::info!( "Found {} workspace dependencies", workspace_dependencies.len() @@ -879,11 +919,16 @@ pub(crate) async fn tarball_workspace( } if include_schedules.unwrap_or(false) { + // Managed ducklake-maintenance schedules are excluded: they are + // derived from the workspace ducklake settings (and admins bypass the + // RLS that hides them), so exporting them would drag unsyncable rows + // into git. let schedules = sqlx::query_as::<_, Schedule>( "SELECT workspace_id, path, edited_by, edited_at, schedule, timezone, enabled, script_path, is_flow, args, extra_perms, email, permissioned_as, error, on_failure, on_failure_times, on_failure_exact, on_failure_extra_args, on_recovery, on_recovery_times, on_recovery_extra_args, on_success, on_success_extra_args, ws_error_handler_muted, retry, no_flow_overlap, summary, description, tag, paused_until, cron_version, dynamic_skip, labels FROM schedule - WHERE workspace_id = $1", + WHERE workspace_id = $1 AND NOT starts_with(path, $2)", ) .bind(&w_id) + .bind(windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX) .fetch_all(&mut *tx) .await?; @@ -1501,6 +1546,34 @@ pub(crate) async fn tarball_workspace( .await?; } + { + // Data table migrations live in the `datatable_migrations` table; surface + // them in the export as `migrations/datatable//_` + // .up.sql (and .down.sql when present) so `wmill sync` treats them like any + // other workspace item. + let migrations = sqlx::query!( + "SELECT datatable, timestamp, name, code_up, code_down FROM datatable_migrations \ + WHERE workspace_id = $1 ORDER BY datatable, timestamp", + &w_id + ) + .fetch_all(&mut *tx) + .await?; + for m in migrations { + let base = format!( + "migrations/datatable/{}/{}_{}", + m.datatable, m.timestamp, m.name + ); + archive + .write_to_archive(&m.code_up, &format!("{base}.up.sql")) + .await?; + if let Some(code_down) = m.code_down { + archive + .write_to_archive(&code_down, &format!("{base}.down.sql")) + .await?; + } + } + } + archive.finish().await?; let file = tokio::fs::File::open(&file_path).await?; diff --git a/backend/windmill-common/src/assets.rs b/backend/windmill-common/src/assets.rs index 2b70227454..62f6d32070 100644 --- a/backend/windmill-common/src/assets.rs +++ b/backend/windmill-common/src/assets.rs @@ -4,9 +4,47 @@ use sqlx::{PgExecutor, Postgres, Transaction}; use crate::{error, scripts::ScriptHash}; -pub use windmill_parser::asset_parser::{parse_pipeline_annotations, TriggerSpec, PARTITION_TOKEN}; +pub use windmill_parser::asset_parser::{ + merge_column_lineage, parse_pipeline_annotations, ColumnLineage, ColumnRef, DataTest, + OnSchemaChange, PartitionKind, PipelineAnnotations, RetrySpec, TriggerSpec, PARTITION_TOKEN, +}; pub use windmill_types::assets::*; +// --- Workspace DuckDB macro registry cache (worker hot path) --- +// Every DuckDB job reads the `macro_definition` registry; cascades can run +// many jobs per second. Primary invalidation is the +// `notify_macro_registry_change` event, emitted transactionally with every +// registry mutation and dispatched to this cache in main.rs — the TTL is a +// backstop for mutation paths that don't emit (manual SQL edits). + +#[derive(Clone, Debug, sqlx::FromRow)] +pub struct MacroRegistryEntry { + pub name: String, + pub params: String, + pub body: String, + pub is_table_macro: bool, + pub provider_path: String, +} + +#[derive(Clone)] +pub struct ExpiringMacroRegistry { + pub rows: std::sync::Arc>, + pub expires_at: std::time::Instant, +} + +pub const MACRO_REGISTRY_TTL: std::time::Duration = std::time::Duration::from_secs(60); + +// Tests sharing one process across several databases must disable the cache: +// it is keyed by workspace id alone, and a notify from one DB can't evict +// entries populated from another (same hazard as DEPLOYED_SCRIPT_CACHE_DISABLED). +pub static MACRO_REGISTRY_CACHE_DISABLED: std::sync::atomic::AtomicBool = + std::sync::atomic::AtomicBool::new(false); + +lazy_static::lazy_static! { + pub static ref MACRO_REGISTRY_CACHE: quick_cache::sync::Cache = + quick_cache::sync::Cache::new(1000); +} + #[derive(sqlx::Type, Debug, Clone, Copy, PartialEq)] #[sqlx(type_name = "SCRIPT_TRIGGER_KIND", rename_all = "lowercase")] pub enum ScriptTriggerKind { @@ -131,6 +169,68 @@ fn is_write_access(access: Option) -> bool { ) } +/// Kinds whose *read* usage auto-derives a cascade trigger edge inside a +/// `// pipeline`. Scoped to the two intra-pipeline data kinds — a ducklake +/// table read (the core case) and an s3 object read (file-ingestion +/// producers). Resource / datatable / volume reads stay explicit-`// on`: +/// a config/lookup read cascading is more often surprising than wanted. +fn is_auto_trigger_kind(kind: AssetKind) -> bool { + matches!(kind, AssetKind::Ducklake | AssetKind::S3Object) +} + +/// Trigger refs auto-derived from a pipeline script's inferred reads, so the +/// FROM clause alone wires the cascade edge (no redundant `// on `). +/// +/// Included: an input read read-*only* (`R`) of a supported kind +/// ([`is_auto_trigger_kind`]). The effective access type is +/// `access_type.or(alt_access_type)` — same precedence as the persisted +/// `asset.usage_access_type` and the frontend mirror's `access_type ?? +/// alt_access_type`, so a manual read override on an ambiguous parse still +/// derives an edge (and the live canvas and the deployed graph agree). +/// Excluded, each for a reason: +/// - `RW` / `W` — the script also writes the asset; an edge would be a +/// self-triggering loop. +/// - `None` access — usage is ambiguous (poisoned merge) with no override; +/// can't confirm a read, so fail safe and don't cascade. +/// - already in `explicit_refs` — the author wrote `// on `, which +/// wins (it carries the per-edge debounce/opts). +/// - in `muted_refs` — a `// mute ` opt-out (lookup / SCD input). +/// +/// `mute_all` (from `// mute all`) short-circuits to no derivation, leaving +/// only the explicit `// on` edges. Returns canonical refs (e.g. +/// `ducklake://main.orders`), deduped, in input order. +pub fn derive_pipeline_asset_trigger_refs( + assets: &[AssetWithAltAccessType], + explicit_refs: &HashSet, + muted_refs: &HashSet, + mute_all: bool, +) -> Vec { + if mute_all { + return vec![]; + } + let mut out = vec![]; + let mut seen = HashSet::new(); + for a in assets { + // Effective access mirrors the persisted `usage_access_type` and the + // frontend derivation: an explicit parse wins, else the manual override. + let access = a.access_type.or(a.alt_access_type); + if access != Some(AssetUsageAccessType::R) || !is_auto_trigger_kind(a.kind) { + continue; + } + let Some(prefix) = a.kind.canonical_prefix() else { + continue; + }; + let r = format!("{}{}", prefix, a.path); + if explicit_refs.contains(&r) || muted_refs.contains(&r) { + continue; + } + if seen.insert(r.clone()) { + out.push(r); + } + } + out +} + /// Clear and reinsert the full static-asset usage set of a script in one tx, /// invalidating the producer-writes cache at most once and only on a real /// change. The cache (asset_dispatch::ASSET_PRODUCER_WRITES_CACHE) keys a @@ -314,6 +414,10 @@ mod debounce_duration_tests { assert_eq!(parse_duration_secs("5m"), Some(300)); assert_eq!(parse_duration_secs("2h"), Some(7200)); assert_eq!(parse_duration_secs(" 1d "), Some(86400)); + // Explicit plus sign comes free with i64 parsing; the TS mirror + // (parseDurationSecs) matches it — keep the two in lockstep. + assert_eq!(parse_duration_secs("+5m"), Some(300)); + assert_eq!(parse_duration_secs("+45"), Some(45)); } #[test] @@ -327,6 +431,173 @@ mod debounce_duration_tests { } } +#[cfg(test)] +mod derive_trigger_tests { + use super::{derive_pipeline_asset_trigger_refs, AssetKind, AssetUsageAccessType}; + use std::collections::HashSet; + use windmill_types::assets::AssetWithAltAccessType; + + fn asset( + kind: AssetKind, + path: &str, + at: Option, + ) -> AssetWithAltAccessType { + AssetWithAltAccessType { + path: path.to_string(), + kind, + access_type: at, + alt_access_type: None, + columns: None, + } + } + + fn derive(assets: &[AssetWithAltAccessType]) -> Vec { + derive_pipeline_asset_trigger_refs(assets, &HashSet::new(), &HashSet::new(), false) + } + + #[test] + fn read_only_ducklake_and_s3_derive_an_edge() { + use AssetUsageAccessType::R; + let a = [ + asset(AssetKind::Ducklake, "main.orders", Some(R)), + asset(AssetKind::S3Object, "raw/events", Some(R)), + ]; + assert_eq!( + derive(&a), + vec![ + "ducklake://main.orders".to_string(), + "s3://raw/events".to_string() + ] + ); + } + + #[test] + fn writes_and_rw_are_skipped_to_avoid_self_edges() { + use AssetUsageAccessType::*; + // W (pure producer) and RW (reads *and* writes the same table — a + // self-cascade if edged) both derive nothing. + let a = [ + asset(AssetKind::Ducklake, "main.out", Some(W)), + asset(AssetKind::Ducklake, "main.self", Some(RW)), + ]; + assert!(derive(&a).is_empty()); + } + + #[test] + fn ambiguous_access_and_unsupported_kinds_are_skipped() { + use AssetUsageAccessType::R; + let a = [ + asset(AssetKind::Ducklake, "main.ambiguous", None), // poisoned merge + asset(AssetKind::Resource, "f/db", Some(R)), // out of scope + asset(AssetKind::DataTable, "main.dt", Some(R)), // out of scope + ]; + assert!(derive(&a).is_empty()); + } + + #[test] + fn manual_read_override_on_ambiguous_parse_derives_an_edge() { + use AssetUsageAccessType::{R, W}; + // Parser can't confirm access (`access_type: None`) but the user manually + // overrode it. Effective access = `access_type.or(alt_access_type)`, the + // same value persisted to `asset.usage_access_type` and used by the + // frontend canvas — so a read override derives an edge (parity, no + // silently-vanishing edge on deploy) and a write override does not. + let read_override = AssetWithAltAccessType { + path: "main.override_r".to_string(), + kind: AssetKind::Ducklake, + access_type: None, + alt_access_type: Some(R), + columns: None, + }; + let write_override = AssetWithAltAccessType { + path: "main.override_w".to_string(), + kind: AssetKind::Ducklake, + access_type: None, + alt_access_type: Some(W), + columns: None, + }; + assert_eq!( + derive(&[read_override, write_override]), + vec!["ducklake://main.override_r".to_string()] + ); + } + + #[test] + fn explicit_and_muted_refs_are_excluded() { + use AssetUsageAccessType::R; + let a = [ + asset(AssetKind::Ducklake, "main.explicit", Some(R)), + asset(AssetKind::Ducklake, "main.muted", Some(R)), + asset(AssetKind::Ducklake, "main.keep", Some(R)), + ]; + let explicit: HashSet = ["ducklake://main.explicit".to_string()].into(); + let muted: HashSet = ["ducklake://main.muted".to_string()].into(); + assert_eq!( + derive_pipeline_asset_trigger_refs(&a, &explicit, &muted, false), + vec!["ducklake://main.keep".to_string()] + ); + } + + #[test] + fn mute_all_derives_nothing() { + use AssetUsageAccessType::R; + let a = [asset(AssetKind::Ducklake, "main.orders", Some(R))]; + assert!( + derive_pipeline_asset_trigger_refs(&a, &HashSet::new(), &HashSet::new(), true) + .is_empty() + ); + } + + #[test] + fn duplicate_reads_dedup() { + use AssetUsageAccessType::R; + let a = [ + asset(AssetKind::Ducklake, "main.orders", Some(R)), + asset(AssetKind::Ducklake, "main.orders", Some(R)), + ]; + assert_eq!(derive(&a), vec!["ducklake://main.orders".to_string()]); + } +} + +#[cfg(test)] +mod trigger_ref_roundtrip_tests { + use super::{parse_asset_trigger_ref, trigger_spec_to_row, AssetKind, ScriptTriggerKind}; + use windmill_parser::asset_parser::{parse_asset_syntax, AssetKind as PAssetKind, TriggerSpec}; + + // `trigger_spec_to_row` rebuilds a stored ref as `s3://`, and + // `parse_asset_trigger_ref` parses it back. The two must be inverse for + // every S3 URI form, or a consumer's `// on` trigger lands on a different + // graph node than the producer's inferred write. Because `parse_asset_syntax` + // strips ALL leading slashes, a canonical path never starts with `/`, so the + // naive `prefix + path` rebuild round-trips — including the `S3Object(s3="/x")` + // quad-slash case that previously desynced (path `/x` rebuilt to `s3:///x`, + // which re-parsed to `x`). + fn roundtrip(uri: &str) -> String { + let (pkind, path) = parse_asset_syntax(uri, false).expect("parse uri"); + assert_eq!(pkind, PAssetKind::S3Object); + let spec = TriggerSpec::Asset { asset_kind: pkind, path: path.to_string(), debounce: None }; + let (kind, stored) = trigger_spec_to_row(&spec).expect("to row"); + assert_eq!(kind, ScriptTriggerKind::Asset); + let (rkind, rpath) = parse_asset_trigger_ref(&stored).expect("parse ref"); + assert_eq!(rkind, AssetKind::S3Object); + // The producer path and the round-tripped consumer path must match. + assert_eq!( + rpath, path, + "round-trip diverged for {uri} (stored {stored})" + ); + rpath + } + + #[test] + fn s3_trigger_ref_roundtrips_for_every_uri_form() { + assert_eq!(roundtrip("s3:///exports/x"), "exports/x"); // SDK default storage + assert_eq!(roundtrip("s3://exports/x"), "exports/x"); // DuckDB / bare + assert_eq!(roundtrip("s3://mybucket/exports/x"), "mybucket/exports/x"); // explicit + assert_eq!(roundtrip("s3:////x"), "x"); // S3Object(s3="/x") quad-slash + assert_eq!(roundtrip("s3:///y=2024/f.parquet"), "y=2024/f.parquet"); // Hive + } +} + // Inverse of trigger_spec_to_row for the Asset variant: parses a stored // trigger_ref (e.g. `s3://foo`, `$res:bar`) back into the (kind, path) pair // used as a graph node id. Returns None for refs that don't match any known diff --git a/backend/windmill-common/src/auth.rs b/backend/windmill-common/src/auth.rs index 7fc6a0825a..bd6bf1ef9d 100644 --- a/backend/windmill-common/src/auth.rs +++ b/backend/windmill-common/src/auth.rs @@ -287,7 +287,7 @@ impl From for Authed { } } -pub async fn is_super_admin_email(db: &DB, email: &str) -> Result { +pub async fn is_super_admin_email<'c>(db: impl sqlx::PgExecutor<'c>, email: &str) -> Result { if email == SUPERADMIN_SECRET_EMAIL || email == SUPERADMIN_NOTIFICATION_EMAIL { return Ok(true); } @@ -413,9 +413,18 @@ async fn fetch_authed_from_permissioned_as_inner( }) } } else { + // Bare (no `u/`|`g/` prefix) permissioned_as is reached for superadmins + // whose identifier is their email (they are not a workspace member). Use + // the instance-derived username when available so no email leaks + // downstream as the acting username. + let username = if is_super_admin && permissioned_as == email { + crate::usernames::get_instance_username_or_fallback_to_email(&mut *conn, email).await? + } else { + permissioned_as.to_string() + }; Ok(Authed { email: email.to_string(), - username: permissioned_as.to_string(), + username, is_admin: is_super_admin, is_operator: true, groups: vec![], diff --git a/backend/windmill-common/src/bench.rs b/backend/windmill-common/src/bench.rs index 1ed2b9aa56..68c3cb5a31 100644 --- a/backend/windmill-common/src/bench.rs +++ b/backend/windmill-common/src/bench.rs @@ -258,6 +258,8 @@ pub async fn benchmark_init(benchmark_jobs: i32, db: &DB) { .execute(db) .await .unwrap_or_else(|e| panic!("failed to clean up concurrency_counter: {e:#}")); + // Benchmark jobs never produce dispatch_event / flow_conversation_message / + // zombie_job_counter rows, so this cleanup needs no side-table deletes (cf. delete_jobs). sqlx::query!("DELETE FROM v2_job WHERE workspace_id = 'admins'") .execute(db) .await diff --git a/backend/windmill-common/src/cache.rs b/backend/windmill-common/src/cache.rs index e39fd6ea43..6c34db8169 100644 --- a/backend/windmill-common/src/cache.rs +++ b/backend/windmill-common/src/cache.rs @@ -1189,11 +1189,25 @@ const _: () = { use std::fs::OpenOptions; use std::io::Write; + // Atomic write: truncate+write a uniquely-named temp file (UUID, not pid — pids + // collide across container PID namespaces on a shared cache volume), fsync, then + // rename(2) over the target. Without this a shorter overwrite leaves a stale tail + // and concurrent writers tear the file — a corrupt entry a reader would import. + let final_path = item.path(self); + let tmp_path = final_path.with_extension(format!("tmp.{}", Uuid::new_v4())); OpenOptions::new() .write(true) .create(true) - .open(item.path(self)) - .and_then(|mut file| file.write_all(data.as_ref())) + .truncate(true) + .open(&tmp_path) + .and_then(|mut file| { + file.write_all(data.as_ref())?; + file.sync_all() + }) + .and_then(|()| std::fs::rename(&tmp_path, &final_path)) + .inspect_err(|_| { + let _ = std::fs::remove_file(&tmp_path); + }) } } @@ -1237,6 +1251,33 @@ mod tests { use super::*; use serde_json::json; + #[test] + fn fs_cache_put_overwrites_without_stale_tail() { + // Regression for the non-truncating, non-atomic `put`: overwriting a value with a + // shorter one must not leave stale trailing bytes (which imported as corrupt/wrong + // content — the #9751 worker-cache hazard). + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + + root.put("k", b"a-long-cached-value-0123456789").unwrap(); + assert_eq!(root.get("k").unwrap(), b"a-long-cached-value-0123456789"); + + root.put("k", b"short").unwrap(); + assert_eq!( + root.get("k").unwrap(), + b"short", + "shorter overwrite must fully replace, no stale tail" + ); + + // No temp files left behind after a successful write. + let leftover: Vec<_> = std::fs::read_dir(root) + .unwrap() + .filter_map(|e| e.ok()) + .filter(|e| e.file_name().to_string_lossy().contains(".tmp.")) + .collect(); + assert!(leftover.is_empty(), "temp files must be renamed/cleaned up"); + } + #[test] fn flow_data_extras_preserves_notes_and_groups() { let raw = serde_json::value::to_raw_value(&json!({ diff --git a/backend/windmill-common/src/ee_oss.rs b/backend/windmill-common/src/ee_oss.rs index e68bc00c6e..26b9c87b7e 100644 --- a/backend/windmill-common/src/ee_oss.rs +++ b/backend/windmill-common/src/ee_oss.rs @@ -20,6 +20,7 @@ lazy_static::lazy_static! { pub static ref LICENSE_KEY: arc_swap::ArcSwap = arc_swap::ArcSwap::from_pointee("".to_string()); pub static ref LICENSE_OFFLINE_METADATA: arc_swap::ArcSwap> = arc_swap::ArcSwap::from_pointee(None); pub static ref LICENSE_OFFLINE_OVER_CU_CAP: AtomicBool = AtomicBool::new(false); + pub static ref LICENSE_OFFLINE_OVER_SEAT_CAP: AtomicBool = AtomicBool::new(false); pub static ref LICENSE_OFFLINE_LAST_STATUS: arc_swap::ArcSwap> = arc_swap::ArcSwap::from_pointee(None); pub static ref LICENSE_OFFLINE_LAST_CHECKED_AT: arc_swap::ArcSwap>> = arc_swap::ArcSwap::from_pointee(None); } @@ -62,6 +63,14 @@ pub async fn check_seat_cap_for_new_user( Ok(None) } +#[cfg(all(feature = "enterprise", not(feature = "private")))] +pub async fn check_seat_cap_for_reactivation( + _db: &DB, + _email: &str, +) -> anyhow::Result> { + Ok(None) +} + #[cfg(all(feature = "enterprise", not(feature = "private")))] pub async fn compute_instance_hash(_db: &DB) -> anyhow::Result> { // Implementation is not open source diff --git a/backend/windmill-common/src/git_sync_oss.rs b/backend/windmill-common/src/git_sync_oss.rs index 4cd7a7c3d1..1e17532ec7 100644 --- a/backend/windmill-common/src/git_sync_oss.rs +++ b/backend/windmill-common/src/git_sync_oss.rs @@ -15,6 +15,29 @@ pub async fn get_github_app_token_internal( )); } +lazy_static::lazy_static! { + /// Matches a `user:password@` (or `user@`) userinfo component right after the URL scheme. + static ref GIT_URL_USERINFO_RE: regex::Regex = + regex::Regex::new(r"://[^/@]+@").unwrap(); +} + +/// Strip embedded credentials (the `user:password@` userinfo component) from a git URL so it can be +/// safely included in error messages and logs. Falls back to a regex when the URL does not parse. +pub fn sanitize_git_url(url: &str) -> String { + if let Ok(mut parsed) = Url::parse(url) { + if !parsed.username().is_empty() || parsed.password().is_some() { + // These setters only fail for cannot-be-a-base URLs, in which case we keep the parsed + // string as-is and let the regex fallback below handle stripping. + let _ = parsed.set_username(""); + let _ = parsed.set_password(None); + } + return GIT_URL_USERINFO_RE + .replace(parsed.as_str(), "://***@") + .into_owned(); + } + GIT_URL_USERINFO_RE.replace(url, "://***@").into_owned() +} + pub fn prepend_token_to_github_url( github_url: &str, installation_token: &str, @@ -32,3 +55,41 @@ pub fn prepend_token_to_github_url( url.path() )) } + +#[cfg(test)] +mod tests { + use super::sanitize_git_url; + + #[test] + fn strips_username_and_password() { + assert_eq!( + sanitize_git_url("https://user:p4ssw0rd@github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn strips_token_only_userinfo() { + assert_eq!( + sanitize_git_url("https://ghp_secrettoken@github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn leaves_credential_free_url_untouched() { + assert_eq!( + sanitize_git_url("https://github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn strips_credentials_from_unparseable_url() { + // scp-like syntax that `url::Url` cannot parse + assert_eq!( + sanitize_git_url("not a url://user:secret@host/repo"), + "not a url://***@host/repo" + ); + } +} diff --git a/backend/windmill-common/src/jobs.rs b/backend/windmill-common/src/jobs.rs index 78f5a6ee5a..2178d9a023 100644 --- a/backend/windmill-common/src/jobs.rs +++ b/backend/windmill-common/src/jobs.rs @@ -343,8 +343,8 @@ lazy_static::lazy_static! { ).unwrap_or(false); } -pub async fn check_tag_available_for_workspace_internal( - db: &DB, +pub async fn check_tag_available_for_workspace_internal<'c>( + db: impl sqlx::PgExecutor<'c>, w_id: &str, tag: &str, email: &str, @@ -458,6 +458,47 @@ pub struct WorkerInternalServerInlineUtils { pub static WORKER_INTERNAL_SERVER_INLINE_UTILS: OnceCell = OnceCell::new(); +/// Deletes the given jobs from `v2_job` together with the side tables that reference it +/// without an `ON DELETE CASCADE` foreign key. +/// +/// **Authorization contract:** this helper does NO authorization and NO workspace scoping — +/// it deletes exactly the `ids` passed, regardless of which workspace they belong to. Callers +/// MUST ensure `ids` only contains jobs the caller is allowed to delete (either a trusted +/// internal id set, e.g. a retention batch, or ids already filtered by `workspace_id`). +/// Passing user-supplied, unvalidated ids would reintroduce the cross-workspace side-row +/// deletion this centralizes. It is deliberately not workspace-scoped at the signature level +/// because its primary caller — retention — deletes expired jobs across every workspace at +/// once; a `workspace_id` parameter cannot express that. (This is the same trust model as the +/// `ON DELETE CASCADE` FK it replaces: given a job id, the row and its side rows go.) +/// +/// Those FKs were removed (migration `drop_v2_job_side_table_cascades`) because they turned +/// every bulk retention delete into a per-row RI trigger; for the unindexed +/// `flow_conversation_message.job_id` that was a sequential scan per deleted row. The +/// set-based deletes below cost one scan per table per call instead. Because the cascade no +/// longer fires, every code path that deletes from `v2_job` by id must go through this helper +/// (or delete these tables itself) or it will leave orphan rows behind. +pub async fn delete_jobs(conn: &mut sqlx::PgConnection, ids: &[uuid::Uuid]) -> error::Result<()> { + sqlx::query!( + "DELETE FROM dispatch_event WHERE producer_job_id = ANY($1)", + ids + ) + .execute(&mut *conn) + .await?; + sqlx::query!( + "DELETE FROM flow_conversation_message WHERE job_id = ANY($1)", + ids + ) + .execute(&mut *conn) + .await?; + sqlx::query!("DELETE FROM zombie_job_counter WHERE job_id = ANY($1)", ids) + .execute(&mut *conn) + .await?; + sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", ids) + .execute(&mut *conn) + .await?; + Ok(()) +} + #[cfg(test)] mod tests { use super::is_safe_log_file_path; diff --git a/backend/windmill-common/src/lib.rs b/backend/windmill-common/src/lib.rs index c501bbed6e..acf97734d8 100644 --- a/backend/windmill-common/src/lib.rs +++ b/backend/windmill-common/src/lib.rs @@ -61,9 +61,11 @@ pub mod indexer; pub mod instance_config; pub mod job_metrics; pub mod log_context; +pub mod materialization; pub mod min_version; pub mod notify_events; pub mod runtime_assets; +pub mod schema_contracts; pub mod workspace_dependencies; #[cfg(feature = "private")] @@ -267,6 +269,10 @@ lazy_static::lazy_static! { pub static ref INSTANCE_NAME: String = rd_string(5); pub static ref DEPLOYED_SCRIPT_HASH_CACHE: Cache<(String, String), ExpiringLatestVersionId> = Cache::new(1000); + // Latest non-archived version per (workspace, path) for bundle cache keying — + // looser predicate than DEPLOYED_SCRIPT_HASH_CACHE (no lock requirement), so + // the two must not share entries. See get_latest_script_hash_for_import_cached. + pub static ref IMPORTED_SCRIPT_HASH_CACHE: Cache<(String, String), ExpiringLatestVersionId> = Cache::new(1000); pub static ref FLOW_VERSION_CACHE: Cache<(String, String), ExpiringLatestVersionId> = Cache::new(1000); pub static ref DYNAMIC_INPUT_CACHE: Cache> = Cache::new(1000); pub static ref DEPLOYED_SCRIPT_INFO_CACHE: Cache<(String, i64), ScriptHashInfo> = Cache::new(1000); @@ -284,6 +290,16 @@ lazy_static::lazy_static! { const LATEST_VERSION_ID_CACHE_TTL: std::time::Duration = std::time::Duration::from_secs(60); +/// Test hook: disables the process-global deployed-script hash/info caches so +/// every resolution reads the current DB. Integration tests use `#[sqlx::test]` +/// isolated DBs that share one workspace id and reuse script paths, so a cache +/// keyed by `(workspace, path)`/`(workspace, hash)` resolves a path to a hash +/// that lives in a *different* test's DB — and when the info cache misses for +/// that foreign hash the lookup 404s in the wrong DB. Always `false` in +/// production (the caches are TTL/LRU-bounded against real deploys). +pub static DEPLOYED_SCRIPT_CACHE_DISABLED: std::sync::atomic::AtomicBool = + std::sync::atomic::AtomicBool::new(false); + pub async fn shutdown_signal( tx: KillpillSender, mut rx: tokio::sync::broadcast::Receiver<()>, @@ -528,6 +544,157 @@ mod classify_python_logging_line_tests { } } +#[cfg(test)] +mod validate_dbname_tests { + use super::validate_dbname; + + #[test] + fn accepts_letters_digits_underscores_and_hyphens() { + assert!(validate_dbname("mydb").is_ok()); + assert!(validate_dbname("my_db").is_ok()); + assert!(validate_dbname("my-database").is_ok()); + assert!(validate_dbname("My-Db_1").is_ok()); + } + + #[test] + fn rejects_invalid_names() { + // Must start with a letter (hyphen/digit/underscore leads are rejected). + assert!(validate_dbname("-db").is_err()); + assert!(validate_dbname("1db").is_err()); + assert!(validate_dbname("_db").is_err()); + // No other special characters or whitespace. + assert!(validate_dbname("my db").is_err()); + assert!(validate_dbname("my;db").is_err()); + assert!(validate_dbname("").is_err()); + } +} + +#[cfg(test)] +mod pg_tls_tests { + use super::PgDatabase; + + // A syntactically valid (self-signed) certificate, used only to exercise the + // "root certificate supplied" branch — its contents are never validated here. + const VALID_PEM: &str = "-----BEGIN CERTIFICATE-----\n\ +MIIDETCCAfmgAwIBAgIUX/yHsMoWBljFzJr5Xh7V2I6ykMEwDQYJKoZIhvcNAQEL\n\ +BQAwGDEWMBQGA1UEAwwNd2luZG1pbGwtdGVzdDAeFw0yNjA2MjkwOTUwNTlaFw0z\n\ +NjA2MjYwOTUwNTlaMBgxFjAUBgNVBAMMDXdpbmRtaWxsLXRlc3QwggEiMA0GCSqG\n\ +SIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvF2hMw8adQGG6EnDk8GsOIoHT+kLN1W0F\n\ +yYFwH1wGVmzVP1YNfUts8aQfMtl/ZjW7SQlvKeK+18id4fVNYvZpbFhj66IsKMOU\n\ +MnJHcC6X/IAdhANyhM1fcrS6YupanAKOhLPk4HYRD5tGI4Y1vzTnQKGffIZ0bof7\n\ +3GtCiJLv8wrJKszeoKPtdFazdW+CYePbFq3Owc7HMo8CwA7A5TsgcowELhCfYwZv\n\ +Pn/9v+NDHQO0jJclH7qK221RkbqZGD+nPJ4rUm7oRi0vfApBQZ0FFJZjiki/Kg2+\n\ +RACb6Ud/LOeRBerKQHbN8KeYnGafCaIC4s/XytVwxAz+kgK1qyl7AgMBAAGjUzBR\n\ +MB0GA1UdDgQWBBRo2Jby4SZlrwMNbhA4bswZcBNRyjAfBgNVHSMEGDAWgBRo2Jby\n\ +4SZlrwMNbhA4bswZcBNRyjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUA\n\ +A4IBAQBlED+FQW3GB3Wa1NdVN252vihuFNnbq81yvhf4T7dfAxwkxI9jiM+ZWCw2\n\ +g59FbLupj8Rwun5gE2H/9M8ZunISdlwaMH5nyDJlbRjttPfY1cEoyGEY+UXIslfg\n\ +BoiI5rOtz9R2qurxEic1VtEVfXhEuWwCG86vCBDdHrL/qqqUJEx/P8qyC7uVc8XC\n\ +uclnJVL7x1ax0jTmEPur9K+DQn2ws01mzpq2QwSunibpDL5D5xM1oYekv0tQFEkT\n\ +ta9ELulniZau8zUAtwqwecxodzl+KO8NYj0a9PGgAM64dMqkRtRA8P4UP350Nag3\n\ ++hOq1qpWD7yPVyycx/KCilICOKVf\n\ +-----END CERTIFICATE-----\n"; + + fn pg(sslmode: Option<&str>, root_cert: Option<&str>) -> PgDatabase { + PgDatabase { + host: "db.example.com".to_string(), + user: Some("u".to_string()), + password: Some("p".to_string()), + port: Some(5432), + sslmode: sslmode.map(|s| s.to_string()), + dbname: "mydb".to_string(), + root_certificate_pem: root_cert.map(|s| s.to_string()), + accept_invalid_certs: None, + use_iam_auth: None, + region: None, + } + } + + /// Whether the connector enforces certificate verification for the given config. + fn verifies( + sslmode: Option<&str>, + root_cert: Option<&str>, + accept_invalid_certs: Option, + ) -> bool { + let mut builder = native_tls::TlsConnector::builder(); + PgDatabase::configure_pg_tls_verification( + &mut builder, + sslmode, + root_cert, + accept_invalid_certs, + ) + .unwrap() + } + + #[test] + fn verify_modes_enforce_verification_when_explicitly_requested() { + // accept_invalid_certs=Some(false) is what newly created resources carry: it + // verifies even with no custom cert (against the OS trust store). + assert!(verifies(Some("verify-full"), None, Some(false))); + assert!(verifies(Some("verify-ca"), None, Some(false))); + assert!(verifies(Some("verify-full"), Some(""), Some(false))); + assert!(verifies(Some("verify-full"), Some(VALID_PEM), Some(false))); + assert!(verifies(Some("verify-ca"), Some(VALID_PEM), Some(false))); + } + + #[test] + fn verify_modes_unset_fall_back_to_legacy_behavior() { + // Unset (None): verify iff a root cert is present — preserves the behavior of + // resources that predate the flag (incl. git-synced), so upgrades don't break. + assert!(!verifies(Some("verify-full"), None, None)); + assert!(!verifies(Some("verify-ca"), None, None)); + assert!(!verifies(Some("verify-full"), Some(""), None)); + assert!(verifies(Some("verify-full"), Some(VALID_PEM), None)); + assert!(verifies(Some("verify-ca"), Some(VALID_PEM), None)); + } + + #[test] + fn accept_invalid_certs_true_disables_verification_for_verify_modes() { + assert!(!verifies(Some("verify-full"), Some(VALID_PEM), Some(true))); + assert!(!verifies(Some("verify-ca"), None, Some(true))); + } + + #[test] + fn accept_invalid_certs_is_ignored_outside_verify_modes() { + // require never consults the flag: it verifies iff a cert is present, and + // encrypts-without-verifying otherwise, regardless of accept_invalid_certs. + assert!(!verifies(Some("require"), None, Some(false))); + assert!(!verifies(Some("require"), None, Some(true))); + assert!(!verifies(None, None, Some(true))); + assert!(verifies(Some("require"), Some(VALID_PEM), Some(true))); + assert!(verifies(Some("require"), Some(VALID_PEM), None)); + } + + #[test] + fn invalid_pem_is_rejected() { + let mut builder = native_tls::TlsConnector::builder(); + let err = PgDatabase::configure_pg_tls_verification( + &mut builder, + Some("verify-full"), + Some("not a certificate"), + Some(false), + ); + assert!(err.is_err()); + } + + #[test] + fn to_uri_collapses_verify_modes_for_tokio_postgres() { + // to_uri() feeds tokio-postgres, which only parses disable/prefer/require; + // verify-* therefore map to require there (verification is connector-driven). + for mode in ["require", "verify-ca", "verify-full"] { + assert!( + pg(Some(mode), None).to_uri().contains("sslmode=require"), + "{mode} should map to sslmode=require in to_uri()" + ); + } + assert!(pg(Some("disable"), None) + .to_uri() + .contains("sslmode=disable")); + assert!(pg(Some("allow"), None).to_uri().contains("sslmode=prefer")); + assert!(pg(None, None).to_uri().contains("sslmode=prefer")); + } +} + #[derive(Serialize, Debug)] pub struct PrepareQueryColumnInfo { pub name: String, @@ -552,6 +719,12 @@ pub struct PgDatabase { pub sslmode: Option, pub dbname: String, pub root_certificate_pem: Option, + /// Only meaningful for sslmode verify-ca/verify-full. `Some(true)` accepts any + /// server certificate (no chain or hostname check); `Some(false)` enforces + /// verification. `None` falls back to legacy behavior — verify only when a root + /// certificate is present — so resources that predate this flag (including + /// git-synced ones, whose source never sets it) keep working unchanged. + pub accept_invalid_certs: Option, pub use_iam_auth: Option, pub region: Option, } @@ -640,10 +813,94 @@ impl PgDatabase { } } + /// True when sslmode requests verification (verify-ca/verify-full) but the + /// effective configuration disables it, so the server's identity is not + /// checked. Mirrors the verify-* decision in `configure_pg_tls_verification`. + pub fn verify_mode_skips_verification(&self) -> bool { + matches!( + self.sslmode.as_deref(), + Some("verify-ca") | Some("verify-full") + ) && self.accept_invalid_certs.unwrap_or( + self.root_certificate_pem + .as_deref() + .unwrap_or("") + .is_empty(), + ) + } + + /// Configure certificate and hostname verification on a native-tls connector + /// according to the requested Postgres `sslmode`. The crates.io tokio-postgres + /// build only parses disable/prefer/require, so verify-ca and verify-full are + /// enforced here, on the connector, rather than through the connection URI. + /// + /// verify-full — verify the certificate chain AND that it matches the host. + /// verify-ca — verify the chain only; libpq does not check the hostname. + /// require / other — encrypt without verifying identity, unless a root + /// certificate is supplied (then verify the chain). + /// + /// `accept_invalid_certs` only applies to verify-ca/verify-full: `Some(true)` + /// accepts any certificate, `Some(false)` enforces verification, and `None` + /// falls back to the legacy behavior — verify only when a root certificate is + /// present — so resources predating the flag (including git-synced ones, whose + /// source never sets it) keep working unchanged. Verification uses the OS trust + /// store plus any supplied root certificate. Returns false when the connector + /// was set to accept any certificate, so callers can surface that an unverified + /// connection is being made. + fn configure_pg_tls_verification( + builder: &mut native_tls::TlsConnectorBuilder, + sslmode: Option<&str>, + root_certificate_pem: Option<&str>, + accept_invalid_certs: Option, + ) -> Result { + use native_tls::Certificate; + + let custom_root = match root_certificate_pem { + Some(pem) if !pem.is_empty() => Some( + Certificate::from_pem(pem.as_bytes()) + .map_err(|e| error::Error::BadConfig(format!("Invalid Certs: {e:#}")))?, + ), + _ => None, + }; + + match sslmode { + Some("verify-full") | Some("verify-ca") => { + // Unset falls back to the legacy behavior: verify iff a cert is present. + if accept_invalid_certs.unwrap_or(custom_root.is_none()) { + builder + .danger_accept_invalid_certs(true) + .danger_accept_invalid_hostnames(true); + return Ok(false); + } + if let Some(cert) = custom_root { + builder.add_root_certificate(cert); + } + if sslmode == Some("verify-ca") { + // verify-ca verifies the chain but, per libpq, not the hostname. + builder.danger_accept_invalid_hostnames(true); + } + Ok(true) + } + _ => { + // "require": accept_invalid_certs does not apply. Encrypt but do not + // verify identity, unless an explicit root certificate was supplied + // (then verify the chain). + if let Some(cert) = custom_root { + builder.add_root_certificate(cert); + Ok(true) + } else { + builder + .danger_accept_invalid_certs(true) + .danger_accept_invalid_hostnames(true); + Ok(false) + } + } + } + } + async fn connect_inner( &self, ) -> Result<(tokio_postgres::Client, TokioPgConnection), error::Error> { - use native_tls::{Certificate, TlsConnector}; + use native_tls::TlsConnector; use postgres_native_tls::MakeTlsConnector; use tokio_postgres::tls::NoTls; let ssl_mode_is_require = matches!( @@ -654,21 +911,17 @@ impl PgDatabase { if ssl_mode_is_require { tracing::info!("Creating new connection"); let mut connector = TlsConnector::builder(); - if let Some(root_certificate_pem) = &self.root_certificate_pem { - if !root_certificate_pem.is_empty() { - connector.add_root_certificate( - Certificate::from_pem(root_certificate_pem.as_bytes()).map_err(|e| { - error::Error::BadConfig(format!("Invalid Certs: {e:#}")) - })?, - ); - } else { - connector.danger_accept_invalid_certs(true); - connector.danger_accept_invalid_hostnames(true); - } - } else { - connector - .danger_accept_invalid_certs(true) - .danger_accept_invalid_hostnames(true); + Self::configure_pg_tls_verification( + &mut connector, + self.sslmode.as_deref(), + self.root_certificate_pem.as_deref(), + self.accept_invalid_certs, + )?; + if self.verify_mode_skips_verification() { + tracing::warn!( + "Postgres connection with sslmode={} is not verifying the server certificate (accept_invalid_certs is set, or no root certificate is configured and the resource predates that flag). Set accept_invalid_certs=false or provide root_certificate_pem to enforce verification.", + self.sslmode.as_deref().unwrap_or("") + ); } let (client, connection) = tokio::time::timeout( @@ -723,23 +976,16 @@ impl PgDatabase { error::Error::InternalErr(format!("IAM token generation failed: {e:#}")) })?; - // RDS IAM auth requires SSL + // RDS IAM auth requires SSL. let mut connector = TlsConnector::builder(); - if let Some(root_certificate_pem) = &self.root_certificate_pem { - if !root_certificate_pem.is_empty() { - connector.add_root_certificate( - native_tls::Certificate::from_pem(root_certificate_pem.as_bytes()) - .map_err(|e| error::Error::BadConfig(format!("Invalid Certs: {e:#}")))?, - ); - } else { - connector.danger_accept_invalid_certs(true); - connector.danger_accept_invalid_hostnames(true); - } - } else { - tracing::warn!("IAM RDS auth without root certificate: TLS certificate verification is disabled. Consider providing root_certificate_pem for production use."); - connector - .danger_accept_invalid_certs(true) - .danger_accept_invalid_hostnames(true); + let verified = Self::configure_pg_tls_verification( + &mut connector, + self.sslmode.as_deref(), + self.root_certificate_pem.as_deref(), + self.accept_invalid_certs, + )?; + if !verified { + tracing::warn!("IAM RDS auth without certificate verification: TLS certificate verification is disabled. Provide root_certificate_pem (and set sslmode=verify-full) to enforce verification."); } tracing::info!("Creating new IAM RDS connection to {}", &self.host); @@ -804,6 +1050,7 @@ impl PgDatabase { dbname, sslmode, root_certificate_pem: None, + accept_invalid_certs: None, use_iam_auth: None, region: None, }) @@ -811,7 +1058,7 @@ impl PgDatabase { } /// Validate a database name to prevent SQL injection. -/// Must start with a letter, contain only alphanumeric characters or underscores, and be <= 63 chars. +/// Must start with a letter, contain only alphanumeric characters, underscores, or hyphens, and be <= 63 chars. pub fn validate_dbname(dbname: &str) -> error::Result<()> { let dbname = dbname.trim(); if dbname.is_empty() { @@ -835,10 +1082,11 @@ pub fn validate_dbname(dbname: &str) -> error::Result<()> { } if !dbname .chars() - .all(|c| c.is_ascii_alphanumeric() || c == '_') + .all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') { return Err(error::Error::BadRequest( - "Database name must contain only alphanumeric characters or underscores".to_string(), + "Database name must contain only alphanumeric characters, underscores, or hyphens" + .to_string(), )); } Ok(()) @@ -1278,8 +1526,12 @@ pub fn get_latest_deployed_hash_for_path<'e>( ) -> impl Future>> + Send + 'e { async move { let cache_key = (w_id.to_string(), script_path.to_string()); + let use_cache = !DEPLOYED_SCRIPT_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); let mut computed_hash = None; - let hash = match DEPLOYED_SCRIPT_HASH_CACHE.get(&cache_key) { + let hash = match DEPLOYED_SCRIPT_HASH_CACHE + .get(&cache_key) + .filter(|_| use_cache) + { Some(cached_hash) if cached_hash.expires_at > std::time::Instant::now() && db.as_ref().is_none_or(|x| { @@ -1322,13 +1574,15 @@ pub fn get_latest_deployed_hash_for_path<'e>( }; let hash = utils::not_found_if_none(hash, "script", script_path)?; - DEPLOYED_SCRIPT_HASH_CACHE.insert( - cache_key, - ExpiringLatestVersionId { - id: hash, - expires_at: std::time::Instant::now() + LATEST_VERSION_ID_CACHE_TTL, - }, - ); + if use_cache { + DEPLOYED_SCRIPT_HASH_CACHE.insert( + cache_key, + ExpiringLatestVersionId { + id: hash, + expires_at: std::time::Instant::now() + LATEST_VERSION_ID_CACHE_TTL, + }, + ); + } hash } @@ -1353,6 +1607,46 @@ pub async fn get_latest_script_hash<'e, E: sqlx::PgExecutor<'e>>( return Ok(hash); } +/// Latest non-archived hash for an imported `path`, for bundle cache keying. +/// MUST select the same row as the bundler's content endpoint +/// (`raw_script_by_path_internal`: `archived = false ORDER BY created_at DESC`, +/// no lock predicate) — a stricter filter here would let the key point at an +/// older version than the content that gets inlined. Cached with the same +/// freshness contract as that endpoint's `RAW_SCRIPT_LATEST_HASH_CACHE`: +/// evicted by `notify_runnable_version_change` events, 60s TTL fallback. +pub async fn get_latest_script_hash_for_import_cached( + db: &DB, + w_id: &str, + script_path: &str, +) -> error::Result> { + let use_cache = !DEPLOYED_SCRIPT_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); + let cache_key = (w_id.to_string(), script_path.to_string()); + if use_cache { + if let Some(cached) = IMPORTED_SCRIPT_HASH_CACHE.get(&cache_key) { + if cached.expires_at > std::time::Instant::now() { + return Ok(Some(cached.id)); + } + } + } + let hash = sqlx::query_scalar!( + "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND archived = false ORDER BY created_at DESC LIMIT 1", + script_path, + w_id + ) + .fetch_optional(db) + .await?; + if let (true, Some(hash)) = (use_cache, hash) { + IMPORTED_SCRIPT_HASH_CACHE.insert( + cache_key, + ExpiringLatestVersionId { + id: hash, + expires_at: std::time::Instant::now() + LATEST_VERSION_ID_CACHE_TTL, + }, + ); + } + Ok(hash) +} + pub async fn get_script_info_for_hash<'e, E: sqlx::PgExecutor<'e>>( db_authed: Option>>, db: E, @@ -1360,9 +1654,10 @@ pub async fn get_script_info_for_hash<'e, E: sqlx::PgExecutor<'e>>( hash: i64, ) -> error::Result> { let key = (w_id.to_string(), hash); + let use_cache = !DEPLOYED_SCRIPT_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); let mut computed_hash = None; - match DEPLOYED_SCRIPT_INFO_CACHE.get(&key) { + match DEPLOYED_SCRIPT_INFO_CACHE.get(&key).filter(|_| use_cache) { Some(info) if db_authed.as_ref().is_none_or(|x| { let r = HASH_PERMS_CACHE.check_perms_in_cache(x.authed, scripts::ScriptHash(hash)); @@ -1391,7 +1686,9 @@ pub async fn get_script_info_for_hash<'e, E: sqlx::PgExecutor<'e>>( let info = utils::not_found_if_none(info, "script", &hash.to_string())?; - DEPLOYED_SCRIPT_INFO_CACHE.insert(key, info.clone()); + if use_cache { + DEPLOYED_SCRIPT_INFO_CACHE.insert(key, info.clone()); + } Ok(info) } diff --git a/backend/windmill-common/src/materialization.rs b/backend/windmill-common/src/materialization.rs new file mode 100644 index 0000000000..236a71c855 --- /dev/null +++ b/backend/windmill-common/src/materialization.rs @@ -0,0 +1,380 @@ +//! CE materialization state — the per-partition status recorded by the managed +//! `// materialize` write (in windmill-worker), read by the partition-status +//! grid and by the EE backfill worklist. +//! +//! The write engine and this state are CE; only automatic partition +//! *resolution* (`partition_ee`) and *backfill* orchestration +//! (`pipeline_advanced_ee`) are enterprise. This module is the shared seam: +//! the EE backfill enumerates the partitions in a range, diffs them against +//! these rows to find the missing/failed set, and pushes one CE materialization +//! job per gap (with an explicit `partition` arg — which runs idempotently and +//! upserts the row here). Nothing about that orchestration lives in this file; +//! it only needs the rows to exist, which is why recording is CE. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use sqlx::types::Json; +use sqlx::{PgExecutor, Postgres, Transaction}; +use uuid::Uuid; + +use crate::assets::AssetKind; +use crate::error::Result; + +/// Sentinel `partition` value for an unpartitioned (whole-table) +/// materialization — partition is part of the primary key and cannot be NULL. +pub const UNPARTITIONED: &str = ""; + +/// Mirrors the `MATERIALIZATION_STATUS` pg enum (see migration +/// `20260619170118_add_materialized_partition`). +#[derive(sqlx::Type, Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[sqlx(type_name = "MATERIALIZATION_STATUS", rename_all = "lowercase")] +#[serde(rename_all = "lowercase")] +pub enum MaterializationStatus { + Running, + Materialized, + Failed, +} + +/// One column of a captured asset output schema: its name and substrate type +/// (e.g. `{"name": "order_id", "type": "BIGINT"}`). `type` is the substrate's +/// own type spelling (DuckDB for ducklake) — kept verbatim so #2b can compare +/// declared vs. captured without a lossy normalization step. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SchemaColumn { + pub name: String, + #[serde(rename = "type")] + pub data_type: String, +} + +/// The materialization outcome an agent worker (`Connection::Http`, no direct +/// DB) sends to the API to be recorded. Mirrors the `record_materialization` +/// args; the API handler unpacks it and calls that function with its own DB. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct RecordMaterializationRequest { + pub asset_kind: AssetKind, + pub asset_path: String, + pub partition: String, + pub status: MaterializationStatus, + pub snapshot_id: Option, + pub row_count: Option, + pub job_id: Option, + pub error: Option, + /// Captured output schema of the materialized asset (`None` when the + /// substrate/run produced no schema, e.g. a failed run or a polyglot helper + /// that doesn't DESCRIBE). When present, the recorder also upserts a + /// `materialized_asset_schema` version. Defaults to `None` so older agents + /// stay wire-compatible. + #[serde(default)] + pub schema: Option>, +} + +/// Upsert the latest materialization state for one (asset, partition) slice. +/// The worker records the terminal outcome once the write finishes: +/// `Materialized` (with the DuckLake `snapshot_id` + `row_count`) or `Failed` +/// (with `error`). `Running` mirrors the pg enum but has no writer in this flow. +/// Idempotent: re-running the same partition overwrites the row — exactly the +/// backfill / failure-recovery contract. +#[allow(clippy::too_many_arguments)] +pub async fn record_materialization<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, + partition: &str, + status: MaterializationStatus, + snapshot_id: Option, + row_count: Option, + job_id: Option, + error: Option<&str>, +) -> Result<()> { + sqlx::query!( + "INSERT INTO materialized_partition + (workspace_id, asset_kind, asset_path, partition, status, + snapshot_id, row_count, job_id, materialized_at, error) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9) + ON CONFLICT (workspace_id, asset_kind, asset_path, partition) + DO UPDATE SET status = EXCLUDED.status, + -- A failed run records no snapshot, but must not erase the last + -- committed one: a physical table from an earlier commit (or from a + -- committed write whose data tests then failed) still exists, and + -- fork defer/graph state keys on that evidence. + snapshot_id = COALESCE(EXCLUDED.snapshot_id, materialized_partition.snapshot_id), + row_count = EXCLUDED.row_count, + job_id = EXCLUDED.job_id, + materialized_at = now(), + error = EXCLUDED.error", + workspace_id, + asset_kind as AssetKind, + asset_path, + partition, + status as MaterializationStatus, + snapshot_id, + row_count, + job_id, + error, + ) + .execute(executor) + .await?; + Ok(()) +} + +/// One materialized-partition row, for the status grid / backfill diff. +#[derive(sqlx::FromRow, Debug, Clone, Serialize)] +pub struct MaterializedPartition { + pub asset_kind: AssetKind, + pub asset_path: String, + pub partition: String, + pub status: MaterializationStatus, + pub snapshot_id: Option, + pub row_count: Option, + pub job_id: Option, + pub materialized_at: DateTime, + pub error: Option, +} + +/// All recorded partitions for one asset, newest first — the grid's data and +/// the backfill worklist's "what already exists" set. +pub async fn list_materialized_partitions<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, +) -> Result> { + let rows = sqlx::query_as!( + MaterializedPartition, + r#"SELECT asset_kind AS "asset_kind: AssetKind", asset_path, partition, + status AS "status: MaterializationStatus", snapshot_id, + row_count, job_id, materialized_at, error + FROM materialized_partition + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY partition DESC"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_all(executor) + .await?; + Ok(rows) +} + +/// One captured schema version of an asset, newest first — the schema-evolution +/// history surfaced on the asset node and read by #2b contract enforcement. +#[derive(sqlx::FromRow, Debug, Clone, Serialize)] +pub struct AssetSchemaVersion { + pub version: i64, + pub columns: Json>, + pub snapshot_id: Option, + pub job_id: Option, + pub captured_at: DateTime, +} + +/// Record the captured output schema of a freshly-materialized asset. +/// +/// **Authorization:** like the sibling `record_materialization`, this performs +/// no access control of its own — it writes the row for whatever `workspace_id` +/// it is given. Callers MUST pass a workspace-authorized executor and a +/// `workspace_id` the caller is allowed to write: an RLS-scoped `user_db` +/// transaction for API / agent-worker entry points, or the trusted worker DB +/// pool for the in-worker recorder. Do not expose it to an unauthenticated path. +/// +/// Versioning across re-materializations: a new `version` row is inserted only +/// when `columns` differs from the latest stored version; an unchanged +/// re-materialize re-affirms the latest row in place (updates its +/// `snapshot_id`/`job_id`/`captured_at`). The result is a compact +/// schema-evolution history where `MAX(version)` is the current contract. +/// +/// Runs in a transaction guarded by a per-asset advisory lock so two concurrent +/// materializations of the same asset can't both insert the same next version +/// or interleave a stale comparison. Returns `true` if a new version was +/// inserted (the schema changed), `false` if the latest was re-affirmed. +pub async fn record_asset_schema( + tx: &mut Transaction<'_, Postgres>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, + columns: &[SchemaColumn], + snapshot_id: Option, + job_id: Option, +) -> Result { + // Serialize concurrent captures of the *same* asset; the lock auto-releases + // at tx end. Hash the identity into the bigint advisory-lock key space. + sqlx::query!( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0::int8))", + format!("materialized_asset_schema:{workspace_id}:{asset_kind:?}:{asset_path}"), + ) + .fetch_one(&mut **tx) + .await?; + + let latest = sqlx::query!( + r#"SELECT version, columns AS "columns: Json>" + FROM materialized_asset_schema + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY version DESC + LIMIT 1"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_optional(&mut **tx) + .await?; + + let columns_json = Json(columns.to_vec()); + let next_version = match latest { + Some(latest) if latest.columns.0.as_slice() == columns => { + // Unchanged schema — re-affirm the latest version in place. + sqlx::query!( + "UPDATE materialized_asset_schema + SET snapshot_id = $5, job_id = $6, captured_at = now() + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + AND version = $4", + workspace_id, + asset_kind as AssetKind, + asset_path, + latest.version, + snapshot_id, + job_id, + ) + .execute(&mut **tx) + .await?; + return Ok(false); + } + Some(latest) => latest.version + 1, + None => 1, + }; + sqlx::query!( + "INSERT INTO materialized_asset_schema + (workspace_id, asset_kind, asset_path, version, columns, + snapshot_id, job_id, captured_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, now())", + workspace_id, + asset_kind as AssetKind, + asset_path, + next_version, + columns_json as Json>, + snapshot_id, + job_id, + ) + .execute(&mut **tx) + .await?; + Ok(true) +} + +/// One table a fork workspace should read from an ancestor through a defer view. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ForkDeferTable { + /// Lake-internal table name: the `asset_path` minus its `/` prefix (may itself be + /// `schema.table`). + pub table: String, + /// The owning ancestor's latest captured schema carries the SCD2 marker column + /// (`is_current`), so that lake also holds a managed `
_current` companion view + /// that consumers read — defer it alongside the table. + #[serde(default)] + pub with_current_view: bool, + /// Index into `DucklakeForkDefer.ancestors` (nearest-first) of the NEAREST ancestor that + /// materialized this table — the defer view must target that ancestor's namespace. In a + /// `fork → parent → root` chain where only root materialized a table, the parent has no + /// physical copy (it defers too), so a view over the parent would not bind. Defaults to 0 + /// (the direct parent) for wire compatibility with agents that predate the field. + #[serde(default)] + pub ancestor_idx: u32, +} + +/// Tables of lake `lake_name` materialized somewhere in the fork's ancestor chain +/// (nearest-first) but not (yet) in the fork — the fork's read-defer set, each mapped to the +/// nearest ancestor that owns a physical copy. Only `Materialized` rows count on every side: a +/// deferred table must physically exist in the targeted ancestor (defer views bind at CREATE +/// and would otherwise fail the whole job), and any successful fork materialization makes the +/// fork's own table authoritative. +/// +/// **Authorization:** performs no access control; trusted server-side callers only. It reads +/// ancestor workspaces' rows on behalf of a fork — acceptable because defer itself exposes +/// the ancestors' table contents to fork members. +pub async fn list_fork_defer_tables<'e>( + executor: impl PgExecutor<'e>, + ancestor_workspace_ids: &[String], + fork_workspace_id: &str, + lake_name: &str, +) -> Result> { + let rows = sqlx::query!( + r#" + WITH ancestor AS ( + SELECT wid, ord FROM unnest($1::text[]) WITH ORDINALITY AS a(wid, ord) + ), anc_mat AS ( + -- Per table, the nearest ancestor (lowest ord) that materialized it. + SELECT DISTINCT ON (mp.asset_path) mp.asset_path, a.wid, a.ord + FROM materialized_partition mp + JOIN ancestor a ON a.wid = mp.workspace_id + WHERE mp.asset_kind = 'ducklake' AND mp.status = 'materialized' + AND split_part(mp.asset_path, '/', 1) = $3 AND mp.asset_path LIKE '%/%' + ORDER BY mp.asset_path, a.ord + ), fork_mat AS ( + -- Fork-OWNED assets: anything whose physical table exists in the fork + -- namespace, not just clean materializations. A committed write whose data + -- tests failed afterwards records status='failed' WITH a snapshot — its table + -- is real, and a defer view emitted over it would silently yield to it + -- (CREATE VIEW IF NOT EXISTS) while claiming the read defers to the parent. + SELECT DISTINCT asset_path FROM materialized_partition + WHERE workspace_id = $2 AND asset_kind = 'ducklake' + AND (status = 'materialized' OR snapshot_id IS NOT NULL) + ), latest_schema AS ( + SELECT DISTINCT ON (workspace_id, asset_path) workspace_id, asset_path, columns + FROM materialized_asset_schema + WHERE workspace_id = ANY($1) AND asset_kind = 'ducklake' + ORDER BY workspace_id, asset_path, version DESC + ) + SELECT am.asset_path AS "asset_path!", + am.ord AS "ord!", + COALESCE(EXISTS ( + SELECT 1 FROM jsonb_array_elements(ls.columns) e + WHERE e->>'name' = 'is_current' + ), false) AS "has_current!" + FROM anc_mat am + LEFT JOIN latest_schema ls + ON ls.asset_path = am.asset_path AND ls.workspace_id = am.wid + WHERE am.asset_path NOT IN (SELECT asset_path FROM fork_mat) + ORDER BY am.asset_path + "#, + ancestor_workspace_ids, + fork_workspace_id, + lake_name, + ) + .fetch_all(executor) + .await?; + Ok(rows + .into_iter() + .map(|r| ForkDeferTable { + table: r.asset_path[lake_name.len() + 1..].to_string(), + with_current_view: r.has_current, + // WITH ORDINALITY is 1-based; ancestors vec is 0-based. + ancestor_idx: (r.ord - 1).max(0) as u32, + }) + .collect()) +} + +/// All captured schema versions for one asset, newest version first. +/// +/// **Authorization:** performs no access control (mirrors +/// `list_materialized_partitions`); the caller must pass a workspace-authorized +/// executor (an RLS-scoped `user_db` transaction on the API read path) and a +/// `workspace_id` it is allowed to read. +pub async fn list_asset_schemas<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, +) -> Result> { + let rows = sqlx::query_as!( + AssetSchemaVersion, + r#"SELECT version, columns AS "columns: Json>", + snapshot_id, job_id, captured_at + FROM materialized_asset_schema + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY version DESC"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_all(executor) + .await?; + Ok(rows) +} diff --git a/backend/windmill-common/src/oidc_oss.rs b/backend/windmill-common/src/oidc_oss.rs index e212558377..60a09ce9b7 100644 --- a/backend/windmill-common/src/oidc_oss.rs +++ b/backend/windmill-common/src/oidc_oss.rs @@ -61,6 +61,8 @@ pub struct JobClaim { pub email: String, pub workspace: String, #[serde(skip_serializing_if = "Option::is_none")] + pub fork_parent_workspace: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub end_user_email: Option, } diff --git a/backend/windmill-common/src/pipeline_advanced_oss.rs b/backend/windmill-common/src/pipeline_advanced_oss.rs index 009e783bd5..3220b251a0 100644 --- a/backend/windmill-common/src/pipeline_advanced_oss.rs +++ b/backend/windmill-common/src/pipeline_advanced_oss.rs @@ -1,16 +1,27 @@ -//! OSS fallback: pipeline freshness/SLA enforcement and partition backfills -//! are enterprise features; their implementations live in windmill-ee-private -//! (see `pipeline_advanced_ee`). In the public build the entry points report -//! that the enterprise edition is required. +//! OSS fallback for enterprise pipeline features (implementations in +//! windmill-ee-private, see `pipeline_advanced_ee`): partition backfill +//! reports that the enterprise edition is required, and materialization-plan +//! assembly runs the plan verbatim — dbt-like commit-then-test instead of the +//! enterprise write-audit-publish. (Freshness lives elsewhere: the fresh/stale +//! badge is CE in the assets API, the active watchdog is windmill-queue's +//! `freshness_watchdog`.) use crate::error::Error; -pub fn freshness_enforcement_todo() -> Error { - Error::internal_err( - "Pipeline freshness/SLA enforcement requires the enterprise edition".to_string(), - ) -} - pub fn backfill_todo() -> Error { Error::internal_err("Pipeline partition backfill requires the enterprise edition".to_string()) } + +/// Assemble a materialization plan into the statement list the DuckDB executor +/// runs. The public build executes the plan verbatim — dbt-like +/// commit-then-test: a failing `// data_test` still fails the run and stops +/// the cascade, but the written slice stays live. The enterprise +/// implementation (`pipeline_advanced_ee`) instead restructures the plan into +/// write-audit-publish, where a failing test rolls the whole write back before +/// anything is published. +pub fn finalize_materialize_query( + plan: windmill_parser::sql_materialize::MaterializePlan, + _asset_path: &str, +) -> Vec { + plan.stmts.into_iter().map(|s| s.sql).collect() +} diff --git a/backend/windmill-common/src/query_builders.rs b/backend/windmill-common/src/query_builders.rs index 2c168cd433..ce7dbc2533 100644 --- a/backend/windmill-common/src/query_builders.rs +++ b/backend/windmill-common/src/query_builders.rs @@ -172,6 +172,19 @@ pub struct SimpleColumn { pub struct SelectOptions { pub limit: Option, pub offset: Option, + /// DuckLake time-travel: when set (DuckDB only), the read is pinned to this + /// catalog snapshot via `AT (VERSION => n)`. Ignored for other db types. + pub version: Option, +} + +/// DuckLake time-travel suffix appended after a table name in a FROM clause. +/// `n` is a server-controlled `i64` (a snapshot id), so inlining it is +/// injection-safe. Empty string when unpinned (reads the latest snapshot). +fn duckdb_version_suffix(version: Option) -> String { + match version { + Some(v) => format!(" AT (VERSION => {})", v), + None => String::new(), + } } // --------------------------------------------------------------------------- @@ -190,6 +203,8 @@ struct SelectPayload { #[serde(rename = "fixPgIntTypes")] fix_pg_int_types: Option, ducklake: Option, + /// DuckLake snapshot to time-travel the read to (DuckDB only). + version: Option, } #[derive(Deserialize)] @@ -200,6 +215,21 @@ struct CountPayload { #[serde(rename = "whereClause")] where_clause: Option, ducklake: Option, + /// DuckLake snapshot to time-travel the count to (DuckDB only). + version: Option, +} + +/// `WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS` payload — lists the time-travel history +/// of a ducklake table. DuckLake snapshots are catalog-wide commits, so without +/// a `table` this lists every commit; with one it is scoped to snapshots where +/// that table exists (see `expand_ducklake_snapshots`). +#[derive(Deserialize)] +struct DucklakeSnapshotsPayload { + ducklake: String, + /// Schema-qualified table name (e.g. `main.events_daily`) to scope the + /// history to. Snapshots predating the table's creation are excluded — a + /// time-travel read can't target a version where the table didn't exist. + table: Option, } #[derive(Deserialize)] @@ -304,6 +334,10 @@ pub fn try_expand_internal_db_query( expand_primary_key_constraint(json_str, db_type).map(ExpandedQuery::sql) } "SNOWFLAKE_PRIMARY_KEYS" => expand_snowflake_primary_keys(json_str).map(ExpandedQuery::sql), + // DuckLake time-travel: list a ducklake's snapshot history + "DUCKLAKE_SNAPSHOTS" => { + expand_ducklake_snapshots(json_str, db_type).map(ExpandedQuery::sql) + } _ => Err(format!("Unknown WM_INTERNAL_DB operation: {}", op)), }; @@ -324,7 +358,8 @@ fn expand_select(json_str: &str, db_type: DbType) -> Result { let payload: SelectPayload = serde_json::from_str(json_str).map_err(|e| format!("Invalid SELECT payload: {}", e))?; - let options = SelectOptions { limit: payload.limit, offset: payload.offset }; + let options = + SelectOptions { limit: payload.limit, offset: payload.offset, version: payload.version }; let breaking = payload .fix_pg_int_types .map(|v| BreakingFeatures { fix_pg_int_types: v }); @@ -350,11 +385,47 @@ fn expand_count(json_str: &str, db_type: DbType) -> Result { &payload.table, payload.where_clause.as_deref(), &payload.column_defs, + payload.version, )?; Ok(maybe_wrap_ducklake(query, payload.ducklake.as_deref())) } +/// Expand `DUCKLAKE_SNAPSHOTS` into the catalog's time-travel history. DuckLake +/// snapshots are catalog-wide commits, so `ducklake_snapshots('dl')` (the alias +/// `maybe_wrap_ducklake` attaches) lists every version any `AT (VERSION => n)` +/// read can target, newest first. +fn expand_ducklake_snapshots(json_str: &str, db_type: DbType) -> Result { + if db_type != DbType::Duckdb { + return Err("DUCKLAKE_SNAPSHOTS is only supported for DuckDB".to_string()); + } + let payload: DucklakeSnapshotsPayload = serde_json::from_str(json_str) + .map_err(|e| format!("Invalid DUCKLAKE_SNAPSHOTS payload: {}", e))?; + // `dl` is the alias `wrap_ducklake_query` attaches and `USE`s below. + let query = match &payload.table { + // Scope to snapshots from the table's first creation onward. A DuckLake + // table created at snapshot N can't be read before N (the catalog-wide + // list would otherwise offer impossible versions). The creation snapshot + // is the earliest whose `changes.tables_created` names the table; + // COALESCE to 0 (show all) if it is never found. + Some(table) => { + let table = escape_sql_literal(table); + format!( + "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') \ + WHERE snapshot_id >= COALESCE((\ + SELECT min(snapshot_id) FROM ducklake_snapshots('dl') \ + WHERE list_contains(changes.tables_created, '{table}')), 0) \ + ORDER BY snapshot_id DESC" + ) + } + None => { + "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') ORDER BY snapshot_id DESC" + .to_string() + } + }; + Ok(maybe_wrap_ducklake(query, Some(&payload.ducklake))) +} + /// Filter columns to primary keys only; fall back to all columns if none are marked. fn pk_columns_or_all(columns: &[ColumnDef]) -> Vec { let pks: Vec = columns.iter().filter(|c| c.isprimarykey).cloned().collect(); @@ -950,9 +1021,10 @@ pub fn make_select_query( ); query.push_str(&format!( - "SELECT {} FROM {}\n", + "SELECT {} FROM {}{}\n", filtered_columns.join(", "), - quote_table_name(table, db_type) + quote_table_name(table, db_type), + duckdb_version_suffix(options.and_then(|o| o.version)) )); query.push_str(&format!( " WHERE {} {}\n", @@ -977,6 +1049,8 @@ pub fn make_count_query( table: &str, where_clause: Option<&str>, column_defs: &[ColumnDef], + // DuckLake time-travel snapshot (DuckDB only); `None` counts the latest. + version: Option, ) -> Result { let where_prefix = " WHERE "; let and_condition = " AND "; @@ -1118,8 +1192,9 @@ pub fn make_count_query( quicksearch_condition.push_str(" ($quicksearch = '' OR 1 = 1)"); } query.push_str(&format!( - "SELECT COUNT(*) as count FROM {}", - quote_table_name(table, db_type) + "SELECT COUNT(*) as count FROM {}{}", + quote_table_name(table, db_type), + duckdb_version_suffix(version) )); } } @@ -2998,7 +3073,7 @@ mod tests { #[test] fn test_select_snowflake_custom_limit() { let cols = vec![col("id", "int")]; - let opts = SelectOptions { limit: Some(50), offset: Some(10) }; + let opts = SelectOptions { limit: Some(50), offset: Some(10), version: None }; let result = make_select_query( "my_table", &cols, @@ -3072,7 +3147,7 @@ mod tests { #[test] fn test_count_postgresql_basic() { let cols = vec![col("id", "int4"), col("name", "text")]; - let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- $1 quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\"")); @@ -3090,6 +3165,7 @@ mod tests { "my_table", Some("status = 'active'"), &cols, + None, ) .unwrap(); @@ -3105,7 +3181,7 @@ mod tests { c.ignored = Some(true); c }]; - let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("($1 = '' OR 1 = 1)")); } @@ -3116,7 +3192,7 @@ mod tests { #[test] fn test_count_mysql_basic() { let cols = vec![col("id", "int"), col("name", "varchar")]; - let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- :quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`")); @@ -3130,7 +3206,7 @@ mod tests { #[test] fn test_count_mssql_basic() { let cols = vec![col("id", "int"), col("name", "nvarchar")]; - let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap(); assert!(result.contains("SELECT COUNT(*) as count FROM [my_table]")); assert!(result.contains("(@p1 = '' OR CONCAT([id], [name]) LIKE '%' + @p1 + '%')")); @@ -3143,7 +3219,7 @@ mod tests { #[test] fn test_count_snowflake_basic() { let cols = vec![col("id", "int"), col("name", "text")]; - let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap(); // Two quicksearch params for snowflake with visible columns assert!(result.contains("-- ? quicksearch (text)\n-- ? quicksearch (text)")); @@ -3158,7 +3234,7 @@ mod tests { c.ignored = Some(true); c }]; - let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap(); // One quicksearch param let param_lines: Vec<&str> = result.lines().filter(|l| l.starts_with("-- ?")).collect(); assert_eq!(param_lines.len(), 1); @@ -3172,7 +3248,7 @@ mod tests { #[test] fn test_count_bigquery_basic() { let cols = vec![col("id", "INTEGER"), col("name", "STRING")]; - let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- @quicksearch (string)")); assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`")); @@ -3182,7 +3258,7 @@ mod tests { #[test] fn test_count_bigquery_json_type() { let cols = vec![col("id", "INTEGER"), col("data", "JSON")]; - let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap(); assert!(result.contains("TO_JSON_STRING(`data`)")); } @@ -3193,7 +3269,7 @@ mod tests { #[test] fn test_count_duckdb_basic() { let cols = vec![col("id", "int"), col("name", "text")]; - let result = make_count_query(DbType::Duckdb, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Duckdb, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- $quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\"")); @@ -3202,6 +3278,74 @@ mod tests { ); } + // ----------------------------------------------------------------------- + // DuckLake time-travel (AT VERSION) + snapshot history + // ----------------------------------------------------------------------- + + #[test] + fn test_select_duckdb_time_travel() { + let cols = vec![col("id", "int"), col("name", "text")]; + let opts = SelectOptions { limit: None, offset: None, version: Some(42) }; + let result = + make_select_query("orders", &cols, None, DbType::Duckdb, Some(&opts), None).unwrap(); + // Read is pinned to the catalog snapshot via AT (VERSION => n). + assert!(result.contains("FROM \"orders\" AT (VERSION => 42)\n")); + } + + #[test] + fn test_select_duckdb_no_version_unpinned() { + let cols = vec![col("id", "int")]; + let result = make_select_query("orders", &cols, None, DbType::Duckdb, None, None).unwrap(); + // Without a version the read targets the latest snapshot — no AT clause. + assert!(result.contains("FROM \"orders\"\n")); + assert!(!result.contains("AT (VERSION")); + } + + #[test] + fn test_count_duckdb_time_travel() { + let cols = vec![col("id", "int")]; + let result = make_count_query(DbType::Duckdb, "orders", None, &cols, Some(7)).unwrap(); + assert!(result.contains("FROM \"orders\" AT (VERSION => 7)")); + } + + #[test] + fn test_version_ignored_for_non_duckdb() { + // AT (VERSION) is DuckLake-only; other dialects must never emit it even + // if a version is somehow passed through. + let cols = vec![col("id", "int4")]; + let opts = SelectOptions { limit: None, offset: None, version: Some(5) }; + let result = + make_select_query("orders", &cols, None, DbType::Postgresql, Some(&opts), None) + .unwrap(); + assert!(!result.contains("AT (VERSION")); + } + + #[test] + fn test_expand_ducklake_snapshots() { + let json = r#"{"ducklake": "analytics"}"#; + let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap(); + assert!(result.contains("ATTACH 'ducklake://analytics' AS dl;USE dl;")); + assert!(result.contains("ducklake_snapshots('dl')")); + assert!(result.contains("ORDER BY snapshot_id DESC")); + // Unscoped: no per-table existence filter. + assert!(!result.contains("tables_created")); + } + + #[test] + fn test_expand_ducklake_snapshots_scoped_to_table() { + let json = r#"{"ducklake": "analytics", "table": "main.events_daily"}"#; + let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap(); + // Scoped to snapshots from the table's first creation onward. + assert!(result.contains("list_contains(changes.tables_created, 'main.events_daily')")); + assert!(result.contains("snapshot_id >= COALESCE")); + } + + #[test] + fn test_expand_ducklake_snapshots_non_duckdb_errors() { + let json = r#"{"ducklake": "analytics"}"#; + assert!(expand_ducklake_snapshots(json, DbType::Postgresql).is_err()); + } + // ----------------------------------------------------------------------- // DELETE - all DB types // ----------------------------------------------------------------------- @@ -3500,7 +3644,7 @@ mod tests { #[test] fn test_count_mssql_no_where() { let cols = vec![col("id", "int")]; - let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap(); // MSSQL uses WHERE directly (no AND replacement) assert!(result.contains("SELECT COUNT(*) as count FROM [my_table] WHERE ")); } @@ -3508,7 +3652,7 @@ mod tests { #[test] fn test_count_mysql_no_where_uses_where_keyword() { let cols = vec![col("id", "int")]; - let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap(); // The AND should be replaced with WHERE assert!(result.contains("FROM `my_table` WHERE ")); assert!(!result.contains("FROM `my_table` AND ")); diff --git a/backend/windmill-common/src/runnable_settings/mod.rs b/backend/windmill-common/src/runnable_settings/mod.rs index c5692d40bd..33d4c1a039 100644 --- a/backend/windmill-common/src/runnable_settings/mod.rs +++ b/backend/windmill-common/src/runnable_settings/mod.rs @@ -52,13 +52,10 @@ pub trait RunnableSettingsTrait: /// get [[Self]] from cache or fetch from db /// if not found, returns Error - fn get<'a>( + fn get<'e>( hash: i64, - db: &'a Pool, - ) -> impl Future> - where - Self: 'a, - { + db: impl sqlx::PgExecutor<'e>, + ) -> impl Future> { async move { let v = RUNNABLE_INDIVIDUAL_SETTINGS .get_or_insert_async(hash, async { diff --git a/backend/windmill-common/src/runnable_settings/settings.rs b/backend/windmill-common/src/runnable_settings/settings.rs index 04b62fbaab..b814646553 100644 --- a/backend/windmill-common/src/runnable_settings/settings.rs +++ b/backend/windmill-common/src/runnable_settings/settings.rs @@ -1,5 +1,3 @@ -use std::future::Future; - use sqlx::{Pool, Postgres}; use crate::{ @@ -38,29 +36,63 @@ pub async fn prefetch_cached_from_handle( prefetch_cached(&rs, db).await } +/// Like [`prefetch_cached`], but reuses a held transaction's connection instead +/// of checking out a second one from the pool. Use this on paths that already +/// hold an open `tx` to avoid dual-connection pool contention. +pub async fn prefetch_cached_tx( + rs: &RunnableSettings, + tx: &mut sqlx::Transaction<'_, Postgres>, +) -> error::Result<(DebouncingSettings, ConcurrencySettings)> { + Ok(( + if let Some(hash) = rs.debouncing_settings { + DebouncingSettings::get(hash, &mut **tx).await? + } else { + Default::default() + }, + if let Some(hash) = rs.concurrency_settings { + ConcurrencySettings::get(hash, &mut **tx).await? + } else { + Default::default() + }, + )) +} + +/// Resolve the retry policy (if any) for a job from its `runnable_settings_handle`. +/// Returns `None` when the job carries no retry policy. Read lazily on the +/// failure path only — never on the hot job-pull path. +pub async fn prefetch_retry_from_handle( + hash: Option, + db: &DB, +) -> error::Result> { + let rs = from_handle(hash, db).await?; + Ok(if let Some(hash) = rs.retry_settings { + Some(RetrySettings::get(hash, db).await?) + } else { + None + }) +} + /// Returns error if provided `hash` has no corresponding entry in db /// If `hash` is None, returns Default -pub fn from_handle<'a>( +pub async fn from_handle<'e>( hash: Option, - db: &'a DB, -) -> impl Future> + 'a { - async move { - if let Some(hash) = hash { - super::RUNNABLE_SETTINGS_REFERENCES - .get_or_insert_async(hash, async { - sqlx::query_as!( - RunnableSettings, - r#"SELECT concurrency_settings, debouncing_settings FROM runnable_settings WHERE hash = $1"#, - hash - ) - .fetch_one(db) - .await - .map_err(error::Error::from) - }) + db: impl sqlx::PgExecutor<'e>, +) -> error::Result { + if let Some(hash) = hash { + super::RUNNABLE_SETTINGS_REFERENCES + .get_or_insert_async(hash, async { + sqlx::query_as!( + RunnableSettings, + r#"SELECT concurrency_settings, debouncing_settings, retry_settings FROM runnable_settings WHERE hash = $1"#, + hash + ) + .fetch_one(db) .await - } else { - Ok(RunnableSettings::default()) - } + .map_err(error::Error::from) + }) + .await + } else { + Ok(RunnableSettings::default()) } } @@ -68,7 +100,9 @@ pub async fn insert_rs(rs: RunnableSettings, db: &Pool) -> error::Resu use std::hash::{Hash, Hasher}; if !min_version_supports_runnable_settings_v0().await - || (rs.debouncing_settings.is_none() && rs.concurrency_settings.is_none()) + || (rs.debouncing_settings.is_none() + && rs.concurrency_settings.is_none() + && rs.retry_settings.is_none()) { return Ok(None); } @@ -82,13 +116,14 @@ pub async fn insert_rs(rs: RunnableSettings, db: &Pool) -> error::Resu super::RUNNABLE_SETTINGS_REFERENCES .get_or_insert_async(hash, async { sqlx::query!( - "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings) - VALUES ($1, $2, $3) + "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings, retry_settings) + VALUES ($1, $2, $3, $4) ON CONFLICT (hash) DO NOTHING", hash, rs.debouncing_settings, - rs.concurrency_settings + rs.concurrency_settings, + rs.retry_settings ) .execute(db) .await?; @@ -132,3 +167,25 @@ impl super::private_mod::RunnableSettingsTraitInternal for ConcurrencySettings { } } impl super::RunnableSettingsTrait for ConcurrencySettings {} +impl super::private_mod::RunnableSettingsTraitInternal for RetrySettings { + const SETTINGS_NAME: &str = "retry_settings"; + const INCLUDE_FIELDS: &[&str] = &[ + "constant_attempts", + "constant_seconds", + "exponential_attempts", + "exponential_multiplier", + "exponential_seconds", + "exponential_random_factor", + "retry_if_expr", + ]; + fn bind_arguments<'a>(&'a self, q: Q<'a>) -> Q<'a> { + q.bind(&self.constant_attempts) + .bind(&self.constant_seconds) + .bind(&self.exponential_attempts) + .bind(&self.exponential_multiplier) + .bind(&self.exponential_seconds) + .bind(&self.exponential_random_factor) + .bind(&self.retry_if_expr) + } +} +impl super::RunnableSettingsTrait for RetrySettings {} diff --git a/backend/windmill-common/src/schema_contracts.rs b/backend/windmill-common/src/schema_contracts.rs new file mode 100644 index 0000000000..b605aae33b --- /dev/null +++ b/backend/windmill-common/src/schema_contracts.rs @@ -0,0 +1,614 @@ +//! Save-time schema-contract check (pipelines gap #2b): validate a consumer +//! script's asset references against the latest *captured* producer schema +//! (`materialized_asset_schema`, written post-materialize by #2a) and return +//! WARNINGS — never errors. A deliberate upstream reshape must not fail every +//! consumer save; blocking (`on_schema_change=fail`) is deliberately not +//! offered in v1. +//! +//! Ducklake-only: `// materialize` targets are ducklake-only in v1, so nothing +//! else has a captured schema to check against; an asset with no captured +//! schema produces no warnings (first deploy, datatable, external tables). +//! +//! The comparison itself (`diff_contract`) is pure so it can be unit-tested +//! and mirrored 1:1 by the editor-side TS check +//! (frontend/src/lib/components/assets/AssetGraph/schemaContracts.ts); the +//! async wrapper owns the DB reads (schemas + producer resolution) and runs on +//! the caller's RLS-scoped transaction. + +use std::collections::{HashMap, HashSet}; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use sqlx::types::Json; +use sqlx::{Postgres, Transaction}; +use windmill_parser::asset_parser::{ + ColumnLineage, DataTest, MaterializeSpec, OnSchemaChange, PARTITION_TOKEN, +}; +use windmill_types::assets::{AssetKind, AssetWithAltAccessType}; + +use crate::error::Result; +use crate::materialization::SchemaColumn; + +/// Columns the materialize engine adds/manages; never part of the captured +/// schema, so consumer reads of them must not warn (`_wm_partition` is +/// filtered out of the DESCRIBE capture on purpose). +const RESERVED_COLUMNS: &[&str] = &["_wm_partition"]; + +#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum ContractWarningKind { + /// A column the body reads/writes is absent from the captured schema. + MissingColumn, + /// A `// column … <- .` source column is absent. + MissingLineageSource, + /// A `// data_test relationships … -> .` ref column is absent. + MissingRelationshipColumn, + /// Relationship join columns have different captured types (may still + /// coerce at run time — phrased as "differs", not "will fail"). + RelationshipTypeMismatch, + /// Warnings for this asset were suppressed by the producer's + /// `on_schema_change=ignore` (one informational entry per asset). + Suppressed, +} + +/// One save-time contract warning. `schema_version`/`captured_at` identify the +/// capture the check ran against, so a stale-capture warning is +/// self-explaining (the schema is as-of the producer's last run, not its +/// latest save). +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] +pub struct ContractWarning { + pub kind: ContractWarningKind, + /// Normalized ducklake asset path (`/
`). + pub asset_path: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub column: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub expected_type: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub found_type: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub schema_version: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub captured_at: Option>, + pub message: String, +} + +/// The latest captured schema of one asset, as loaded by the wrapper. +#[derive(Debug, Clone)] +pub struct CapturedSchema { + pub columns: Vec, + pub version: i64, + pub captured_at: DateTime, +} + +impl CapturedSchema { + /// Case-insensitive column lookup — DuckDB matches unquoted identifiers + /// case-insensitively, and the body parser preserves source casing while + /// DESCRIBE returns stored casing. + fn find(&self, name: &str) -> Option<&SchemaColumn> { + self.columns + .iter() + .find(|c| c.name.eq_ignore_ascii_case(name)) + } +} + +fn is_reserved(name: &str) -> bool { + RESERVED_COLUMNS + .iter() + .any(|r| r.eq_ignore_ascii_case(name)) +} + +/// Strip the `{partition}` token a declared URI may carry (`// on +/// ducklake://lake/t/{partition}` or pasted refs) so lookups hit the captured +/// path. Body-inferred paths never carry it, but annotation refs can. +pub fn normalize_asset_path(path: &str) -> String { + path.replace(&format!("/{}", PARTITION_TOKEN), "") + .replace(PARTITION_TOKEN, "") + .trim_end_matches('/') + .to_string() +} + +fn column_list(schema: &CapturedSchema) -> String { + schema + .columns + .iter() + .map(|c| c.name.as_str()) + .collect::>() + .join(", ") +} + +/// Pure comparison: consumer refs vs captured schemas. `schemas` is keyed by +/// normalized ducklake path (the `_current` → base-table fallback is resolved +/// by the wrapper before this runs); `ignored` holds paths whose producer +/// declared `on_schema_change=ignore`. +pub fn diff_contract( + assets: &[AssetWithAltAccessType], + column_lineage: &[ColumnLineage], + data_tests: &[DataTest], + materialize: Option<&MaterializeSpec>, + schemas: &HashMap, + ignored: &HashSet, +) -> Vec { + let mut warnings: Vec = vec![]; + + // W1 — body-read/written columns missing from the captured schema. Assets + // whose column set the parser could not derive (`columns: None`, e.g. + // wildcard SELECT or non-SQL access) are skipped fail-safe; a literal "*" + // key is skipped defensively for the same reason. + for a in assets { + if a.kind != AssetKind::Ducklake { + continue; + } + let Some(columns) = a.columns.as_ref() else { + continue; + }; + let path = normalize_asset_path(&a.path); + let Some(schema) = schemas.get(&path) else { + continue; + }; + for col in columns.keys() { + if col == "*" || is_reserved(col) { + continue; + } + if schema.find(col).is_none() { + warnings.push(ContractWarning { + kind: ContractWarningKind::MissingColumn, + asset_path: path.clone(), + column: Some(col.clone()), + expected_type: None, + found_type: None, + schema_version: Some(schema.version), + captured_at: Some(schema.captured_at), + message: format!( + "column `{col}` of ducklake://{path} is not in its captured schema \ + (v{}, columns: {})", + schema.version, + column_list(schema) + ), + }); + } + } + } + + // W2 — `// column` lineage source refs. Only annotation-declared lineage + // reaches this fn (AST-inferred lineage is redundant with W1 and can + // mis-attribute aliases). + for cl in column_lineage { + for input in &cl.inputs { + if input.from_kind != windmill_parser::asset_parser::AssetKind::Ducklake { + continue; + } + let path = normalize_asset_path(&input.from_path); + let Some(schema) = schemas.get(&path) else { + continue; + }; + if is_reserved(&input.from_column) { + continue; + } + if schema.find(&input.from_column).is_none() { + warnings.push(ContractWarning { + kind: ContractWarningKind::MissingLineageSource, + asset_path: path.clone(), + column: Some(input.from_column.clone()), + expected_type: None, + found_type: None, + schema_version: Some(schema.version), + captured_at: Some(schema.captured_at), + message: format!( + "`// column {}` reads `{}` from ducklake://{path}, which is not in \ + its captured schema (v{})", + cl.column, input.from_column, schema.version + ), + }); + } + } + } + + // W3 — relationships data-test refs: the referenced column must exist; + // when both sides have captured types, flag a difference. Types come from + // DuckDB DESCRIBE on both sides so verbatim spellings are comparable; the + // runtime probe's IN-subquery still coerces, so a difference is "differs", + // never "will fail". + let own_schema = materialize + .filter(|m| m.target_kind == windmill_parser::asset_parser::AssetKind::Ducklake) + .and_then(|m| schemas.get(&normalize_asset_path(&m.target_path))); + for dt in data_tests { + let DataTest::Relationships { column, to_kind, to_path, to_column } = dt else { + continue; + }; + if *to_kind != windmill_parser::asset_parser::AssetKind::Ducklake { + continue; + } + let path = normalize_asset_path(to_path); + let Some(schema) = schemas.get(&path) else { + continue; + }; + match schema.find(to_column) { + None => { + warnings.push(ContractWarning { + kind: ContractWarningKind::MissingRelationshipColumn, + asset_path: path.clone(), + column: Some(to_column.clone()), + expected_type: None, + found_type: None, + schema_version: Some(schema.version), + captured_at: Some(schema.captured_at), + message: format!( + "`// data_test relationships {column}` references \ + ducklake://{path}.{to_column}, which is not in its captured schema \ + (v{})", + schema.version + ), + }); + } + Some(ref_col) => { + if let Some(own_col) = own_schema.and_then(|s| s.find(column)) { + if !own_col.data_type.eq_ignore_ascii_case(&ref_col.data_type) { + warnings.push(ContractWarning { + kind: ContractWarningKind::RelationshipTypeMismatch, + asset_path: path.clone(), + column: Some(to_column.clone()), + expected_type: Some(own_col.data_type.clone()), + found_type: Some(ref_col.data_type.clone()), + schema_version: Some(schema.version), + captured_at: Some(schema.captured_at), + message: format!( + "`// data_test relationships {column}` joins `{}` ({}) to \ + ducklake://{path}.{to_column} ({}) — captured types differ", + column, own_col.data_type, ref_col.data_type + ), + }); + } + } + } + } + } + + // W4 — producer opted the asset out (`on_schema_change=ignore`): drop its + // warnings, leaving one informational entry per suppressed asset so the + // response still records that a mismatch exists but was muted upstream. + if !ignored.is_empty() { + let mut suppressed_assets: Vec = vec![]; + warnings.retain(|w| { + if ignored.contains(&w.asset_path) { + if !suppressed_assets.contains(&w.asset_path) { + suppressed_assets.push(w.asset_path.clone()); + } + false + } else { + true + } + }); + for path in suppressed_assets { + warnings.push(ContractWarning { + kind: ContractWarningKind::Suppressed, + asset_path: path.clone(), + column: None, + expected_type: None, + found_type: None, + schema_version: None, + captured_at: None, + message: format!( + "schema mismatches on ducklake://{path} suppressed by its producer's \ + `on_schema_change=ignore`" + ), + }); + } + } + + warnings +} + +/// Load captured schemas + producer modes and run the contract check for one +/// consumer script's parsed refs. +/// +/// Runs on the caller's RLS-scoped transaction (`user_db`), consistent with +/// the `listAssetSchemas` read path: a producer script the caller cannot read +/// simply stays unresolved and keeps the default `warn` behavior. Draft-only +/// producers have no `asset` write edges yet and likewise default to `warn`. +pub async fn check_schema_contracts( + tx: &mut Transaction<'_, Postgres>, + workspace_id: &str, + assets: &[AssetWithAltAccessType], + column_lineage: &[ColumnLineage], + data_tests: &[DataTest], + materialize: Option<&MaterializeSpec>, +) -> Result> { + // Referenced ducklake paths (normalized) across every ref family the diff + // inspects — plus the consumer's own materialize target (for W3 types). + let mut paths: HashSet = HashSet::new(); + for a in assets { + if a.kind == AssetKind::Ducklake && a.columns.is_some() { + paths.insert(normalize_asset_path(&a.path)); + } + } + for cl in column_lineage { + for input in &cl.inputs { + if input.from_kind == windmill_parser::asset_parser::AssetKind::Ducklake { + paths.insert(normalize_asset_path(&input.from_path)); + } + } + } + for dt in data_tests { + if let DataTest::Relationships { to_kind, to_path, .. } = dt { + if *to_kind == windmill_parser::asset_parser::AssetKind::Ducklake { + paths.insert(normalize_asset_path(to_path)); + } + } + } + if let Some(m) = materialize { + if m.target_kind == windmill_parser::asset_parser::AssetKind::Ducklake { + paths.insert(normalize_asset_path(&m.target_path)); + } + } + if paths.is_empty() { + return Ok(vec![]); + } + + // A managed scd2 producer (re)creates a `_current` view with the base + // table's columns; only the base table's schema is captured. Include the + // base path in the lookup so `_current` readers can fall back to it (the + // fallback itself is gated on the producer's spec below). + let mut lookup_paths: HashSet = paths.clone(); + for p in &paths { + if let Some(base) = p.strip_suffix("_current") { + if !base.is_empty() { + lookup_paths.insert(base.to_string()); + } + } + } + let lookup_vec: Vec = lookup_paths.into_iter().collect(); + + let schema_rows = sqlx::query!( + r#"SELECT DISTINCT ON (asset_path) + asset_path, version, columns AS "columns: Json>", captured_at + FROM materialized_asset_schema + WHERE workspace_id = $1 AND asset_kind = 'ducklake' AND asset_path = ANY($2) + ORDER BY asset_path, version DESC"#, + workspace_id, + &lookup_vec, + ) + .fetch_all(&mut **tx) + .await?; + let mut schemas: HashMap = schema_rows + .into_iter() + .map(|r| { + ( + r.asset_path, + CapturedSchema { + columns: r.columns.0, + version: r.version, + captured_at: r.captured_at, + }, + ) + }) + .collect(); + + // Producer resolution: write edges on the referenced assets → latest + // non-archived producer content → parsed `// materialize` spec. Drives + // both the `on_schema_change=ignore` suppression and the `_current` + // fallback. Flow writers are excluded — they cannot carry the annotation. + let paths_vec: Vec = paths.iter().cloned().collect(); + let producer_edges = sqlx::query!( + r#"SELECT DISTINCT path AS "asset_path!", usage_path AS "producer_path!" + FROM asset + WHERE workspace_id = $1 AND kind = 'ducklake' AND path = ANY($2) + AND usage_kind = 'script' AND usage_access_type IN ('w', 'rw')"#, + workspace_id, + &paths_vec, + ) + .fetch_all(&mut **tx) + .await?; + + let mut ignored: HashSet = HashSet::new(); + if !producer_edges.is_empty() { + let producer_paths: Vec = producer_edges + .iter() + .map(|e| e.producer_path.clone()) + .collect::>() + .into_iter() + .collect(); + // Same latest-content pattern as the asset-graph endpoint; NOT + // `get_latest_script_hash`, whose `lock IS NOT NULL` filter transiently + // excludes a just-deployed producer pending its dependency job. + let producer_rows = sqlx::query!( + r#"SELECT DISTINCT ON (path) path, content + FROM script + WHERE workspace_id = $1 AND path = ANY($2) + AND archived = false AND deleted = false + ORDER BY path, created_at DESC"#, + workspace_id, + &producer_paths, + ) + .fetch_all(&mut **tx) + .await?; + let producer_specs: HashMap> = producer_rows + .into_iter() + .map(|r| { + ( + r.path, + windmill_parser::asset_parser::parse_pipeline_annotations(&r.content) + .materialize, + ) + }) + .collect(); + + for edge in &producer_edges { + let Some(Some(spec)) = producer_specs.get(&edge.producer_path) else { + continue; + }; + if spec.target_kind != windmill_parser::asset_parser::AssetKind::Ducklake { + continue; + } + let target = normalize_asset_path(&spec.target_path); + // `on_schema_change=ignore` — any producer declaring it wins. + if spec.on_schema_change == OnSchemaChange::Ignore + && (target == edge.asset_path + || (spec.scd2 && format!("{target}_current") == edge.asset_path)) + { + ignored.insert(edge.asset_path.clone()); + } + // `_current` fallback: a managed scd2 producer's view has exactly + // the base table's columns. + if spec.scd2 + && !spec.manual + && format!("{target}_current") == edge.asset_path + && !schemas.contains_key(&edge.asset_path) + { + if let Some(base) = schemas.get(&target).cloned() { + schemas.insert(edge.asset_path.clone(), base); + } + } + } + } + + Ok(diff_contract( + assets, + column_lineage, + data_tests, + materialize, + &schemas, + &ignored, + )) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::BTreeMap; + use windmill_parser::asset_parser::{parse_pipeline_annotations, ColumnRef}; + use windmill_types::assets::AssetUsageAccessType; + + fn schema(cols: &[(&str, &str)]) -> CapturedSchema { + CapturedSchema { + columns: cols + .iter() + .map(|(n, t)| SchemaColumn { name: n.to_string(), data_type: t.to_string() }) + .collect(), + version: 2, + captured_at: DateTime::::MIN_UTC, + } + } + + fn read_asset(path: &str, cols: &[&str]) -> AssetWithAltAccessType { + AssetWithAltAccessType { + path: path.to_string(), + kind: AssetKind::Ducklake, + access_type: Some(AssetUsageAccessType::R), + alt_access_type: None, + columns: Some( + cols.iter() + .map(|c| (c.to_string(), AssetUsageAccessType::R)) + .collect::>(), + ), + } + } + + #[test] + fn missing_read_column_warns_case_insensitively() { + let schemas = HashMap::from([( + "lake/orders".to_string(), + schema(&[("Order_ID", "BIGINT"), ("amount_usd", "DOUBLE")]), + )]); + let assets = vec![read_asset("lake/orders", &["order_id", "amount"])]; + let w = diff_contract(&assets, &[], &[], None, &schemas, &HashSet::new()); + // order_id matches case-insensitively; amount is gone + assert_eq!(w.len(), 1); + assert_eq!(w[0].kind, ContractWarningKind::MissingColumn); + assert_eq!(w[0].column.as_deref(), Some("amount")); + assert_eq!(w[0].schema_version, Some(2)); + } + + #[test] + fn unknown_columns_wildcard_and_reserved_are_skipped() { + let schemas = HashMap::from([("lake/orders".to_string(), schema(&[("id", "BIGINT")]))]); + // columns: None (wildcard SELECT) — skipped entirely + let mut a = read_asset("lake/orders", &[]); + a.columns = None; + assert!(diff_contract(&[a], &[], &[], None, &schemas, &HashSet::new()).is_empty()); + // literal "*" and the reserved partition column are skipped + let a = read_asset("lake/orders", &["*", "_wm_partition", "id"]); + assert!(diff_contract(&[a], &[], &[], None, &schemas, &HashSet::new()).is_empty()); + } + + #[test] + fn asset_without_captured_schema_is_silent() { + let assets = vec![read_asset("lake/unknown", &["whatever"])]; + assert!( + diff_contract(&assets, &[], &[], None, &HashMap::new(), &HashSet::new()).is_empty() + ); + } + + #[test] + fn partition_token_is_normalized() { + let schemas = HashMap::from([("lake/orders".to_string(), schema(&[("id", "BIGINT")]))]); + let assets = vec![read_asset("lake/orders/{partition}", &["gone"])]; + let w = diff_contract(&assets, &[], &[], None, &schemas, &HashSet::new()); + assert_eq!(w.len(), 1); + assert_eq!(w[0].asset_path, "lake/orders"); + } + + #[test] + fn lineage_ref_missing_column_warns() { + let schemas = HashMap::from([("lake/orders".to_string(), schema(&[("id", "BIGINT")]))]); + let lineage = vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: windmill_parser::asset_parser::AssetKind::Ducklake, + from_path: "lake/orders".to_string(), + from_column: "amount".to_string(), + }], + }]; + let w = diff_contract(&[], &lineage, &[], None, &schemas, &HashSet::new()); + assert_eq!(w.len(), 1); + assert_eq!(w[0].kind, ContractWarningKind::MissingLineageSource); + } + + #[test] + fn relationships_missing_and_type_mismatch() { + let schemas = HashMap::from([ + ("lake/customers".to_string(), schema(&[("id", "VARCHAR")])), + ( + "lake/orders".to_string(), + schema(&[("customer_id", "BIGINT")]), + ), + ]); + let ann = parse_pipeline_annotations( + "// materialize ducklake://lake/orders\n\ + // data_test relationships customer_id -> ducklake://lake/customers.id\n\ + // data_test relationships customer_id -> ducklake://lake/customers.uuid\n\ + SELECT 1;", + ); + let w = diff_contract( + &[], + &[], + &ann.data_tests, + ann.materialize.as_ref(), + &schemas, + &HashSet::new(), + ); + assert_eq!(w.len(), 2); + assert!(w + .iter() + .any(|w| w.kind == ContractWarningKind::RelationshipTypeMismatch + && w.expected_type.as_deref() == Some("BIGINT") + && w.found_type.as_deref() == Some("VARCHAR"))); + assert!(w + .iter() + .any(|w| w.kind == ContractWarningKind::MissingRelationshipColumn + && w.column.as_deref() == Some("uuid"))); + } + + #[test] + fn ignored_asset_suppresses_to_single_note() { + let schemas = HashMap::from([("lake/orders".to_string(), schema(&[("id", "BIGINT")]))]); + let assets = vec![read_asset("lake/orders", &["a", "b"])]; + let ignored = HashSet::from(["lake/orders".to_string()]); + let w = diff_contract(&assets, &[], &[], None, &schemas, &ignored); + assert_eq!(w.len(), 1); + assert_eq!(w[0].kind, ContractWarningKind::Suppressed); + // and nothing at all when there was nothing to suppress + let assets = vec![read_asset("lake/orders", &["id"])]; + assert!(diff_contract(&assets, &[], &[], None, &schemas, &ignored).is_empty()); + } +} diff --git a/backend/windmill-common/src/scripts.rs b/backend/windmill-common/src/scripts.rs index 0cbcb4a1c1..f3abf14e1c 100644 --- a/backend/windmill-common/src/scripts.rs +++ b/backend/windmill-common/src/scripts.rs @@ -345,10 +345,10 @@ pub async fn clone_script<'c>( ))); }; - let rs = - runnable_settings::from_handle(s.runnable_settings.runnable_settings_handle, db).await?; + let rs = runnable_settings::from_handle(s.runnable_settings.runnable_settings_handle, &mut *tx) + .await?; let (debouncing_settings, concurrency_settings) = - runnable_settings::prefetch_cached(&rs, db).await?; + runnable_settings::prefetch_cached_tx(&rs, &mut tx).await?; let ns = NewScript { path: s.path.clone(), diff --git a/backend/windmill-common/src/secret_backend/mod.rs b/backend/windmill-common/src/secret_backend/mod.rs index 71fa2ea999..596539cd08 100644 --- a/backend/windmill-common/src/secret_backend/mod.rs +++ b/backend/windmill-common/src/secret_backend/mod.rs @@ -13,6 +13,9 @@ //! vaults like HashiCorp Vault (Enterprise Edition). pub mod database; +pub mod resolver; + +pub use resolver::*; #[cfg(feature = "private")] pub mod vault_ee; diff --git a/backend/windmill-common/src/secret_backend/resolver.rs b/backend/windmill-common/src/secret_backend/resolver.rs new file mode 100644 index 0000000000..949300a3d6 --- /dev/null +++ b/backend/windmill-common/src/secret_backend/resolver.rs @@ -0,0 +1,324 @@ +/* + * Author: Ruben Fiszel + * Copyright: Windmill Labs, Inc 2024 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +//! Resolution of the configured secret backend. +//! +//! Lives in `windmill-common` (rather than the API/store crates) so that +//! lower-level helpers such as [`crate::variables::get_variable_or_self`] can +//! route secret reads through the configured backend. With an external backend +//! (Vault / Azure Key Vault / AWS Secrets Manager), the `variable.value` column +//! holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64 +//! ciphertext, so decrypting it directly fails — reads must go through the +//! backend instead. +//! +//! Note: external backends require Enterprise Edition. The OSS version only +//! supports the database backend. + +use std::sync::Arc; + +use crate::{ + db::DB, + error::{Error, Result}, + secret_backend::{database::DatabaseBackend, SecretBackend}, + variables::{build_crypt, decrypt}, +}; + +#[cfg(all(feature = "private", feature = "enterprise"))] +use crate::{ + global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, + secret_backend::{ + AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, + AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, + }, +}; + +#[cfg(all(feature = "private", feature = "enterprise"))] +use tokio::sync::RwLock; + +// Cached Vault backend to avoid recreating it for every request +// This enables connection pooling and avoids repeated setup overhead +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedVaultBackend { + backend: Arc, + settings: VaultSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedAzureKvBackend { + backend: Arc, + settings: AzureKeyVaultSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +// Cached AWS Secrets Manager backend +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedAwsSmBackend { + backend: Arc, + settings: AwsSecretsManagerSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +/// Get the current secret backend based on global settings +/// +/// OSS: Always returns DatabaseBackend +/// EE: Returns configured backend (Database or Vault) +#[cfg(not(all(feature = "private", feature = "enterprise")))] +pub async fn get_secret_backend(db: &DB) -> Result> { + Ok(Arc::new(DatabaseBackend::new(db.clone()))) +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +pub async fn get_secret_backend(db: &DB) -> Result> { + let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { + Some(value) => serde_json::from_value::(value).unwrap_or_default(), + None => SecretBackendConfig::default(), + }; + + match config { + SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), + SecretBackendConfig::HashiCorpVault(settings) => { + get_or_create_vault_backend(db, settings).await + } + SecretBackendConfig::AzureKeyVault(settings) => { + get_or_create_azure_kv_backend(db, settings).await + } + SecretBackendConfig::AwsSecretsManager(settings) => { + get_or_create_aws_sm_backend(db, settings).await + } + } +} + +/// Get a cached Vault backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_vault_backend( + _db: &DB, + settings: VaultSettings, +) -> Result> { + // Check if we have a cached backend with matching settings (read lock) + { + let cache = VAULT_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + // Need to create a new backend - acquire write lock + let mut cache = VAULT_BACKEND_CACHE.write().await; + + // Double-check (another task may have created it while we waited) + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + // Create new backend + let backend: Arc = { + #[cfg(feature = "openidconnect")] + if settings.token.is_none() { + Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) + } else { + Arc::new(VaultBackend::new(settings.clone())) + } + + #[cfg(not(feature = "openidconnect"))] + Arc::new(VaultBackend::new(settings.clone())) + }; + + // Cache it + *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Get a cached Azure Key Vault backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_azure_kv_backend( + _db: &DB, + settings: AzureKeyVaultSettings, +) -> Result> { + // Check if we have a cached backend with matching settings (read lock) + { + let cache = AZURE_KV_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + // Need to create a new backend - acquire write lock + let mut cache = AZURE_KV_BACKEND_CACHE.write().await; + + // Double-check (another task may have created it while we waited) + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + // Create new backend + let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); + + // Cache it + *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Get a cached AWS SM backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_aws_sm_backend( + _db: &DB, + settings: AwsSecretsManagerSettings, +) -> Result> { + { + let cache = AWS_SM_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + let mut cache = AWS_SM_BACKEND_CACHE.write().await; + + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + let backend: Arc = + Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); + + *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Check if a Vault backend is currently configured +/// +/// OSS: Always returns false +/// EE: Checks global settings +#[cfg(not(all(feature = "private", feature = "enterprise")))] +pub async fn is_vault_backend_configured(_db: &DB) -> Result { + Ok(false) +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +pub async fn is_vault_backend_configured(db: &DB) -> Result { + let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { + Some(value) => serde_json::from_value::(value).unwrap_or_default(), + None => SecretBackendConfig::default(), + }; + + Ok(matches!( + config, + SecretBackendConfig::HashiCorpVault(_) + | SecretBackendConfig::AzureKeyVault(_) + | SecretBackendConfig::AwsSecretsManager(_) + )) +} + +/// Get a secret value using the configured backend +/// +/// For database backend: decrypts `encrypted_value` using the workspace key +/// For external backends (EE only): fetches from the backend at `path`, +/// ignoring `encrypted_value` (which holds only a `$...:` marker) +pub async fn get_secret_value( + db: &DB, + workspace_id: &str, + path: &str, + encrypted_value: &str, +) -> Result { + let backend = get_secret_backend(db).await?; + + match backend.backend_name() { + "database" => { + // Use existing database decryption + let mc = build_crypt(db, workspace_id).await?; + decrypt(&mc, encrypted_value.to_string()).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) + }) + } + "hashicorp_vault" => { + // Fetch from Vault directly + backend.get_secret(workspace_id, path).await + } + "azure_key_vault" => backend.get_secret(workspace_id, path).await, + "aws_secrets_manager" => backend.get_secret(workspace_id, path).await, + _ => Err(Error::internal_err(format!( + "Unknown backend: {}", + backend.backend_name() + ))), + } +} + +/// Check if a value is stored in Vault (indicated by the $vault: prefix) +pub fn is_vault_stored_value(value: &str) -> bool { + value.starts_with("$vault:") +} + +/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) +pub fn is_azure_kv_stored_value(value: &str) -> bool { + value.starts_with("$azure_kv:") +} + +/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix) +pub fn is_aws_sm_stored_value(value: &str) -> bool { + value.starts_with("$aws_sm:") +} + +/// Check if a value is stored in any external secret backend +pub fn is_external_stored_value(value: &str) -> bool { + is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn external_markers_are_detected() { + assert!(is_external_stored_value("$vault:u/admin/secret")); + assert!(is_external_stored_value("$azure_kv:u/admin/secret")); + assert!(is_external_stored_value("$aws_sm:u/admin/secret")); + } + + #[test] + fn base64_ciphertext_is_not_treated_as_external() { + // A base64 magic_crypt blob must route through `decrypt`, never the + // external backend. The leading `$` is what distinguishes a marker from + // ciphertext; decrypting a marker fails with "Invalid byte 36" (`$`), + // which is the bug this gate prevents. + for v in [ + "bm90LWEtbWFya2Vy", + "AAAA1234+/abcd==", + "", + "$something_else", + ] { + assert!(!is_external_stored_value(v), "unexpected external: {v:?}"); + } + } +} diff --git a/backend/windmill-common/src/user_drafts.rs b/backend/windmill-common/src/user_drafts.rs index 620cbe2a5f..7e0c1f3f55 100644 --- a/backend/windmill-common/src/user_drafts.rs +++ b/backend/windmill-common/src/user_drafts.rs @@ -237,18 +237,22 @@ async fn fetch_other_drafts_users( kind: UserDraftItemKind, path: &str, ) -> Result> { - // The `admins` workspace has no `usr` rows (username IS the email there), - // so fall back to `d.email` when the join misses, else a real teammate's - // draft renders as a phantom "Legacy draft". The genuine NULL-email legacy - // row keeps `username = None` (its `d.email` is NULL, so the CASE yields NULL). + // A superadmin authoring in a workspace they are not a member of has no `usr` + // row: fall back to their instance-derived username (`password.username`), or + // their email when derivation is disabled. Else a real teammate's draft renders + // as a phantom "Legacy draft". The genuine NULL-email legacy row keeps + // `username = None` (no `usr`/`password` match and `d.email` is NULL). let rows = sqlx::query_as!( OtherDraftUser, - r#"SELECT COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) as "username?", + r#"SELECT COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN d.email END) as "username?", d.created_at as "draft_saved_at!" FROM draft d LEFT JOIN usr u ON u.workspace_id = d.workspace_id AND u.email = d.email + LEFT JOIN password p + ON p.email = d.email + AND p.super_admin = true WHERE d.workspace_id = $1 AND d.path = $2 AND d.typ = $3 diff --git a/backend/windmill-common/src/usernames.rs b/backend/windmill-common/src/usernames.rs index 2501841cac..6fde2a8186 100644 --- a/backend/windmill-common/src/usernames.rs +++ b/backend/windmill-common/src/usernames.rs @@ -94,6 +94,38 @@ pub async fn generate_instance_username_for_all_users(db: &DB) -> error::Result< Ok(()) } +/// Resolve the username to use for a user (typically a superadmin) accessing a +/// workspace they are not a member of. When instance-wide username derivation is +/// enabled (`automate_username_creation`, the default on almost all instances), +/// `password.username` is populated, so we return that derived username instead +/// of leaking the raw email address downstream (as job/script author, audit +/// actor, etc.). Falls back to the email when no derived username exists (the few +/// instances that keep `automate_username_creation` disabled). +/// +/// The hot callers (token auth, `fetch_api_authed`) are already behind their own +/// 120s auth caches, so this deliberately does not add another email->username +/// cache (which would only help the cold JWT-mint / job-perms paths while risking +/// cross-DB contamination in the shared-process integration tests). +/// +/// A DB error is propagated rather than swallowed into the email fallback: falling +/// back to the email on a transient failure would reintroduce the very email leak +/// this path exists to prevent, so callers fail closed instead. +pub async fn get_instance_username_or_fallback_to_email<'e, E>( + db: E, + email: &str, +) -> error::Result +where + E: sqlx::PgExecutor<'e>, +{ + let derived = sqlx::query_scalar!("SELECT username FROM password WHERE email = $1", email) + .fetch_optional(db) + .await? + .flatten(); + // No derived username (automate_username_creation disabled) → fall back to the + // email. This is the only legitimate fallback; a query error propagates above. + Ok(derived.unwrap_or_else(|| email.to_string())) +} + pub async fn get_instance_username_or_create_pending<'c>( tx: &mut Transaction<'c, Postgres>, email: &str, diff --git a/backend/windmill-common/src/users.rs b/backend/windmill-common/src/users.rs index f91c21701a..738dc84d14 100644 --- a/backend/windmill-common/src/users.rs +++ b/backend/windmill-common/src/users.rs @@ -46,14 +46,40 @@ lazy_static::lazy_static! { const EMAIL_CACHE_TTL_SECS: u64 = 60; +/// Resolve a workspace-scoped username to its email. +/// +/// Members are found in `usr`. A superadmin acting in a workspace they are *not* +/// a member of has no `usr` row; they carry either their instance-derived +/// username (`password.username`, when `automate_username_creation` is enabled) +/// or their email (when it is disabled), so fall back to `password` on both, +/// gated on `super_admin` since only superadmins can act without membership. +/// Returns `None` when the username resolves to nobody. +pub async fn resolve_username_to_email<'c>( + workspace_id: &str, + username: &str, + db: impl sqlx::PgExecutor<'c>, +) -> crate::error::Result> { + Ok(sqlx::query_scalar!( + "SELECT COALESCE( + (SELECT email FROM usr WHERE workspace_id = $1 AND username = $2), + (SELECT email FROM password WHERE (username = $2 OR email = $2) AND super_admin = true) + )", + workspace_id, + username + ) + .fetch_optional(db) + .await? + .flatten()) +} + /// Get email from permissioned_as string. -/// - "u/{username}" → lookup email from usr table (cached) +/// - "u/{username}" → resolve via [`resolve_username_to_email`] (cached) /// - "g/{group}" → "group-{group}@windmill.dev" /// - raw email → return as-is -pub async fn get_email_from_permissioned_as( +pub async fn get_email_from_permissioned_as<'c>( permissioned_as: &str, workspace_id: &str, - db: &sqlx::Pool, + db: impl sqlx::PgExecutor<'c>, ) -> crate::error::Result { if let Some(username) = permissioned_as.strip_prefix(PERMISSIONED_AS_USER_PREFIX) { let lookup = EmailCacheKey(workspace_id, username); @@ -62,14 +88,9 @@ pub async fn get_email_from_permissioned_as( return Ok(email); } } - let email = sqlx::query_scalar!( - "SELECT email FROM usr WHERE username = $1 AND workspace_id = $2", - username, - workspace_id - ) - .fetch_optional(db) - .await? - .unwrap_or_else(|| format!("{}@unknown.windmill.dev", username)); + let email = resolve_username_to_email(workspace_id, username, db) + .await? + .unwrap_or_else(|| format!("{}@unknown.windmill.dev", username)); let key = (workspace_id.to_string(), username.to_string()); EMAIL_CACHE.insert(key, (email.clone(), std::time::Instant::now())); Ok(email) diff --git a/backend/windmill-common/src/variables.rs b/backend/windmill-common/src/variables.rs index 0d42b95a5a..b43eb07e9c 100644 --- a/backend/windmill-common/src/variables.rs +++ b/backend/windmill-common/src/variables.rs @@ -9,6 +9,7 @@ use crate::db::{Authable, UserDB}; use crate::error::{self, Error}; use crate::scripts::ScriptHash; +use crate::secret_backend::{get_secret_value, is_external_stored_value}; use crate::utils::WarnAfterExt; use crate::worker::Connection; use crate::{worker::WORKER_GROUP, BASE_URL, DB}; @@ -234,13 +235,17 @@ pub async fn get_secret_value_as_admin( let r = if variable.is_secret { let value = variable.value; if !value.is_empty() { - let mc = build_crypt(db, w_id).await?; - decrypt(&mc, value).map_err(|e| { - crate::error::Error::internal_err(format!( - "Error decrypting variable {}: {}", - variable.path, e - )) - })? + if is_external_stored_value(&value) { + get_secret_value(db, w_id, &variable.path, &value).await? + } else { + let mc = build_crypt(db, w_id).await?; + decrypt(&mc, value).map_err(|e| { + crate::error::Error::internal_err(format!( + "Error decrypting variable {}: {}", + variable.path, e + )) + })? + } } else { "".to_string() } @@ -546,10 +551,14 @@ pub async fn get_variable_or_self( if let Some(record) = record { let mut value = record.value; if record.is_secret { - let mc = build_crypt(db, w_id).await?; - value = decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) - })?; + if is_external_stored_value(&value) { + value = get_secret_value(db, w_id, &path, &value).await?; + } else { + let mc = build_crypt(db, w_id).await?; + value = decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) + })?; + } } Ok(value) @@ -591,10 +600,14 @@ pub async fn get_variable_or_self_as( if let Some(record) = record { let mut value = record.value; if record.is_secret { - let mc = build_crypt(db, w_id).await?; - value = decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e)) - })?; + if is_external_stored_value(&value) { + value = get_secret_value(db, w_id, &var_path, &value).await?; + } else { + let mc = build_crypt(db, w_id).await?; + value = decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e)) + })?; + } } Ok(value) diff --git a/backend/windmill-common/src/worker.rs b/backend/windmill-common/src/worker.rs index e3bd7ef7ef..ea2b5455f3 100644 --- a/backend/windmill-common/src/worker.rs +++ b/backend/windmill-common/src/worker.rs @@ -273,6 +273,14 @@ lazy_static::lazy_static! { /// production `app.windmill.dev` cluster, not on staging or self-hosted. pub static ref CLOUD_PRODUCTION_HOST: &'static str = "app.windmill.dev"; + /// `--no-auth` mode: when set, every API request is treated as + /// authenticated as the `admin@windmill.dev` superadmin and no login is + /// ever required. Meant for self-hosted deployments that front Windmill + /// with their own authenticating gateway. Never honored on the managed + /// cloud (`CLOUD_HOSTED`), which must always enforce real authentication. + pub static ref NO_AUTH: bool = !*CLOUD_HOSTED + && std::env::var("NO_AUTH").ok().is_some_and(|x| x == "1" || x == "true"); + pub static ref CUSTOM_TAGS: Vec = std::env::var("CUSTOM_TAGS") .ok() .map(|x| x.split(',').map(|x| x.to_string()).collect::>()).unwrap_or_default(); @@ -693,8 +701,6 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error let full_path = job_dir.join(&user_path); - // let normalized_job_dir = std::fs::canonicalize(job_dir)?; - // let normalized_full_path = std::fs::canonicalize(&full_path)?; let normalized_job_dir = normalize_path(job_dir); let normalized_full_path = normalize_path(&full_path); @@ -706,6 +712,36 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error .into()); } + // The lexical check above cannot see symlinks: a symlink planted inside the + // job dir - e.g. by an earlier Ansible `git_repos` clone whose tracked + // content includes one - would let a later `git clone` or file write follow + // it out of the job dir while still passing the textual `starts_with` check. + // Walk the *normalized* relative path (`..`/`.` already collapsed) so each + // step matches the real on-disk resolution, and reject any existing component + // that is a symlink. Walking the raw user path would drift on an in-bounds + // `..` (e.g. `foo/../link`, which normalizes back inside the job dir) and miss + // the real symlinked component. Not-yet-existing components are safe: a path + // that does not exist cannot itself be a symlink. + let relative = normalized_full_path + .strip_prefix(&normalized_job_dir) + .unwrap_or(&normalized_full_path); + let mut current = normalized_job_dir.clone(); + for component in relative.components() { + if let Component::Normal(c) = component { + current.push(c); + if std::fs::symlink_metadata(¤t) + .map(|m| m.file_type().is_symlink()) + .unwrap_or(false) + { + return Err(std::io::Error::new( + std::io::ErrorKind::PermissionDenied, + "Path traverses a symlink, which is not allowed.", + ) + .into()); + } + } + } + Ok(normalized_full_path) } @@ -2828,4 +2864,71 @@ mod tests { let _ = std::fs::remove_dir_all(&base); } + + #[test] + fn test_is_allowed_file_location_allows_plain_relative() { + let base = std::env::temp_dir().join(format!("wm_allowed_loc_ok_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + let out = is_allowed_file_location(job_dir_str, "repo/sub/playbook.yml").unwrap(); + assert_eq!(out, normalize_path(&job_dir.join("repo/sub/playbook.yml"))); + + let _ = std::fs::remove_dir_all(&base); + } + + #[test] + fn test_is_allowed_file_location_rejects_parent_and_absolute() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_esc_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + assert!(is_allowed_file_location(job_dir_str, "../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "a/../../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "/etc/passwd").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } + + // Regression for GHSA-v934-cvpf-6fjw: a symlink planted inside the job dir + // (e.g. by an earlier `git_repos` clone) must not let a later target traverse + // it out of the job dir, even though the lexical path stays "inside". + #[cfg(unix)] + #[test] + fn test_is_allowed_file_location_rejects_symlink_traversal() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_symlink_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + // Stand-in for the shared cache dir living outside the job dir. + let outside = base.join("outside"); + std::fs::create_dir_all(&outside).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + // Plant `job/repo` -> `../outside`, as a malicious first clone would. + let planted = job_dir.join("repo"); + std::os::unix::fs::symlink(&outside, &planted).unwrap(); + + // Both the symlink itself and any path traversing it are rejected. + assert!(is_allowed_file_location(job_dir_str, "repo").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/payload").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/sub/payload").is_err()); + + // An in-bounds `..` must not bypass the check: `foo/../repo/payload` + // normalizes back to `repo/payload` and still traverses the symlink. + assert!(is_allowed_file_location(job_dir_str, "foo/../repo/payload").is_err()); + std::fs::create_dir(job_dir.join("real")).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "real/../repo/payload").is_err()); + + // A dangling symlink (target does not exist yet) is still caught: + // `symlink_metadata` does not follow the link. + let dangling = job_dir.join("dangling"); + std::os::unix::fs::symlink(base.join("nonexistent"), &dangling).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "dangling/payload").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } } diff --git a/backend/windmill-common/src/workspaces.rs b/backend/windmill-common/src/workspaces.rs index 3d4402decb..a8c6e26f0e 100644 --- a/backend/windmill-common/src/workspaces.rs +++ b/backend/windmill-common/src/workspaces.rs @@ -8,6 +8,7 @@ use strum::AsRefStr; use crate::{ error::{self, to_anyhow, Error, Result}, get_database_url, + secret_backend::{get_secret_value, is_external_stored_value}, utils::get_custom_pg_instance_password, variables::{build_crypt, decrypt}, PgDatabase, DB, @@ -163,6 +164,7 @@ pub enum ObjectType { Settings, Key, WorkspaceDependencies, + DatatableMigration, } pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28719/sync-script-to-git-repo-windmill"; @@ -178,7 +180,37 @@ pub const WM_FORK_PREFIX: &str = "wm-fork-"; /// layer because the actual branch creation runs in a deferred git-sync worker job — without /// this check, the API returns 200 and the failure only surfaces later in the worker. pub fn validate_fork_workspace_id(id: &str) -> error::Result<()> { - if !id.starts_with(WM_FORK_PREFIX) { + validate_workspace_branch_id(id, true) +} + +/// Like [`validate_fork_workspace_id`] but does not require the `wm-fork-` prefix. Used for dev +/// workspaces, whose id is an ordinary (prefix-less) workspace id but must still be git-branch-safe +/// because it is interpolated into a `wm-fork//` branch name like any fork. +pub fn validate_dev_workspace_id(id: &str) -> error::Result<()> { + validate_workspace_branch_id(id, false) +} + +/// The `workspace.name` column is `character varying(50)`, so a name longer than 50 characters +/// triggers a raw `value too long for type character varying(50)` SQL error on insert. Validate +/// up front to return a clear message instead. +pub fn validate_workspace_name(name: &str) -> error::Result<()> { + if name.chars().count() > 50 { + return Err(Error::BadRequest(format!( + "Workspace name is too long ({} chars). Maximum length is 50 characters.", + name.chars().count() + ))); + } + Ok(()) +} + +fn validate_workspace_branch_id(id: &str, require_fork_prefix: bool) -> error::Result<()> { + if id.is_empty() { + return Err(Error::BadRequest( + "Workspace id cannot be empty".to_string(), + )); + } + + if require_fork_prefix && !id.starts_with(WM_FORK_PREFIX) { return Err(Error::BadRequest(format!( "The id `{}` is invalid for a forked workspace. It should be prefixed by {}", id, WM_FORK_PREFIX @@ -187,8 +219,9 @@ pub fn validate_fork_workspace_id(id: &str) -> error::Result<()> { if id.len() > 50 { return Err(Error::BadRequest(format!( - "Fork workspace id `{}` is too long ({} chars). Maximum length is 50 characters (including the '{}' prefix).", - id, id.len(), WM_FORK_PREFIX + "Workspace id `{}` is too long ({} chars). Maximum length is 50 characters.", + id, + id.len() ))); } @@ -315,9 +348,215 @@ lazy_static::lazy_static! { pub static ref PUBLIC_APP_RATE_LIMIT_CACHE: Cache, i64)> = Cache::new(1000); } +#[cfg(feature = "cloud")] +lazy_static::lazy_static! { + // Maps a workspace id to its root (billing) ancestor. Value: (root_id, expiry_timestamp). + // Reparenting (attach/detach dev) is rare and self-heals via the 60s TTL, so a brief stale + // mapping only mis-attributes usage for <60s across other instances. + pub static ref BILLING_WORKSPACE_CACHE: Cache = Cache::new(5000); +} + +/// Resolve the "billing" workspace for `w_id`: the root ancestor of the fork/dev chain (the +/// workspace whose plan and usage a fork draws from). Returns `w_id` unchanged for a standalone +/// workspace, an unknown id, or a (malformed) cyclic chain. +/// +/// Unauthenticated metering helper: it only reads the parent chain and returns another workspace id, +/// so callers must already be authorized for `w_id` (or run in trusted server-side code); `w_id` is +/// expected to be a server-side id, not raw user input. +#[cfg(feature = "cloud")] +pub async fn get_billing_workspace_id(db: &crate::DB, w_id: &str) -> Result { + let now = chrono::Utc::now().timestamp(); + if let Some((root, expiry)) = BILLING_WORKSPACE_CACHE.get(w_id) { + if expiry > now { + return Ok(root); + } + } + + // The depth bound is a cycle-safety backstop kept well above the enforced `MAX_FORK_DEPTH`, so a + // truncated (root-not-found) result — which would fall back to `w_id` and mis-attribute billing — + // is unreachable for any real hierarchy; only a malformed cycle could hit it. + let root = sqlx::query_scalar!( + r#" + WITH RECURSIVE chain AS ( + SELECT id, parent_workspace_id, 0 AS depth + FROM workspace WHERE id = $1 + UNION ALL + SELECT w.id, w.parent_workspace_id, chain.depth + 1 + FROM workspace w + JOIN chain ON w.id = chain.parent_workspace_id + WHERE chain.depth < 20 + ) + SELECT id AS "id!" FROM chain WHERE parent_workspace_id IS NULL LIMIT 1 + "#, + w_id + ) + .fetch_optional(db) + .await + .map_err(|e| Error::internal_err(format!("resolving billing workspace for {w_id}: {e:#}")))? + .unwrap_or_else(|| w_id.to_string()); + + BILLING_WORKSPACE_CACHE.insert(w_id.to_string(), (root.clone(), now + 60)); + Ok(root) +} + +/// Invalidate the billing-workspace mapping for a workspace (call after reparenting it). +#[cfg(feature = "cloud")] +pub fn invalidate_billing_workspace_cache(w_id: &str) { + BILLING_WORKSPACE_CACHE.remove(w_id); +} + +/// Invalidate the cached team-plan (premium/past-due) status for a workspace. `TEAM_PLAN_CACHE` has +/// no TTL — it's only evicted by the premium-change NOTIFY — so call this when a workspace id is +/// permanently deleted, otherwise a reused id could inherit the old workspace's premium status. +#[cfg(feature = "cloud")] +pub fn invalidate_team_plan_cache(w_id: &str) { + TEAM_PLAN_CACHE.remove(w_id); +} + +/// Depth of `w_id` in its fork chain: 0 for a root (no parent), 1 for a direct fork, and so on. Walks +/// the parent chain up to the root. The recursion bound is a cycle-safety backstop set well above the +/// enforced `MAX_FORK_DEPTH`; a (malformed) cyclic chain saturates it and so reads as "too deep", +/// which safely rejects rather than allows. +/// +/// Unauthenticated helper: reads workspace hierarchy for any `w_id`, so callers must already be +/// authorized for that workspace (or run in trusted server-side code). +pub async fn fork_chain_depth(db: &crate::DB, w_id: &str) -> Result { + let depth = sqlx::query_scalar!( + r#" + WITH RECURSIVE chain AS ( + SELECT id, parent_workspace_id, 0 AS depth + FROM workspace WHERE id = $1 + UNION ALL + SELECT w.id, w.parent_workspace_id, chain.depth + 1 + FROM workspace w + JOIN chain ON w.id = chain.parent_workspace_id + WHERE chain.depth < 20 + ) + SELECT COALESCE(MAX(depth), 0)::bigint AS "depth!" FROM chain + "#, + w_id + ) + .fetch_one(db) + .await + .map_err(|e| Error::internal_err(format!("computing fork depth for {w_id}: {e:#}")))?; + Ok(depth) +} + +/// Height of the fork subtree rooted at `w_id`: 0 when it has no live child forks, 1 with direct +/// children, and so on. Used so that attaching a candidate which already has its own child forks can't +/// push the family past the depth limit. +/// +/// Unauthenticated helper: reads workspace hierarchy for any `w_id`, so callers must already be +/// authorized for that workspace (or run in trusted server-side code). +pub async fn fork_subtree_height(db: &crate::DB, w_id: &str) -> Result { + // The `deleted` filter is applied in the outer aggregation (not the recursive step, matching + // count_workspace_forks) so a live descendant under a soft-deleted intermediate is still measured + // at its true depth rather than pruned — otherwise the height could be underestimated and let the + // resulting chain exceed the depth limit. + let height = sqlx::query_scalar!( + r#" + WITH RECURSIVE tree AS ( + SELECT id, deleted, 0 AS depth FROM workspace WHERE id = $1 + UNION ALL + SELECT w.id, w.deleted, tree.depth + 1 FROM workspace w + JOIN tree ON w.parent_workspace_id = tree.id + WHERE tree.depth < 20 + ) + SELECT COALESCE(MAX(depth) FILTER (WHERE NOT deleted), 0)::bigint AS "height!" FROM tree + "#, + w_id + ) + .fetch_one(db) + .await + .map_err(|e| Error::internal_err(format!("computing fork subtree height for {w_id}: {e:#}")))?; + Ok(height) +} + +/// Ids of every fork/dev workspace anywhere under `w_id` (excludes `w_id` itself), including live +/// descendants beneath a soft-deleted intermediate. Used to invalidate per-workspace caches for a +/// whole subtree after its ancestor is reparented. +/// +/// Unauthenticated helper: reads workspace hierarchy for any `w_id`, so callers must already be +/// authorized for that workspace (or run in trusted server-side code). +pub async fn list_fork_descendants(db: &crate::DB, w_id: &str) -> Result> { + let ids = sqlx::query_scalar!( + r#" + WITH RECURSIVE tree AS ( + SELECT id, 0 AS depth FROM workspace WHERE id = $1 + UNION ALL + SELECT w.id, tree.depth + 1 FROM workspace w + JOIN tree ON w.parent_workspace_id = tree.id + WHERE tree.depth < 20 + ) + SELECT id AS "id!" FROM tree WHERE id != $1 + "#, + w_id + ) + .fetch_all(db) + .await + .map_err(|e| Error::internal_err(format!("listing fork descendants of {w_id}: {e:#}")))?; + Ok(ids) +} + +/// Count non-deleted fork/dev workspaces anywhere under `root` (excludes `root` itself). +/// +/// Unauthenticated metering helper: it reads workspace hierarchy for any `root` id, so callers must +/// already be authorized for that workspace (or run in trusted server-side code). `root` is expected +/// to be a server-resolved id, never raw user input. +#[cfg(feature = "cloud")] +pub async fn count_workspace_forks(db: &crate::DB, root: &str) -> Result { + // The `deleted` filter is on the outer SELECT (not the recursive step) so that a live sub-fork + // whose intermediate parent was soft-deleted is still counted rather than pruned with it. The + // depth bound is a cycle-safety backstop kept well above the enforced `MAX_FORK_DEPTH`, so + // descendants are never silently dropped from the cap count. + let count = sqlx::query_scalar!( + r#" + WITH RECURSIVE tree AS ( + SELECT id, deleted, 0 AS depth FROM workspace WHERE id = $1 + UNION ALL + SELECT w.id, w.deleted, tree.depth + 1 FROM workspace w + JOIN tree ON w.parent_workspace_id = tree.id + WHERE tree.depth < 20 + ) + SELECT COUNT(DISTINCT id) AS "count!" FROM tree WHERE id != $1 AND NOT deleted + "#, + root + ) + .fetch_one(db) + .await + .map_err(|e| Error::internal_err(format!("counting forks of {root}: {e:#}")))?; + Ok(count) +} + +/// Approximate paid seats of a workspace as `ceil(developers + operators/2)`, excluding disabled and +/// service-account members. Reuses billing's author/operator weighting, but counts provisioned +/// members rather than the active-user population billing meters, so it only ever loosens the fork +/// cap (never blocks a paid seat) — good enough for a soft guardrail. +/// +/// Unauthenticated metering helper: reads member counts for any `w_id`, so callers must already be +/// authorized for that workspace (or run in trusted server-side code). +#[cfg(feature = "cloud")] +pub async fn count_paid_seats(db: &crate::DB, w_id: &str) -> Result { + let row = sqlx::query!( + r#"SELECT + COUNT(*) FILTER (WHERE NOT operator AND NOT disabled AND NOT is_service_account) AS "developers!", + COUNT(*) FILTER (WHERE operator AND NOT disabled AND NOT is_service_account) AS "operators!" + FROM usr WHERE workspace_id = $1"#, + w_id + ) + .fetch_one(db) + .await + .map_err(|e| Error::internal_err(format!("counting paid seats of {w_id}: {e:#}")))?; + Ok(((row.developers as f64) + 0.5 * (row.operators as f64)).ceil() as i64) +} + #[cfg(feature = "cloud")] pub async fn get_team_plan_status(_db: &crate::DB, _w_id: &str) -> Result { - let cached = TEAM_PLAN_CACHE.get(_w_id); + // A fork/dev workspace draws its plan from the root (billing) workspace. Resolve to the root and + // key the cache by it: the premium-change NOTIFY is keyed by the workspace whose premium row + // changed (the root), so keying by root keeps invalidation correct and lets forks share it. + let billing_w_id = get_billing_workspace_id(_db, _w_id).await?; + let cached = TEAM_PLAN_CACHE.get(&billing_w_id); if let Some(cached) = cached { return Ok(cached); } @@ -336,7 +575,7 @@ pub async fn get_team_plan_status(_db: &crate::DB, _w_id: &str) -> Result Result Result, + /// Whether the SQL-migrations feature is opted in for this data table. + /// Absent on data tables created before the feature: treated as enabled only + /// when migrations already exist (see `datatable_migrations_enabled`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub migrations_enabled: Option, } #[derive(Deserialize, Serialize, Debug)] @@ -553,25 +802,55 @@ pub enum DataTableCatalogResourceType { Instance, } +/// Build a self-teaching error for an unresolved `datatable://` reference. +/// The raw "not found" gives the user no way forward — the datatable substrate has +/// no auto-provisioning (unlike a DuckLake catalog), so the fix is always to create +/// one in workspace settings. Surface the available names (to catch typos) and point +/// at the settings page so the message is actionable wherever it bubbles up +/// (pipeline `ATTACH`, schema fetch, postgres executor, ...). +fn datatable_not_found_error(name: &str, datatables: Option<&serde_json::Value>) -> Error { + let available: Vec<&str> = datatables + .and_then(|d| d.as_object()) + .map(|o| o.keys().map(String::as_str).collect()) + .unwrap_or_default(); + + let hint = if available.is_empty() { + "No data table is configured in this workspace yet.".to_string() + } else { + format!("Configured data tables: {}.", available.join(", ")) + }; + + Error::NotFound(format!( + "Data table '{name}' not found. {hint} \ + Create one in workspace settings under the \"Data tables\" tab \ + (/workspace_settings?tab=windmill_data_tables) — the name \"main\" is the default \ + used by `datatable://main`." + )) +} + pub async fn get_datatable_resource_from_db_unchecked( db: &DB, w_id: &str, name: &str, ) -> Result { - let datatable = sqlx::query_scalar!( + let datatables = sqlx::query_scalar!( r#" - SELECT ws.datatable->'datatables'->$2 AS config + SELECT ws.datatable->'datatables' AS datatables FROM workspace_settings ws WHERE ws.workspace_id = $1 "#, &w_id, - name ) .fetch_one(db) .await - .map_err(|err| Error::internal_err(format!("getting datatable {name}: {err}")))? - .ok_or_else(|| Error::internal_err(format!("datatable {name} not found")))?; - let datatable = serde_json::from_value::(datatable)?; + .map_err(|err| Error::internal_err(format!("getting datatable {name}: {err}")))?; + + let datatable = datatables + .as_ref() + .and_then(|d| d.get(name)) + .filter(|v| !v.is_null()) + .ok_or_else(|| datatable_not_found_error(name, datatables.as_ref()))?; + let datatable = serde_json::from_value::(datatable.clone())?; let db_resource = if datatable.database.resource_type == DataTableCatalogResourceType::Instance { @@ -599,6 +878,97 @@ pub struct Ducklake { pub storage: DucklakeStorage, #[serde(skip_serializing_if = "Option::is_none")] pub extra_args: Option, + /// How this lake behaves when the workspace is a fork/dev workspace. Only meaningful in a + /// fork's own settings; stamped at fork creation from the user's per-lake choice. Absent = + /// `Isolated` — the safe default, so forks created before this field existed (and API + /// callers that omit it) never write the parent's lake. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub fork_behavior: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub maintenance: Option, +} + +/// Per-lake fork data-environment choice, made at fork creation. +#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum DucklakeForkBehavior { + /// Fork-scoped namespace + read-defer to the parent (default). + Isolated, + /// The fork reads AND WRITES the parent's lake directly — explicit opt-out of isolation + /// (e.g. a fork meant to run prod-equivalent backfills). + Shared, +} + +/// Scheduled maintenance for a ducklake (enterprise): snapshot expiry, +/// adjacent-file compaction and orphaned-file cleanup, run as a managed +/// per-lake schedule. Not mirrored in `instance_config::Ducklake`: +/// instance-level lakes have no workspace to schedule into. +#[derive(Deserialize, Serialize, Debug, Clone)] +pub struct DucklakeMaintenance { + pub enabled: bool, + /// Cron (v2/croner, seconds optional). None → daily at 03:00 UTC with a + /// deterministic per-(workspace, lake) minute offset. + #[serde(skip_serializing_if = "Option::is_none")] + pub schedule: Option, + /// Snapshot retention window in days (default 7). Snapshots older than + /// this are expired: time-travel reads (`AT (VERSION => n)`) older than + /// the window stop working. 0 keeps only the current snapshot. + #[serde(skip_serializing_if = "Option::is_none")] + pub retention_days: Option, + /// Merge adjacent small parquet files (default true). + #[serde(skip_serializing_if = "Option::is_none")] + pub compaction: Option, + /// Delete orphaned files older than max(retention, 1 day) (default true). + #[serde(skip_serializing_if = "Option::is_none")] + pub orphan_cleanup: Option, +} + +impl DucklakeMaintenance { + pub const DEFAULT_RETENTION_DAYS: u32 = 7; + + pub fn retention_days(&self) -> u32 { + self.retention_days.unwrap_or(Self::DEFAULT_RETENTION_DAYS) + } + pub fn compaction(&self) -> bool { + self.compaction.unwrap_or(true) + } + pub fn orphan_cleanup(&self) -> bool { + self.orphan_cleanup.unwrap_or(true) + } +} + +/// Reserved schedule path namespace for managed ducklake maintenance +/// schedules. Must satisfy the `schedule.path` CHECK constraint +/// (`^[ufg](\/[\w-]+){2,}$`), hence the `f/` prefix; the folder itself never +/// exists. The schedule API rejects user mutations under this prefix and the +/// list/export endpoints filter it out — the lifecycle is owned by the +/// workspace ducklake settings. +/// +/// Accepted limitation: a schedule that pre-dated this namespace under a real +/// `ducklake_maintenance` folder keeps running (tick dispatch falls through +/// to its script when its path's lake has no enabled maintenance config, and +/// the settings sync only touches rows derived from config) but stays hidden +/// from list/export and immutable via the schedule API until renamed out of +/// the namespace. Judged unlikely enough to not warrant a discriminator +/// column or a rename migration. +pub const DUCKLAKE_MAINTENANCE_PATH_PREFIX: &str = "f/ducklake_maintenance/"; + +pub fn ducklake_maintenance_schedule_path(lake: &str) -> String { + format!("{DUCKLAKE_MAINTENANCE_PATH_PREFIX}{lake}") +} + +pub fn lake_from_ducklake_maintenance_path(path: &str) -> Option<&str> { + path.strip_prefix(DUCKLAKE_MAINTENANCE_PATH_PREFIX) +} + +/// Lake names are interpolated into `ATTACH 'ducklake://'`, generated +/// maintenance SQL and the reserved schedule path (CHECK-constrained to +/// `[\w-]+` segments), so they must stay to this charset. +pub fn is_valid_ducklake_name(name: &str) -> bool { + !name.is_empty() + && name + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') } #[derive(Deserialize, Serialize, Debug)] @@ -632,6 +1002,213 @@ pub struct DucklakeWithConnData { pub storage: DucklakeStorage, #[serde(skip_serializing_if = "Option::is_none")] pub extra_args: Option, + /// Present when the resolved workspace is a fork/dev workspace. Carries the read-defer + /// context (ancestor namespaces + tables to expose as views over the direct parent). The + /// fork *write* redirect is folded into `storage.path`/`extra_args` above, so an agent + /// worker that predates this field still writes the fork namespace (it only misses the + /// defer views). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub fork_defer: Option, +} + +/// Read-defer context for a fork workspace's ducklake: which ancestor namespaces to attach +/// read-only, and which tables to expose as views over the direct parent because the fork has +/// not materialized them yet. `ancestors` is empty when defer is unavailable (an ancestor no +/// longer defines the lake) — the fork namespace still isolates writes in that case. +#[derive(Deserialize, Serialize)] +pub struct DucklakeForkDefer { + /// Nearest-first (direct parent … root), each resolved from that workspace's own settings. + pub ancestors: Vec, + pub defer_tables: Vec, + /// Views currently live in the fork namespace (read from the catalog's `ducklake_view` at + /// resolution time, lake-internal names). The worker's view→table transition (DROP VIEW + /// before a managed materialize) keys on THIS, not on recorded materialization status: + /// after a failed run the status can't distinguish a defer view from a real table, and + /// `DROP VIEW` against a table (or `CREATE TABLE` against a view) errors — either guess + /// would wedge the asset until manual repair. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub fork_views: Vec, +} + +/// Connection data for one ancestor namespace of a fork's ducklake. +#[derive(Deserialize, Serialize)] +pub struct DucklakeAncestorAttach { + pub workspace_id: String, + /// DuckDB catalog alias this namespace must be attached under. Persisted defer-view SQL + /// references it, so it is a pure function of (lake name, ancestor workspace id) and every + /// session reading those views attaches the ancestor under this exact alias. + pub alias: String, + pub catalog: DucklakeCatalog, + pub catalog_resource: serde_json::Value, + pub storage: DucklakeStorage, + /// None = the lake's default metadata schema (the ancestor is a root/non-fork workspace). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub metadata_schema: Option, + /// The ancestor config's own non-reserved ATTACH args (e.g. `ENCRYPTED true`), already + /// stripped of the fork-owned `METADATA_SCHEMA`/`DATA_PATH`/`OVERRIDE_DATA_PATH` — an + /// option-dependent lake would otherwise fail its read-only ancestor attach even though + /// the same lake attaches fine everywhere else. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub extra_args: Option, +} + +/// Prefix of fork-scoped ducklake metadata schemas. Cleanup refuses to drop any pg schema not +/// carrying it, mirroring the `wm_fork_` guard on forked datatable databases. +pub const FORK_DUCKLAKE_SCHEMA_PREFIX: &str = "wm_fork_"; + +/// Bucket-root directory holding all fork namespaces' data files: each fork writes under +/// `{FORK_DUCKLAKE_DATA_DIR}//` where the segment is +/// [`fork_data_dir_segment`] (see [`fork_data_path`] for why it wraps the lake's path instead +/// of nesting under it). +pub const FORK_DUCKLAKE_DATA_DIR: &str = "__wm_forks"; + +fn mangle_identifier(s: &str, max: usize) -> String { + s.chars() + .take(max) + .map(|c| { + let c = c.to_ascii_lowercase(); + if c.is_ascii_alphanumeric() { + c + } else { + '_' + } + }) + .collect() +} + +/// Deterministic, injective pg-schema name for a fork workspace's namespace of ONE lake: +/// `wm_fork___<8-hex sha256>`, ≤58 chars (pg limit 63). +/// Lake-scoped, not just workspace-scoped: two lakes of one workspace may share a catalog +/// database, and a per-workspace schema would merge their namespaces in the fork (tables and +/// snapshots colliding across `ducklake://a/…` and `ducklake://b/…`). The hash keeps distinct +/// (workspace, lake) pairs distinct after mangling. The registry row records the computed name +/// for cleanup, but ATTACH recomputes it — so this function must stay stable across releases +/// or existing forks would silently lose their namespace. +pub fn fork_ducklake_metadata_schema(w_id: &str, lake_name: &str) -> String { + use sha2::{Digest, Sha256}; + let hash = hex::encode(&Sha256::digest(format!("{w_id}\0{lake_name}").as_bytes())[..4]); + format!( + "{FORK_DUCKLAKE_SCHEMA_PREFIX}{}_{}_{hash}", + mangle_identifier(w_id, 24), + mangle_identifier(lake_name, 16) + ) +} + +/// The `METADATA_SCHEMA ''` value carried in a lake config's `extra_args`, if any — the +/// schema the lake's OWN catalog namespace lives in (how one catalog database hosts several +/// lakes). Ancestor read-only attaches must preserve it or they'd bind the wrong namespace. +pub fn extract_metadata_schema_arg(extra_args: &str) -> Option { + lazy_static::lazy_static! { + static ref MS: regex::Regex = regex::Regex::new( + r"(?i)\bMETADATA_SCHEMA\s*(?:'([^']*)'|([A-Za-z0-9_]+))" + ) + .unwrap(); + } + // Last occurrence wins, matching DuckDB's duplicate-option semantics. + MS.captures_iter(extra_args) + .last() + .and_then(|c| c.get(1).or_else(|| c.get(2))) + .map(|m| m.as_str().to_string()) +} + +/// Deterministic DuckDB attach alias for an ancestor namespace of a fork's lake. Persisted +/// defer-view SQL references it (same stability requirement as +/// [`fork_ducklake_metadata_schema`]). +pub fn fork_ducklake_ancestor_alias(lake_name: &str, ancestor_w_id: &str) -> String { + use sha2::{Digest, Sha256}; + let hash = + hex::encode(&Sha256::digest(format!("{lake_name}\0{ancestor_w_id}").as_bytes())[..4]); + format!( + "__wm_dl_{}_{}_{hash}", + mangle_identifier(lake_name, 20), + mangle_identifier(ancestor_w_id, 20) + ) +} + +/// Strip `METADATA_SCHEMA` / `DATA_PATH` / `OVERRIDE_DATA_PATH` tokens from ducklake ATTACH +/// extra args. In a fork these options are injected by the fork resolution; a user- or +/// settings-supplied duplicate silently wins (DuckDB keeps the last occurrence) and would +/// escape the fork namespace back to the parent's, so they are removed rather than overridden. +pub fn strip_fork_reserved_attach_args(extra_args: &str) -> String { + lazy_static::lazy_static! { + static ref RESERVED: regex::Regex = regex::Regex::new( + r"(?i)\b(METADATA_SCHEMA|DATA_PATH|OVERRIDE_DATA_PATH)\s*('[^']*'|[A-Za-z0-9_]+)" + ) + .unwrap(); + } + RESERVED + .replace_all(extra_args, "") + .split(',') + .map(str::trim) + .filter(|s| !s.is_empty()) + .collect::>() + .join(", ") +} + +lazy_static::lazy_static! { + /// fork workspace id -> (ancestor chain nearest-first, expiry ts). Empty chain = not a fork. + /// `parent_workspace_id` only changes on dev-workspace attach/detach, so a short TTL is safe. + static ref FORK_ANCESTOR_CHAIN_CACHE: Cache, i64)> = Cache::new(5000); + /// fork workspace id -> (locations recently upserted into `fork_ducklake_namespace`, + /// expiry ts), so the registry write doesn't run on every job of a fork. Keyed by + /// workspace id so cleanup can drop a fork's whole entry: a same-id fork recreated within + /// the TTL must re-register, or its materializations would carry no registry row and leak + /// at ITS deletion. TTL'd as a second line of defense for cleanup paths that bypass + /// `cleanup_fork_ducklake_namespaces` (e.g. manual registry edits). + static ref FORK_DUCKLAKE_REGISTERED: Cache, i64)> = + Cache::new(5000); +} + +/// Drop the "already registered" once-cache for a workspace. MUST be called whenever +/// `fork_ducklake_namespace` rows for that workspace are deleted (namespace cleanup, workspace +/// deletion): a surviving entry would make a same-id fork recreated within the TTL skip +/// re-registration, orphaning its namespace at deletion time. +pub fn invalidate_fork_ducklake_registration_cache(w_id: &str) { + FORK_DUCKLAKE_REGISTERED.remove(w_id); +} + +/// Drop the cached ancestor chain for a workspace. MUST be called wherever +/// `parent_workspace_id` lineage changes (fork creation, dev-workspace attach, reparenting +/// rename, deletion): a cached EMPTY chain reads as "not a fork" and would bypass ducklake +/// fork isolation for the TTL — the first jobs after a dev-workspace attach would write the +/// shared lake. +pub fn invalidate_fork_ancestor_chain_cache(w_id: &str) { + FORK_ANCESTOR_CHAIN_CACHE.remove(w_id); +} + +/// Ancestors of `w_id`, nearest-first (direct parent … root). Empty for a non-fork workspace or +/// an unknown id. The depth bound is a cycle-safety backstop, same convention as +/// [`fork_chain_depth`]. +/// +/// Unauthenticated helper: reads workspace hierarchy for any `w_id`, so callers must already be +/// authorized for that workspace (or run in trusted server-side code). +pub async fn fork_ancestor_chain(db: &crate::DB, w_id: &str) -> Result> { + let now = chrono::Utc::now().timestamp(); + if let Some((chain, expiry)) = FORK_ANCESTOR_CHAIN_CACHE.get(w_id) { + if expiry > now { + return Ok(chain); + } + } + let chain = sqlx::query_scalar!( + r#" + WITH RECURSIVE chain AS ( + SELECT id, parent_workspace_id, 0 AS depth + FROM workspace WHERE id = $1 + UNION ALL + SELECT w.id, w.parent_workspace_id, chain.depth + 1 + FROM workspace w + JOIN chain ON w.id = chain.parent_workspace_id + WHERE chain.depth < 20 + ) + SELECT id AS "id!" FROM chain WHERE depth > 0 ORDER BY depth + "#, + w_id + ) + .fetch_all(db) + .await + .map_err(|e| Error::internal_err(format!("resolving fork ancestors of {w_id}: {e:#}")))?; + FORK_ANCESTOR_CHAIN_CACHE.insert(w_id.to_string(), (chain.clone(), now + 60)); + Ok(chain) } pub async fn get_ducklake_from_db_unchecked( @@ -639,6 +1216,29 @@ pub async fn get_ducklake_from_db_unchecked( w_id: &str, db: &DB, ) -> Result { + let (base, fork_behavior) = ducklake_conn_data(name, w_id, db).await?; + let chain = fork_ancestor_chain(db, w_id).await?; + // `Shared` is the explicit fork-creation opt-out of isolation: the fork reads and writes + // the parent's lake through its own (cloned) config, exactly like a non-fork workspace. + // An empty chain alone does NOT mean "not a fork": `parent_workspace_id` is ON DELETE SET + // NULL, so a `wm-fork-*` workspace can outlive its parent — its cloned config still points + // at the shared lake, so the prefix keeps it isolated (mirrors `workspace_is_fork`). No + // ancestors ⇒ no defer, write redirect only. Prefix-less dev workspaces can't be orphaned + // (deletion of their prod is blocked while attached). + if (chain.is_empty() && !w_id.starts_with(WM_FORK_PREFIX)) + || fork_behavior == Some(DucklakeForkBehavior::Shared) + { + return Ok(base); + } + fork_scoped_ducklake(name, w_id, base, chain, db).await +} + +/// Resolve one workspace's own config for lake `name` — no fork awareness. +async fn ducklake_conn_data( + name: &str, + w_id: &str, + db: &DB, +) -> Result<(DucklakeWithConnData, Option)> { let ducklake = sqlx::query_scalar!( r#" SELECT ws.ducklake->'ducklakes'->$2 AS config @@ -671,13 +1271,447 @@ pub async fn get_ducklake_from_db_unchecked( ) .await? }; + let fork_behavior = ducklake.fork_behavior; let ducklake = DucklakeWithConnData { catalog_resource, catalog: ducklake.catalog, storage: ducklake.storage, extra_args: ducklake.extra_args, + fork_defer: None, }; - Ok(ducklake) + Ok((ducklake, fork_behavior)) +} + +/// The fork's directory segment under `__wm_forks/`: mangled + hashed into ONE path component +/// (same recipe and stability requirement as [`fork_ducklake_metadata_schema`]). Fork/dev +/// workspace ids are only git-branch-safe and may contain `/` (e.g. `wm-fork-a/b`) — used +/// raw, such an id would nest inside the sibling `wm-fork-a`'s prefix and be swept by ITS +/// cleanup. The hash keeps distinct ids distinct after mangling. +pub fn fork_data_dir_segment(fork_w_id: &str) -> String { + use sha2::{Digest, Sha256}; + let hash = hex::encode(&Sha256::digest(fork_w_id.as_bytes())[..4]); + format!("{}_{hash}", mangle_identifier(fork_w_id, 40)) +} + +/// The fork namespace's data path within the same bucket: a bucket-root +/// `__wm_forks//` prefix wrapping the lake's own path, NOT a sub-path under it — +/// the parent lake's maintenance (snapshot expiry / `ducklake_delete_orphaned_files`) scans +/// everything under the parent's DATA_PATH and would treat live fork files nested there as +/// orphans and delete them. +pub fn fork_data_path(base_path: &str, fork_w_id: &str) -> String { + let segment = fork_data_dir_segment(fork_w_id); + let base = base_path.trim_matches('/'); + if base.is_empty() { + format!("{FORK_DUCKLAKE_DATA_DIR}/{segment}") + } else { + format!("{FORK_DUCKLAKE_DATA_DIR}/{segment}/{base}") + } +} + +/// Redirect a fork workspace's lake to its fork-scoped namespace (same catalog DB, fork +/// metadata schema + data sub-path) and assemble the read-defer context over its ancestors. +async fn fork_scoped_ducklake( + name: &str, + w_id: &str, + mut base: DucklakeWithConnData, + chain: Vec, + db: &DB, +) -> Result { + if base.catalog.resource_type == DucklakeCatalogResourceType::Mysql { + return Err(Error::BadRequest(format!( + "ducklake {name}: mysql-catalog lakes are not supported in fork workspaces — \ + running against the shared catalog would write the parent workspace's data" + ))); + } + let metadata_schema = fork_ducklake_metadata_schema(w_id, name); + let fork_path = fork_data_path(&base.storage.path, w_id); + let catalog_identity = ducklake_catalog_identity(&base.catalog); + register_fork_ducklake_namespace( + db, + w_id, + name, + &metadata_schema, + &catalog_identity, + base.storage.storage.as_deref(), + &fork_path, + ) + .await?; + + // Ancestor namespaces, nearest-first, each from its own settings so a fork-side settings + // edit can't silently repoint what "parent" means. All-or-nothing: a broken link anywhere + // in the chain disables defer entirely (a defer view over a missing ancestor attach fails + // at bind time and would kill unrelated jobs), but write isolation still applies. + let mut ancestors = Vec::with_capacity(chain.len()); + for (i, ancestor_id) in chain.iter().enumerate() { + match ducklake_conn_data(name, ancestor_id, db).await { + Ok((mut a, ancestor_behavior)) => { + // A fork ancestor lives in its own namespace UNLESS its lake is `Shared` — + // then it never redirected and its data sits at its config's default + // location, exactly like a root workspace. Chain position alone can't tell + // the two apart: an orphaned `wm-fork-*` ancestor (its own parent deleted, + // `parent_workspace_id` SET NULL) ends the chain like a root but its data + // lives in ITS fork namespace — key on the prefix too, as in resolution. + let is_isolated_fork = (i + 1 < chain.len() + || ancestor_id.starts_with(WM_FORK_PREFIX)) + && ancestor_behavior != Some(DucklakeForkBehavior::Shared); + let metadata_schema = if is_isolated_fork { + a.storage.path = fork_data_path(&a.storage.path, ancestor_id); + Some(fork_ducklake_metadata_schema(ancestor_id, name)) + } else { + // Root / shared ancestors live in their config's OWN catalog namespace — + // which may be a non-default schema when one catalog database hosts + // several lakes (`extra_args METADATA_SCHEMA '…'`). Preserve it, or the + // read-only attach would bind the wrong (or a nonexistent) lake. + a.extra_args + .as_deref() + .and_then(extract_metadata_schema_arg) + }; + let extra_args = a + .extra_args + .as_deref() + .map(strip_fork_reserved_attach_args) + .filter(|s| !s.is_empty()); + ancestors.push(DucklakeAncestorAttach { + workspace_id: ancestor_id.clone(), + alias: fork_ducklake_ancestor_alias(name, ancestor_id), + catalog: a.catalog, + catalog_resource: a.catalog_resource, + storage: a.storage, + metadata_schema, + extra_args, + }); + } + Err(e) => { + tracing::warn!( + "fork {w_id}: ducklake {name} not resolvable in ancestor {ancestor_id} \ + ({e:#}); read-defer disabled, fork namespace still isolated" + ); + ancestors.clear(); + break; + } + } + } + // Drop fork ancestors whose namespace was never bootstrapped (e.g. a fresh intermediate + // fork that never attached this lake): their READ_ONLY attach would fail the whole job + // ("DuckLake does not exist" + creation disabled), and a nonexistent namespace can't own + // tables or be referenced by any persisted view. Checked against the fork's catalog DB — + // must happen BEFORE defer discovery so `ancestor_idx` values index the filtered list. + let (existing_schemas, fork_views, fork_tables) = + inspect_fork_catalog(&base, &metadata_schema, &ancestors, db).await?; + ancestors.retain(|a| { + a.metadata_schema + .as_ref() + .map_or(true, |s| existing_schemas.contains(s)) + }); + let defer_tables = if ancestors.is_empty() { + vec![] + } else { + // Discovery walks the WHOLE chain: a table only a grandparent materialized has no + // physical copy in the direct parent (it defers too), so its view must target the + // nearest ancestor that owns one. + let ancestor_ids: Vec = ancestors.iter().map(|a| a.workspace_id.clone()).collect(); + let mut tables = + crate::materialization::list_fork_defer_tables(db, &ancestor_ids, w_id, name).await?; + // Catalog truth beats recorded status: any table live in the fork namespace is + // fork-owned, whatever its materialized_partition row says (failed-after-commit, + // raw SQL) — a defer view over it would silently yield to it. + tables.retain(|t| !fork_tables.contains(&t.table)); + tables + }; + + base.storage.path = fork_path; + base.extra_args = Some(match base.extra_args.take() { + Some(e) => { + let sanitized = strip_fork_reserved_attach_args(&e); + if sanitized.is_empty() { + format!("METADATA_SCHEMA '{metadata_schema}'") + } else { + format!("{sanitized}, METADATA_SCHEMA '{metadata_schema}'") + } + } + None => format!("METADATA_SCHEMA '{metadata_schema}'"), + }); + base.fork_defer = Some(DucklakeForkDefer { ancestors, defer_tables, fork_views }); + Ok(base) +} + +/// One round trip to the fork's catalog DB for both namespace introspections: which fork +/// ancestors' metadata schemas actually exist (a fresh intermediate fork may never have +/// bootstrapped its namespace), and the views currently live in THIS fork's namespace +/// (straight from DuckLake's `ducklake_view` metadata — missing metadata tables read as "no +/// views", correct since nothing can exist there). Each ancestor's schema is checked in the +/// ancestor's OWN catalog database — a fork whose catalog drifted away from an ancestor's +/// must not misread that ancestor's (existing, elsewhere) namespace as missing. Ancestors +/// whose catalog is unreachable read as missing — the safe direction: their defer is skipped +/// (loud absent-table reads) rather than emitting a READ_ONLY attach that would fail every +/// job in this fork. +async fn inspect_fork_catalog( + base: &DucklakeWithConnData, + metadata_schema: &str, + ancestors: &[DucklakeAncestorAttach], + db: &DB, +) -> Result<( + std::collections::HashSet, + Vec, + std::collections::HashSet, +)> { + // Ancestor fork-schemas grouped by which catalog database they live in. + let mut groups: std::collections::HashMap)> = + std::collections::HashMap::new(); + for a in ancestors { + if let Some(ms) = &a.metadata_schema { + groups + .entry(ducklake_catalog_identity(&a.catalog)) + .or_insert_with(|| (a.catalog_resource.clone(), vec![])) + .1 + .push(ms.clone()); + } + } + + async fn query_schemas( + client: &tokio_postgres::Client, + schemas: Vec, + ) -> std::result::Result, tokio_postgres::Error> { + client + .query( + "SELECT schema_name::text FROM information_schema.schemata + WHERE schema_name = ANY($1)", + &[&schemas], + ) + .await + .map(|rows| rows.iter().map(|r| r.get::<_, String>(0)).collect()) + } + + let mut existing_schemas: std::collections::HashSet = Default::default(); + + // The fork's own catalog: same-catalog ancestor schemas + this fork's live views. + let pg: crate::PgDatabase = + serde_json::from_value(base.catalog_resource.clone()).map_err(|e| { + Error::internal_err(format!("ducklake catalog resource is not postgres: {e}")) + })?; + let (client, connection) = pg.connect(Some(db)).await?; + let join_handle = tokio::spawn(async move { connection.await }); + let same_catalog = groups.remove(&ducklake_catalog_identity(&base.catalog)); + let same_catalog_res = match same_catalog { + Some((_, schemas)) => query_schemas(&client, schemas).await.map(Some), + None => Ok(None), + }; + // Identifier-quoted schema: the name is server-derived (mangled + hashed) but quote anyway. + let q = format!( + r#"SELECT CASE WHEN s.schema_name = 'main' THEN v.view_name + ELSE s.schema_name || '.' || v.view_name END AS name + FROM "{ms}".ducklake_view v + JOIN "{ms}".ducklake_schema s ON s.schema_id = v.schema_id AND s.end_snapshot IS NULL + WHERE v.end_snapshot IS NULL"#, + ms = metadata_schema.replace('"', "\"\"") + ); + let view_rows = client.query(&q, &[]).await; + // Live TABLES in the fork's namespace, same name shape as the views — the defer list is + // filtered against them: `CREATE VIEW IF NOT EXISTS` silently yields to an existing + // table, so emitting a defer view over one would leave reads on the fork table while + // claiming they defer (recorded status can't tell — a committed write whose data tests + // failed, or a table left by raw SQL, has no `materialized` row). + let qt = format!( + r#"SELECT CASE WHEN s.schema_name = 'main' THEN t.table_name + ELSE s.schema_name || '.' || t.table_name END AS name + FROM "{ms}".ducklake_table t + JOIN "{ms}".ducklake_schema s ON s.schema_id = t.schema_id AND s.end_snapshot IS NULL + WHERE t.end_snapshot IS NULL"#, + ms = metadata_schema.replace('"', "\"\"") + ); + let table_rows = client.query(&qt, &[]).await; + drop(client); + let _ = join_handle.await; + + existing_schemas.extend( + same_catalog_res + .map_err(|e| { + Error::internal_err(format!("checking fork ducklake ancestor schemas: {e}")) + })? + .unwrap_or_default(), + ); + let fork_views = match view_rows { + Ok(rows) => rows.iter().map(|r| r.get::<_, String>(0)).collect(), + // 42P01 undefined_table / 3F000 invalid_schema_name: namespace not bootstrapped yet. + Err(e) + if e.code().map_or(false, |c| { + c == &tokio_postgres::error::SqlState::UNDEFINED_TABLE + || c == &tokio_postgres::error::SqlState::INVALID_SCHEMA_NAME + }) => + { + vec![] + } + Err(e) => { + return Err(Error::internal_err(format!( + "listing fork ducklake views in {metadata_schema}: {e}" + ))) + } + }; + let fork_tables = match table_rows { + Ok(rows) => rows.iter().map(|r| r.get::<_, String>(0)).collect(), + Err(e) + if e.code().map_or(false, |c| { + c == &tokio_postgres::error::SqlState::UNDEFINED_TABLE + || c == &tokio_postgres::error::SqlState::INVALID_SCHEMA_NAME + }) => + { + Default::default() + } + Err(e) => { + return Err(Error::internal_err(format!( + "listing fork ducklake tables in {metadata_schema}: {e}" + ))) + } + }; + + // Ancestors living in OTHER catalog databases (this fork's catalog drifted after forking): + // one connection per distinct catalog, best-effort — an unreachable ancestor catalog only + // disables that ancestor's defer. + for (identity, (resource, schemas)) in groups { + let checked = async { + let pg: crate::PgDatabase = serde_json::from_value(resource) + .map_err(|e| Error::internal_err(format!("not a postgres resource: {e}")))?; + let (client, connection) = pg.connect(Some(db)).await?; + let join_handle = tokio::spawn(async move { connection.await }); + let res = query_schemas(&client, schemas).await; + drop(client); + let _ = join_handle.await; + res.map_err(|e| Error::internal_err(format!("{e}"))) + } + .await; + match checked { + Ok(found) => existing_schemas.extend(found), + Err(e) => { + tracing::warn!( + "fork ancestor catalog `{identity}` unreachable while checking ducklake \ + namespaces ({e:#}); its ancestors' defer is disabled for this resolution" + ); + } + } + } + Ok((existing_schemas, fork_views, fork_tables)) +} + +/// Canonical identity of a lake's catalog database (`:`) — used +/// for the cleanup registry and for grouping ancestors by which catalog their namespace lives +/// in. Identifies the database *pointer*, not its (live-resolved) credentials. +pub fn ducklake_catalog_identity(catalog: &DucklakeCatalog) -> String { + format!( + "{}:{}", + catalog.resource_type.as_ref(), + catalog.resource_path + ) +} + +/// Record that a fork attached this lake at this physical location, so fork deletion knows +/// exactly which pg metadata schema and which storage prefixes to clean up. One row per +/// (lake, storage, data path) EVER attached — if the fork's lake settings drift, later +/// attaches add rows rather than replace them, so cleanup covers every prefix the fork wrote. +/// The once-cache is keyed on the full location for the same reason. Re-registering an +/// existing row resets its `schema_dropped` cleanup phase: attaching recreates the metadata +/// schema, so a stale "already dropped" marker would make the eventual cleanup skip a live +/// schema. +async fn register_fork_ducklake_namespace( + db: &DB, + w_id: &str, + name: &str, + metadata_schema: &str, + catalog: &str, + storage: Option<&str>, + data_path: &str, +) -> Result<()> { + // '' = default storage (the column is part of the PK, which cannot hold NULL). + let storage = storage.unwrap_or(""); + // Resolve the logical storage name to its identity NOW: cleanup must delete from the + // storage that was active when the data was written, not whatever the name points at by + // deletion time. '' = unresolvable (no LFS configured — the write itself will fail at the + // proxy, so nothing lands anywhere). + let storage_ref = fork_storage_ref(db, w_id, storage) + .await + .unwrap_or_default(); + let location = format!("{name}\0{catalog}\0{storage}\0{storage_ref}\0{data_path}"); + let now = chrono::Utc::now().timestamp(); + if FORK_DUCKLAKE_REGISTERED + .get(w_id) + .is_some_and(|(locations, exp)| exp > now && locations.contains(&location)) + { + return Ok(()); + } + sqlx::query!( + "INSERT INTO fork_ducklake_namespace + (workspace_id, ducklake_name, metadata_schema, catalog, storage, storage_ref, data_path) + VALUES ($1, $2, $3, $4, $5, $6, $7) + ON CONFLICT (workspace_id, ducklake_name, catalog, storage, storage_ref, data_path) + DO UPDATE SET schema_dropped = false", + w_id, + name, + metadata_schema, + catalog, + storage, + &storage_ref, + data_path, + ) + .execute(db) + .await + .map_err(|e| Error::internal_err(format!("registering fork ducklake namespace: {e:#}")))?; + let mut locations = FORK_DUCKLAKE_REGISTERED + .get(w_id) + .filter(|(_, exp)| *exp > now) + .map(|(locations, _)| locations) + .unwrap_or_default(); + locations.insert(location); + FORK_DUCKLAKE_REGISTERED.insert(w_id.to_string(), (locations, now + 60)); + Ok(()) +} + +/// Canonical identity of a workspace storage (`:`) as +/// configured RIGHT NOW — recorded in the fork namespace registry so cleanup targets the +/// storage the data was actually written to. `storage` = '' for the primary storage, else a +/// `secondary_storage` name. Returns None when no matching storage is configured. +async fn fork_storage_ref(db: &DB, w_id: &str, storage: &str) -> Option { + let lfs_json = sqlx::query_scalar!( + "SELECT large_file_storage FROM workspace_settings WHERE workspace_id = $1", + w_id + ) + .fetch_optional(db) + .await + .ok() + .flatten() + .flatten()?; + let entry = if storage.is_empty() || storage == "_default_" { + lfs_json.clone() + } else { + lfs_json.get("secondary_storage")?.get(storage)?.clone() + }; + lfs_entry_storage_ref(&entry) +} + +/// The pure part of [`fork_storage_ref`]: one `LargeFileStorage` JSON entry → its canonical +/// `:` descriptor. The `$res:` prefix on resource paths is normalized away (stored +/// configs are inconsistent about it); cleanup re-adds it when resolving. +pub fn lfs_entry_storage_ref(entry: &serde_json::Value) -> Option { + let typ = entry.get("type")?.as_str()?; + let path_field = match typ { + "S3Storage" | "S3AwsOidc" => "s3_resource_path", + "AzureBlobStorage" | "AzureWorkloadIdentity" => "azure_blob_resource_path", + "GoogleCloudStorage" => "gcs_resource_path", + "FilesystemStorage" => "root_path", + _ => return None, + }; + let path = entry.get(path_field)?.as_str()?; + let path = path.strip_prefix("$res:").unwrap_or(path); + Some(format!("{typ}:{path}")) +} + +/// Resolve a `$res:`/`$var:` reference tree to its concrete value (recursively, secrets +/// decrypted). No permission checks — trusted server-side callers only; never echo the result +/// to a user. +pub async fn transform_json_value_unchecked( + value: &serde_json::Value, + w_id: &str, + db: &DB, +) -> Result { + transform_json_unchecked(value, w_id, db).await } // This does not check for any permission. Should never be displayed to a user. @@ -725,10 +1759,14 @@ async fn transform_json_unchecked( .await .map_err(to_anyhow)?; let value = if is_secret { - let mc = build_crypt(&db, &w_id).await?; - decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", &s, e)) - })? + if is_external_stored_value(&value) { + get_secret_value(db, w_id, &s[5..], &value).await? + } else { + let mc = build_crypt(&db, &w_id).await?; + decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", &s, e)) + })? + } } else { value }; @@ -790,9 +1828,211 @@ mod tests { assert!(validate_fork_workspace_id("wm-fork-foo/bar.lock").is_err()); } + #[test] + fn test_validate_dev_workspace_id_accepts_prefixless_valid() { + // Dev workspaces use ordinary, prefix-less ids but must stay git-branch-safe. + validate_dev_workspace_id("dev").unwrap(); + validate_dev_workspace_id("my-dev-workspace").unwrap(); + validate_dev_workspace_id("staging.42").unwrap(); + // The fork prefix is allowed but not required. + validate_dev_workspace_id("wm-fork-dev").unwrap(); + } + + #[test] + fn test_validate_dev_workspace_id_rejects_empty_and_git_unsafe() { + assert!(validate_dev_workspace_id("").is_err()); + assert!(validate_dev_workspace_id("dev workspace").is_err()); + assert!(validate_dev_workspace_id("dev..staging").is_err()); + assert!(validate_dev_workspace_id("dev/.x").is_err()); + assert!(validate_dev_workspace_id("dev.lock").is_err()); + } + + #[test] + fn test_validate_fork_workspace_id_rejects_empty() { + assert!(validate_fork_workspace_id("").is_err()); + } + #[test] fn test_validate_fork_workspace_id_rejects_too_long() { let long_id = format!("wm-fork-{}", "a".repeat(43)); assert!(validate_fork_workspace_id(&long_id).is_err()); } + + #[test] + fn test_fork_ducklake_metadata_schema_shape() { + let s = fork_ducklake_metadata_schema("wm-fork-my-feature-42", "main"); + assert!(s.starts_with(FORK_DUCKLAKE_SCHEMA_PREFIX), "{s}"); + assert!(s.len() <= 63, "pg schema name limit: {s}"); + assert!( + s.chars().all(|c| c.is_ascii_alphanumeric() || c == '_'), + "{s}" + ); + // Deterministic (persisted view SQL / registry rows depend on it). + assert_eq!( + s, + fork_ducklake_metadata_schema("wm-fork-my-feature-42", "main") + ); + } + + #[test] + fn test_fork_ducklake_metadata_schema_injective_after_mangling() { + // `-` and `_` mangle to the same char; the hash suffix must keep them distinct. + let a = fork_ducklake_metadata_schema("wm-fork-a-b", "main"); + let b = fork_ducklake_metadata_schema("wm-fork-a_b", "main"); + assert_ne!(a, b); + // Lake-scoped: two lakes of one workspace may share a catalog database, so their fork + // namespaces must be distinct schemas. + assert_ne!( + fork_ducklake_metadata_schema("wm-fork-a-b", "lake_a"), + fork_ducklake_metadata_schema("wm-fork-a-b", "lake_b") + ); + // Long ids truncate to the same mangled prefix; hash must still differ. + let long_a = fork_ducklake_metadata_schema(&format!("wm-fork-{}x", "a".repeat(40)), "main"); + let long_b = fork_ducklake_metadata_schema(&format!("wm-fork-{}y", "a".repeat(40)), "main"); + assert_ne!(long_a, long_b); + let long_lake = + fork_ducklake_metadata_schema(&format!("wm-fork-{}", "a".repeat(42)), &"l".repeat(40)); + assert!(long_a.len() <= 63 && long_b.len() <= 63 && long_lake.len() <= 63); + } + + #[test] + fn test_fork_ducklake_ancestor_alias_valid_identifier() { + let a = fork_ducklake_ancestor_alias("analytics", "wm-fork-dev-1"); + assert!(a.starts_with("__wm_dl_"), "{a}"); + assert!( + a.chars().all(|c| c.is_ascii_alphanumeric() || c == '_'), + "{a}" + ); + // Distinct per lake for the same ancestor (a script can attach several lakes). + assert_ne!(a, fork_ducklake_ancestor_alias("staging", "wm-fork-dev-1")); + assert_ne!( + a, + fork_ducklake_ancestor_alias("analytics", "wm-fork-dev-2") + ); + } + + #[test] + fn test_extract_metadata_schema_arg() { + assert_eq!( + extract_metadata_schema_arg("METADATA_SCHEMA 'lake_b_ns', ENCRYPTED true"), + Some("lake_b_ns".to_string()) + ); + assert_eq!( + extract_metadata_schema_arg("metadata_schema bare_ident"), + Some("bare_ident".to_string()) + ); + // Last occurrence wins (DuckDB duplicate-option semantics). + assert_eq!( + extract_metadata_schema_arg("METADATA_SCHEMA 'a', METADATA_SCHEMA 'b'"), + Some("b".to_string()) + ); + assert_eq!(extract_metadata_schema_arg("ENCRYPTED true"), None); + } + + #[test] + fn test_fork_data_path_prefix_isolation() { + // Fork/dev ids are git-branch-safe and may contain `/`: `wm-fork-a/b` is a valid id. + // Its data prefix must NOT nest inside `wm-fork-a`'s, or deleting `wm-fork-a` would + // sweep the sibling's files via the object-store prefix listing. + assert!(validate_fork_workspace_id("wm-fork-a/b").is_ok()); + let a = fork_data_path("lake", "wm-fork-a"); + let ab = fork_data_path("lake", "wm-fork-a/b"); + assert!( + !format!("{ab}/").starts_with(&format!( + "{FORK_DUCKLAKE_DATA_DIR}/{}/", + fork_data_dir_segment("wm-fork-a") + )), + "{ab} nests under {a}'s cleanup prefix" + ); + // Single path component: the segment itself contains no separator. + assert!(!fork_data_dir_segment("wm-fork-a/b").contains('/')); + // Injective after mangling (`/` and `_` both mangle to `_`). + assert_ne!( + fork_data_dir_segment("wm-fork-a/b"), + fork_data_dir_segment("wm-fork-a_b") + ); + // Deterministic (registry rows + cleanup guard recompute it). + assert_eq!(a, fork_data_path("lake", "wm-fork-a")); + // Empty base path still yields a well-formed prefix. + assert_eq!( + fork_data_path("", "wm-fork-a"), + format!( + "{FORK_DUCKLAKE_DATA_DIR}/{}", + fork_data_dir_segment("wm-fork-a") + ) + ); + } + + #[test] + fn test_lfs_entry_storage_ref() { + assert_eq!( + lfs_entry_storage_ref(&serde_json::json!({ + "type": "S3Storage", "s3_resource_path": "$res:u/admin/minio" + })), + Some("S3Storage:u/admin/minio".to_string()) + ); + // The `$res:` prefix is optional in stored configs; normalized either way. + assert_eq!( + lfs_entry_storage_ref(&serde_json::json!({ + "type": "AzureBlobStorage", "azure_blob_resource_path": "u/admin/az" + })), + Some("AzureBlobStorage:u/admin/az".to_string()) + ); + assert_eq!( + lfs_entry_storage_ref(&serde_json::json!({ + "type": "FilesystemStorage", "root_path": "/data/lfs" + })), + Some("FilesystemStorage:/data/lfs".to_string()) + ); + assert_eq!( + lfs_entry_storage_ref(&serde_json::json!({"type": "SomethingNew"})), + None + ); + assert_eq!(lfs_entry_storage_ref(&serde_json::json!({})), None); + } + + #[test] + fn test_strip_fork_reserved_attach_args() { + // Reserved options removed wherever they appear, others preserved. + assert_eq!( + strip_fork_reserved_attach_args("METADATA_SCHEMA 'main', ENCRYPTED true"), + "ENCRYPTED true" + ); + assert_eq!( + strip_fork_reserved_attach_args( + "ENCRYPTED true, DATA_PATH 's3://b/prod', OVERRIDE_DATA_PATH FALSE" + ), + "ENCRYPTED true" + ); + // Case-insensitive, and quoted values may contain commas/spaces. + assert_eq!( + strip_fork_reserved_attach_args("data_path 's3://b/x, y', SNAPSHOT_VERSION 3"), + "SNAPSHOT_VERSION 3" + ); + assert_eq!(strip_fork_reserved_attach_args(""), ""); + assert_eq!( + strip_fork_reserved_attach_args("METADATA_SCHEMA 'wm_fork_evil'"), + "" + ); + } + + #[test] + fn test_datatable_not_found_error_no_datatables() { + let msg = datatable_not_found_error("main", None).to_string(); + assert!(msg.contains("'main' not found"), "{msg}"); + assert!(msg.contains("No data table is configured"), "{msg}"); + // Always signposts the settings tab so the message is actionable. + assert!(msg.contains("tab=windmill_data_tables"), "{msg}"); + } + + #[test] + fn test_datatable_not_found_error_lists_available() { + let configured = serde_json::json!({ "analytics": {}, "staging": {} }); + let msg = datatable_not_found_error("main", Some(&configured)).to_string(); + assert!(msg.contains("'main' not found"), "{msg}"); + // Surface configured names to catch typos. + assert!(msg.contains("analytics"), "{msg}"); + assert!(msg.contains("staging"), "{msg}"); + assert!(msg.contains("tab=windmill_data_tables"), "{msg}"); + } } diff --git a/backend/windmill-common/tests/asset_schema_capture.rs b/backend/windmill-common/tests/asset_schema_capture.rs new file mode 100644 index 0000000000..f12ff3cc14 --- /dev/null +++ b/backend/windmill-common/tests/asset_schema_capture.rs @@ -0,0 +1,105 @@ +/*! + * Tests the schema-capture versioning contract (gap #2a): + * `record_asset_schema` inserts a new `materialized_asset_schema` version only + * when the captured column set changes, re-affirms the latest row in place when + * it doesn't, and `list_asset_schemas` returns the evolution history newest + * first. + */ + +use sqlx::{Pool, Postgres}; +use windmill_common::assets::AssetKind; +use windmill_common::materialization::{list_asset_schemas, record_asset_schema, SchemaColumn}; + +const WS: &str = "test-workspace"; +const PATH: &str = "analytics/orders"; + +fn col(name: &str, ty: &str) -> SchemaColumn { + SchemaColumn { name: name.to_string(), data_type: ty.to_string() } +} + +async fn record(db: &Pool, cols: &[SchemaColumn], snapshot_id: i64) -> bool { + let mut tx = db.begin().await.expect("begin"); + let inserted = record_asset_schema( + &mut tx, + WS, + AssetKind::Ducklake, + PATH, + cols, + Some(snapshot_id), + None, + ) + .await + .expect("record asset schema"); + tx.commit().await.expect("commit"); + inserted +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn first_capture_inserts_version_one(db: Pool) { + let cols = [col("order_id", "BIGINT"), col("status", "VARCHAR")]; + assert!( + record(&db, &cols, 10).await, + "first capture inserts a version" + ); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 1); + assert_eq!(versions[0].version, 1); + assert_eq!(versions[0].snapshot_id, Some(10)); + assert_eq!(versions[0].columns.0, cols); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn unchanged_schema_reaffirms_without_new_version(db: Pool) { + let cols = [col("order_id", "BIGINT")]; + record(&db, &cols, 10).await; + // Identical column set on a later snapshot: no new version, but the latest + // row's snapshot_id advances. + assert!( + !record(&db, &cols, 20).await, + "unchanged schema must not insert a new version" + ); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 1, "still a single version"); + assert_eq!(versions[0].version, 1); + assert_eq!(versions[0].snapshot_id, Some(20), "snapshot re-affirmed"); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn changed_schema_bumps_version_newest_first(db: Pool) { + record(&db, &[col("order_id", "BIGINT")], 10).await; + // A column added → schema changed → new version. + let evolved = [col("order_id", "BIGINT"), col("amount", "DOUBLE")]; + assert!(record(&db, &evolved, 20).await, "changed schema inserts v2"); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 2); + // Newest first. + assert_eq!(versions[0].version, 2); + assert_eq!(versions[0].columns.0, evolved); + assert_eq!(versions[1].version, 1); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn column_order_change_is_a_new_version(db: Pool) { + let a = [col("a", "BIGINT"), col("b", "VARCHAR")]; + let reordered = [col("b", "VARCHAR"), col("a", "BIGINT")]; + record(&db, &a, 10).await; + // Same columns, different physical order: the captured list is ordered, so a + // reorder is a real schema change (downstream `SELECT *` consumers see it). + assert!( + record(&db, &reordered, 20).await, + "column reorder is a distinct schema version" + ); + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 2); +} diff --git a/backend/windmill-common/tests/billing_workspace.rs b/backend/windmill-common/tests/billing_workspace.rs new file mode 100644 index 0000000000..1b0186a5cf --- /dev/null +++ b/backend/windmill-common/tests/billing_workspace.rs @@ -0,0 +1,211 @@ +#![cfg(feature = "cloud")] +//! Tests for the fork/dev "billing workspace" resolution and the fork-cap seat/count helpers. + +use sqlx::{Pool, Postgres}; +use windmill_common::workspaces::{ + count_paid_seats, count_workspace_forks, fork_chain_depth, fork_subtree_height, + get_billing_workspace_id, invalidate_billing_workspace_cache, list_fork_descendants, +}; + +async fn insert_ws(db: &Pool, id: &str, parent: Option<&str>, deleted: bool) { + sqlx::query( + "INSERT INTO workspace (id, name, owner, parent_workspace_id, deleted) + VALUES ($1, $1, 'test-user', $2, $3)", + ) + .bind(id) + .bind(parent) + .bind(deleted) + .execute(db) + .await + .expect("insert workspace"); + // The resolver caches per id (60s TTL) in a process-global cache shared across tests, so drop + // any stale mapping for this id before the test reads it. + invalidate_billing_workspace_cache(id); +} + +async fn insert_member( + db: &Pool, + w_id: &str, + email: &str, + operator: bool, + disabled: bool, + is_service_account: bool, +) { + // `usr.username` has a `proper_username` check (no `@`), so derive one from the email prefix. + let username = email.split('@').next().unwrap(); + sqlx::query( + "INSERT INTO usr (workspace_id, email, username, is_admin, operator, disabled, is_service_account, role) + VALUES ($1, $2, $3, false, $4, $5, $6, 'Developer')", + ) + .bind(w_id) + .bind(email) + .bind(username) + .bind(operator) + .bind(disabled) + .bind(is_service_account) + .execute(db) + .await + .expect("insert usr"); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn billing_workspace_resolves_to_root(db: Pool) { + insert_ws(&db, "bwt-root", None, false).await; + insert_ws(&db, "bwt-fork", Some("bwt-root"), false).await; + insert_ws(&db, "bwt-grandchild", Some("bwt-fork"), false).await; + + assert_eq!( + get_billing_workspace_id(&db, "bwt-root").await.unwrap(), + "bwt-root" + ); + assert_eq!( + get_billing_workspace_id(&db, "bwt-fork").await.unwrap(), + "bwt-root" + ); + assert_eq!( + get_billing_workspace_id(&db, "bwt-grandchild") + .await + .unwrap(), + "bwt-root" + ); + // Unknown / orphaned ids resolve to themselves. + assert_eq!( + get_billing_workspace_id(&db, "bwt-missing").await.unwrap(), + "bwt-missing" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn billing_workspace_survives_cycles(db: Pool) { + insert_ws(&db, "bwc-a", None, false).await; + insert_ws(&db, "bwc-b", Some("bwc-a"), false).await; + // Introduce a cycle a -> b -> a; no row has a NULL parent, so resolution falls back to the input + // and the depth guard keeps it from looping forever. + sqlx::query("UPDATE workspace SET parent_workspace_id = 'bwc-b' WHERE id = 'bwc-a'") + .execute(&db) + .await + .unwrap(); + invalidate_billing_workspace_cache("bwc-a"); + invalidate_billing_workspace_cache("bwc-b"); + + assert_eq!( + get_billing_workspace_id(&db, "bwc-a").await.unwrap(), + "bwc-a" + ); + assert_eq!( + get_billing_workspace_id(&db, "bwc-b").await.unwrap(), + "bwc-b" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn paid_seats_and_fork_count(db: Pool) { + insert_ws(&db, "seat-root", None, false).await; + // 2 developers + 2 operators counted -> ceil(2 + 0.5*2) = 3. + insert_member(&db, "seat-root", "dev1@w.dev", false, false, false).await; + insert_member(&db, "seat-root", "dev2@w.dev", false, false, false).await; + insert_member(&db, "seat-root", "op1@w.dev", true, false, false).await; + insert_member(&db, "seat-root", "op2@w.dev", true, false, false).await; + // These must NOT count towards seats. + insert_member(&db, "seat-root", "disabled@w.dev", false, true, false).await; + insert_member(&db, "seat-root", "svc@w.dev", false, false, true).await; + + assert_eq!(count_paid_seats(&db, "seat-root").await.unwrap(), 3); + + insert_ws(&db, "seat-fork1", Some("seat-root"), false).await; + insert_ws(&db, "seat-fork2", Some("seat-root"), false).await; + // A deleted fork itself is not counted... + insert_ws(&db, "seat-fork-deleted", Some("seat-root"), true).await; + // ...but a live sub-fork under it still is (deleted filter is on the outer SELECT, not the walk). + insert_ws(&db, "seat-deleted-child", Some("seat-fork-deleted"), false).await; + // A grandchild fork still counts. + insert_ws(&db, "seat-fork1-child", Some("seat-fork1"), false).await; + + // Live: fork1, fork2, fork1-child, deleted-child -> 4 (seat-fork-deleted excluded). + assert_eq!(count_workspace_forks(&db, "seat-root").await.unwrap(), 4); + // A standalone workspace has no forks. + assert_eq!(count_workspace_forks(&db, "seat-fork2").await.unwrap(), 0); + + // list_fork_descendants returns every descendant id (deleted included, for cache invalidation): + // fork1, fork2, fork-deleted, deleted-child, fork1-child -> 5. + let mut descendants = list_fork_descendants(&db, "seat-root").await.unwrap(); + descendants.sort(); + assert_eq!( + descendants, + vec![ + "seat-deleted-child", + "seat-fork-deleted", + "seat-fork1", + "seat-fork1-child", + "seat-fork2", + ] + ); + assert!(list_fork_descendants(&db, "seat-fork2") + .await + .unwrap() + .is_empty()); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn billing_cache_invalidation_reflects_reparent(db: Pool) { + insert_ws(&db, "inv-root-a", None, false).await; + insert_ws(&db, "inv-root-b", None, false).await; + insert_ws(&db, "inv-fork", Some("inv-root-a"), false).await; + + // Resolve + cache: fork -> root-a. + assert_eq!( + get_billing_workspace_id(&db, "inv-fork").await.unwrap(), + "inv-root-a" + ); + + // Reparent in the DB, as delete+recreate-under-another-root (or attach) would. + sqlx::query("UPDATE workspace SET parent_workspace_id = 'inv-root-b' WHERE id = 'inv-fork'") + .execute(&db) + .await + .unwrap(); + + // The cached mapping survives until invalidated (this is the staleness the delete/attach/rename + // paths must clear). + assert_eq!( + get_billing_workspace_id(&db, "inv-fork").await.unwrap(), + "inv-root-a" + ); + + // After invalidation (what delete_workspace / attach_dev_workspace / change_workspace_id now call), + // it re-resolves to the new root. + invalidate_billing_workspace_cache("inv-fork"); + assert_eq!( + get_billing_workspace_id(&db, "inv-fork").await.unwrap(), + "inv-root-b" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn fork_depth_and_subtree_height(db: Pool) { + // Chain: root -> f1 -> f2 -> f3 + insert_ws(&db, "fd-root", None, false).await; + insert_ws(&db, "fd-f1", Some("fd-root"), false).await; + insert_ws(&db, "fd-f2", Some("fd-f1"), false).await; + insert_ws(&db, "fd-f3", Some("fd-f2"), false).await; + + // Depth walks up to the root: root = 0, each fork adds one. + assert_eq!(fork_chain_depth(&db, "fd-root").await.unwrap(), 0); + assert_eq!(fork_chain_depth(&db, "fd-f1").await.unwrap(), 1); + assert_eq!(fork_chain_depth(&db, "fd-f3").await.unwrap(), 3); + // An unknown id has no chain, so depth 0 (treated as a root by the guard). + assert_eq!(fork_chain_depth(&db, "fd-missing").await.unwrap(), 0); + + // Height walks down: the deepest live descendant below the node. + assert_eq!(fork_subtree_height(&db, "fd-root").await.unwrap(), 3); + assert_eq!(fork_subtree_height(&db, "fd-f2").await.unwrap(), 1); + assert_eq!(fork_subtree_height(&db, "fd-f3").await.unwrap(), 0); + + // A deleted leaf doesn't add to the height. + insert_ws(&db, "fd-f3-del", Some("fd-f3"), true).await; + assert_eq!(fork_subtree_height(&db, "fd-f3").await.unwrap(), 0); + + // ...but a LIVE descendant below a soft-deleted intermediate still counts at its true depth + // (the walk traverses through the deleted node; only the aggregation filters deleted). + insert_ws(&db, "fd-f3-live-gc", Some("fd-f3-del"), false).await; + assert_eq!(fork_subtree_height(&db, "fd-f3").await.unwrap(), 2); +} diff --git a/backend/windmill-duckdb-ffi-internal/.gitignore b/backend/windmill-duckdb-ffi-internal/.gitignore new file mode 100644 index 0000000000..ea8c4bf7f3 --- /dev/null +++ b/backend/windmill-duckdb-ffi-internal/.gitignore @@ -0,0 +1 @@ +/target diff --git a/backend/windmill-duckdb-ffi-internal/build_dev.sh b/backend/windmill-duckdb-ffi-internal/build_dev.sh index d4ae2b7583..f3ce520c68 100755 --- a/backend/windmill-duckdb-ffi-internal/build_dev.sh +++ b/backend/windmill-duckdb-ffi-internal/build_dev.sh @@ -1,3 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +# The `duckdb` "bundled" feature compiles the whole DuckDB C++ library from +# source (~minutes) and dominates this crate's build. The crate changes very +# rarely, so by default build into a per-user cache shared across worktrees and +# keyed by the dependency lock: a fresh worktree reuses the already-compiled +# DuckDB instead of rebuilding it, and distinct DuckDB versions don't collide. +# Override the cache root with WINDMILL_DUCKDB_FFI_TARGET. +# +# If you are actively editing this crate in a worktree, uncommitted changes to +# its source switch to an isolated per-worktree ./target so your incremental +# builds neither disturb nor are disturbed by the shared cache. + +src_dirty="$(git status --porcelain -- src Cargo.toml Cargo.lock build.rs 2>/dev/null || true)" + +if [ -n "$src_dirty" ]; then + export CARGO_TARGET_DIR="$PWD/target" + echo "duckdb-ffi: local crate changes detected -> isolated target $CARGO_TARGET_DIR" +else + cache_root="${WINDMILL_DUCKDB_FFI_TARGET:-${XDG_CACHE_HOME:-$HOME/.cache}/windmill/duckdb-ffi-target}" + if command -v sha256sum >/dev/null 2>&1; then hash_cmd=sha256sum; else hash_cmd="shasum -a 256"; fi + key="$(cat Cargo.lock build.rs | $hash_cmd | cut -c1-16)" + export CARGO_TARGET_DIR="$cache_root/$key" + echo "duckdb-ffi: shared cache $CARGO_TARGET_DIR" + # The cache key covers only the dependency inputs (Cargo.lock + build.rs), so + # the expensive bundled-DuckDB build is shared even when this crate's own + # source differs between checkouts. But the shared dir's uplifted cdylib is a + # single fixed-name artifact written by whichever worktree built last. Touch + # our sources so cargo re-links THIS checkout's cdylib and re-uplifts it + # before we copy — the copied artifact then always matches this worktree, + # never a sibling's. The bundled dependency is a separate crate and stays + # cached, so this only costs a ~1s relink. + find src -type f -exec touch {} + +fi + CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --release -p windmill_duckdb_ffi_internal mkdir -p ../target/debug/ -cp target/release/libwindmill_duckdb_ffi_internal.* ../target/debug/ +cp "$CARGO_TARGET_DIR/release/"libwindmill_duckdb_ffi_internal.* ../target/debug/ diff --git a/backend/windmill-duckdb-ffi-internal/src/lib.rs b/backend/windmill-duckdb-ffi-internal/src/lib.rs index f3ee59cab7..a7543620b6 100644 --- a/backend/windmill-duckdb-ffi-internal/src/lib.rs +++ b/backend/windmill-duckdb-ffi-internal/src/lib.rs @@ -148,6 +148,23 @@ fn is_setup_statement(query: &str) -> bool { || upper.starts_with("RESET") || upper.starts_with("CREATE OR REPLACE SECRET") || upper.starts_with("CREATE SECRET") + || is_create_temp_macro(&upper) +} + +/// `CREATE [OR REPLACE] TEMP|TEMPORARY MACRO` — a connection-scoped definition +/// that returns no result set and must be *executed* (not just prepared) during +/// the prepare/diagnostics pass, so later blocks that call the macro bind +/// against it instead of failing "function does not exist". Both the materialize +/// runtime (`wm_partition`) and the workspace-macro splicer inject these ahead +/// of the query that uses them, so — like ATTACH — they are connection setup, +/// not user queries (they must not add a `PrepareQueryResult` entry either). +/// Persistent `CREATE MACRO` (no TEMP) is a real user statement and is excluded. +fn is_create_temp_macro(upper: &str) -> bool { + let norm = upper.split_whitespace().collect::>().join(" "); + norm.starts_with("CREATE TEMP MACRO") + || norm.starts_with("CREATE TEMPORARY MACRO") + || norm.starts_with("CREATE OR REPLACE TEMP MACRO") + || norm.starts_with("CREATE OR REPLACE TEMPORARY MACRO") } /// Returns true if the query is expected to return a result set and can be wrapped with DESCRIBE. @@ -635,7 +652,9 @@ fn duckdb_value_to_json_value( .ok_or_else(|| "Could not convert to f64".to_string())?, ), duckdb::types::Value::Decimal(d) => serde_json::Value::String(d.to_string()), - duckdb::types::Value::Timestamp(_, ts) => serde_json::Value::String(ts.to_string()), + duckdb::types::Value::Timestamp(unit, ts) => { + serde_json::Value::String(duckdb_timestamp_to_iso(unit, ts)) + } duckdb::types::Value::Text(s) if type_alias.as_deref().unwrap_or_default() == "JSON" => { serde_json::from_str(&s) .map_err(|e| format!("Error parsing JSON text: {}", e.to_string()))? @@ -646,8 +665,15 @@ fn duckdb_value_to_json_value( .map(|byte| serde_json::Value::Number(byte.into())) .collect(), ), - duckdb::types::Value::Date32(d) => serde_json::Value::Number(d.into()), - duckdb::types::Value::Time64(_, t) => serde_json::Value::String(t.to_string()), + duckdb::types::Value::Date32(d) => { + match chrono::DateTime::from_timestamp(i64::from(d) * 86_400, 0) { + Some(dt) => serde_json::Value::String(dt.date_naive().to_string()), + None => serde_json::Value::Number(d.into()), + } + } + duckdb::types::Value::Time64(unit, t) => { + serde_json::Value::String(duckdb_time_to_iso(unit, t).unwrap_or_else(|| t.to_string())) + } duckdb::types::Value::Interval { months, days, nanos } => serde_json::json!({ "months": months, "days": days, @@ -688,6 +714,413 @@ fn duckdb_value_to_json_value( Ok(json_value) } +// DuckDB surfaces TIMESTAMP[_S/_MS/_NS] values as a raw count since the epoch; +// stringifying that count leaks values like "1782974022218435" into job +// results. Render ISO-8601 instead (the Postgres executor's +// "2024-01-15T10:30:00" shape); a count outside chrono's representable range +// falls back to the raw number. +fn duckdb_timestamp_to_iso(unit: duckdb::types::TimeUnit, ts: i64) -> String { + let dt = match unit { + duckdb::types::TimeUnit::Second => chrono::DateTime::from_timestamp(ts, 0), + duckdb::types::TimeUnit::Millisecond => chrono::DateTime::from_timestamp_millis(ts), + duckdb::types::TimeUnit::Microsecond => chrono::DateTime::from_timestamp_micros(ts), + duckdb::types::TimeUnit::Nanosecond => Some(chrono::DateTime::from_timestamp_nanos(ts)), + }; + dt.map(|dt| dt.naive_utc().format("%Y-%m-%dT%H:%M:%S%.f").to_string()) + .unwrap_or_else(|| ts.to_string()) +} + +// Same story for TIME: a raw count since midnight. None when out of range +// (caller falls back to the raw number). +fn duckdb_time_to_iso(unit: duckdb::types::TimeUnit, t: i64) -> Option { + let (secs, nanos) = match unit { + duckdb::types::TimeUnit::Second => (t, 0), + duckdb::types::TimeUnit::Millisecond => (t / 1_000, (t % 1_000) * 1_000_000), + duckdb::types::TimeUnit::Microsecond => (t / 1_000_000, (t % 1_000_000) * 1_000), + duckdb::types::TimeUnit::Nanosecond => (t / 1_000_000_000, t % 1_000_000_000), + }; + chrono::NaiveTime::from_num_seconds_from_midnight_opt( + u32::try_from(secs).ok()?, + u32::try_from(nanos).ok()?, + ) + .map(|t| t.format("%H:%M:%S%.f").to_string()) +} + +#[cfg(test)] +mod temporal_json_tests { + use super::*; + use duckdb::types::TimeUnit; + + #[test] + fn timestamp_micros_renders_iso() { + // 2026-07-01 23:13:42.218435 UTC + assert_eq!( + duckdb_timestamp_to_iso(TimeUnit::Microsecond, 1_782_947_622_218_435), + "2026-07-01T23:13:42.218435" + ); + } + + #[test] + fn timestamp_seconds_renders_iso_without_subseconds() { + assert_eq!( + duckdb_timestamp_to_iso(TimeUnit::Second, 1_735_689_600), + "2025-01-01T00:00:00" + ); + } + + #[test] + fn out_of_range_timestamp_falls_back_to_raw() { + assert_eq!( + duckdb_timestamp_to_iso(TimeUnit::Second, i64::MAX), + i64::MAX.to_string() + ); + } + + #[test] + fn time_micros_renders_iso() { + assert_eq!( + duckdb_time_to_iso(TimeUnit::Microsecond, 37_800_500_000).as_deref(), + Some("10:30:00.500") + ); + } + + #[test] + fn temporal_values_render_iso_through_real_query() { + let conn = duckdb::Connection::open_in_memory().unwrap(); + let mut stmt = conn + .prepare( + "SELECT TIMESTAMP '2026-07-01 23:13:42.218435' AS ts, + TIMESTAMPTZ '2026-07-01 23:13:42+00' AS tstz, + TIMESTAMP_NS '2026-07-01 23:13:42.218435678' AS ts_ns, + DATE '2026-07-01' AS d, + TIME '10:30:00' AS t", + ) + .unwrap(); + let mut rows = stmt.query([]).unwrap(); + let row = rows.next().unwrap().unwrap(); + let json_of = |i: usize| { + let v: duckdb::types::Value = row.get(i).unwrap(); + duckdb_value_to_json_value(v, &None).unwrap() + }; + assert_eq!(json_of(0), serde_json::json!("2026-07-01T23:13:42.218435")); + assert_eq!(json_of(1), serde_json::json!("2026-07-01T23:13:42")); + assert_eq!( + json_of(2), + serde_json::json!("2026-07-01T23:13:42.218435678") + ); + assert_eq!(json_of(3), serde_json::json!("2026-07-01")); + assert_eq!(json_of(4), serde_json::json!("10:30:00")); + } + + // The data-test sample probe shape emitted by + // `windmill-parser::sql_materialize::build_data_test_checks`: one scan + // yielding the violating-row count plus a bounded `to_json` sample of the + // rows as a VARCHAR. These tests gate that design against the *bundled* + // engine (json extension availability, row-as-struct alias reference, + // NULL degrade on zero rows / oversized samples, exotic column types). + fn sample_probe_sql(rows_query: &str, max_len: usize) -> String { + format!( + "SELECT v, CASE WHEN strlen(s_raw) <= {max_len} THEN s_raw END AS s \ + FROM (SELECT count(*) AS v, \ + to_json(list(_wm_v ORDER BY _wm_rn) FILTER (WHERE _wm_rn <= 20))::VARCHAR AS s_raw \ + FROM (SELECT _wm_v, row_number() OVER () AS _wm_rn FROM ({rows_query}) _wm_v))" + ) + } + + fn run_sample_probe( + conn: &duckdb::Connection, + rows_query: &str, + max_len: usize, + ) -> (i64, Option) { + let mut stmt = conn + .prepare(&sample_probe_sql(rows_query, max_len)) + .unwrap(); + let mut rows = stmt.query([]).unwrap(); + let row = rows.next().unwrap().unwrap(); + (row.get(0).unwrap(), row.get(1).unwrap()) + } + + #[test] + fn data_test_sample_probe_counts_and_samples() { + let conn = duckdb::Connection::open_in_memory().unwrap(); + conn.execute_batch( + "CREATE TABLE t (id INT, name VARCHAR); \ + INSERT INTO t VALUES (1, 'a'), (2, NULL), (3, NULL);", + ) + .unwrap(); + let (v, s) = run_sample_probe(&conn, "SELECT * FROM t WHERE name IS NULL", 51200); + assert_eq!(v, 2); + let parsed: serde_json::Value = serde_json::from_str(&s.unwrap()).unwrap(); + assert_eq!( + parsed, + serde_json::json!([{"id": 2, "name": null}, {"id": 3, "name": null}]) + ); + } + + #[test] + fn data_test_sample_probe_zero_rows_yields_null_sample() { + let conn = duckdb::Connection::open_in_memory().unwrap(); + conn.execute_batch("CREATE TABLE t (id INT); INSERT INTO t VALUES (1);") + .unwrap(); + let (v, s) = run_sample_probe(&conn, "SELECT * FROM t WHERE id IS NULL", 51200); + assert_eq!(v, 0); + assert!(s.is_none()); + } + + #[test] + fn data_test_sample_probe_caps_at_20_rows_but_counts_all() { + let conn = duckdb::Connection::open_in_memory().unwrap(); + conn.execute_batch("CREATE TABLE t AS SELECT range AS id FROM range(50);") + .unwrap(); + let (v, s) = run_sample_probe(&conn, "SELECT * FROM t", 51200); + assert_eq!(v, 50); + let parsed: serde_json::Value = serde_json::from_str(&s.unwrap()).unwrap(); + assert_eq!(parsed.as_array().unwrap().len(), 20); + } + + #[test] + fn data_test_sample_probe_oversized_sample_degrades_to_null() { + let conn = duckdb::Connection::open_in_memory().unwrap(); + conn.execute_batch("CREATE TABLE t AS SELECT repeat('x', 1000) AS big FROM range(5);") + .unwrap(); + let (v, s) = run_sample_probe(&conn, "SELECT * FROM t", 100); + assert_eq!(v, 5); + assert!(s.is_none()); + } + + #[test] + fn data_test_sample_probe_codegen_row_query_shapes() { + // The exact rows-query shapes emitted by `build_data_test_checks`: + // unique's `{value, count}` grain and the star-EXCLUDE forms used on + // partitioned targets (plain and `_wm_src.`-qualified). + let conn = duckdb::Connection::open_in_memory().unwrap(); + conn.execute_batch( + "CREATE TABLE t (id INT, name VARCHAR, _wm_partition VARCHAR); \ + INSERT INTO t VALUES (1, 'a', 'p'), (1, 'b', 'p'), (2, NULL, 'p');", + ) + .unwrap(); + let (v, s) = run_sample_probe( + &conn, + "SELECT \"id\" AS \"value\", count(*) AS \"count\" FROM t \ + WHERE \"id\" IS NOT NULL GROUP BY \"id\" HAVING count(*) > 1", + 51200, + ); + assert_eq!(v, 1); + let parsed: serde_json::Value = serde_json::from_str(&s.unwrap()).unwrap(); + assert_eq!(parsed, serde_json::json!([{"value": 1, "count": 2}])); + + let (v, s) = run_sample_probe( + &conn, + "SELECT * EXCLUDE (\"_wm_partition\") FROM t WHERE name IS NULL", + 51200, + ); + assert_eq!(v, 1); + assert_eq!( + serde_json::from_str::(&s.unwrap()).unwrap(), + serde_json::json!([{"id": 2, "name": null}]) + ); + + let (v, s) = run_sample_probe( + &conn, + "SELECT _wm_src.* EXCLUDE (\"_wm_partition\") FROM t _wm_src WHERE _wm_src.name IS NULL", + 51200, + ); + assert_eq!(v, 1); + assert_eq!( + serde_json::from_str::(&s.unwrap()).unwrap(), + serde_json::json!([{"id": 2, "name": null}]) + ); + } + + #[test] + fn data_test_sample_probe_survives_exotic_types() { + let conn = duckdb::Connection::open_in_memory().unwrap(); + conn.execute_batch( + "CREATE TABLE t AS SELECT \ + INTERVAL 3 DAY AS iv, \ + 12345678901234567890123456789::HUGEINT AS hi, \ + '\\xDE\\xAD'::BLOB AS bl, \ + [1, 2, 3] AS li, \ + {'a': 1, 'b': 'x'} AS st, \ + TIMESTAMPTZ '2026-07-01 23:13:42+00' AS tstz, \ + DECIMAL '12.34' AS dec;", + ) + .unwrap(); + let (v, s) = run_sample_probe(&conn, "SELECT * FROM t", 51200); + assert_eq!(v, 1); + let parsed: serde_json::Value = serde_json::from_str(&s.unwrap()).unwrap(); + let row = &parsed.as_array().unwrap()[0]; + // Exact renderings are DuckDB's to_json choices — the gate is only + // that every type serializes without error and parses back as JSON. + assert!(row.get("iv").is_some()); + assert!(row.get("hi").is_some()); + assert!(row.get("bl").is_some()); + assert_eq!(row["li"], serde_json::json!([1, 2, 3])); + assert_eq!(row["st"], serde_json::json!({"a": 1, "b": "x"})); + assert!(row.get("tstz").is_some()); + assert!(row.get("dec").is_some()); + } +} + +// Cross-engine parity for the `wm_partition` materialize macro. The macro is +// `strftime(ts, '')` where `` is single-sourced in windmill-parser's +// `PartitionKind::default_time_format` — the SAME format the EE resolver +// (`partition_ee.rs`) uses (via chrono) to stamp the `{partition}` identity. +// The macro only works if the *bundled* DuckDB engine renders that format +// byte-for-byte identically to chrono. Weekly `%G-W%V` (ISO year/week) is the +// one that can diverge at a year boundary, so it's exercised head-on here — a +// gap a pure-Rust unit test can't reach. +#[cfg(test)] +mod wm_partition_parity_tests { + use super::*; + + // Mirror of windmill-parser `PartitionKind::default_time_format`. + const FORMATS: &[&str] = &["%Y-%m-%d", "%Y-%m-%dT%H", "%G-W%V", "%Y-%m"]; + + fn scalar_str(conn: &duckdb::Connection, sql: &str) -> String { + let mut stmt = conn.prepare(sql).unwrap(); + let mut rows = stmt.query([]).unwrap(); + row_string(rows.next().unwrap().unwrap()) + } + fn row_string(row: &Row) -> String { + row.get::(0).unwrap() + } + + #[test] + fn duckdb_strftime_matches_chrono_for_every_grain_format() { + use chrono::NaiveDate; + let conn = duckdb::Connection::open_in_memory().unwrap(); + // A normal instant, two same-hour instants, and ISO-week year boundaries + // where chrono's and DuckDB's ISO rules must agree: 2026-01-01 (Thu) is + // in 2026-W01 and makes 2026 a 53-week ISO year, so 2027-01-01 (Fri) + // belongs to 2026-W53; 2023-01-01 (Sun) → 2022-W52; 2021-01-03 (Sun) → + // 2020-W53. + let instants = [ + (2026, 7, 5, 23, 10, 0), + (2026, 7, 5, 23, 55, 0), + (2026, 1, 1, 0, 0, 0), + (2027, 1, 1, 0, 0, 0), + (2023, 1, 1, 0, 0, 0), + (2021, 1, 3, 23, 59, 0), + (2020, 12, 31, 12, 0, 0), + ]; + for (y, mo, d, h, mi, s) in instants { + let dt = NaiveDate::from_ymd_opt(y, mo, d) + .unwrap() + .and_hms_opt(h, mi, s) + .unwrap(); + let ts_lit = dt.format("%Y-%m-%d %H:%M:%S").to_string(); + for fmt in FORMATS { + let chrono_val = dt.format(fmt).to_string(); + let duck = scalar_str( + &conn, + &format!("SELECT strftime(TIMESTAMP '{ts_lit}', '{fmt}')"), + ); + assert_eq!( + duck, chrono_val, + "DuckDB strftime disagrees with chrono for {ts_lit} / {fmt}" + ); + } + } + // Explicit year-boundary pin: guards against BOTH engines drifting the + // same way (2027-01-01 is ISO week 53 of 2026). + assert_eq!( + scalar_str(&conn, "SELECT strftime(TIMESTAMP '2027-01-01', '%G-W%V')"), + "2027-01-01" + .parse::() + .unwrap() + .format("%G-W%V") + .to_string() + ); + assert_eq!( + scalar_str(&conn, "SELECT strftime(TIMESTAMP '2027-01-01', '%G-W%V')"), + "2026-W53" + ); + } + + #[test] + fn temp_macro_is_classified_as_setup() { + // The macro the executor injects, plus whitespace/keyword variants the + // workspace-macro splicer can emit. + assert!(is_setup_statement( + "CREATE OR REPLACE TEMP MACRO wm_partition(ts) AS strftime(ts, '%Y-%m-%dT%H')" + )); + assert!(is_setup_statement(" create temp macro foo(x) as x + 1")); + assert!(is_setup_statement("CREATE TEMPORARY MACRO foo(x) AS x")); + assert!(is_setup_statement( + "CREATE OR REPLACE\n TEMPORARY MACRO foo(x) AS x" + )); + // Persistent (non-TEMP) macros are real user statements, not setup. + assert!(!is_setup_statement( + "CREATE OR REPLACE MACRO safe_div(a, b) AS a / b" + )); + assert!(!is_setup_statement("CREATE MACRO foo(x) AS x")); + } + + #[test] + fn prepare_pass_binds_consumer_only_when_macro_runs_as_setup() { + // Mirror prepare_duckdb_internal's per-block contract: setup statements + // are EXECUTED, everything else is only prepared. A block that calls + // wm_partition binds at prepare time, so it resolves only if the macro + // block already ran as setup. + let consumer = "SELECT wm_partition(TIMESTAMP '2026-07-05 23:10:00') AS p"; + let macro_stmt = + "CREATE OR REPLACE TEMP MACRO wm_partition(ts) AS strftime(ts, '%Y-%m-%dT%H')"; + + let conn = duckdb::Connection::open_in_memory().unwrap(); + for block in [macro_stmt, consumer] { + if is_setup_statement(block) { + conn.execute_batch(block).unwrap(); + } else { + conn.prepare(block) + .expect("consumer must prepare once the macro ran as setup"); + } + } + + // Guard: if the macro were NOT setup (the bug), the consumer's prepare + // fails "function does not exist" — proving the classification matters. + let fresh = duckdb::Connection::open_in_memory().unwrap(); + assert!( + fresh.prepare(consumer).is_err(), + "consumer must fail to prepare when wm_partition was never created" + ); + } + + #[test] + fn wm_partition_macro_buckets_whole_slice_and_naive_cast_errors() { + let conn = duckdb::Connection::open_in_memory().unwrap(); + // Exactly the statement the executor injects for an hourly materialize. + conn.execute_batch( + "CREATE OR REPLACE TEMP MACRO wm_partition(ts) AS strftime(ts, '%Y-%m-%dT%H');", + ) + .unwrap(); + conn.execute_batch( + "CREATE TABLE t(ts TIMESTAMP); INSERT INTO t VALUES \ + (TIMESTAMP '2026-07-05 23:10:00'), (TIMESTAMP '2026-07-05 23:55:00'), \ + (TIMESTAMP '2026-07-05 22:30:00'), (TIMESTAMP '2026-07-06 00:05:00');", + ) + .unwrap(); + // Filtering with the injected macro against the identity string selects + // the WHOLE hour bucket (both 23:10 and 23:55) — not just the boundary + // instant a `= TIMESTAMP {partition}` cast could ever match. + let n = scalar_str( + &conn, + "SELECT count(*)::VARCHAR FROM t WHERE wm_partition(ts) = '2026-07-05T23'", + ); + assert_eq!(n, "2"); + // The footgun the macro exists to avoid: the weekly/monthly identity + // strings are not valid TIMESTAMP literals, so the naive cast errors. + assert!( + conn.execute_batch("SELECT TIMESTAMP '2026-W27';").is_err(), + "expected `TIMESTAMP '2026-W27'` to be a Conversion Error" + ); + assert!( + conn.execute_batch("SELECT TIMESTAMP '2026-07';").is_err(), + "expected `TIMESTAMP '2026-07'` to be a Conversion Error" + ); + } +} + fn json_value_to_duckdb_value( json_value: &serde_json::Value, arg_type: &str, diff --git a/backend/windmill-git-sync/src/lib.rs b/backend/windmill-git-sync/src/lib.rs index b10f62e7a5..6c1b9bf89b 100644 --- a/backend/windmill-git-sync/src/lib.rs +++ b/backend/windmill-git-sync/src/lib.rs @@ -24,30 +24,102 @@ pub use git_sync_oss::{ #[derive(Clone, Debug)] pub enum DeployedObject { - Script { hash: ScriptHash, path: String, parent_path: Option }, - Flow { path: String, parent_path: Option, version: i64 }, - App { path: String, version: i64, parent_path: Option }, - RawApp { path: String, version: i64, parent_path: Option }, - Folder { path: String }, - Resource { path: String, parent_path: Option }, - Variable { path: String, parent_path: Option }, - Schedule { path: String }, - ResourceType { path: String }, - User { email: String }, - Group { name: String }, - HttpTrigger { path: String, parent_path: Option }, - WebsocketTrigger { path: String, parent_path: Option }, - KafkaTrigger { path: String, parent_path: Option }, - NatsTrigger { path: String, parent_path: Option }, - PostgresTrigger { path: String, parent_path: Option }, - MqttTrigger { path: String, parent_path: Option }, - SqsTrigger { path: String, parent_path: Option }, - GcpTrigger { path: String, parent_path: Option }, - AzureTrigger { path: String, parent_path: Option }, - EmailTrigger { path: String, parent_path: Option }, - Settings { setting_type: String }, - Key { key_type: String }, - WorkspaceDependencies { path: String }, + Script { + hash: ScriptHash, + path: String, + parent_path: Option, + }, + Flow { + path: String, + parent_path: Option, + version: i64, + }, + App { + path: String, + version: i64, + parent_path: Option, + }, + RawApp { + path: String, + version: i64, + parent_path: Option, + }, + Folder { + path: String, + }, + Resource { + path: String, + parent_path: Option, + }, + Variable { + path: String, + parent_path: Option, + }, + Schedule { + path: String, + }, + ResourceType { + path: String, + }, + User { + email: String, + }, + Group { + name: String, + }, + HttpTrigger { + path: String, + parent_path: Option, + }, + WebsocketTrigger { + path: String, + parent_path: Option, + }, + KafkaTrigger { + path: String, + parent_path: Option, + }, + NatsTrigger { + path: String, + parent_path: Option, + }, + PostgresTrigger { + path: String, + parent_path: Option, + }, + MqttTrigger { + path: String, + parent_path: Option, + }, + SqsTrigger { + path: String, + parent_path: Option, + }, + GcpTrigger { + path: String, + parent_path: Option, + }, + AzureTrigger { + path: String, + parent_path: Option, + }, + EmailTrigger { + path: String, + parent_path: Option, + }, + Settings { + setting_type: String, + }, + Key { + key_type: String, + }, + WorkspaceDependencies { + path: String, + }, + /// A single data table migration, identified by `/_`. + DatatableMigration { + path: String, + }, } impl DeployedObject { @@ -77,6 +149,7 @@ impl DeployedObject { DeployedObject::Settings { .. } => "settings.yaml".to_string(), DeployedObject::Key { .. } => "encryption_key.yaml".to_string(), DeployedObject::WorkspaceDependencies { path, .. } => path.to_owned(), + DeployedObject::DatatableMigration { path } => path.to_owned(), } } @@ -118,6 +191,7 @@ impl DeployedObject { DeployedObject::Settings { .. } => None, DeployedObject::Key { .. } => None, DeployedObject::WorkspaceDependencies { .. } => None, + DeployedObject::DatatableMigration { .. } => None, } } @@ -147,6 +221,7 @@ impl DeployedObject { DeployedObject::Settings { .. } => "settings", DeployedObject::Key { .. } => "key", DeployedObject::WorkspaceDependencies { .. } => "workspace_dependencies", + DeployedObject::DatatableMigration { .. } => "datatable_migration", } .to_string() } diff --git a/backend/windmill-mcp/src/common/scope.rs b/backend/windmill-mcp/src/common/scope.rs index 7da4e2cebc..f43095d740 100644 --- a/backend/windmill-mcp/src/common/scope.rs +++ b/backend/windmill-mcp/src/common/scope.rs @@ -38,6 +38,71 @@ impl McpScopeConfig { is_resource_allowed(path, patterns) } + + /// Directional subset check: does this config grant at least everything + /// `requested` grants? Used to enforce monotonic containment when an MCP + /// OAuth approval mints a token (the granted scopes must be within the + /// approving token's own scopes). + /// + /// Unlike `is_allowed` (which tests a single concrete path with OR + /// semantics), this requires every requested pattern to be covered by some + /// caller pattern — so `mcp:scripts:f/x` cannot widen into `mcp:scripts:*`. + pub fn contains(&self, requested: &McpScopeConfig) -> bool { + if self.all { + return true; + } + if requested.all { + return false; + } + if requested.favorites && !self.favorites { + return false; + } + if let Some(req_hub) = requested.hub_apps.as_ref() { + match self.hub_apps.as_ref() { + Some(caller_hub) => { + let caller_apps: std::collections::HashSet<&str> = + caller_hub.split(',').map(|s| s.trim()).collect(); + if !req_hub + .split(',') + .map(|s| s.trim()) + .all(|a| caller_apps.contains(a)) + { + return false; + } + } + None => return false, + } + } + resource_list_covers(&self.scripts, &requested.scripts) + && resource_list_covers(&self.flows, &requested.flows) + && resource_list_covers(&self.endpoints, &requested.endpoints) + } +} + +/// Every requested pattern must be covered by some caller pattern. +fn resource_list_covers(caller: &[String], requested: &[String]) -> bool { + requested + .iter() + .all(|req| caller.iter().any(|c| pattern_covers(c, req))) +} + +/// Directional: does the single caller pattern cover `requested`? `caller` may +/// be `*`, an exact path/name, or a `/*` subtree; `requested` may itself +/// be a subtree wildcard, in which case the whole requested subtree must fall +/// within the caller's. Mirrors the route-scope containment in windmill-api-auth. +fn pattern_covers(caller: &str, requested: &str) -> bool { + if caller == "*" || caller == requested { + return true; + } + // An exact caller pattern only covers itself (handled above); a wildcard + // requested can never be covered by a non-`*` exact caller. + let Some(prefix) = caller.strip_suffix("/*") else { + return false; + }; + let requested_base = requested.strip_suffix("/*").unwrap_or(requested); + requested_base == prefix + || (requested_base.starts_with(prefix) + && requested_base.as_bytes().get(prefix.len()) == Some(&b'/')) } /// Parse MCP scopes from token scope strings @@ -254,4 +319,51 @@ mod tests { assert!(config.is_allowed("flow", "f/automation/test")); assert!(!config.is_allowed("flow", "f/other/test")); } + + fn cfg(scopes: &[&str]) -> McpScopeConfig { + parse_mcp_scopes(&scopes.iter().map(|s| s.to_string()).collect::>()).unwrap() + } + + #[test] + fn test_contains_subset_and_widening() { + // mcp:all contains anything. + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:all"]))); + + // A wildcard caller covers narrower requests, but not other domains/all. + let star = cfg(&["mcp:scripts:*"]); + assert!(star.contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(star.contains(&cfg(&["mcp:scripts:*"]))); + assert!(!star.contains(&cfg(&["mcp:all"]))); + assert!(!star.contains(&cfg(&["mcp:flows:f/x"]))); + + // The core regression: a single-path caller must NOT widen into `*` or + // into another path. + let narrow = cfg(&["mcp:scripts:f/x"]); + assert!(narrow.contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(!narrow.contains(&cfg(&["mcp:scripts:*"]))); + assert!(!narrow.contains(&cfg(&["mcp:scripts:f/y"]))); + assert!(!narrow.contains(&cfg(&["mcp:all"]))); + + // Subtree wildcard covers paths within it but not a sibling subtree. + let subtree = cfg(&["mcp:scripts:f/team/*"]); + assert!(subtree.contains(&cfg(&["mcp:scripts:f/team/sub"]))); + assert!(subtree.contains(&cfg(&["mcp:scripts:f/team/sub/*"]))); + assert!(!subtree.contains(&cfg(&["mcp:scripts:f/other/x"]))); + } + + #[test] + fn test_contains_favorites_and_endpoints() { + assert!(cfg(&["mcp:favorites"]).contains(&cfg(&["mcp:favorites"]))); + // A caller without favorites cannot grant favorites. + assert!(!cfg(&["mcp:scripts:*"]).contains(&cfg(&["mcp:favorites"]))); + + // Endpoint names match exactly (or via `*`). + let ep = cfg(&["mcp:endpoints:getVariable"]); + assert!(ep.contains(&cfg(&["mcp:endpoints:getVariable"]))); + assert!(!ep.contains(&cfg(&["mcp:endpoints:getResource"]))); + assert!(!ep.contains(&cfg(&["mcp:all"]))); + // mcp:all grants all endpoints. + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:endpoints:getResource"]))); + } } diff --git a/backend/windmill-oauth/src/lib.rs b/backend/windmill-oauth/src/lib.rs index 2f28a6d4c7..e314f13d75 100644 --- a/backend/windmill-oauth/src/lib.rs +++ b/backend/windmill-oauth/src/lib.rs @@ -452,12 +452,12 @@ pub async fn build_client_credentials_oauth_client( let caller_supplied_creds = !client_id.is_empty() && !client_secret.is_empty(); - // Apply the server-resolved concrete token URL. Instance-templated providers - // (e.g. Coupa) carry an empty or `{instance}`-templated token URL in their - // registry config; the resolved value (host-pinned for bring-your-own, - // persisted on the row for refresh) is what completes it. The caller never - // supplies a free-form token URL: this value always comes from - // `resolve_cc_token_url_input` or a previously-resolved persisted URL. + // Apply the resolved concrete token URL. Instance-templated providers (e.g. + // Coupa) carry an empty or `{instance}`-templated token URL in their registry + // config; the resolved value (host-pinned for instance-name connections, + // persisted on the row for refresh) is what completes it. For bring-your-own + // connections this value may instead be a caller-supplied override — safe + // because only the caller's own credentials are ever sent to it. if let Some(url) = resolved_token_url { connect_config.token_url = url.to_string(); } @@ -652,6 +652,28 @@ pub fn resolve_cc_token_url_input( Ok(template.replace("{instance}", value)) } +/// Whether a built-in provider's client-credentials token URL is host-pinned via +/// an `{instance}` template (e.g. servicenow, snowflake, coupa). Such providers +/// only accept an instance name substituted into a fixed-host template, so a +/// free-form caller token URL override must be rejected for them — otherwise the +/// exchange host could be redirected, which is exactly what the template pins. +/// Fixed-host registry providers and custom (non-registry) providers return +/// `false`: an override is allowed there. +pub fn is_instance_templated_cc(connect_configs_json: &str, client_name: &str) -> bool { + serde_json::from_str::>(connect_configs_json) + .ok() + .and_then(|m| resolve_registry_config(&m, client_name)) + .map(|cfg| { + cfg.connect_config_template + .as_ref() + .map(|t| t.token_url.clone()) + .filter(|u| !u.is_empty()) + .unwrap_or(cfg.token_url) + .contains("{instance}") + }) + .unwrap_or(false) +} + /// Resolve the concrete bring-your-own client-credentials token URL for any /// provider, never from a caller-supplied URL: /// - **Built-in registry providers** resolve from the registry via @@ -1350,4 +1372,20 @@ mod tests { assert!(resolve_cc_token_url_input(CC_REGISTRY, "bad_host_tpl", Some("evil.com")).is_err()); assert!(resolve_cc_token_url_input(CC_REGISTRY, "bad_mid_tpl", Some("evil")).is_err()); } + + #[test] + fn instance_templated_cc_true_for_templated_providers() { + // Host-pinned via `{instance}`: a bring-your-own token URL override must be + // refused for these (only the instance-name path may set their URL). + assert!(is_instance_templated_cc(CC_REGISTRY, "coupa")); + assert!(is_instance_templated_cc(CC_REGISTRY, "servicenow")); + } + + #[test] + fn instance_templated_cc_false_for_fixed_host_and_unknown() { + // Fixed-host registry provider and custom (non-registry) provider both allow + // an override, so neither is reported as instance-templated. + assert!(!is_instance_templated_cc(CC_REGISTRY, "visma")); + assert!(!is_instance_templated_cc(CC_REGISTRY, "my_custom_thing")); + } } diff --git a/backend/windmill-object-store/src/lib.rs b/backend/windmill-object-store/src/lib.rs index cf30efe116..553eb69917 100644 --- a/backend/windmill-object-store/src/lib.rs +++ b/backend/windmill-object-store/src/lib.rs @@ -494,6 +494,23 @@ fn build_azure_blob_client( return Ok(Arc::new(store)); } +/// Whether a GCS `service_account_key` carries no static credentials, in which case the client +/// should fall back to the instance's ambient credentials (GKE Workload Identity / metadata server) +/// instead of being handed an unparseable key. Besides an empty/whitespace string, the settings UI +/// stores "no key" as an empty JSON object `{}` (and `serde_json` may yield `null`), so treat those +/// as absent too. Shared with the connectivity-test SSRF guard so both agree on what "no key" means. +pub fn gcs_service_account_key_is_blank(service_account_key: &str) -> bool { + let trimmed = service_account_key.trim(); + if trimmed.is_empty() { + return true; + } + match serde_json::from_str::(trimmed) { + Ok(serde_json::Value::Null) => true, + Ok(serde_json::Value::Object(map)) => map.is_empty(), + _ => false, + } +} + #[cfg(feature = "parquet")] async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result> { let gcs_resource = gcs_resource_ref.clone(); @@ -509,7 +526,12 @@ async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result n)`). Nothing pins the consumer's reads to it. +//! //! Errors are logged but never bubble up to fail the producer's job. use crate::{push, MiniCompletedJob, PushArgs, PushIsolationLevel}; +use serde::Serialize; use serde_json::value::RawValue; use sqlx::types::Json; use sqlx::{Pool, Postgres}; use std::collections::HashMap; use std::sync::Arc; use uuid::Uuid; -use windmill_common::assets::AssetKind; +use windmill_common::assets::{parse_asset_trigger_ref, AssetKind}; use windmill_common::error::{self, Result}; use windmill_common::get_latest_deployed_hash_for_path; use windmill_common::jobs::{JobKind, JobPayload, JobTriggerKind}; @@ -226,6 +236,15 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result if !is_eligible_kind(job) { return Ok(DispatchResult::default()); } + // A parented script is dispatch-eligible only as a native retry attempt — a + // re-run of the SAME runnable as its chain parent. Schedule/error/recovery + // handlers are also parented `Script` children but run a DIFFERENT script; + // excluding them stops a handler that happens to declare assets from + // triggering a cascade (the pre-native-retry `parent_job IS NULL` guard + // excluded every parented child). + if job.parent_job.is_some() && !is_native_retry_attempt(db, job).await? { + return Ok(DispatchResult::default()); + } let runnable_path = match job.runnable_path.as_deref() { Some(p) if !p.is_empty() => p, _ => return Ok(DispatchResult::default()), @@ -277,6 +296,10 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result // subscriber × asset write). The mid-pass join-slot writes // (record_and_check_join_slot) are a separate table and unaffected. let mut events: Vec = Vec::new(); + // A subscriber listening to several of this producer's writes is pushed + // once per edge; its upstream-snapshot record is identical across those + // pushes (same instant, same trigger set), so resolve it once per pass. + let mut snapshot_memo: HashMap>> = HashMap::new(); for (asset_kind, asset_path) in writes { let Some(prefix) = asset_kind.canonical_prefix() else { continue; @@ -352,6 +375,29 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result } } } + // Forensic upstream-state capture, resolved at dispatch time (a + // debounced job that gets superseded is re-pushed by the later + // arrival, which re-resolves — the surviving job records what its + // own dispatch saw). Best-effort: a lookup failure must not stop + // the cascade. + let snapshots = match snapshot_memo.get(&sub_path) { + Some(s) => s.clone(), + None => { + let s = Arc::new( + upstream_snapshots(db, &job.workspace_id, &sub_path) + .await + .unwrap_or_else(|e| { + tracing::error!( + "upstream-snapshot lookup failed for {}: {e:#}", + sub_path + ); + Vec::new() + }), + ); + snapshot_memo.insert(sub_path.clone(), s.clone()); + s + } + }; match push_subscriber( db, job, @@ -364,6 +410,7 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result debounce_s, retry_count, retry_delay_s, + &snapshots, ) .await { @@ -406,12 +453,28 @@ fn is_eligible_kind(job: &MiniCompletedJob) -> bool { if !matches!(job.kind, JobKind::Script | JobKind::Preview) { return false; } - if job.parent_job.is_some() || job.flow_step_id.is_some() { + // Flow steps (and sub-flow jobs) carry `flow_step_id` and are ineligible. + // Native script-retry attempts carry `parent_job` (the chain root) but no + // `flow_step_id`; whether a parented job is actually a retry attempt (vs a + // schedule/error handler child) is decided in `try_dispatch`. + if job.flow_step_id.is_some() { return false; } true } +// Native retry attempts carry an explicit `native_retry_attempt` marker; no +// other parented `Script` child (schedule handlers, WAC inline children, flow +// steps) does. One indexed point lookup, only for parented jobs. +async fn is_native_retry_attempt(db: &DB, job: &MiniCompletedJob) -> Result { + Ok(sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = $1) AS \"exists!\"", + job.id, + ) + .fetch_one(db) + .await?) +} + async fn fetch_args( db: &Pool, workspace_id: &str, @@ -527,6 +590,91 @@ async fn workspace_producer_writes( Ok(map) } +/// Forensic record of one direct upstream's state at dispatch time: the +/// latest captured materialization snapshot of an asset in the subscriber's +/// `// on` trigger set. Serialized into the dispatched job's `trigger` arg +/// (`upstream_snapshots`) so a failing consumer run stays debuggable against +/// DuckLake time-travel. Record-only — the consumer's reads are not pinned. +#[derive(Debug, Serialize)] +struct UpstreamSnapshot { + /// Canonical asset uri, e.g. `ducklake://analytics/orders_daily`. + asset: String, + snapshot_id: i64, + /// Partition whose write produced this snapshot — i.e. the latest slice + /// written, not necessarily the slice this consumer processes. The + /// snapshot itself is table-global. Omitted for whole-table + /// materializations. + #[serde(skip_serializing_if = "Option::is_none")] + partition: Option, +} + +/// Latest captured snapshot per direct upstream of `subscriber_path`: its +/// asset trigger set joined against `materialized_partition`, keeping the +/// highest `snapshot_id` per asset (the newest substrate version the consumer +/// could read). Assets with no captured snapshot (non-materialized upstreams) +/// simply produce no entry. Two queries total regardless of upstream count. +async fn upstream_snapshots( + db: &Pool, + workspace_id: &str, + subscriber_path: &str, +) -> Result> { + let refs = sqlx::query_scalar!( + r#"SELECT DISTINCT trigger_ref AS "trigger_ref!" + FROM script_trigger + WHERE workspace_id = $1 + AND runnable_path = $2 + AND trigger_kind = 'asset' + AND runnable_kind = 'script' + ORDER BY trigger_ref"#, + workspace_id, + subscriber_path, + ) + .fetch_all(db) + .await?; + // Keep only refs with a recognized asset prefix, preserving ref order so + // the recorded list is deterministic. + let parsed: Vec<(String, AssetKind, String)> = refs + .into_iter() + .filter_map(|r| parse_asset_trigger_ref(&r).map(|(k, p)| (r, k, p))) + .collect(); + if parsed.is_empty() { + return Ok(Vec::new()); + } + let kinds: Vec = parsed.iter().map(|(_, k, _)| *k).collect(); + let paths: Vec = parsed.iter().map(|(_, _, p)| p.clone()).collect(); + let rows = sqlx::query!( + r#"SELECT DISTINCT ON (mp.asset_kind, mp.asset_path) + mp.asset_kind AS "asset_kind: AssetKind", mp.asset_path, + mp.snapshot_id AS "snapshot_id!", mp.partition + FROM materialized_partition mp + JOIN unnest($2::ASSET_KIND[], $3::text[]) AS u(kind, path) + ON mp.asset_kind = u.kind AND mp.asset_path = u.path + WHERE mp.workspace_id = $1 + AND mp.status = 'materialized' AND mp.snapshot_id IS NOT NULL + ORDER BY mp.asset_kind, mp.asset_path, mp.snapshot_id DESC"#, + workspace_id, + kinds as Vec, + &paths, + ) + .fetch_all(db) + .await?; + let mut latest: HashMap<(AssetKind, String), (i64, String)> = rows + .into_iter() + .map(|r| ((r.asset_kind, r.asset_path), (r.snapshot_id, r.partition))) + .collect(); + Ok(parsed + .into_iter() + .filter_map(|(trigger_ref, kind, path)| { + let (snapshot_id, partition) = latest.remove(&(kind, path))?; + Some(UpstreamSnapshot { + asset: trigger_ref, + snapshot_id, + partition: (!partition.is_empty()).then_some(partition), + }) + }) + .collect()) +} + /// A subscriber row resolved from `script_trigger`. Bundles the per-edge /// options (debounce) and the script-level policy fields (`join_all`, /// retry) that travel together to dispatch. @@ -589,6 +737,7 @@ async fn push_subscriber( debounce_s: Option, retry_count: Option, retry_delay_s: Option, + upstream_snapshots: &[UpstreamSnapshot], ) -> Result { // Same resolution as every other trigger path (`script_path_to_payload`): // latest deployed hash plus the script's own runnable settings @@ -618,14 +767,16 @@ async fn push_subscriber( script.runnable_settings.debouncing_settings, ); - // Retry is only available via the flow runtime — wrap the script in a - // one-step flow when the cascade declares one. No retry = - // unwrapped `ScriptHash` push. + // When the cascade declares a retry, hand `push` a one-step-flow request + // carrying the policy + `language`; `push` materializes it into a native + // retryable `Script` (not a flow), so a failed/recovered subscriber stays + // eligible to trigger its own downstream. No retry = plain `ScriptHash`. let payload = if let Some(retry) = crate::cascade::cascade_retry(retry_count, retry_delay_s) { JobPayload::SingleStepFlow { path: subscriber_path.to_string(), hash: Some(hash), flow_version: None, + language: Some(script.language), args: HashMap::new(), retry: Some(retry), error_handler_path: None, @@ -673,7 +824,7 @@ async fn push_subscriber( }; let mut args: HashMap> = HashMap::new(); - let trigger_payload = serde_json::json!({ + let mut trigger_payload = serde_json::json!({ "kind": "asset", "asset_kind": serde_json::to_value(&asset_kind).expect("AssetKind serializes"), "asset_path": asset_path, @@ -682,6 +833,10 @@ async fn push_subscriber( CHAIN_KEY: chain, PARTITION_ARG: partition, }); + if !upstream_snapshots.is_empty() { + trigger_payload["upstream_snapshots"] = + serde_json::to_value(upstream_snapshots).expect("UpstreamSnapshot serializes"); + } args.insert(TRIGGER_ARG.to_string(), to_raw_value(&trigger_payload)); // Carry the producer's resolved partition forward as a top-level arg so // the subscriber's body can read it and the next cascade hop's diff --git a/backend/windmill-queue/src/ducklake_maintenance_oss.rs b/backend/windmill-queue/src/ducklake_maintenance_oss.rs new file mode 100644 index 0000000000..ab6e035847 --- /dev/null +++ b/backend/windmill-queue/src/ducklake_maintenance_oss.rs @@ -0,0 +1,96 @@ +//! OSS fallback: scheduled ducklake maintenance (snapshot expiry, adjacent-file +//! compaction, orphaned-file cleanup via managed per-lake schedules) is an +//! enterprise feature; the implementation lives in windmill-ee-private +//! (see `ducklake_maintenance_ee`). In the public build the entry points report +//! that the enterprise edition is required. + +use std::collections::HashMap; + +use sqlx::{Postgres, Transaction}; +use windmill_common::{ + error::{Error, Result}, + jobs::JobPayload, + schedule::Schedule, + workspaces::Ducklake, + DB, +}; + +/// Reconcile the managed `f/ducklake_maintenance/` schedule rows with +/// the ducklake settings, inside the caller's transaction. +/// +/// Not an authorization boundary: it mutates schedule rows for `w_id` on +/// behalf of `edited_by`/`email`, so the caller MUST already have enforced +/// workspace-admin on `w_id` and that the identity is the authenticated +/// caller's (as `edit_ducklake_config` does via `require_admin`). +// Only newly-enabled maintenance is rejected: a config that already had it +// enabled (e.g. an enterprise license lapsed) must not make every unrelated +// ducklake settings save fail, and the admin must be able to save it off. +pub async fn sync_ducklake_maintenance_schedules<'c>( + _db: &DB, + mut tx: Transaction<'c, Postgres>, + w_id: &str, + ducklakes: &HashMap, + previous: &HashMap, + _edited_by: &str, + _email: &str, +) -> Result> { + let enabled = |dl: &Ducklake| -> bool { dl.maintenance.as_ref().is_some_and(|m| m.enabled) }; + if ducklakes + .iter() + .any(|(name, dl)| enabled(dl) && !previous.get(name).is_some_and(|prev| enabled(prev))) + { + return Err(Error::BadRequest( + "Ducklake scheduled maintenance is only available in the enterprise edition" + .to_string(), + )); + } + + // Saving maintenance off (e.g. after an enterprise license lapsed) must + // remove the managed row AND its already-queued occurrence here too — + // otherwise a maintenance job pushed under the enterprise edition still + // runs once after the admin disabled it. Like the enterprise + // implementation, the removed set is derived from config, never from the + // path prefix. + let removed = previous + .iter() + .filter(|(name, dl)| enabled(dl) && !ducklakes.get(*name).is_some_and(|cur| enabled(cur))) + .map(|(name, _)| windmill_common::workspaces::ducklake_maintenance_schedule_path(name)) + .collect::>(); + for path in removed.iter() { + crate::schedule::clear_schedule(&mut tx, path, w_id).await?; + } + sqlx::query!( + "DELETE FROM schedule WHERE workspace_id = $1 AND path = ANY($2)", + w_id, + &removed + ) + .execute(&mut *tx) + .await?; + + Ok(tx) +} + +/// Build the job payload for one occurrence of a managed maintenance schedule +/// (`push_scheduled_job` calls this for reserved-prefix schedule paths). +/// Returns `(payload, tag, timeout, on_behalf_of_email, created_by)`. +/// +/// Always `Ok(None)` in the public build: the caller falls through to normal +/// script resolution, so a pre-existing user schedule under a real +/// `ducklake_maintenance` folder keeps running, while a managed row left over +/// from an enterprise period fails script resolution with NotFound and is +/// auto-disabled with `schedule.error` recorded by the post-completion +/// scheduler. +pub async fn build_maintenance_schedule_payload<'c>( + _tx: &mut Transaction<'c, Postgres>, + _schedule: &Schedule, +) -> Result< + Option<( + JobPayload, + Option, + Option, + Option, + String, + )>, +> { + Ok(None) +} diff --git a/backend/windmill-queue/src/freshness_watchdog_oss.rs b/backend/windmill-queue/src/freshness_watchdog_oss.rs new file mode 100644 index 0000000000..319fb25f62 --- /dev/null +++ b/backend/windmill-queue/src/freshness_watchdog_oss.rs @@ -0,0 +1,9 @@ +//! OSS fallback for the pipeline freshness watchdog. The active backstop — +//! re-running a `// freshness`-annotated producer whose output aged past its +//! window — is an enterprise feature (see `freshness_watchdog_ee`). In the +//! public build the tick is a no-op; CE keeps the passive fresh/stale badge +//! on the asset graph. + +use windmill_common::DB; + +pub async fn tick(_db: &DB) {} diff --git a/backend/windmill-queue/src/jobs.rs b/backend/windmill-queue/src/jobs.rs index 5cb8911c7e..99f33da3bf 100644 --- a/backend/windmill-queue/src/jobs.rs +++ b/backend/windmill-queue/src/jobs.rs @@ -45,8 +45,8 @@ use windmill_common::min_version::{ MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2, }; use windmill_common::runnable_settings::{ - ConcurrencySettings, ConcurrencySettingsWithCustom, DebouncingSettings, RunnableSettings, - RunnableSettingsTrait, + ConcurrencySettings, ConcurrencySettingsWithCustom, DebouncingSettings, RetrySettings, + RunnableSettings, RunnableSettingsTrait, }; use windmill_common::triggers::TriggerMetadata; use windmill_common::utils::{calculate_hash, configure_client, now_from_db}; @@ -68,7 +68,7 @@ use windmill_common::{ }, flows::{ add_virtual_items_if_necessary, FlowModule, FlowModuleValue, FlowValue, InputTransform, - StopAfterIf, + Retry, StopAfterIf, }, jobs::{get_payload_tag_from_prefixed_path, JobKind, JobPayload, QueuedJob, RawCode}, min_version::{MIN_VERSION_IS_AT_LEAST_1_432, MIN_VERSION_IS_AT_LEAST_1_440}, @@ -958,6 +958,28 @@ pub async fn add_completed_job( )); } + // Native script retry: a failed `Script` job that carries a retry policy and + // has attempts left gets its next attempt enqueued here — before the queue + // row (which holds the attempt counter) is removed by commit. The failed + // attempt is still recorded as a completed job below. `maybe_enqueue_…` + // self-guards on kind/cancellation/policy, so the success path is unaffected. + let retry_pending = if !success && !skipped && !from_cache { + // Serialized lazily, and only when a `retry_if` policy actually needs it. + let result_fn = || serde_json::value::to_raw_value(&result).ok(); + match maybe_enqueue_native_script_retry(db, completed_job, &canceled_by, &result_fn).await { + Ok(enqueued) => enqueued, + Err(e) => { + tracing::error!( + "native retry enqueue failed for {}: {e:#}", + completed_job.id + ); + false + } + } + } else { + false + }; + let result_columns = result_columns.as_ref(); let (opt_uuid, duration, _skip_downstream_error_handlers, wac_job_ids) = (|| { commit_completed_job( @@ -972,6 +994,7 @@ pub async fn add_completed_job( flow_is_done, duration, from_cache, + retry_pending, ) .warn_after_seconds(10) }) @@ -1031,6 +1054,9 @@ async fn commit_completed_job( flow_is_done: bool, duration: Option, from_cache: bool, + // True when a native script retry was enqueued for this failed attempt, i.e. + // this is not the terminal attempt — schedule completion handlers must wait. + retry_pending: bool, ) -> windmill_common::error::Result<(Option, i64, bool, Option)> { // let start = std::time::Instant::now(); @@ -1050,6 +1076,19 @@ async fn commit_completed_job( return value; } + // Resolve the concurrency-limit settings on the pool *before* opening the + // completion transaction: doing it inside the tx would hold a second + // simultaneous connection from the small per-worker pool. + let has_concurrent_limit = completed_job.concurrent_limit.is_some() + || windmill_common::runnable_settings::prefetch_cached_from_handle( + completed_job.runnable_settings_handle, + db, + ) + .await? + .1 + .concurrent_limit + .is_some(); + let mut tx = db.begin().warn_after_seconds(10).await?; let duration = sqlx::query_scalar!( @@ -1258,11 +1297,17 @@ async fn commit_completed_job( // for flows, only try to schedule next tick here if flow failed and because first handle_flow failed (step = 0, modules[0] = {type: 'Failure', 'job': uuid::nil()}) // or job was cancelled before first handle_flow was called (step = 0, modules = [] OR modules[0].type == 'WaitingForPriorSteps') // otherwise flow rescheduling is done inside handle_flow - let schedule_next_tick = !completed_job.is_flow() - || from_cache - || !success - && sqlx::query_scalar!( - "SELECT + // Native retry attempts carry the schedule trigger (so the + // terminal attempt can drive handlers) but `parent_job` is set — + // they must not each push the next cron tick (the root already + // did), so gate next-tick on a top-level (`parent_job IS NULL`) + // occurrence. + let schedule_next_tick = completed_job.parent_job.is_none() + && (!completed_job.is_flow() + || from_cache + || !success + && sqlx::query_scalar!( + "SELECT flow_status->>'step' = '0' AND ( jsonb_array_length(flow_status->'modules') = 0 @@ -1273,15 +1318,15 @@ async fn commit_completed_job( ) ) FROM v2_job_completed WHERE id = $2 AND workspace_id = $3", - Uuid::nil().to_string(), - &completed_job.id, - &completed_job.workspace_id - ) - .fetch_optional(&mut *tx) - .warn_after_seconds(10) - .await? - .flatten() - .unwrap_or(false); + Uuid::nil().to_string(), + &completed_job.id, + &completed_job.workspace_id + ) + .fetch_optional(&mut *tx) + .warn_after_seconds(10) + .await? + .flatten() + .unwrap_or(false)); if schedule_next_tick { let (returned_tx, schedule_push_err) = @@ -1292,42 +1337,55 @@ async fn commit_completed_job( } } + // Defer schedule completion handlers (on_failure/on_success/ + // on_recovery) while a native retry is pending: only the terminal + // attempt should drive them. apply_schedule_handlers resolves + // per-occurrence failure/recovery status across the whole retry + // chain, so multi-count/exact handler policies work even though + // each attempt is its own completed job. #[cfg(all(feature = "enterprise", feature = "private"))] - if let Err(err) = crate::jobs_ee::apply_schedule_handlers( - db, - &schedule, - &script_path, - &completed_job.workspace_id, - success, - result, - job_id, - completed_job.started_at.unwrap_or(chrono::Utc::now()), - completed_job.priority, - ) - .warn_after_seconds(10) - .await - { - if !success { - tracing::error!("Could not apply schedule error handler: {}", err); - let base_url = windmill_common::BASE_URL.load(); - let w_id: &String = &completed_job.workspace_id; - if !matches!(err, Error::QuotaExceeded(_)) { - report_error_to_workspace_handler_or_critical_side_channel( - &completed_job, - db, - format!( - "Failed to push schedule error handler job to handle failed job ({base_url}/run/{}?workspace={w_id}): {}", - completed_job.id, - err - ), - ) - .warn_after_seconds(10) - .await; + if !retry_pending { + if let Err(err) = crate::jobs_ee::apply_schedule_handlers( + db, + &schedule, + &script_path, + &completed_job.workspace_id, + success, + result, + job_id, + // Current occurrence's root: the terminal native-retry + // attempt's parent, else the job itself. + completed_job.parent_job.unwrap_or(job_id), + completed_job.started_at.unwrap_or(chrono::Utc::now()), + completed_job.priority, + ) + .warn_after_seconds(10) + .await + { + if !success { + tracing::error!("Could not apply schedule error handler: {}", err); + let base_url = windmill_common::BASE_URL.load(); + let w_id: &String = &completed_job.workspace_id; + if !matches!(err, Error::QuotaExceeded(_)) { + report_error_to_workspace_handler_or_critical_side_channel( + &completed_job, + db, + format!( + "Failed to push schedule error handler job to handle failed job ({base_url}/run/{}?workspace={w_id}): {}", + completed_job.id, + err + ), + ) + .warn_after_seconds(10) + .await; + } + } else { + tracing::error!("Could not apply schedule recovery handler: {}", err); } - } else { - tracing::error!("Could not apply schedule recovery handler: {}", err); - } - }; + }; + } + #[cfg(not(all(feature = "enterprise", feature = "private")))] + let _ = retry_pending; } else { tracing::error!( "Schedule {schedule_path} in {} not found. Impossible to schedule again and apply schedule handlers", @@ -1337,16 +1395,7 @@ async fn commit_completed_job( } } - if completed_job.concurrent_limit.is_some() - || windmill_common::runnable_settings::prefetch_cached_from_handle( - completed_job.runnable_settings_handle, - db, - ) - .await? - .1 - .concurrent_limit - .is_some() - { + if has_concurrent_limit { let concurrency_key = sqlx::query_scalar!( "SELECT key FROM concurrency_key WHERE job_id = $1", &completed_job.id @@ -1589,6 +1638,267 @@ async fn restart_job_if_perpetual_inner( Ok(()) } +/// Evaluate a `retry_if` JS expression. `result`/`previous_result` are the +/// failure output and `flow_input` the job args. Defaults to retrying on eval +/// error (an unevaluable gate shouldn't silently swallow retries). +#[cfg(feature = "quickjs")] +async fn eval_retry_if( + expr: &str, + result: Option<&serde_json::value::RawValue>, + args: &HashMap>, +) -> bool { + let result_val = result + .and_then(|r| serde_json::from_str::(r.get()).ok()) + .unwrap_or(serde_json::Value::Null); + let mut globals = HashMap::new(); + globals.insert("result".to_string(), result_val.clone()); + globals.insert("previous_result".to_string(), result_val); + globals.insert( + "flow_input".to_string(), + serde_json::to_value(args).unwrap_or(serde_json::Value::Null), + ); + match windmill_jseval::eval_simple_js(format!("Boolean({expr})"), globals).await { + Ok(v) => v.get() == "true", + Err(e) => { + tracing::warn!("Failed to evaluate retry_if expression, retrying anyway: {e:#}"); + true + } + } +} + +/// `retry_if` is unsupported on a worker built without the `quickjs` feature +/// (the expression cannot be evaluated). Such a worker can't run JS jobs either, +/// so this is not reached in practice; we fail closed and do not retry. +#[cfg(not(feature = "quickjs"))] +async fn eval_retry_if( + _expr: &str, + _result: Option<&serde_json::value::RawValue>, + _args: &HashMap>, +) -> bool { + tracing::warn!("retry_if is unsupported without the quickjs feature; not retrying"); + false +} + +/// Native script retry. When a failed `Script` job carries a retry policy (via +/// `runnable_settings_handle`) and has attempts left, enqueue a fresh attempt of +/// the same script after the policy's backoff delay — instead of having wrapped +/// it in a one-step flow. Each attempt is a real `Script` job; the attempt +/// counter lives in the `native_retry_attempt` marker, written here and read only +/// on the next failure (never on the hot job-pull path). +/// +/// Returns `true` if a retry was enqueued. +/// +/// Authorization: this performs no auth check by design. It is `pub` only so the +/// integration test can reach it; the sole production caller is the worker +/// job-completion path (`add_completed_job`), which passes a `MiniCompletedJob` +/// built from a real, already-persisted completed job — its workspace/identity +/// fields come from the DB, not from request input. Callers MUST uphold this: +/// never invoke it with caller-supplied or unauthorized job identity. +pub async fn maybe_enqueue_native_script_retry( + db: &Pool, + job: &MiniCompletedJob, + canceled_by: &Option, + // Lazily serialize the failure result: only `retry_if` policies need it, so + // the common (no-retry_if) failure never pays the serialization cost. + result_fn: &(dyn Fn() -> Option> + Sync), +) -> Result { + // Only plain top-level scripts retry natively; cancellation always wins. + if canceled_by.is_some() || !matches!(job.kind, JobKind::Script) || job.is_flow_step() { + return Ok(false); + } + + let Some(retry_settings) = windmill_common::runnable_settings::prefetch_retry_from_handle( + job.runnable_settings_handle, + db, + ) + .await? + else { + return Ok(false); + }; + let policy: Retry = retry_settings.into(); + if !policy.has_attempts() { + return Ok(false); + } + + // Attempt counter for this job: its `native_retry_attempt` marker, or 0 for + // the first (un-marked) attempt. The marker is persistent (unlike the queue + // row), so it doubles as the explicit "this job is a retry attempt" signal + // consumers key off of. + let prev_attempts = sqlx::query_scalar!( + "SELECT attempt FROM native_retry_attempt WHERE job_id = $1", + job.id, + ) + .fetch_optional(db) + .await? + .unwrap_or(0) as u32; + let root = job.parent_job.unwrap_or(job.id); + // Scheduled chains keep the schedule trigger so the terminal attempt drives + // the schedule completion handlers (on_failure/on_success); `parent_job` + // keeps every retry out of the per-occurrence handler counting queries. + let trigger = job + .schedule_path() + .map(|sp| TriggerMetadata::new(Some(sp), JobTriggerKind::Schedule)); + + let Some(delay) = policy.interval(prev_attempts, false) else { + // Attempts exhausted — let the failure finalize normally. + return Ok(false); + }; + // Cap the backoff to match the flow-runtime retry path (evaluate_retry). + let delay = std::cmp::min(delay, MAX_RETRY_INTERVAL); + let scheduled_for = chrono::Utc::now() + + chrono::Duration::from_std(delay).unwrap_or_else(|_| chrono::Duration::zero()); + + let args = sqlx::query_scalar!( + "SELECT args as \"args: sqlx::types::Json>>\" FROM v2_job WHERE id = $1 AND workspace_id = $2", + job.id, + job.workspace_id, + ) + .fetch_optional(db) + .await? + .flatten() + .unwrap_or_default(); + + // Optional `retry_if`: gate the retry on a JS expression over the failure + // `result` and `flow_input` (the job args). Evaluated by `eval_retry_if`, + // which on a worker built without the `quickjs` feature cannot evaluate the + // expression and fails closed (no retry). + if let Some(retry_if) = policy.retry_if.as_ref() { + let result = result_fn(); + if !eval_retry_if(&retry_if.expr, result.as_deref(), &args.0).await { + return Ok(false); + } + } + + // Re-push as a one-step-flow request; `push` materializes it back into a + // native retryable `Script` carrying the policy again. `parent_job = root` + // links the chain (and excludes retries from schedule-handler counting); the + // schedule trigger, when present, lets the terminal attempt fire handlers. + // + // Idempotent retry id: deterministic per (root, next attempt). If a worker + // dies between this push and the current attempt's finalization, the reaper + // re-handles the un-finalized attempt and we land here again with the same + // id, so the retry is enqueued exactly once (no double-retry). + let retry_job_id = { + use std::hash::{Hash, Hasher}; + let next_attempt = prev_attempts + 1; + let mut high = std::hash::DefaultHasher::new(); + root.hash(&mut high); + next_attempt.hash(&mut high); + let mut low = std::hash::DefaultHasher::new(); + "native-retry".hash(&mut low); + next_attempt.hash(&mut low); + root.hash(&mut low); + Uuid::from_u64_pair(high.finish(), low.finish()) + }; + // If that retry already exists (the crash-and-reaper-replay case above), + // report it as pending WITHOUT re-pushing. Deriving `retry_pending` from the + // push *result* would otherwise flip to false on the duplicate-id error and + // let the schedule completion handlers fire for this non-terminal attempt. + if sqlx::query_scalar!("SELECT 1 FROM v2_job WHERE id = $1", retry_job_id) + .fetch_optional(db) + .await? + .is_some() + { + return Ok(true); + } + // Carry forward the failed attempt's concurrency/debouncing settings (same + // runnable_settings_handle as the retry policy) so a retry of a concurrency- + // limited script still inserts its concurrency_key and respects the limit, + // rather than running unbounded with only the retry policy. + let (debouncing_settings, concurrency_settings) = + windmill_common::runnable_settings::prefetch_cached_from_handle( + job.runnable_settings_handle, + db, + ) + .await?; + let tx = PushIsolationLevel::IsolatedRoot(db.clone()); + let (new_id, mut tx) = match push( + db, + tx, + &job.workspace_id, + JobPayload::SingleStepFlow { + path: job.runnable_path.clone().unwrap_or_default(), + hash: job.runnable_id, + flow_version: None, + language: job.script_lang.clone(), + args: HashMap::new(), + retry: Some(policy), + error_handler_path: None, + error_handler_args: None, + skip_handler: None, + cache_ttl: job.cache_ttl, + cache_ignore_s3_path: job.cache_ignore_s3_path, + priority: job.priority, + tag_override: Some(job.tag.clone()), + trigger_path: None, + apply_preprocessor: false, + concurrency_settings, + debouncing_settings, + }, + PushArgs::from(&args.0), + &job.created_by, + &job.permissioned_as_email, + job.permissioned_as.clone(), + Some(&format!("retry.{}", job.id)), + Some(scheduled_for), + None, + Some(root), + None, + None, + Some(retry_job_id), + false, + false, + None, + true, + Some(job.tag.clone()), + None, + None, + job.priority, + None, + false, + None, + trigger, + None, + ) + .await + { + Ok(v) => v, + Err(e) => { + // Race with a concurrent completion of the same attempt: it may have + // inserted the deterministic retry id between our pre-check and this + // push, so the push fails on the duplicate id. The retry IS pending — + // re-check and report it as such instead of propagating the error + // (which would flip `retry_pending` to false and fire the schedule + // completion handlers for this non-terminal attempt). + if sqlx::query_scalar!("SELECT 1 FROM v2_job WHERE id = $1", retry_job_id) + .fetch_optional(db) + .await? + .is_some() + { + return Ok(true); + } + return Err(e); + } + }; + + sqlx::query!( + "INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, $2)", + new_id, + (prev_attempts + 1) as i32, + ) + .execute(&mut *tx) + .await?; + + tx.commit().await?; + tracing::info!( + "Native retry: enqueued attempt {} of script {:?} (root {root}) in {}s as {new_id}", + prev_attempts + 1, + job.runnable_path, + delay.as_secs(), + ); + Ok(true) +} + #[cfg(feature = "cloud")] fn apply_completed_job_cloud_usage( db: &Pool, @@ -1604,22 +1914,31 @@ fn apply_completed_job_cloud_usage( tokio::task::spawn(async move { let additional_usage = _duration / 1000; let result = tokio::time::timeout(std::time::Duration::from_secs(10), async move { + // Fork/dev execution-seconds meter against the root (billing) workspace; resolves to + // `w_id` itself off-fork. + let billing_w_id = + windmill_common::workspaces::get_billing_workspace_id(&db, &w_id) + .await + .unwrap_or_else(|e| { + tracing::error!("Failed to resolve billing workspace for {w_id}: {e:#}"); + w_id.clone() + }); // Update workspace usage let workspace_result = sqlx::query!( "INSERT INTO usage (id, is_workspace, month_, usage) VALUES ($1, TRUE, EXTRACT(YEAR FROM current_date) * 12 + EXTRACT(MONTH FROM current_date), $2) ON CONFLICT (id, is_workspace, month_) DO UPDATE SET usage = usage.usage + EXCLUDED.usage", - &w_id, + &billing_w_id, additional_usage as i32 ) .execute(&db) .await; if let Err(e) = workspace_result { - tracing::error!("Failed to update workspace usage for {}: {:#}", w_id, e); + tracing::error!("Failed to update workspace usage for {}: {:#}", billing_w_id, e); } - match windmill_common::workspaces::get_team_plan_status(&db, &w_id).await { + match windmill_common::workspaces::get_team_plan_status(&db, &billing_w_id).await { Ok(team_plan_status) => { // Update user usage for non-premium workspaces if !team_plan_status.premium { @@ -1995,7 +2314,7 @@ pub async fn try_schedule_next_job<'c>( let email = match windmill_common::users::get_email_from_permissioned_as( &permissioned_as, &job.workspace_id, - db, + &mut *tx, ) .await { @@ -3172,108 +3491,189 @@ impl PulledJobResult { if let Some(args) = &mut j.args { args.remove(field_name); } + + // No accumulation on this path: just clean up the batch rows. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j_id, + ) + .execute(db) + .await?; } else if let Some(arg_name_to_accumulate) = // TODO: Maybe support multiple arguments in future debounce_args_to_accumulate.as_ref().and_then(|v| v.get(0)) { - tracing::debug!( - job_id = %j_id, - job_kind = ?kind, - arg_name = arg_name_to_accumulate, - "Accumulating debounced arguments from batch" - ); - let mut accumulated_arg: Vec> = vec![]; - for str_o in sqlx::query_scalar!( - "WITH ids AS ( - SELECT id as job_id FROM v2_job_debounce_batch WHERE debounce_batch = ( - SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 - ) - ) SELECT args->>$2 FROM ids LEFT JOIN v2_job ON v2_job.id = ids.job_id + // Claim this job's contribution to its debounce batch exactly once. + // Instead of deleting the batch rows, mark them consumed (stamping + // consumed_by = this job). A batch normally has a single survivor that + // sweeps every row; only a narrow push/pull race can leave two survivors + // on one batch. The claim lets the second survivor tell apart: + // - already swept in by the other survivor -> run empty (no duplicate), + // - its own earlier claim on re-pull -> keep its accumulated args, + // - never batched (CE / workers behind v2) -> keep its own args. + // Consumed rows are GC'd by the monitor. + // Claim + accumulate + persist atomically: a crash between stamping the + // batch rows consumed_by=self and persisting the merged args would + // otherwise let a zombie re-pull see its own prior claim and keep only + // its own args (dropping the siblings it had claimed). One transaction + // makes the claim and the merged-args write commit together (or neither). + let mut tx = db.begin().await?; + // Emitted AFTER the transaction commits — writing logs via a second pool + // connection while the claim tx + row locks are held risks pool-exhaustion + // stalls under concurrent debounced pulls. + let mut accumulation_log: Option = None; + let claim = sqlx::query!( + "WITH mine AS ( + SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1 + ), claimed AS ( + -- Claim the whole batch in ONE update so concurrent same-batch + -- survivors lock rows in identical scan order (no lock-ordering + -- deadlock); each re-evaluates `consumed_at IS NULL` under EvalPlanQual + -- and skips rows the other already took. A claim therefore consumes + -- every still-unclaimed row of the batch atomically. + UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1 + WHERE debounce_batch = (SELECT debounce_batch FROM mine) + AND consumed_at IS NULL + RETURNING id + ) + SELECT + EXISTS (SELECT 1 FROM mine) AS \"had_row!\", + (SELECT consumed_by FROM mine) AS prev_consumed_by, + ARRAY(SELECT id FROM claimed) AS \"claimed_ids!\", + EXISTS (SELECT 1 FROM claimed WHERE id = $1) AS \"claimed_self!\" ", j_id, - arg_name_to_accumulate, ) - .fetch_all(db) - .await? - .into_iter() - { - if let Some(s) = str_o.as_ref() { - match serde_json::from_str::>>(s) { - Ok(ref mut vec) => accumulated_arg.append(vec), - Err(_) => { - // Value is not an array — wrap the scalar into a - // single-element array. This supports union types - // like T | T[] where the caller may pass a bare T. - match RawValue::from_string(s.to_string()) { - Ok(raw) => accumulated_arg.push(raw), - Err(e) => { - return Err(error::Error::ArgumentErr(format!("cannot consolidate argument `{arg_name_to_accumulate}`: value is neither a valid list nor a valid JSON value\nUnwrapped Error: {e}"))); + .fetch_one(&mut *tx) + .await?; + + if !claim.had_row { + // Never batched (CE / workers behind v2): keep the job's own args. + tracing::debug!( + job_id = %j_id, + "Debounce: no batch row, keeping original args" + ); + } else if claim.claimed_self { + // We claimed our own row; since a claim takes the whole batch, this also + // swept any not-yet-claimed siblings. Accumulate exactly the rows we own. + let ids = claim.claimed_ids; + + tracing::debug!( + job_id = %j_id, + job_kind = ?kind, + arg_name = arg_name_to_accumulate, + claimed = ids.len(), + "Accumulating debounced arguments from claimed batch rows" + ); + + let mut accumulated_arg: Vec> = vec![]; + for str_o in sqlx::query_scalar!( + "SELECT args->>$2 FROM v2_job WHERE id = ANY($1)", + &ids, + arg_name_to_accumulate, + ) + .fetch_all(&mut *tx) + .await? + .into_iter() + { + if let Some(s) = str_o.as_ref() { + match serde_json::from_str::>>(s) { + Ok(ref mut vec) => accumulated_arg.append(vec), + Err(_) => { + // Value is not an array — wrap the scalar into a + // single-element array. This supports union types + // like T | T[] where the caller may pass a bare T. + match RawValue::from_string(s.to_string()) { + Ok(raw) => accumulated_arg.push(raw), + Err(e) => { + return Err(error::Error::ArgumentErr(format!("cannot consolidate argument `{arg_name_to_accumulate}`: value is neither a valid list nor a valid JSON value\nUnwrapped Error: {e}"))); + } } } } } } - } - tracing::debug!( - job_id = %j_id, - arg_name = arg_name_to_accumulate, - accumulated_count = accumulated_arg.len(), - "Accumulated arguments from debounced jobs in batch" - ); + if !accumulated_arg.is_empty() { + let new_value = to_raw_value(&accumulated_arg); - // If the batch query returned no entries (e.g. CE where - // v2_job_debounce_batch is never populated), keep the - // original value unchanged instead of replacing it with []. - if !accumulated_arg.is_empty() { - let new_value = to_raw_value(&accumulated_arg); + let original_value = j + .args + .as_ref() + .and_then(|a| a.get(arg_name_to_accumulate)) + .map(|v| v.get().to_string()) + .unwrap_or_else(|| "null".to_string()); - let original_value = j - .args - .as_ref() - .and_then(|a| a.get(arg_name_to_accumulate)) - .map(|v| v.get().to_string()) - .unwrap_or_else(|| "null".to_string()); - - append_logs( - &j_id, - &j.workspace_id, - format!( + accumulation_log = Some(format!( "Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n", &new_value - ), - &(db.into()), - ) - .await; + )); + j.args + .get_or_insert(Json(Default::default())) + .as_mut() + .insert(arg_name_to_accumulate.to_owned(), new_value); + + // Persist accumulated args to v2_job so that flow steps + // re-reading from the DB (via get_mini_pulled_job) see them + if let Some(ref args) = j.args { + sqlx::query!( + "UPDATE v2_job SET args = $2 WHERE id = $1", + j_id, + args as &Json>>, + ) + .execute(&mut *tx) + .await?; + } + } + } else if claim.prev_consumed_by == Some(j_id) { + // Our own prior claim seen again on a re-pull (e.g. crash recovery): + // keep the args we already persisted on the first pull. + } else { + // Another survivor already accumulated this job's contribution + // (consumed_by a different job); run as a no-op so its items are not + // reprocessed. + tracing::info!( + job_id = %j_id, + arg_name = arg_name_to_accumulate, + "Debounce: contribution already consumed by a concurrent survivor, running empty" + ); j.args .get_or_insert(Json(Default::default())) .as_mut() - .insert(arg_name_to_accumulate.to_owned(), new_value); - - // Persist accumulated args to v2_job so that flow steps - // re-reading from the DB (via get_mini_pulled_job) see them + .insert( + arg_name_to_accumulate.to_owned(), + to_raw_value(&Vec::>::new()), + ); if let Some(ref args) = j.args { sqlx::query!( "UPDATE v2_job SET args = $2 WHERE id = $1", j_id, args as &Json>>, ) - .execute(db) + .execute(&mut *tx) .await?; } } - } + tx.commit().await?; - // Clean up the debounce batch entries now that the job has been pulled - sqlx::query!( - "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( - SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 - )", - j_id, - ) - .execute(db) - .await?; + if let Some(msg) = accumulation_log { + append_logs(&j_id, &j.workspace_id, msg, &(db.into())).await; + } + } else { + // Debounced but no args to accumulate (plain debounce / dependency job): + // consume the batch by removing this job's rows. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j_id, + ) + .execute(db) + .await?; + } // Handle dependency job debouncing cleanup when a job is pulled for execution if is_djob_to_debounce { @@ -4733,8 +5133,13 @@ async fn push_inner<'c, 'd>( ) -> Result<(Uuid, Transaction<'c, Postgres>), Error> { #[cfg(feature = "cloud")] if *CLOUD_HOSTED { + // A fork/dev workspace draws its plan and usage from the root (billing) workspace, so its + // executions are metered against the parent's quota/bill. Resolves to `workspace_id` itself + // for a standalone workspace (no behavior change off-fork). + let billing_w_id = + windmill_common::workspaces::get_billing_workspace_id(db, workspace_id).await?; let team_plan_status = - windmill_common::workspaces::get_team_plan_status(db, workspace_id).await?; + windmill_common::workspaces::get_team_plan_status(db, &billing_w_id).await?; // we track only non flow steps let (workspace_usage, user_usage) = if !matches!( job_payload, @@ -4743,12 +5148,12 @@ async fn push_inner<'c, 'd>( // Check current usage with SELECT (fast, no row locks) // Only check user usage for non-premium workspaces let (current_workspace_usage, current_user_usage) = - check_usage_limits(db, workspace_id, email, !team_plan_status.premium).await?; + check_usage_limits(db, &billing_w_id, email, !team_plan_status.premium).await?; // Spawn async task to update usage counters in the background increment_usage_async( db.clone(), - workspace_id.to_string(), + billing_w_id.clone(), if !team_plan_status.premium { Some(email.to_string()) } else { @@ -4851,7 +5256,7 @@ async fn push_inner<'c, 'd>( WHERE is_workspace IS TRUE AND month_ = EXTRACT(YEAR FROM current_date) * 12 + EXTRACT(MONTH FROM current_date) AND id = $1", - workspace_id + billing_w_id ) .fetch_optional(db) .await? @@ -4878,6 +5283,13 @@ async fn push_inner<'c, 'd>( ))); } + // These two burst guards intentionally stay keyed to `workspace_id`, not the + // billing root: the shared caps are the monthly usage above (metered to the + // root) and the per-user in-queue/concurrent guards further up (keyed by email, + // global across the whole family). Keying these to the root would count the + // root's own queue rather than this workspace's load or the family's; a true + // family-shared burst cap would need a family-wide subquery, not worth the + // hot-path cost for this downgrade-only soft guard. let in_queue_workspace = sqlx::query_scalar!( "SELECT COUNT(id) FROM v2_job_queue WHERE workspace_id = $1", workspace_id @@ -4926,6 +5338,7 @@ async fn push_inner<'c, 'd>( _low_level_priority: Option, concurrency_settings: ConcurrencySettings, debouncing_settings: DebouncingSettings, + retry_settings: RetrySettings, labels: Option>, } let mut preprocessed = None; @@ -4944,6 +5357,7 @@ async fn push_inner<'c, 'd>( _low_level_priority, mut concurrency_settings, debouncing_settings, + retry_settings, labels, } = match job_payload { JobPayload::ScriptHash { @@ -5250,6 +5664,7 @@ async fn push_inner<'c, 'd>( path, hash, flow_version, + language, retry, error_handler_path, error_handler_args, @@ -5263,10 +5678,71 @@ async fn push_inner<'c, 'd>( apply_preprocessor, debouncing_settings, concurrency_settings, - } => { + } => 'ssf: { // Determine if this is a flow or a script let is_flow = flow_version.is_some(); + // Native retry: a bare script wrapped only to gain a retry (no + // skip/error-handler modules) is pushed as a real `Script` job that + // carries the policy in `runnable_settings`. This avoids spawning a + // one-step flow — and its extra job rows, flow_status, and UI + // projection — for the common schedule/pipeline retry case. The flow + // path below is kept only for handler-bearing or flow-wrapping cases. + // + // Gated on the runnable-settings min version: on a mixed-version + // fleet the policy can't be persisted (`insert_rs` would drop it), so + // we fall back to the flow wrapper to preserve retry semantics. + // `retry_if` is always materialized natively and evaluated on the + // failure path (see `eval_retry_if`). On a worker built without the + // `quickjs` feature it cannot be evaluated and fails closed (no retry); + // the flow path is not a fallback, since the flow runtime needs quickjs + // too. + let native_retry = !is_flow + && skip_handler.is_none() + && error_handler_path.is_none() + && hash.is_some() + && language.is_some() + && windmill_common::runnable_settings::min_version_supports_runnable_settings_v0() + .await; + if native_retry { + if apply_preprocessor { + preprocessed = Some(false); + } + // Preserve worker affinity: normal ScriptHash pushes carry the + // script's `dedicated_worker` (it drives the dedicated tag below), + // but the SingleStepFlow payload doesn't — resolve it from the + // script row so a dedicated-worker script keeps its dedicated pool. + let dedicated_worker = if let Some(h) = &hash { + // Read on the non-RLS pool: push_inner is also entered with RLS + // isolation variants under which the script row may be invisible, + // which would mis-resolve dedicated_worker routing. + sqlx::query_scalar::<_, Option>( + "SELECT dedicated_worker FROM script WHERE hash = $1 AND workspace_id = $2", + ) + .bind(h.0) + .bind(workspace_id) + .fetch_optional(db) + .await? + .flatten() + } else { + None + }; + break 'ssf JobPayloadUntagged { + runnable_id: hash.map(|h| h.0), + runnable_path: Some(path), + job_kind: JobKind::Script, + language, + dedicated_worker, + concurrency_settings, + debouncing_settings, + retry_settings: retry.as_ref().map(RetrySettings::from).unwrap_or_default(), + cache_ttl, + cache_ignore_s3_path, + _low_level_priority: priority, + ..Default::default() + }; + } + // Build modules list let mut modules = vec![]; @@ -5903,6 +6379,7 @@ async fn push_inner<'c, 'd>( RunnableSettings { debouncing_settings: debouncing_settings.insert_cached(db).await?, concurrency_settings: concurrency_settings.insert_cached(db).await?, + retry_settings: retry_settings.insert_cached(db).await?, }, db, ) diff --git a/backend/windmill-queue/src/lib.rs b/backend/windmill-queue/src/lib.rs index 6f689c7026..8d8534c926 100644 --- a/backend/windmill-queue/src/lib.rs +++ b/backend/windmill-queue/src/lib.rs @@ -14,6 +14,20 @@ pub mod cascade_oss; pub use cascade_ee as cascade; #[cfg(not(feature = "private"))] pub use cascade_oss as cascade; +#[cfg(feature = "private")] +pub mod ducklake_maintenance_ee; +pub mod ducklake_maintenance_oss; +#[cfg(feature = "private")] +pub use ducklake_maintenance_ee as ducklake_maintenance; +#[cfg(not(feature = "private"))] +pub use ducklake_maintenance_oss as ducklake_maintenance; +#[cfg(feature = "private")] +pub mod freshness_watchdog_ee; +pub mod freshness_watchdog_oss; +#[cfg(feature = "private")] +pub use freshness_watchdog_ee as freshness_watchdog; +#[cfg(not(feature = "private"))] +pub use freshness_watchdog_oss as freshness_watchdog; pub mod jobs; #[cfg(feature = "private")] pub mod jobs_ee; diff --git a/backend/windmill-queue/src/schedule.rs b/backend/windmill-queue/src/schedule.rs index 3f6c587be2..2b5aef96a5 100644 --- a/backend/windmill-queue/src/schedule.rs +++ b/backend/windmill-queue/src/schedule.rs @@ -227,11 +227,30 @@ pub async fn push_scheduled_job<'c>( } } + // Managed ducklake maintenance schedule (enterprise): the runnable is a + // generated DuckDB script, not a deployed one — built in the EE module. + // None (CE build, or no enabled maintenance config for the path's lake) + // falls through to normal script resolution, so a user schedule that + // pre-dates the reserved prefix keeps running its script and a stale + // managed row fails resolution with NotFound (auto-disabling it with + // schedule.error recorded). + let maintenance_payload = + if windmill_common::workspaces::lake_from_ducklake_maintenance_path(&schedule.path) + .is_some() + { + crate::ducklake_maintenance::build_maintenance_schedule_payload(&mut tx, schedule) + .await? + } else { + None + }; + // If schedule handler is defined, wrap the scheduled job in a synthetic flow // with the handler as the first step (with stop_after_if to skip if handler returns false) - let (payload, tag, timeout, on_behalf_of_email, created_by) = if let Some(handler_path) = - &schedule.dynamic_skip + let (payload, tag, timeout, on_behalf_of_email, created_by) = if let Some(maintenance_payload) = + maintenance_payload { + maintenance_payload + } else if let Some(handler_path) = &schedule.dynamic_skip { // Build skip handler args let mut skip_handler_args = HashMap::>::new(); skip_handler_args.insert( @@ -255,6 +274,7 @@ pub async fn push_scheduled_job<'c>( path: schedule.script_path.clone(), hash, flow_version, + language: None, args: args.clone(), retry, error_handler_path: None, @@ -352,6 +372,11 @@ pub async fn push_scheduled_job<'c>( .warn_after_seconds_with_sql(1, "get_latest_hash_for_path".to_string()) .await?; + // NB: read on the non-RLS pool (`db`), not `tx`. push_scheduled_job is + // also invoked with an RLS user_db transaction (api-schedule/api-flows), + // under which these lookups would resolve against the caller's row + // visibility rather than the full table. The dual-connection here is + // intentional and required for correctness. let (debouncing_settings, concurrency_settings) = windmill_common::runnable_settings::prefetch_cached_from_handle( runnable_settings_handle, @@ -371,12 +396,20 @@ pub async fn push_scheduled_job<'c>( for (arg_name, arg_value) in args.clone() { static_args.insert(arg_name, arg_value); } - // if retry is set, we wrap the script into a one step flow with a retry on the module + // A retry on a scheduled script is materialized into a native retry + // (see `push`): `Some(language)` opts in. Completion handlers are + // driven from the terminal attempt, and the per-occurrence + // failure/recovery counting queries (apply_schedule_handlers) resolve + // terminal status across the retry chain — so on_failure/on_recovery + // (incl. multi-count/exact) are all handled. A `retry_if` gate is + // evaluated at failure time; on a worker built without quickjs it + // cannot be evaluated and fails closed (no retry). ( JobPayload::SingleStepFlow { path: schedule.script_path.clone(), hash: Some(hash), flow_version: None, + language: Some(language), retry: Some(parsed_retry), error_handler_path: None, error_handler_args: None, @@ -388,8 +421,12 @@ pub async fn push_scheduled_job<'c>( tag_override: schedule.tag.clone(), trigger_path: None, apply_preprocessor: false, - concurrency_settings: ConcurrencySettings::default(), - debouncing_settings: DebouncingSettings::default(), + // Carry the script's concurrency/debounce settings (fetched + // above) into the native retry materialization, so a retrying + // concurrency-limited scheduled script still inserts its + // concurrency_key instead of running unbounded. + concurrency_settings, + debouncing_settings, }, if schedule.tag.as_ref().is_some_and(|x| x != "") { schedule.tag.clone() @@ -497,7 +534,7 @@ pub async fn push_scheduled_job<'c>( if let Some(tag) = tag.as_deref().filter(|t| !t.is_empty()) { check_tag_available_for_workspace_internal( - &db, + db, &schedule.workspace_id, &tag, &email, @@ -600,7 +637,9 @@ pub async fn clear_schedule<'c>( w_id: &str, ) -> Result<()> { tracing::info!("Clearing schedule {}", path); - sqlx::query!( + // Delete the queued jobs (cascading their v2_job_queue-keyed side tables), then route the + // freed ids through delete_jobs so v2_job and its no-longer-cascading side tables go too. + let deleted_ids: Vec = sqlx::query_scalar!( "WITH to_delete AS ( SELECT id FROM v2_job_queue JOIN v2_job j USING (id) @@ -610,15 +649,16 @@ pub async fn clear_schedule<'c>( AND flow_step_id IS NULL AND running = false FOR UPDATE - ), deleted AS ( - DELETE FROM v2_job_queue - WHERE id IN (SELECT id FROM to_delete) - RETURNING id - ) DELETE FROM v2_job WHERE id IN (SELECT id FROM deleted)", + ) + DELETE FROM v2_job_queue + WHERE id IN (SELECT id FROM to_delete) + RETURNING id", path, w_id ) - .execute(&mut **tx) + .fetch_all(&mut **tx) .await?; + + windmill_common::jobs::delete_jobs(&mut **tx, &deleted_ids).await?; Ok(()) } diff --git a/backend/windmill-queue/src/tags.rs b/backend/windmill-queue/src/tags.rs index a0cc3a305e..d80e50d8b2 100644 --- a/backend/windmill-queue/src/tags.rs +++ b/backend/windmill-queue/src/tags.rs @@ -8,13 +8,21 @@ const FORK_PARENT_CACHE_TTL_SECS: u64 = 300; lazy_static::lazy_static! { // Cache of fork workspace id -> (parent_workspace_id, cached_at). - // `parent_workspace_id` is essentially immutable once a fork is created, so a multi-minute TTL - // is safe. `None` means the lookup found no parent (or the DB call failed); we still cache it - // briefly so that forks missing a parent do not hammer the DB. + // `parent_workspace_id` is stable for the lifetime of a fork EXCEPT across attach/detach of a + // dev workspace, which set/keep it; those paths call `invalidate_fork_parent_cache` so routing + // doesn't lag. `None` means the lookup found no parent (or the DB call failed); we still cache + // it briefly so that forks missing a parent do not hammer the DB. static ref FORK_PARENT_CACHE: quick_cache::sync::Cache, std::time::Instant)> = quick_cache::sync::Cache::new(500); } +/// Drop the cached fork->parent mapping for a workspace. Call after mutating `parent_workspace_id` +/// (attaching/detaching a dev workspace) so per-workspace job tags resolve to the new parent +/// immediately instead of after the cache TTL. +pub fn invalidate_fork_parent_cache(workspace_id: &str) { + FORK_PARENT_CACHE.remove(workspace_id); +} + /// Returns `Some(effective_workspace_tag_id)` if jobs of `workspace_id` should use workspace- /// specific tags, where `effective_workspace_tag_id` is the string embedded in the tag. For forks, /// this is always the parent workspace id, optionally suffixed with `-fork` (controlled by the @@ -26,16 +34,14 @@ pub async fn per_workspace_tag(workspace_id: &str, db: &Pool) -> Optio return None; } - let is_fork = workspace_id.starts_with(WM_FORK_PREFIX); - - // For forks, always resolve to the parent workspace id; regular workspaces avoid the lookup. - let effective_ws_id: String = if is_fork { - lookup_fork_parent(workspace_id, db) - .await - .unwrap_or_else(|| workspace_id.to_string()) // no parent found -> fall back to fork's own id - } else { - workspace_id.to_string() - }; + // Resolve to the parent workspace id when the workspace is a fork or dev workspace (both set + // parent_workspace_id). The lookup caches its `None` result, so non-forks stay cheap after warmup + // (and the common case is already short-circuited by the global toggle above). + let parent = lookup_fork_parent(workspace_id, db).await; + // A `wm-fork-` workspace can outlive its parent (the FK is `ON DELETE SET NULL`), so keep + // treating the prefix as fork-ness for the `-fork` suffix even when the parent link is gone. + let is_fork = parent.is_some() || workspace_id.starts_with(WM_FORK_PREFIX); + let effective_ws_id: String = parent.unwrap_or_else(|| workspace_id.to_string()); // Whitelist check is against the resolved (parent) id so that including a parent in the // whitelist transparently covers all of its forks. @@ -62,8 +68,10 @@ pub async fn per_workspace_tag(workspace_id: &str, db: &Pool) -> Optio }) } -/// Returns the parent workspace id for a fork, or `None` if the fork has no parent set (or the -/// DB lookup failed). Backed by a short-TTL cache to avoid a DB round-trip per job push. +/// Returns the parent workspace id for a fork, or `None` if the fork has no parent set. Backed by a +/// short-TTL cache to avoid a DB round-trip per job push. A transient DB error returns `None` for +/// this call but is NOT cached, so the next push retries instead of misrouting a (prefix-less) dev +/// workspace's jobs for the whole TTL. async fn lookup_fork_parent(fork_id: &str, db: &Pool) -> Option { if let Some((parent, cached_at)) = FORK_PARENT_CACHE.get(fork_id) { if cached_at.elapsed().as_secs() < FORK_PARENT_CACHE_TTL_SECS { @@ -78,8 +86,11 @@ async fn lookup_fork_parent(fork_id: &str, db: &Pool) -> Option Some(parent), - _ => None, + Ok(opt) => opt.flatten(), + Err(e) => { + tracing::warn!("failed to look up fork parent for {fork_id}: {e:#}"); + return None; + } }; FORK_PARENT_CACHE.insert( diff --git a/backend/windmill-queue/tests/debounce_test.rs b/backend/windmill-queue/tests/debounce_test.rs index a2c6971c1c..d74f20ff3f 100644 --- a/backend/windmill-queue/tests/debounce_test.rs +++ b/backend/windmill-queue/tests/debounce_test.rs @@ -3364,6 +3364,7 @@ mod debounce { let rs = RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: concurrency_hash, + retry_settings: None, }; let rs_handle = insert_rs(rs, &db).await?; @@ -3741,6 +3742,7 @@ mod debounce { RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: concurrency_hash, + retry_settings: None, }, db, ) @@ -3878,6 +3880,1577 @@ mod debounce { Ok(()) } + /// Helper: push a script job through push-time `maybe_debounce` with the given key. + /// Returns the args JSON it was pushed with. + async fn push_debounced_script( + db: &Pool, + id: Uuid, + items: Vec, + settings: &DebouncingSettings, + rs_handle: Option, + ) -> serde_json::Value { + let args_val = serde_json::json!({ "items": items }); + insert_script_job_with_args(db, id, "test-workspace", "f/test/script", &args_val).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(db) + .await + .unwrap(); + let args_hm: HashMap> = + serde_json::from_value(args_val.clone()).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &push_args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + args_val + } + + /// Regression test for the "running survivor" data-loss bug. + /// + /// When a debounce survivor has already been pulled and is executing (running), it + /// has committed its batch and can no longer accumulate later arrivals. The old + /// behavior superseded the running survivor anyway: it was completed/skipped + /// ("Debounced Running by ...") and deleted from the queue, silently dropping its + /// accumulated work, while the late arrival could not merge into it. + /// + /// Fix: a late arrival that finds the current survivor already running starts a + /// FRESH debounce window. The running survivor is left to finish with its own + /// accumulated batch; the late arrival accumulates only its own batch. No job is + /// killed and no item is dropped or double-run. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J1, J2 share a window; J2 is the survivor with batch {J1, J2}. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + assert!(is_completed(&db, &j1).await, "J1 should be debounced by J2"); + assert!(is_queued(&db, &j2).await, "J2 should be the survivor"); + + // Worker pulls J2 and marks it running: the window where the key still points + // to J2 and its batch is intact, but J2 can no longer accumulate new arrivals. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3 while J2 is running. + let j3 = Uuid::new_v4(); + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + + // The running survivor J2 must NOT be superseded: still queued, not completed. + assert!( + is_queued(&db, &j2).await, + "running survivor J2 must stay in the queue" + ); + assert!( + !is_completed(&db, &j2).await, + "running survivor J2 must not be completed/skipped" + ); + + // J3 must own the debounce key as the head of a FRESH window (no previous job). + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3, "J3 should hold the debounce key"); + assert!( + dk_prev.is_none(), + "J3 should start a fresh window with no previous job (got {dk_prev:?})" + ); + assert_eq!( + dk_times, 0, + "fresh window should reset debounced_times to 0" + ); + + // The running survivor J2 accumulates only its own committed batch: [1, 2]. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &serde_json::json!({"items": [2]}), + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert!( + j2_res.job.is_some(), + "running survivor J2 must still execute (not nulled out)" + ); + assert_accumulated_items(&j2_res, &[1, 2], "items"); + + // The late arrival J3 accumulates only its own batch: [3]. No overlap with J2. + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert!(j3_res.job.is_some(), "J3 must execute"); + assert_accumulated_items(&j3_res, &[3], "items"); + + Ok(()) + } + + /// A second arrival debouncing a NON-running survivor must keep accumulating into + /// the same batch (the normal debounce behavior must be unchanged by the + /// running-survivor guard). + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_queued_survivor_still_accumulates( + db: Pool, + ) -> anyhow::Result<()> { + let key = "queued_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Three arrivals, none running: classic debounce, all accumulate into J3. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + let j3 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + + assert!(is_completed(&db, &j1).await, "J1 debounced"); + assert!(is_completed(&db, &j2).await, "J2 debounced"); + assert!(is_queued(&db, &j3).await, "J3 is the survivor"); + + // Window keeps growing: J3 is the third arrival in the same batch. + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3); + assert_eq!(dk_prev, Some(j2), "previous job should be J2"); + assert_eq!(dk_times, 2, "debounced_times should keep incrementing"); + + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j3_res, &[1, 2, 3], "items"); + + Ok(()) + } + + /// The running-survivor guard must also apply to plain debounce (delay only, no + /// argument accumulation): a running survivor must never be completed/skipped. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_no_accumulation( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_no_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // No debounce_args_to_accumulate. + ..Default::default() + }; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + insert_noop_job(&db, j1, "test-workspace").await; + insert_noop_job(&db, j2, "test-workspace").await; + + let push = |id: Uuid| { + let settings = settings.clone(); + let db = db.clone(); + async move { + let args_hm = empty_args(); + let args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Noop, + id, + &args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + }; + + push(j1).await; + push(j2).await; + assert!(is_completed(&db, &j1).await, "J1 debounced by J2"); + + // J2 starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3. + let j3 = Uuid::new_v4(); + insert_noop_job(&db, j3, "test-workspace").await; + push(j3).await; + + // Running survivor J2 is preserved; J3 takes over a fresh window. + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3); + assert!(dk_prev.is_none(), "fresh window: no previous job"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + + Ok(()) + } + + /// Once a survivor has fully been pulled (batch + key consumed by + /// maybe_apply_debouncing) and is running, a later arrival naturally starts a new + /// window. This locks in that the committed-running case stays correct alongside + /// the in-flight-running guard. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_committed_running_survivor_independent( + db: Pool, + ) -> anyhow::Result<()> { + let key = "committed_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // J2 is pulled: accumulate its batch and consume key + batch. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j2_res, &[1, 2], "items"); + assert!( + get_debounce_key(&db, key).await.is_none(), + "key consumed when survivor pulled" + ); + + // J2 now running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3: fresh window, independent batch, J2 untouched. + let j3 = Uuid::new_v4(); + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + assert!(is_queued(&db, &j2).await, "running J2 untouched"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, _, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, j3); + assert_eq!(dk_times, 0); + + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j3_res, &[3], "items"); + + Ok(()) + } + + /// Flow post-preprocessing debounce must apply the same running-survivor guard: + /// a running flow survivor must not be completed/skipped by a late flow arrival. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_post_preprocessing_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "pp_running_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + ..Default::default() + }; + let args_hm = empty_args(); + + let flow1 = Uuid::new_v4(); + let flow2 = Uuid::new_v4(); + insert_flow_job(&db, flow1, "test-workspace", "f/test/flow").await; + insert_flow_job(&db, flow2, "test-workspace", "f/test/flow").await; + + let pp = |id: Uuid| { + let settings = settings.clone(); + let db = db.clone(); + let args_hm = args_hm.clone(); + async move { + let args = PushArgs::from(&args_hm); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow".to_string()), + "test-workspace", + id, + &args, + &db, + ) + .await + .unwrap() + } + }; + + pp(flow1).await; + pp(flow2).await; + assert!(is_completed(&db, &flow1).await, "flow1 debounced by flow2"); + + // flow2 (the survivor) starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + flow2 + ) + .execute(&db) + .await?; + + // Late flow3 arrival while flow2 is running. + let flow3 = Uuid::new_v4(); + insert_flow_job(&db, flow3, "test-workspace", "f/test/flow").await; + let sched = pp(flow3).await; + assert!(sched.is_some(), "flow3 should be debounced (fresh window)"); + + // Running flow2 must be preserved; flow3 owns a fresh window. + assert!(is_queued(&db, &flow2).await, "running flow2 stays queued"); + assert!( + !is_completed(&db, &flow2).await, + "running flow2 must not be completed" + ); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, flow3, "flow3 holds the key"); + assert!(dk_prev.is_none(), "fresh window: no previous job"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + + Ok(()) + } + + /// A running survivor resets the debounce window for the late arrival, so an + /// inherited high `debounced_times` cannot push the new arrival over + /// max_total_debounces_amount and force an immediate (un-debounced) run. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_resets_limit_window( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_limit_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // Limit of 3: J1, J2 stay debounced; a third arrival in the SAME window + // would trip the limit (current_amount + 1 >= 3) and fire immediately. + max_total_debounces_amount: Some(3), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Build up the window close to the limit: J1, J2 (debounced_times = 1 on J2). + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let (_, _, times_before) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(times_before, 1); + + // J2 starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // J3 arrives. Without the reset it would inherit debounced_times and could trip + // the max-count limit and fire immediately, killing running J2. With the guard + // it starts a fresh window (debounced_times = 0) and is debounced normally. + let j3 = Uuid::new_v4(); + let mut scheduled_for = None; + { + let args_val = serde_json::json!({ "items": [3] }); + insert_script_job_with_args(&db, j3, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j3, + ) + .execute(&db) + .await?; + let args_hm: HashMap> = serde_json::from_value(args_val).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + j3, + &push_args, + &mut tx, + ) + .await?; + tx.commit().await?; + } + + // J3 is debounced (scheduled_for set, not fired immediately) and J2 survives. + assert!( + scheduled_for.is_some(), + "J3 should be debounced, not fired immediately" + ); + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, j3); + assert!(dk_prev.is_none()); + assert_eq!(dk_times, 0, "fresh window resets the limit counter"); + + Ok(()) + } + + /// Concurrency regression: two late arrivals racing AFTER the survivor started + /// running must not both spawn independent windows. The running check reads the + /// post-conflict-lock holder (`debounce_key.job_id`), so the row lock serializes the + /// two upserts: the first observes the running survivor and opens a fresh window; + /// the second observes that fresh-window head (queued, not running) and debounces + /// into it. Exactly one late arrival survives, the other is debounced — never both. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_concurrent_arrivals_after_running_survivor( + db: Pool, + ) -> anyhow::Result<()> { + let key = "concurrent_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J2 is the survivor and starts running. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Insert the two late arrivals up front, then race only their maybe_debounce calls. + let j3 = Uuid::new_v4(); + let j4 = Uuid::new_v4(); + for (id, items) in [(j3, 3i64), (j4, 4i64)] { + let args_val = serde_json::json!({ "items": [items] }); + insert_script_job_with_args(&db, id, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(&db) + .await?; + } + + let race = |id: Uuid, items: i64| { + let db = db.clone(); + let settings = settings.clone(); + async move { + let args_hm: HashMap> = + serde_json::from_value(serde_json::json!({ "items": [items] })).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &push_args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + }; + tokio::join!(race(j3, 3), race(j4, 4)); + + // The running survivor is untouched. + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + + // Exactly one late arrival survives; the other is debounced into the same window. + let j3q = is_queued(&db, &j3).await; + let j4q = is_queued(&db, &j4).await; + let j3c = is_completed(&db, &j3).await; + let j4c = is_completed(&db, &j4).await; + assert!( + (j3q && j4c && !j4q && !j3c) || (j4q && j3c && !j3q && !j4c), + "exactly one late arrival must survive and the other be debounced \ + (not two independent windows); got j3 queued={j3q} completed={j3c}, \ + j4 queued={j4q} completed={j4c}" + ); + + // The surviving holder chained the debounced arrival into one window. + let (holder, prev, times) = get_debounce_key(&db, key).await.expect("key exists"); + let (survivor, debounced) = if j3q { (j3, j4) } else { (j4, j3) }; + assert_eq!(holder, survivor, "key points to the surviving late arrival"); + assert_eq!( + prev, + Some(debounced), + "the surviving window debounced the other late arrival" + ); + assert_eq!(times, 1, "single fresh window with one debounce"); + + // Both late arrivals must share a batch: when the survivor is pulled, its + // accumulation must include the debounced arrival's items, not just its own. + let survivor_args = serde_json::json!({ "items": [if j3q { 3 } else { 4 }] }); + let mut survivor_res = make_pulled_job_result( + survivor, + "test-workspace", + "f/test/script", + &survivor_args, + JobKind::Script, + "deno", + rs_handle, + ); + survivor_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&survivor_res, &[3, 4], "items"); + + Ok(()) + } + + /// Regression: a push that would chain onto a queued holder must not error if the + /// worker pull path concurrently deletes that holder's debounce_key + /// (`DELETE ... WHERE job_id = ...`, which does NOT take the push advisory lock). + /// The upsert is a single atomic `INSERT ... ON CONFLICT`, so a deleted holder simply + /// yields a fresh window rather than a "no row updated" failure. Races the two and + /// asserts the push always succeeds and leaves a consistent key. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_push_races_key_deletion_by_pull( + db: Pool, + ) -> anyhow::Result<()> { + let key = "push_vs_pull_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // 50 rounds to give the interleaving a chance to land in the read/write window + // that the old split read+UPDATE path would have failed on. + for round in 0..50 { + sqlx::query!("DELETE FROM debounce_key WHERE key = $1", key) + .execute(&db) + .await?; + let holder = Uuid::new_v4(); + push_debounced_script(&db, holder, vec![round], &settings, rs_handle).await; + + let late = Uuid::new_v4(); + let late_args = serde_json::json!({ "items": [round * 1000] }); + insert_script_job_with_args(&db, late, "test-workspace", "f/test/script", &late_args) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + late, + ) + .execute(&db) + .await?; + + // Race: push the late arrival (chains onto `holder`) against the worker pull + // cleanup deleting `holder`'s key. + let push = { + let db = db.clone(); + let settings = settings.clone(); + async move { + let args_hm: HashMap> = + serde_json::from_value(serde_json::json!({ "items": [round * 1000] })) + .unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + let res = windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + late, + &push_args, + &mut tx, + ) + .await; + if res.is_ok() { + tx.commit().await.unwrap(); + } + res + } + }; + let delete = { + let db = db.clone(); + async move { + sqlx::query!("DELETE FROM debounce_key WHERE job_id = $1", holder) + .execute(&db) + .await + } + }; + let (push_res, _) = tokio::join!(push, delete); + assert!( + push_res.is_ok(), + "round {round}: push must not error when the holder key is concurrently deleted: {push_res:?}" + ); + } + + Ok(()) + } + + /// Claim-based exactly-once: if two survivors end up on the same batch (only + /// possible in a narrow push/pull race), the args of each member are accumulated + /// into exactly ONE run. The survivor that claims the batch first accumulates + /// everyone; the second survivor finds its contribution already consumed and runs + /// empty — no item is dropped and none is processed twice. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_batch_consumed_exactly_once(db: Pool) -> anyhow::Result<()> { + let key = "exactly_once_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J1 superseded, J2 the (first) survivor of batch B = {J1, J2}. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // Simulate the race outcome: a second survivor J3 ended up on the SAME batch B. + let j3 = Uuid::new_v4(); + let j3_args = serde_json::json!({ "items": [3] }); + insert_script_job_with_args(&db, j3, "test-workspace", "f/test/script", &j3_args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j3, + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch) + SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + j3, + j2, + ) + .execute(&db) + .await?; + + // J2 pulled first: claims the whole batch, accumulates everyone's items. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert!(j2_res.job.is_some(), "J2 runs"); + assert_accumulated_items(&j2_res, &[1, 2, 3], "items"); + + // J3 pulled next: its contribution was already consumed by J2 -> runs empty, + // so [3] is not processed a second time. + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + let job = j3_res.job.as_ref().expect("J3 still runs (empty)"); + let items: Vec = + serde_json::from_str(job.job.args.as_ref().unwrap().get("items").unwrap().get())?; + assert!( + items.is_empty(), + "J3's items must be empty (already consumed by J2), got {items:?}" + ); + + Ok(()) + } + + /// A survivor re-pulled (e.g. crash recovery) must NOT mistake its own earlier + /// claim for a sibling's and wipe its accumulated args. consumed_by = self is + /// distinguished from consumed_by = another job. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_repull_keeps_accumulated(db: Pool) -> anyhow::Result<()> { + let key = "repull_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // First pull: J2 claims its batch and accumulates [1, 2]. + let mut first = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + first.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&first, &[1, 2], "items"); + + // Re-pull with the args persisted by the first pull: J2 sees its OWN prior claim + // (consumed_by = j2), so it keeps the accumulated args rather than running empty. + let persisted = first.job.as_ref().unwrap().job.args.as_ref().unwrap(); + let persisted_json = serde_json::to_value(persisted).unwrap(); + let mut second = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &persisted_json, + JobKind::Script, + "deno", + rs_handle, + ); + second.maybe_apply_debouncing(&db).await?; + assert!(second.job.is_some(), "re-pulled J2 still runs"); + assert_accumulated_items(&second, &[1, 2], "items"); + + Ok(()) + } + + /// Helper: insert a script job and put it on the SAME debounce batch as `of_job` + /// (simulating a chained survivor). Returns its args JSON. + async fn add_survivor_to_batch_of( + db: &Pool, + id: Uuid, + items: Vec, + of_job: Uuid, + rs_handle: Option, + ) -> serde_json::Value { + let args = serde_json::json!({ "items": items }); + insert_script_job_with_args(db, id, "test-workspace", "f/test/script", &args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(db) + .await + .unwrap(); + let inserted = sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch) + SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + id, + of_job, + ) + .execute(db) + .await + .unwrap(); + // `of_job` must already have a batch row, else this no-ops and the test would + // pass vacuously (the job would end up never-batched, keeping its own args). + assert_eq!( + inserted.rows_affected(), + 1, + "add_survivor_to_batch_of: {of_job} has no batch row to share" + ); + args + } + + /// Helper: read the accumulated `items` of a pulled job as a sorted Vec. + fn items_of(result: &windmill_queue::PulledJobResult) -> Vec { + let job = result.job.as_ref().expect("job present"); + let raw = job.job.args.as_ref().unwrap().get("items").unwrap(); + let mut v: Vec = serde_json::from_str::>(raw.get()) + .unwrap() + .iter() + .map(|x| x.as_i64().unwrap()) + .collect(); + v.sort(); + v + } + + /// Edge: an accumulate-debounced job that was NEVER batched (CE / workers behind v2: + /// no v2_job_debounce_batch row) must keep its own args, not be emptied. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_never_batched_keeps_own_args(db: Pool) -> anyhow::Result<()> { + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("never_batched_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Insert a job with the debounce handle but DO NOT push through maybe_debounce, + // so it has no batch row at all. + let j = Uuid::new_v4(); + let args = serde_json::json!({ "items": [7, 8] }); + insert_script_job_with_args(&db, j, "test-workspace", "f/test/script", &args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j, + ) + .execute(&db) + .await?; + + let mut res = make_pulled_job_result( + j, + "test-workspace", + "f/test/script", + &args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + assert!(res.job.is_some(), "never-batched job still runs"); + assert_accumulated_items(&res, &[7, 8], "items"); + Ok(()) + } + + /// Edge: two survivors of one batch pulled CONCURRENTLY. The atomic claim must + /// partition the batch disjointly — the union of what they each accumulate is the + /// full set, with NO item processed by both. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_concurrent_claim_disjoint(db: Pool) -> anyhow::Result<()> { + let key = "concurrent_claim_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; // superseded + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; // survivor 1 + let j3 = Uuid::new_v4(); + let j3_args = add_survivor_to_batch_of(&db, j3, vec![3], j2, rs_handle).await; // survivor 2 + + let pull = |id: Uuid, args: serde_json::Value| { + let db = db.clone(); + async move { + let mut res = make_pulled_job_result( + id, + "test-workspace", + "f/test/script", + &args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await.unwrap(); + res + } + }; + let (r2, r3) = tokio::join!(pull(j2, j2_args), pull(j3, j3_args)); + + let mut union = items_of(&r2); + union.extend(items_of(&r3)); + union.sort(); + assert_eq!( + union, + vec![1, 2, 3], + "every item accumulated exactly once across the two concurrent survivors" + ); + // disjoint: no overlap between the two survivors' items + let i2 = items_of(&r2); + let i3 = items_of(&r3); + assert!( + !i2.iter().any(|x| i3.contains(x)), + "no item processed by both survivors; got j2={i2:?} j3={i3:?}" + ); + Ok(()) + } + + /// Edge: three survivors on one batch pulled in sequence. The first claims the whole + /// batch; the rest find themselves consumed and run empty. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_three_survivors_first_takes_all( + db: Pool, + ) -> anyhow::Result<()> { + let key = "three_survivors_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let j3 = Uuid::new_v4(); + let j3_args = add_survivor_to_batch_of(&db, j3, vec![3], j2, rs_handle).await; + let j4 = Uuid::new_v4(); + let j4_args = add_survivor_to_batch_of(&db, j4, vec![4], j2, rs_handle).await; + + let mk = |id, args: &serde_json::Value| { + make_pulled_job_result( + id, + "test-workspace", + "f/test/script", + args, + JobKind::Script, + "deno", + rs_handle, + ) + }; + let (mut r2, mut r3, mut r4) = (mk(j2, &j2_args), mk(j3, &j3_args), mk(j4, &j4_args)); + r2.maybe_apply_debouncing(&db).await?; + r3.maybe_apply_debouncing(&db).await?; + r4.maybe_apply_debouncing(&db).await?; + + assert_eq!( + items_of(&r2), + vec![1, 2, 3, 4], + "first survivor takes the whole batch" + ); + assert!(items_of(&r3).is_empty(), "second survivor runs empty"); + assert!(items_of(&r4).is_empty(), "third survivor runs empty"); + Ok(()) + } + + /// Edge: plain debounce (no accumulate args) must HARD-DELETE its batch rows on pull + /// (not leave consumed rows lingering), so the non-accumulate path doesn't leak. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_non_accumulate_deletes_batch(db: Pool) -> anyhow::Result<()> { + let key = "non_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // no debounce_args_to_accumulate + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + // push_debounced_script sends {items:[...]} but with no accumulate arg configured, + // the batch is created yet never accumulated. + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + let batch_rows_before: i64 = + sqlx::query_scalar!("SELECT count(*) as \"c!\" FROM v2_job_debounce_batch") + .fetch_one(&db) + .await?; + assert!(batch_rows_before >= 2, "batch rows exist before pull"); + + let mut res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + + let remaining: i64 = + sqlx::query_scalar!("SELECT count(*) as \"c!\" FROM v2_job_debounce_batch") + .fetch_one(&db) + .await?; + assert_eq!( + remaining, 0, + "non-accumulate pull hard-deletes the batch rows" + ); + Ok(()) + } + + /// Edge: GC sweep deletes consumed rows past the grace period but keeps recently + /// consumed and not-yet-consumed rows. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_gc_consumed_batches(db: Pool) -> anyhow::Result<()> { + let old = Uuid::new_v4(); + let recent = Uuid::new_v4(); + let unconsumed = Uuid::new_v4(); + // A consumed-long-ago sibling that is STILL QUEUED (e.g. stuck behind a + // concurrency limit): its marker must survive GC so its eventual pull still sees + // "already consumed" and runs empty (no duplicate). + let queued_old = Uuid::new_v4(); + insert_script_job_with_args( + &db, + queued_old, + "test-workspace", + "f/test/script", + &serde_json::json!({ "items": [9] }), + ) + .await; + sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES + ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'), + ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'), + ($3, nextval('debounce_batch_seq'), NULL), + ($4, nextval('debounce_batch_seq'), now() - interval '20 minutes')", + old, + recent, + unconsumed, + queued_old, + ) + .execute(&db) + .await?; + + // Mirror the monitor GC sweep (age floor + only-if-no-longer-queued). + let deleted = sqlx::query_scalar!( + "WITH del AS ( + DELETE FROM v2_job_debounce_batch + WHERE consumed_at IS NOT NULL + AND consumed_at < now() - interval '10 minutes' + AND id NOT IN (SELECT id FROM v2_job_queue) + RETURNING 1 + ) SELECT count(*) as \"c!\" FROM del" + ) + .fetch_one(&db) + .await?; + assert_eq!( + deleted, 1, + "only the old, no-longer-queued consumed row is GC'd" + ); + + let exists = |id: Uuid, db: Pool| async move { + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM v2_job_debounce_batch WHERE id = $1) as \"e!\"", + id + ) + .fetch_one(&db) + .await + .unwrap() + }; + assert!(!exists(old, db.clone()).await, "old consumed row gone"); + assert!( + exists(recent, db.clone()).await, + "recently consumed row kept" + ); + assert!(exists(unconsumed, db.clone()).await, "unconsumed row kept"); + assert!( + exists(queued_old, db.clone()).await, + "old consumed row whose job is still queued must be kept" + ); + Ok(()) + } + + /// Helper: mark a queued job as running (simulates a survivor that the + /// concurrency limiter has just started executing). + async fn set_running(db: &Pool, job_id: &Uuid) { + sqlx::query!( + "UPDATE v2_job_queue SET running = true WHERE id = $1", + job_id + ) + .execute(db) + .await + .expect("set running"); + } + + /// Regression (ref #9781): post-preprocessing debounce with + /// `debounce_args_to_accumulate` under a concurrency limit. A survivor accumulates + /// its own element and starts running; a later same-key message must start a NEW + /// batch (survive) rather than be folded into the running survivor and silently + /// dropped. Exercises the full EE path via `jobs_ee::maybe_debounce_post_preprocessing`. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_post_preprocessing_debounce_into_running_survivor_loses_message( + db: Pool, + ) -> anyhow::Result<()> { + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("ported_running_survivor_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // --- Wave 1: a single message becomes the survivor and starts running. --- + let survivor = Uuid::new_v4(); + let survivor_args = serde_json::json!({ "items": [1] }); + insert_flow_job_with_preprocessor( + &db, + survivor, + "test-workspace", + "f/test/flow_run", + true, + 0, + ) + .await; + sqlx::query!( + "UPDATE v2_job SET args = $2 WHERE id = $1", + survivor, + survivor_args + ) + .execute(&db) + .await?; + + let survivor_args_hm: HashMap> = + serde_json::from_value(survivor_args.clone()).unwrap(); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow_run".to_string()), + "test-workspace", + survivor, + &PushArgs::from(&survivor_args_hm), + &db, + ) + .await?; + assert!(is_queued(&db, &survivor).await, "survivor should be queued"); + + // Worker pulls the survivor: accumulate its own [1], consume the batch, run. + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + survivor, + ) + .execute(&db) + .await?; + let mut pulled = make_pulled_job_result( + survivor, + "test-workspace", + "f/test/flow_run", + &survivor_args, + JobKind::Flow, + "flow", + rs_handle, + ); + pulled.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&pulled, &[1], "items"); + set_running(&db, &survivor).await; // survivor is now RUNNING + + // --- Wave 2: a new message arrives while the survivor is running. --- + let late = Uuid::new_v4(); + let late_args = serde_json::json!({ "items": [2] }); + insert_flow_job_with_preprocessor(&db, late, "test-workspace", "f/test/flow_run", true, 0) + .await; + sqlx::query!("UPDATE v2_job SET args = $2 WHERE id = $1", late, late_args) + .execute(&db) + .await?; + + let late_args_hm: HashMap> = + serde_json::from_value(late_args.clone()).unwrap(); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow_run".to_string()), + "test-workspace", + late, + &PushArgs::from(&late_args_hm), + &db, + ) + .await?; + + // The late message must survive (new batch): still queued, and the debounce_key + // moved off the already-running survivor. + let late_survived = is_queued(&db, &late).await && !is_completed(&db, &late).await; + let dk = get_debounce_key(&db, "ported_running_survivor_key").await; + let key_moved_off_running_survivor = + dk.map(|(job_id, _, _)| job_id != survivor).unwrap_or(true); + assert!( + late_survived && key_moved_off_running_survivor, + "message arriving while the survivor is running must start a new batch \ + (survive), not be folded into the running survivor and dropped. \ + late_survived={late_survived}, key_moved_off={key_moved_off_running_survivor}" + ); + Ok(()) + } + + /// Flow-node debounce (third EE entry point, `jobs_ee::maybe_debounce_flow_node`): + /// a running survivor child must not be superseded by a later same-key child; the + /// late child starts a fresh window and the running child is left to finish. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_flow_node_debounce_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "flow_node_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + ..Default::default() + }; + let args_hm = empty_args(); + + let flow1 = Uuid::new_v4(); + let child1 = Uuid::new_v4(); + insert_flow_job(&db, flow1, "test-workspace", "f/test/my_flow").await; + insert_child_job_with_parent(&db, child1, flow1, "test-workspace").await; + + // child1 becomes the survivor, then starts running. + { + let args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce_flow_node( + &settings, + child1, + flow1, + "f/test/my_flow", + "step_a", + "test-workspace", + &args, + &mut tx, + &db, + ) + .await?; + tx.commit().await?; + } + set_running(&db, &child1).await; + + // child2 (later same-key child) arrives while child1 is running. + let flow2 = Uuid::new_v4(); + let child2 = Uuid::new_v4(); + insert_flow_job(&db, flow2, "test-workspace", "f/test/my_flow").await; + insert_child_job_with_parent(&db, child2, flow2, "test-workspace").await; + { + let args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce_flow_node( + &settings, + child2, + flow2, + "f/test/my_flow", + "step_a", + "test-workspace", + &args, + &mut tx, + &db, + ) + .await?; + tx.commit().await?; + } + + // The running child1 (and its parent flow1) must be left alone; child2 owns a + // fresh window. + assert!(is_queued(&db, &child1).await, "running child1 stays queued"); + assert!( + !is_completed(&db, &child1).await, + "running child1 not completed" + ); + assert!( + !is_completed(&db, &flow1).await, + "flow1 of running child not completed" + ); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, child2, "child2 holds the key"); + assert!(dk_prev.is_none(), "fresh window: no previous child"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + Ok(()) + } + + /// Edge: accumulate values that are bare scalars (not arrays) — the `T | T[]` union + /// case. Each scalar contribution must be wrapped into a single-element list so the + /// survivor accumulates them all. Exercises the non-array fallback in the claim path. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_accumulate_scalar_values(db: Pool) -> anyhow::Result<()> { + let key = "scalar_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Push two jobs whose `items` is a BARE SCALAR, not an array. + let push_scalar = |id: Uuid, v: i64, db: Pool, settings: DebouncingSettings| async move { + let args_val = serde_json::json!({ "items": v }); + insert_script_job_with_args(&db, id, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(&db) + .await + .unwrap(); + let hm: HashMap> = serde_json::from_value(args_val).unwrap(); + let mut sf = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut sf, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &PushArgs::from(&hm), + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + }; + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_scalar(j1, 1, db.clone(), settings.clone()).await; + push_scalar(j2, 2, db.clone(), settings.clone()).await; + + let mut res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &serde_json::json!({ "items": 2 }), + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + // Both bare scalars are wrapped and accumulated into a list. + assert_accumulated_items(&res, &[1, 2], "items"); + Ok(()) + } + + /// Edge: GC reclaiming a survivor's consumed row before a re-pull must NOT lose data — + /// the re-pull finds no row (had_row=false) and keeps its already-persisted accumulated + /// args, rather than running empty. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_repull_after_gc_keeps_accumulated( + db: Pool, + ) -> anyhow::Result<()> { + let key = "repull_gc_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + let mut first = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + first.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&first, &[1, 2], "items"); + + // Simulate the GC sweep reclaiming the (now consumed) batch rows for this batch. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j2, + ) + .execute(&db) + .await + .ok(); + // (and any that were already consumed elsewhere) + sqlx::query!("DELETE FROM v2_job_debounce_batch WHERE consumed_at IS NOT NULL") + .execute(&db) + .await?; + + // Re-pull with the args persisted on the first pull: no batch row now, so it must + // fall back to its own (already-accumulated) args — no loss. + let persisted = + serde_json::to_value(first.job.as_ref().unwrap().job.args.as_ref().unwrap()).unwrap(); + let mut second = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &persisted, + JobKind::Script, + "deno", + rs_handle, + ); + second.maybe_apply_debouncing(&db).await?; + assert!(second.job.is_some(), "re-pulled survivor still runs"); + assert_accumulated_items(&second, &[1, 2], "items"); + Ok(()) + } + + /// Throughput benchmark for the FULL debounce path (EE push + + /// `jobs_ee::maybe_debounce`/`complete_debounced_job`/`upsert_debounce_key`, then OSS + /// `maybe_apply_debouncing` claim/accumulate/consume). #[ignore]d — run manually: + /// cargo test -p windmill-queue --test debounce_test --features private,enterprise \ + /// bench_debounce_full_path -- --ignored --nocapture --test-threads=1 + /// Each cycle = a burst of BURST pushes to one key (debounced) + one survivor pull + /// (accumulate+consume), run across CONCURRENCY tasks. Compare before/after by running + /// it on each code revision. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + #[ignore] + async fn bench_debounce_full_path(db: Pool) -> anyhow::Result<()> { + const CONCURRENCY: usize = 4; + const CYCLES_PER_TASK: usize = 300; + const BURST: usize = 3; + + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("bench_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let start = std::time::Instant::now(); + let tasks: Vec<_> = (0..CONCURRENCY) + .map(|t| { + let db = db.clone(); + let settings = DebouncingSettings { + // distinct key space per task so bursts collapse independently + debounce_key: Some(format!("bench_key_{t}")), + ..settings.clone() + }; + tokio::spawn(async move { + for c in 0..CYCLES_PER_TASK { + // Fresh key per cycle so each cycle is one full collapse+pull. + let key = format!("bench_{t}_{c}"); + let settings = DebouncingSettings { + debounce_key: Some(key.clone()), + ..settings.clone() + }; + let mut survivor = Uuid::new_v4(); + for b in 0..BURST { + let id = Uuid::new_v4(); + survivor = id; + let args_val = serde_json::json!({ "items": [b as i64] }); + insert_script_job_with_args( + &db, id, "test-workspace", "f/test/script", &args_val, + ) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, id, + ) + .execute(&db) + .await + .unwrap(); + let hm: HashMap> = + serde_json::from_value(args_val).unwrap(); + let mut sf = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, &mut sf, &Some("f/test/script".to_string()), + "test-workspace", JobKind::Script, id, &PushArgs::from(&hm), &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + // Survivor pull: claim + accumulate + consume. + let mut res = make_pulled_job_result( + survivor, "test-workspace", "f/test/script", + &serde_json::json!({ "items": [] }), + JobKind::Script, "deno", rs_handle, + ); + res.maybe_apply_debouncing(&db).await.unwrap(); + } + }) + }) + .collect(); + for t in tasks { + t.await.unwrap(); + } + let elapsed = start.elapsed(); + let cycles = CONCURRENCY * CYCLES_PER_TASK; + let jobs = cycles * BURST; + eprintln!( + "BENCH full debounce path: {cycles} cycles ({jobs} pushed jobs + {cycles} pulls) in {:.2?} | {:.0} pushes/s | {:.0} pulls/s", + elapsed, + jobs as f64 / elapsed.as_secs_f64(), + cycles as f64 / elapsed.as_secs_f64(), + ); + Ok(()) + } + /// Test: Push-time (script) debounce with max_total_debounces_amount=2. /// 5 calls, each sending {x: [i]}. Expected: /// Call 1: debounced (scheduled_for set) diff --git a/backend/windmill-queue/tests/native_retry_test.rs b/backend/windmill-queue/tests/native_retry_test.rs new file mode 100644 index 0000000000..fb8046149b --- /dev/null +++ b/backend/windmill-queue/tests/native_retry_test.rs @@ -0,0 +1,376 @@ +// Integration tests for native single-script retry (no one-step-flow wrapping). +// +// These use the *runtime* sqlx API (`sqlx::query`/`query_as`, not the `!` macros) +// like schedule_push.rs, so they need no `.sqlx` offline cache entry. +mod native_retry { + use sqlx::{Pool, Postgres}; + use uuid::Uuid; + + use windmill_common::flows::{ConstantDelay, Retry}; + use windmill_common::jobs::{JobKind, JobTriggerKind}; + use windmill_common::runnable_settings::{ + from_handle, insert_rs, ConcurrencySettings, RetrySettings, RunnableSettings, + RunnableSettingsTrait, + }; + use windmill_common::scripts::{ScriptHash, ScriptLang}; + use windmill_common::users::username_to_permissioned_as; + use windmill_queue::jobs::{maybe_enqueue_native_script_retry, MiniCompletedJob}; + + const WS: &str = "test-workspace"; + const SCHED: &str = "f/system/test_schedule"; + const SCRIPT: &str = "f/system/test_script"; + + fn mini(id: Uuid, parent_job: Option, handle: Option) -> MiniCompletedJob { + MiniCompletedJob { + id, + workspace_id: WS.to_string(), + runnable_id: Some(ScriptHash(100001)), + scheduled_for: chrono::Utc::now(), + parent_job, + flow_innermost_root_job: None, + runnable_path: Some(SCRIPT.to_string()), + kind: JobKind::Script, + started_at: Some(chrono::Utc::now()), + permissioned_as: username_to_permissioned_as("test-user"), + created_by: "test-user".to_string(), + script_lang: Some(ScriptLang::Deno), + permissioned_as_email: "test@windmill.dev".to_string(), + flow_step_id: None, + trigger_kind: Some(JobTriggerKind::Schedule), + trigger: Some(SCHED.to_string()), + priority: None, + concurrent_limit: None, + tag: "deno".to_string(), + cache_ttl: None, + cache_ignore_s3_path: None, + runnable_settings_handle: handle, + } + } + + async fn count_retries(db: &Pool, root: Uuid) -> i64 { + sqlx::query_scalar::<_, i64>("SELECT count(*) FROM v2_job WHERE parent_job = $1") + .bind(root) + .fetch_one(db) + .await + .unwrap() + } + + /// The queued retry with the given attempt number, if any: (id, kind, parent_job, backoff_s, handle). + async fn retry_by_attempt( + db: &Pool, + root: Uuid, + attempt: i64, + ) -> Option<(Uuid, String, Option, f64, Option)> { + sqlx::query_as::<_, (Uuid, String, Option, Option, Option)>( + "SELECT j.id, j.kind::text, j.parent_job, + EXTRACT(EPOCH FROM (q.scheduled_for - now()))::float8, + q.runnable_settings_handle + FROM v2_job j + JOIN v2_job_queue q ON q.id = j.id + JOIN native_retry_attempt nra ON nra.job_id = j.id + WHERE j.parent_job = $1 AND nra.attempt = $2", + ) + .bind(root) + .bind(attempt) + .fetch_optional(db) + .await + .unwrap() + .map(|(id, kind, parent, backoff, handle)| { + (id, kind, parent, backoff.unwrap_or(0.0), handle) + }) + } + + fn no_result() -> Option> { + None + } + + // attempt0 -> retry1 -> retry2 -> exhausted, with crash-replay idempotency. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn chains_attempts_and_is_idempotent(db: Pool) -> anyhow::Result<()> { + // Policy: 2 constant attempts, 1s apart. + let retry = Retry { + constant: ConstantDelay { attempts: 2, seconds: 1 }, + exponential: Default::default(), + retry_if: None, + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: None, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + assert!(handle.is_some(), "a retry policy must produce a handle"); + + // attempt 0 (the schedule root); maybe_enqueue tolerates the absent queue row (counter -> 0). + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + + // First failure -> retry 1. + assert!( + maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?, + "first failure enqueues a retry" + ); + let (r1_id, kind, parent, backoff, r1_handle) = retry_by_attempt(&db, root_id, 1) + .await + .expect("retry attempt 1 exists"); + assert_eq!( + kind, "script", + "retry is a native Script, not a singlestepflow" + ); + assert_eq!(parent, Some(root_id), "retry links to the chain root"); + assert!( + r1_handle.is_some(), + "retry carries the policy for further chaining" + ); + assert!( + backoff > 0.0 && backoff <= 3.0, + "constant 1s backoff, got {backoff}s" + ); + + // Crash-replay: the SAME completion again must not double-enqueue, and must + // still report pending (so schedule handlers stay deferred). Regression for P1. + assert!( + maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?, + "replay still reports the retry as pending" + ); + assert_eq!( + count_retries(&db, root_id).await, + 1, + "no double retry on crash replay" + ); + + // retry 1 fails -> retry 2 (still within attempts = 2). + let r1 = mini(r1_id, Some(root_id), r1_handle); + assert!(maybe_enqueue_native_script_retry(&db, &r1, &None, &no_result).await?); + assert_eq!(count_retries(&db, root_id).await, 2); + + // retry 2 fails -> attempts exhausted, no retry 3. + let (r2_id, _, _, _, r2_handle) = retry_by_attempt(&db, root_id, 2) + .await + .expect("retry attempt 2 exists"); + let r2 = mini(r2_id, Some(root_id), r2_handle); + assert!( + !maybe_enqueue_native_script_retry(&db, &r2, &None, &no_result).await?, + "exhausted policy does not enqueue" + ); + assert_eq!( + count_retries(&db, root_id).await, + 2, + "no retry past max attempts" + ); + Ok(()) + } + + // Cancellation always wins over a pending retry. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn canceled_job_does_not_retry(db: Pool) -> anyhow::Result<()> { + let retry = Retry { + constant: ConstantDelay { attempts: 3, seconds: 1 }, + exponential: Default::default(), + retry_if: None, + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: None, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + let canceled = Some(windmill_queue::jobs::CanceledBy { + username: Some("test-user".to_string()), + reason: Some("manual".to_string()), + }); + assert!(!maybe_enqueue_native_script_retry(&db, &root, &canceled, &no_result).await?); + assert_eq!( + count_retries(&db, root_id).await, + 0, + "canceled job must not retry" + ); + Ok(()) + } + + // A concurrency-limited script that also retries must carry its concurrency + // settings into each retry, otherwise the retry runs unbounded. Regression: P1. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn retry_preserves_concurrency_settings(db: Pool) -> anyhow::Result<()> { + let retry = Retry { + constant: ConstantDelay { attempts: 1, seconds: 0 }, + exponential: Default::default(), + retry_if: None, + }; + let concurrency = ConcurrencySettings { + concurrency_key: Some("f/system/test_script".to_string()), + concurrent_limit: Some(1), + concurrency_time_window_s: Some(60), + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: concurrency.insert_cached(&db).await?, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + assert!(maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?); + + let (_id, _kind, _parent, _backoff, r1_handle) = retry_by_attempt(&db, root_id, 1) + .await + .expect("retry attempt 1 exists"); + // The retry's own handle must resolve to the same concurrency settings, not + // just the retry policy — otherwise it would run with no concurrency_key. + let rs = from_handle(r1_handle, &db).await?; + let resolved = ConcurrencySettings::get( + rs.concurrency_settings + .expect("retry must carry concurrency settings forward"), + &db, + ) + .await?; + assert_eq!( + resolved.concurrency_key.as_deref(), + Some("f/system/test_script") + ); + assert_eq!(resolved.concurrent_limit, Some(1)); + assert!( + rs.retry_settings.is_some(), + "retry policy is still carried for further chaining" + ); + Ok(()) + } + + // ------------------------------------------------------------------ + // Per-occurrence terminal status (drives on_failure_times / on_recovery). + // Mirrors the exact query in windmill-ee-private jobs_ee::apply_schedule_handlers. + // ------------------------------------------------------------------ + // `is_retry` marks the seeded job as a native retry attempt (the explicit + // native_retry_attempt marker), the same signal `apply_schedule_handlers` + // keys off. Handlers / WAC inline children are seeded without it. + async fn seed_job( + db: &Pool, + id: Uuid, + parent: Option, + is_retry: bool, + status: &str, + ) { + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, trigger_kind, trigger, parent_job) + VALUES ($1, $2, 'script', $3, 'schedule', $4, $5)", + ) + .bind(id) + .bind(WS) + .bind(SCRIPT) + .bind(SCHED) + .bind(parent) + .execute(db) + .await + .unwrap(); + sqlx::query( + "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, status) + VALUES ($1, $2, 1, $3::job_status)", + ) + .bind(id) + .bind(WS) + .bind(status) + .execute(db) + .await + .unwrap(); + if is_retry { + sqlx::query("INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, 1)") + .bind(id) + .execute(db) + .await + .unwrap(); + } + } + + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn per_occurrence_status_counts_recovered_as_success(db: Pool) { + // A native retry attempt is marked (native_retry_attempt); handler and WAC + // inline children are parented Script children WITHOUT the marker. + // a: root fails, a marked retry succeeds -> RECOVERED -> success + // b: root fails, retry also fails -> failure + // c: root succeeds directly -> success + // e: root fails, only its on_failure HANDLER (unmarked) succeeds + // -> failure: handler child must NOT count as a recovery + // f: root fails, only a WAC inline child (unmarked) succeeds + // -> failure: WAC inline child must NOT count as a recovery + // d: the current occurrence (excluded by `j.id != $4`) + let (a, b, c, e, f, d) = ( + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + ); + seed_job(&db, a, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(a), true, "success").await; // a's retry attempt (marked) + seed_job(&db, b, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(b), true, "failure").await; // b's failed retry + seed_job(&db, c, None, false, "success").await; + seed_job(&db, e, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(e), false, "success").await; // e's handler child (unmarked) + seed_job(&db, f, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(f), false, "success").await; // f's WAC inline child (unmarked) + seed_job(&db, d, None, false, "failure").await; // current occurrence + + // Exact expression from jobs_ee::apply_schedule_handlers: the EXISTS counts + // only marked native retry children, so neither handler nor WAC inline + // children count as a recovery. + let rows = sqlx::query_as::<_, (Uuid, bool)>( + "SELECT j.id, (status = 'success' OR EXISTS ( + SELECT 1 FROM native_retry_attempt nra + JOIN v2_job jc ON jc.id = nra.job_id + JOIN v2_job_completed cc ON cc.id = nra.job_id + WHERE jc.parent_job = j.id AND cc.status = 'success' + )) + FROM v2_job j JOIN v2_job_completed USING (id) + WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2 + AND parent_job IS NULL AND runnable_path = $3 AND j.id != $4 + ORDER BY created_at DESC", + ) + .bind(WS) + .bind(SCHED) + .bind(SCRIPT) + .bind(d) + .fetch_all(&db) + .await + .unwrap(); + + let status: std::collections::HashMap = rows.into_iter().collect(); + // Retries/handlers/WAC children (parent_job set) are NOT occurrences, and the + // current one is excluded: exactly the five roots a, b, c, e, f remain. + assert_eq!( + status.len(), + 5, + "child jobs excluded from occurrence counting; current excluded" + ); + assert_eq!( + status[&a], true, + "recovered occurrence (same-runnable retry succeeded) = success" + ); + assert_eq!( + status[&b], false, + "all-attempts-failed occurrence = failure" + ); + assert_eq!(status[&c], true, "direct success"); + assert_eq!( + status[&e], false, + "on_failure handler success must NOT count as a recovery" + ); + assert_eq!( + status[&f], false, + "WAC inline child success must NOT count as a recovery" + ); + } +} diff --git a/backend/windmill-queue/tests/schedule_push.rs b/backend/windmill-queue/tests/schedule_push.rs index c0e09003f0..e11ced23bc 100644 --- a/backend/windmill-queue/tests/schedule_push.rs +++ b/backend/windmill-queue/tests/schedule_push.rs @@ -3,6 +3,9 @@ mod schedule_push { use sqlx::{Pool, Postgres}; use windmill_common::db::Authed; use windmill_common::jobs::{JobKind, JobTriggerKind}; + use windmill_common::runnable_settings::{ + from_handle, insert_rs, ConcurrencySettings, RunnableSettings, RunnableSettingsTrait, + }; use windmill_common::schedule::Schedule; use windmill_common::scripts::ScriptHash; use windmill_common::users::username_to_permissioned_as; @@ -233,13 +236,78 @@ mod schedule_push { assert_eq!(count_queued_jobs(&db).await, 1); - // When retry is set, the job kind is singlescriptflow (SingleStepFlow wraps it) - let kind = sqlx::query_scalar::<_, String>( - "SELECT kind::text FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + // Native retry: a scheduled script with a retry policy is pushed as a plain + // Script carrying the policy via runnable_settings_handle — no SingleStepFlow. + let (kind, handle) = sqlx::query_as::<_, (String, Option)>( + "SELECT kind::text, q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", ) .fetch_one(&db) .await?; - assert_eq!(kind, "singlestepflow"); + assert_eq!(kind, "script"); + assert!( + handle.is_some(), + "retry policy carried via runnable_settings_handle" + ); + Ok(()) + } + + // A scheduled, concurrency-limited script with a retry policy: the materialized + // root attempt's handle must resolve to BOTH the retry policy and the script's + // concurrency settings — otherwise the retry chain runs unbounded. Regression: P1. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn test_push_script_with_retry_keeps_concurrency( + db: Pool, + ) -> anyhow::Result<()> { + let concurrency = ConcurrencySettings { + concurrency_key: Some("f/system/test_script".to_string()), + concurrent_limit: Some(1), + concurrency_time_window_s: Some(60), + }; + let script_handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: concurrency.insert_cached(&db).await?, + retry_settings: None, + }, + &db, + ) + .await?; + sqlx::query( + "UPDATE script SET runnable_settings_handle = $1 WHERE workspace_id = 'test-workspace' AND path = 'f/system/test_script'", + ) + .bind(script_handle) + .execute(&db) + .await?; + + let schedule = make_schedule(|s| { + s.retry = Some(serde_json::json!({ "constant": { "attempts": 3, "seconds": 10 } })); + }); + let authed = make_authed(); + let tx = db.begin().await?; + let tx = push_scheduled_job(&db, tx, &schedule, Some(&authed), None).await?; + tx.commit().await?; + + let handle = sqlx::query_scalar::<_, Option>( + "SELECT q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + ) + .fetch_one(&db) + .await?; + let rs = from_handle(handle, &db).await?; + assert!( + rs.retry_settings.is_some(), + "root attempt carries the retry policy" + ); + let resolved = ConcurrencySettings::get( + rs.concurrency_settings + .expect("root attempt must carry concurrency settings, not just retry"), + &db, + ) + .await?; + assert_eq!(resolved.concurrent_limit, Some(1)); + assert_eq!( + resolved.concurrency_key.as_deref(), + Some("f/system/test_script") + ); Ok(()) } @@ -779,7 +847,7 @@ mod schedule_push { } // ----------------------------------------------------------------------- - // try_schedule_next_job: script with retry wraps in SingleStepFlow + // try_schedule_next_job: script with retry is a native Script (no wrapping) // ----------------------------------------------------------------------- #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] @@ -798,12 +866,16 @@ mod schedule_push { assert!(err.is_none()); assert_eq!(count_queued_jobs(&db).await, 1); - let kind = sqlx::query_scalar::<_, String>( - "SELECT kind::text FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + let (kind, handle) = sqlx::query_as::<_, (String, Option)>( + "SELECT kind::text, q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", ) .fetch_one(&db) .await?; - assert_eq!(kind, "singlestepflow"); + assert_eq!(kind, "script"); + assert!( + handle.is_some(), + "retry policy carried via runnable_settings_handle" + ); Ok(()) } @@ -1520,4 +1592,174 @@ mod schedule_push { Ok(()) } + + // ----------------------------------------------------------------------- + // push_scheduled_job: reserved ducklake-maintenance prefix + // ----------------------------------------------------------------------- + + // A schedule that pre-dates the reserved prefix (a user schedule under a + // real `ducklake_maintenance` folder) must fall through to normal script + // resolution when its path's lake has no enabled maintenance config — + // never be hijacked into the maintenance payload builder and auto-disabled. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn test_push_reserved_prefix_no_config_falls_through_to_script( + db: Pool, + ) -> anyhow::Result<()> { + let schedule = make_schedule(|s| { + s.path = "f/ducklake_maintenance/legacy".to_string(); + }); + let authed = make_authed(); + + let tx = db.begin().await?; + let tx = push_scheduled_job(&db, tx, &schedule, Some(&authed), None).await?; + tx.commit().await?; + + assert_eq!(count_queued_jobs(&db).await, 1); + let (ws, path, trigger, _) = get_queued_job(&db).await.unwrap(); + assert_eq!(ws, "test-workspace"); + assert_eq!( + path.as_deref(), + Some("f/system/test_script"), + "must resolve the schedule's script_path, not the maintenance builder" + ); + assert_eq!(trigger.as_deref(), Some("f/ducklake_maintenance/legacy")); + Ok(()) + } + + // With maintenance enabled for the path's lake, the occurrence is a + // raw-code duckdb job (kind preview, runnable_path = schedule path, + // duckdb tag pinned) — enterprise builds only. + #[cfg(feature = "private")] + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn test_push_reserved_prefix_with_config_builds_maintenance_job( + db: Pool, + ) -> anyhow::Result<()> { + sqlx::query( + r#"UPDATE workspace_settings SET ducklake = '{"ducklakes": {"legacy": { + "catalog": {"resource_type": "postgresql", "resource_path": "u/test/pg"}, + "storage": {"path": "legacy"}, + "maintenance": {"enabled": true, "retention_days": 3} + }}}'::jsonb WHERE workspace_id = 'test-workspace'"#, + ) + .execute(&db) + .await?; + + let schedule = make_schedule(|s| { + s.path = "f/ducklake_maintenance/legacy".to_string(); + s.script_path = "f/ducklake_maintenance/legacy".to_string(); + s.tag = Some("duckdb".to_string()); + }); + let authed = make_authed(); + + let tx = db.begin().await?; + let tx = push_scheduled_job(&db, tx, &schedule, Some(&authed), None).await?; + tx.commit().await?; + + assert_eq!(count_queued_jobs(&db).await, 1); + let (kind, path, tag, raw_code) = + sqlx::query_as::<_, (String, Option, String, Option)>( + "SELECT j.kind::text, j.runnable_path, j.tag, j.raw_code + FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + ) + .fetch_one(&db) + .await?; + assert_eq!(kind, "preview"); + assert_eq!(path.as_deref(), Some("f/ducklake_maintenance/legacy")); + assert_eq!(tag, "duckdb"); + let raw_code = raw_code.expect("maintenance job must carry generated SQL"); + assert!(raw_code.contains("ducklake_expire_snapshots")); + assert!(raw_code.contains("INTERVAL '3 days'")); + Ok(()) + } + + // Saving maintenance off must remove the managed row AND its queued + // occurrence in BOTH builds: the enterprise sync reconciles, and the + // public stub must not leave a job pushed under the enterprise edition + // to run after the admin disabled maintenance (EE-to-CE downgrade). + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn test_sync_disable_clears_managed_row_and_queued_occurrence( + db: Pool, + ) -> anyhow::Result<()> { + use std::collections::HashMap; + use windmill_common::workspaces::{ + Ducklake, DucklakeCatalog, DucklakeCatalogResourceType, DucklakeMaintenance, + DucklakeStorage, + }; + use windmill_queue::ducklake_maintenance::sync_ducklake_maintenance_schedules; + + fn lake(maintenance_enabled: bool) -> Ducklake { + Ducklake { + catalog: DucklakeCatalog { + resource_type: DucklakeCatalogResourceType::Postgresql, + resource_path: "u/test/pg".to_string(), + }, + storage: DucklakeStorage { storage: None, path: "legacy".to_string() }, + extra_args: None, + fork_behavior: None, + maintenance: Some(DucklakeMaintenance { + enabled: maintenance_enabled, + schedule: None, + retention_days: None, + compaction: None, + orphan_cleanup: None, + }), + } + } + + // a managed row with a queued occurrence (queued via fall-through: no + // lake config exists yet, so the push resolves the script path) + let schedule = make_schedule(|s| { + s.path = "f/ducklake_maintenance/legacy".to_string(); + }); + sqlx::query( + "INSERT INTO schedule (workspace_id, path, schedule, timezone, edited_by, script_path, + is_flow, enabled, email, permissioned_as, cron_version) + VALUES ($1, $2, $3, 'UTC', $4, $5, false, true, $6, $7, 'v2')", + ) + .bind(&schedule.workspace_id) + .bind(&schedule.path) + .bind(&schedule.schedule) + .bind(&schedule.edited_by) + .bind(&schedule.script_path) + .bind(&schedule.email) + .bind(&schedule.permissioned_as) + .execute(&db) + .await?; + let authed = make_authed(); + let tx = db.begin().await?; + let tx = push_scheduled_job(&db, tx, &schedule, Some(&authed), None).await?; + tx.commit().await?; + assert_eq!(count_queued_jobs(&db).await, 1); + + // maintenance saved off + let previous = HashMap::from([("legacy".to_string(), lake(true))]); + let current = HashMap::from([("legacy".to_string(), lake(false))]); + let tx = db.begin().await?; + let tx = sync_ducklake_maintenance_schedules( + &db, + tx, + &schedule.workspace_id, + ¤t, + &previous, + "test-user", + "test@windmill.dev", + ) + .await?; + tx.commit().await?; + + assert_eq!( + count_queued_jobs(&db).await, + 0, + "queued occurrence must be cleared when maintenance is saved off" + ); + let row_exists: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM schedule WHERE workspace_id = $1 AND path = $2)", + ) + .bind(&schedule.workspace_id) + .bind(&schedule.path) + .fetch_one(&db) + .await?; + assert!(!row_exists, "managed schedule row must be deleted"); + Ok(()) + } } diff --git a/backend/windmill-store/Cargo.toml b/backend/windmill-store/Cargo.toml index b3aca5e669..fdd82a7e5d 100644 --- a/backend/windmill-store/Cargo.toml +++ b/backend/windmill-store/Cargo.toml @@ -53,3 +53,7 @@ futures.workspace = true chrono.workspace = true reqwest.workspace = true anyhow.workspace = true +base64.workspace = true + +[dev-dependencies] +magic-crypt.workspace = true diff --git a/backend/windmill-store/src/resources.rs b/backend/windmill-store/src/resources.rs index 44db0a1b68..98f91065e4 100644 --- a/backend/windmill-store/src/resources.rs +++ b/backend/windmill-store/src/resources.rs @@ -1088,21 +1088,48 @@ async fn create_resource( .execute(&db) .await?; } - sqlx::query!( - "INSERT INTO resource - (workspace_id, path, value, description, resource_type, created_by, edited_at, labels) - VALUES ($1, $2, $3, $4, $5, $6, now(), $7) ON CONFLICT (workspace_id, path) - DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description, resource_type = EXCLUDED.resource_type, edited_at = now(), labels = EXCLUDED.labels", - w_id, - resource.path, - raw_json as sqlx::types::Json<&RawValue>, - resource.description, - resource.resource_type, - authed.username, - resource.labels.as_deref() as Option<&[String]> - ) - .execute(&mut *tx) - .await?; + if update_if_exists { + sqlx::query!( + "INSERT INTO resource + (workspace_id, path, value, description, resource_type, created_by, edited_at, labels) + VALUES ($1, $2, $3, $4, $5, $6, now(), $7) ON CONFLICT (workspace_id, path) + DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description, resource_type = EXCLUDED.resource_type, edited_at = now(), labels = EXCLUDED.labels", + w_id, + resource.path, + raw_json as sqlx::types::Json<&RawValue>, + resource.description, + resource.resource_type, + authed.username, + resource.labels.as_deref() as Option<&[String]> + ) + .execute(&mut *tx) + .await?; + } else { + // Create-only (the default): DO NOTHING + a row-count guard, so a path that appears between + // check_path_conflict above and this insert is rejected rather than overwritten. A plain + // DO UPDATE here would clobber a concurrently-created resource, breaking create-only callers + // (e.g. Compare & Deploy "Create in "). + let inserted = sqlx::query!( + "INSERT INTO resource + (workspace_id, path, value, description, resource_type, created_by, edited_at, labels) + VALUES ($1, $2, $3, $4, $5, $6, now(), $7) ON CONFLICT (workspace_id, path) DO NOTHING", + w_id, + resource.path, + raw_json as sqlx::types::Json<&RawValue>, + resource.description, + resource.resource_type, + authed.username, + resource.labels.as_deref() as Option<&[String]> + ) + .execute(&mut *tx) + .await?; + if inserted.rows_affected() == 0 { + return Err(Error::BadRequest(format!( + "Resource {} already exists", + resource.path + ))); + } + } // Mirror update_resource: Some(true) inserts, Some(false) clears (only // meaningful on the upsert path, since a pure create has no existing row), @@ -1385,7 +1412,12 @@ fn collect_var_refs(value: &serde_json::Value, out: &mut Vec) { } } -async fn mark_linked_variables_ws_specific( +/// Marks every variable referenced by the resource at `resource_path` as workspace-specific. +/// +/// AUTH CONTRACT: this mutates `ws_specific` and does NOT check authorization itself. The caller +/// MUST verify that `authed` has write access to the resource at `resource_path` in `w_id` (e.g. via +/// `require_owner_of_path`) before calling it. +pub async fn mark_linked_variables_ws_specific( tx: &mut Transaction<'_, Postgres>, authed: &ApiAuthed, w_id: &str, @@ -1634,6 +1666,12 @@ async fn update_resource( let path = path.to_path(); check_scopes(&authed, || format!("resources:write:{}", path))?; + // A rename moves the resource (and its linked variable) to ns.path, so the + // destination must also be within the token's write scope, not just the + // source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("resources:write:{}", npath))?; + } if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, AuditAuthorable::username(&authed), diff --git a/backend/windmill-store/src/secret_backend_ext.rs b/backend/windmill-store/src/secret_backend_ext.rs index 1ae32c80ce..d5272384c0 100644 --- a/backend/windmill-store/src/secret_backend_ext.rs +++ b/backend/windmill-store/src/secret_backend_ext.rs @@ -6,269 +6,27 @@ * LICENSE-AGPL for a copy of the license. */ -//! Secret backend extension for the API layer +//! Secret backend extension for the store layer //! -//! This module provides helper functions for integrating the SecretBackend -//! trait with variable operations in the API. +//! Write-side helpers for integrating the SecretBackend trait with variable +//! operations. Backend resolution and read helpers live in +//! `windmill_common::secret_backend` (so lower-level crates can resolve secrets +//! too) and are re-exported here for existing callers. //! //! Note: HashiCorp Vault integration requires Enterprise Edition. //! The OSS version only supports the database backend. -use std::sync::Arc; - use windmill_common::{ db::DB, error::{Error, Result}, - secret_backend::{database::DatabaseBackend, SecretBackend}, - variables::{build_crypt, decrypt, encrypt}, + variables::{build_crypt, encrypt}, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::{ - global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, - secret_backend::{ - AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, - AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, - }, +pub use windmill_common::secret_backend::{ + get_secret_backend, get_secret_value, is_aws_sm_stored_value, is_azure_kv_stored_value, + is_external_stored_value, is_vault_backend_configured, is_vault_stored_value, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use tokio::sync::RwLock; - -// Cached Vault backend to avoid recreating it for every request -// This enables connection pooling and avoids repeated setup overhead -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedVaultBackend { - backend: Arc, - settings: VaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAzureKvBackend { - backend: Arc, - settings: AzureKeyVaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached AWS Secrets Manager backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAwsSmBackend { - backend: Arc, - settings: AwsSecretsManagerSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -/// Get the current secret backend based on global settings -/// -/// OSS: Always returns DatabaseBackend -/// EE: Returns configured backend (Database or Vault) -#[cfg(not(all(feature = "private", feature = "enterprise")))] -pub async fn get_secret_backend(db: &DB) -> Result> { - Ok(Arc::new(DatabaseBackend::new(db.clone()))) -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -pub async fn get_secret_backend(db: &DB) -> Result> { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - match config { - SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), - SecretBackendConfig::HashiCorpVault(settings) => { - get_or_create_vault_backend(db, settings).await - } - SecretBackendConfig::AzureKeyVault(settings) => { - get_or_create_azure_kv_backend(db, settings).await - } - SecretBackendConfig::AwsSecretsManager(settings) => { - get_or_create_aws_sm_backend(db, settings).await - } - } -} - -/// Get a cached Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_vault_backend( - _db: &DB, - settings: VaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = VAULT_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = VAULT_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = { - #[cfg(feature = "openidconnect")] - if settings.token.is_none() { - Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) - } else { - Arc::new(VaultBackend::new(settings.clone())) - } - - #[cfg(not(feature = "openidconnect"))] - Arc::new(VaultBackend::new(settings.clone())) - }; - - // Cache it - *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached Azure Key Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_azure_kv_backend( - _db: &DB, - settings: AzureKeyVaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = AZURE_KV_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = AZURE_KV_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); - - // Cache it - *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached AWS SM backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_aws_sm_backend( - _db: &DB, - settings: AwsSecretsManagerSettings, -) -> Result> { - { - let cache = AWS_SM_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - let mut cache = AWS_SM_BACKEND_CACHE.write().await; - - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - let backend: Arc = - Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); - - *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Check if a Vault backend is currently configured -/// -/// OSS: Always returns false -/// EE: Checks global settings -#[cfg(not(all(feature = "private", feature = "enterprise")))] -pub async fn is_vault_backend_configured(_db: &DB) -> Result { - Ok(false) -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -pub async fn is_vault_backend_configured(db: &DB) -> Result { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - Ok(matches!( - config, - SecretBackendConfig::HashiCorpVault(_) - | SecretBackendConfig::AzureKeyVault(_) - | SecretBackendConfig::AwsSecretsManager(_) - )) -} - -/// Get a secret value using the configured backend -/// -/// For database backend: decrypts using workspace key -/// For vault backend (EE only): fetches from Vault directly -pub async fn get_secret_value( - db: &DB, - workspace_id: &str, - path: &str, - encrypted_value: &str, -) -> Result { - let backend = get_secret_backend(db).await?; - - match backend.backend_name() { - "database" => { - // Use existing database decryption - let mc = build_crypt(db, workspace_id).await?; - decrypt(&mc, encrypted_value.to_string()).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) - }) - } - "hashicorp_vault" => { - // Fetch from Vault directly - backend.get_secret(workspace_id, path).await - } - "azure_key_vault" => backend.get_secret(workspace_id, path).await, - "aws_secrets_manager" => backend.get_secret(workspace_id, path).await, - _ => Err(Error::internal_err(format!( - "Unknown backend: {}", - backend.backend_name() - ))), - } -} - /// Store a secret value using the configured backend /// /// For database backend: encrypts using workspace key and returns encrypted value @@ -412,26 +170,6 @@ pub async fn delete_secret_from_backend(db: &DB, workspace_id: &str, path: &str) } } -/// Check if a value is stored in Vault (indicated by the $vault: prefix) -pub fn is_vault_stored_value(value: &str) -> bool { - value.starts_with("$vault:") -} - -/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) -pub fn is_azure_kv_stored_value(value: &str) -> bool { - value.starts_with("$azure_kv:") -} - -/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix) -pub fn is_aws_sm_stored_value(value: &str) -> bool { - value.starts_with("$aws_sm:") -} - -/// Check if a value is stored in any external secret backend -pub fn is_external_stored_value(value: &str) -> bool { - is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) -} - /// Rename a secret in Vault when a variable path changes (EE only) #[cfg(not(all(feature = "private", feature = "enterprise")))] pub async fn rename_vault_secret( diff --git a/backend/windmill-store/src/variables.rs b/backend/windmill-store/src/variables.rs index 72ed20cb71..55047ec0c2 100644 --- a/backend/windmill-store/src/variables.rs +++ b/backend/windmill-store/src/variables.rs @@ -14,8 +14,8 @@ use windmill_common::db::DB; use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult}; use crate::secret_backend_ext::{ - delete_secret_from_backend, get_secret_value, is_vault_stored_value, rename_vault_secret, - store_secret_value, + delete_secret_from_backend, get_secret_value, is_external_stored_value, is_vault_stored_value, + rename_vault_secret, store_secret_value, }; use windmill_common::utils::{escape_ilike_pattern, BulkDeleteRequest}; use windmill_common::webhook::{WebhookMessage, WebhookShared}; @@ -25,6 +25,7 @@ use axum::{ routing::{delete, get, post}, Json, Router, }; +use base64::{engine::general_purpose::STANDARD, Engine as _}; use futures::future::try_join_all; use hyper::StatusCode; use serde_json::Value; @@ -535,6 +536,35 @@ async fn check_path_conflict(db: &DB, w_id: &str, path: &str) -> Result<()> { return Ok(()); } +/// Reject a secret value flagged as already-encrypted (`already_encrypted=true`) +/// that is not actually workspace-key ciphertext — e.g. plaintext mistakenly +/// pushed as encrypted. Storing plaintext in the encrypted `value` column +/// silently bricks the variable: every later read fails to decrypt it. +/// +/// The check is purely structural and never decrypts, so it cannot act as a +/// decryption/padding oracle for a caller who can write but not read secrets. +/// `encrypt` (AES-256-CBC) always yields standard base64 decoding to a non-zero +/// multiple of the 16-byte block size; anything else cannot be our ciphertext. +/// Values stored by an external backend ($vault:/$aws_sm:/$azure_kv: markers) +/// are not workspace ciphertext and are passed through untouched. +fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> { + if is_external_stored_value(value) { + return Ok(()); + } + let looks_like_ciphertext = STANDARD + .decode(value) + .map(|bytes| !bytes.is_empty() && bytes.len() % 16 == 0) + .unwrap_or(false); + if !looks_like_ciphertext { + return Err(Error::BadRequest(format!( + "Variable {path} was sent as already-encrypted (already_encrypted=true) but its \ + value is not valid workspace-encrypted ciphertext. To push a plaintext secret, \ + send it without already_encrypted (CLI: use --plain-secrets) so it gets encrypted." + ))); + } + Ok(()) +} + async fn create_variable( authed: ApiAuthed, Extension(db): Extension, @@ -585,6 +615,11 @@ async fn create_variable( // Use secret backend for encryption (supports both DB and Vault) store_secret_value(&db, &w_id, &variable.path, &plain).await? } else { + if variable.is_secret { + // already_encrypted == true: value is stored verbatim, so it must be + // ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(&variable.path, &variable.value)?; + } variable.value }; @@ -1037,6 +1072,12 @@ async fn update_variable( let path = path.to_path(); check_scopes(&authed, || format!("variables:write:{}", path))?; + // A rename moves the (possibly secret) variable to ns.path, so the + // destination must also be within the token's write scope, not just the + // source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("variables:write:{}", npath))?; + } let authed = maybe_refresh_folders(&path, &w_id, authed, &db).await; let mut sqlb = SqlBuilder::update_table("variable"); @@ -1076,6 +1117,11 @@ async fn update_variable( // Store at target_path (new path if renaming, otherwise current path) store_secret_value(&db, &w_id, target_path, &plain).await? } else { + if is_secret { + // already_encrypted == true: value is stored verbatim, so it must + // be ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(target_path, &nvalue)?; + } nvalue }; sqlb.set_str("value", &value); @@ -1507,3 +1553,61 @@ pub async fn get_value_internal<'a>( Ok(r) } + +#[cfg(test)] +mod tests { + use super::*; + use magic_crypt::MagicCryptTrait; + + #[test] + fn accepts_real_workspace_ciphertext() { + // The exact shape produced by `encrypt` (AES-256-CBC, base64). + let mc = magic_crypt::new_magic_crypt!("a-test-workspace-key", 256); + for plain in [ + "", + "original-secret", + "some: plaintext\n", + "a".repeat(500).as_str(), + ] { + let ciphertext = mc.encrypt_str_to_base64(plain); + assert!( + validate_already_encrypted_secret("f/x/cfg", &ciphertext).is_ok(), + "should accept genuine ciphertext for plaintext {plain:?}: {ciphertext}" + ); + } + } + + #[test] + fn rejects_plaintext_mislabeled_as_encrypted() { + // Plaintext mislabeled as encrypted: storing it verbatim would make the + // variable undecryptable on every read, so it must be rejected. + for plaintext in [ + "some: plaintext\n", + "original-secret", + "hunter2", + "{\"a\": 1}", + "not base64!!", + " leading-space", + ] { + assert!( + validate_already_encrypted_secret("f/x/cfg", plaintext).is_err(), + "should reject plaintext mislabeled as encrypted: {plaintext:?}" + ); + } + } + + #[test] + fn rejects_empty_and_non_block_aligned() { + // Valid base64 but not a whole number of AES blocks -> cannot be our ciphertext. + assert!(validate_already_encrypted_secret("p", "").is_err()); + assert!(validate_already_encrypted_secret("p", "dGVzdA==").is_err()); // "test" -> 4 bytes + } + + #[test] + fn passes_through_external_backend_markers() { + // External secret backends store $-prefixed markers, not workspace ciphertext. + for marker in ["$vault:f/x/cfg", "$aws_sm:f/x/cfg", "$azure_kv:f/x/cfg"] { + assert!(validate_already_encrypted_secret("f/x/cfg", marker).is_ok()); + } + } +} diff --git a/backend/windmill-trigger-http/src/handler.rs b/backend/windmill-trigger-http/src/handler.rs index 7b617e1b8d..17321b7775 100644 --- a/backend/windmill-trigger-http/src/handler.rs +++ b/backend/windmill-trigger-http/src/handler.rs @@ -7,7 +7,7 @@ use axum::{extract::Path, routing::post, Extension, Json, Router}; use http::StatusCode; use sqlx::PgConnection; use std::collections::HashSet; -use windmill_api_auth::ApiAuthed; +use windmill_api_auth::{check_scopes, ApiAuthed}; use windmill_audit::{audit_oss::audit_log, ActionKind}; use windmill_common::global_settings::HTTP_ROUTE_WORKSPACED_ROUTE; use windmill_common::{ @@ -262,6 +262,12 @@ pub async fn create_many_http_triggers( let mut route_path_keys = Vec::with_capacity(new_http_triggers.len()); for new_http_trigger in new_http_triggers.iter() { + // Per-item write scope, matching the single-create handler. The bulk + // endpoint must not let a path-scoped token create triggers outside it. + check_scopes(&authed, || { + format!("http_triggers:write:{}", &new_http_trigger.base.path) + })?; + handler .validate_new(&db, &w_id, &new_http_trigger.config) .await @@ -373,7 +379,8 @@ impl TriggerCrud for HttpTrigger { const TABLE_NAME: &'static str = "http_trigger"; const TRIGGER_TYPE: &'static str = "http"; - const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerHttp; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = + windmill_common::user_drafts::UserDraftItemKind::TriggerHttp; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/http_triggers"; diff --git a/backend/windmill-trigger-websocket/src/handler.rs b/backend/windmill-trigger-websocket/src/handler.rs index ed8e95b164..529e5902cc 100644 --- a/backend/windmill-trigger-websocket/src/handler.rs +++ b/backend/windmill-trigger-websocket/src/handler.rs @@ -4,7 +4,7 @@ use async_trait::async_trait; use itertools::Itertools; use serde_json::value::RawValue; use sqlx::{types::Json as SqlxJson, PgConnection}; -use windmill_api_auth::ApiAuthed; +use windmill_api_auth::{check_scopes, ApiAuthed}; use windmill_common::DB; use windmill_common::{ db::UserDB, @@ -15,10 +15,39 @@ use windmill_git_sync::DeployedObject; use windmill_trigger::{Trigger, TriggerCrud, TriggerData}; use super::{ - get_url_from_runnable_value, proxy::connect_async_with_proxy, validate_websocket_url_for_ssrf, - TestWebsocketConfig, WebsocketConfig, WebsocketConfigRequest, WebsocketTrigger, + get_url_from_runnable_value, listener::InitialMessage, proxy::connect_async_with_proxy, + validate_websocket_url_for_ssrf, TestWebsocketConfig, WebsocketConfig, WebsocketConfigRequest, + WebsocketTrigger, }; +/// A websocket_triggers:write token can configure secondary runnables that the +/// listener later executes under the trigger owner's identity: a `$flow:`/ +/// `$script:` URL resolver and `initial_messages` of kind `runnable_result`. +/// That execution happens in a background task where the reconstructed authed is +/// scopeless (so its check_scopes is a no-op), so enforce run scope here, at +/// create/update time, against the API caller's token. +fn check_secondary_runnable_scopes( + authed: &ApiAuthed, + config: &WebsocketConfigRequest, +) -> Result<()> { + if let Some(rest) = config.url.strip_prefix("$flow:") { + check_scopes(authed, || format!("jobs:run:flows:{}", rest))?; + } else if let Some(rest) = config.url.strip_prefix("$script:") { + check_scopes(authed, || format!("jobs:run:scripts:{}", rest))?; + } + if let Some(messages) = config.initial_messages.as_ref() { + for msg in messages { + if let Ok(InitialMessage::RunnableResult { path, is_flow, .. }) = + serde_json::from_value::(msg.clone()) + { + let kind = if is_flow { "flows" } else { "scripts" }; + check_scopes(authed, || format!("jobs:run:{}:{}", kind, path))?; + } + } + } + Ok(()) +} + #[async_trait] impl TriggerCrud for WebsocketTrigger { type TriggerConfig = WebsocketConfig; @@ -101,6 +130,7 @@ impl TriggerCrud for WebsocketTrigger { w_id: &str, trigger: TriggerData, ) -> Result<()> { + check_secondary_runnable_scopes(authed, &trigger.config)?; let resolved_edited_by = trigger.base.resolve_edited_by(authed); let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed); let filters = trigger @@ -178,6 +208,7 @@ impl TriggerCrud for WebsocketTrigger { path: &str, trigger: TriggerData, ) -> Result<()> { + check_secondary_runnable_scopes(authed, &trigger.config)?; let resolved_edited_by = trigger.base.resolve_edited_by(authed); let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed); let filters = trigger diff --git a/backend/windmill-trigger-websocket/src/listener.rs b/backend/windmill-trigger-websocket/src/listener.rs index 31e3f83995..e34fe6e4f6 100644 --- a/backend/windmill-trigger-websocket/src/listener.rs +++ b/backend/windmill-trigger-websocket/src/listener.rs @@ -11,7 +11,7 @@ use serde::Deserialize; use serde_json::value::RawValue; use std::{borrow::Cow, collections::HashMap, sync::Arc}; use tokio::{net::TcpStream, sync::RwLock}; -use tokio_tungstenite::{tungstenite::Message, MaybeTlsStream, WebSocketStream}; +use tokio_tungstenite::{tungstenite, tungstenite::Message, MaybeTlsStream, WebSocketStream}; use windmill_common::{ error::{to_anyhow, Error, Result}, jobs::JobTriggerKind, @@ -22,13 +22,30 @@ use windmill_common::{ }; use windmill_queue::PushArgsOwned; use windmill_trigger::filter::{check_filters, Filter}; -use windmill_trigger::listener::ListeningTrigger; +use windmill_trigger::listener::{update_rw_lock, ListeningTrigger}; use windmill_trigger::trigger_helpers::{ trigger_runnable, trigger_runnable_and_wait_for_raw_result, trigger_runnable_and_wait_for_raw_result_with_error_ctx, TriggerJobArgs, }; use windmill_trigger::Listener; +const MAX_CONNECT_ATTEMPTS: u32 = 5; + +/// Whether a failed WebSocket connect is worth retrying: network-level IO +/// errors and HTTP 5xx/429 handshake responses are typically transient (edge +/// proxies like Cloudflare return 502/520 sporadically), while other errors +/// (bad URL, protocol mismatch, other 4xx) point at configuration and would +/// fail identically on every attempt. +fn is_transient_connect_error(err: &tungstenite::Error) -> bool { + match err { + tungstenite::Error::Io(_) => true, + tungstenite::Error::Http(resp) => { + resp.status().is_server_error() || resp.status().as_u16() == 429 + } + _ => false, + } +} + async fn send_initial_messages( listening_trigger: &ListeningTrigger, writer: &mut SplitSink>, Message>, @@ -176,12 +193,50 @@ impl Listener for WebsocketTrigger { validate_websocket_url_for_ssrf(&connect_url).await?; - let connection = connect_async_with_proxy(&*connect_url) - .await - .map(|conn| Some(conn)) - .map_err(|err| to_anyhow(err).into()); - - connection + // Gateway endpoints are often fronted by an edge proxy (e.g. Cloudflare) + // that sporadically answers the upgrade request with a transient 5xx + // instead of `101 Switching Protocols`, and a `get_consumer` error + // disables the trigger until a human re-enables it — so retry transient + // failures with backoff before giving up. The caller runs `loop_ping` + // concurrently so `last_server_ping` stays alive across the sleeps, and + // killpill cancels this future between awaits. + let mut attempt = 0; + loop { + attempt += 1; + match connect_async_with_proxy(&*connect_url).await { + Ok(conn) => return Ok(Some(conn)), + // Only retry in trigger mode: a failed connect there disables the + // trigger until a human re-enables it, while capture mode is an + // interactive test where instant feedback beats resilience (an + // `Io` error can also be a permanent misconfiguration, e.g. a + // typo'd host, which should surface immediately when iterating). + Err(err) + if listening_trigger.trigger_mode + && attempt < MAX_CONNECT_ATTEMPTS + && is_transient_connect_error(&err) => + { + let delay_secs = 1u64 << attempt; + tracing::warn!( + "Transient error connecting to WebSocket for trigger {} (attempt {}/{}), retrying in {}s: {}", + listening_trigger.path, + attempt, + MAX_CONNECT_ATTEMPTS, + delay_secs, + err + ); + update_rw_lock( + err_message.clone(), + Some(format!( + "Connection attempt {}/{} failed ({}), retrying in {}s...", + attempt, MAX_CONNECT_ATTEMPTS, err, delay_secs + )), + ) + .await; + tokio::time::sleep(std::time::Duration::from_secs(delay_secs)).await; + } + Err(err) => return Err(to_anyhow(err).into()), + } + } } async fn consume( &self, @@ -509,7 +564,7 @@ impl Clone for ReturnMessageChannels { } #[derive(Debug, Deserialize)] -enum InitialMessage { +pub(crate) enum InitialMessage { #[serde(rename = "raw_message")] RawMessage(String), #[serde(rename = "runnable_result")] diff --git a/backend/windmill-trigger-websocket/tests/connect_retry.rs b/backend/windmill-trigger-websocket/tests/connect_retry.rs new file mode 100644 index 0000000000..2d21512692 --- /dev/null +++ b/backend/windmill-trigger-websocket/tests/connect_retry.rs @@ -0,0 +1,153 @@ +//! End-to-end tests for the websocket trigger connect-retry behavior: a mock +//! TCP server rejects the websocket upgrade with a configurable HTTP status a +//! number of times before completing a real handshake, and the tests assert +//! which failures `get_consumer` retries. +//! +//! This lives in an integration-test binary (own process) because it sets +//! ALLOW_PRIVATE_WEBSOCKET_URLS — the mock server listens on 127.0.0.1, which +//! the SSRF check blocks — and that process-global env var must not leak into +//! the crate's unit tests, which assert loopback URLs are rejected. + +use std::sync::{ + atomic::{AtomicU32, Ordering}, + Arc, +}; + +use tokio::{ + io::{AsyncBufReadExt, AsyncWriteExt, BufReader}, + net::TcpListener, + sync::{broadcast, RwLock}, +}; +use windmill_trigger::{listener::ListeningTrigger, Listener}; +use windmill_trigger_websocket::{ + WebsocketConfig, WebsocketTrigger, ALLOW_PRIVATE_WEBSOCKET_URLS_ENV, +}; + +/// Mock server: rejects the first `failures` upgrade requests with +/// `status_line` and closes, then completes real websocket handshakes and +/// parks the connection open. Returns the bound address and the +/// connection-attempt counter. +async fn mock_ws_server( + failures: u32, + status_line: &'static str, +) -> (std::net::SocketAddr, Arc) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let attempts = Arc::new(AtomicU32::new(0)); + let served = attempts.clone(); + tokio::spawn(async move { + loop { + let (mut socket, _) = listener.accept().await.unwrap(); + let n = served.fetch_add(1, Ordering::SeqCst); + if n < failures { + // Drain the request head, then reject the upgrade. + let mut reader = BufReader::new(&mut socket); + let mut line = String::new(); + loop { + line.clear(); + let read = reader.read_line(&mut line).await.unwrap_or(0); + if read == 0 || line == "\r\n" { + break; + } + } + socket + .write_all( + format!("HTTP/1.1 {status_line}\r\nContent-Length: 0\r\n\r\n").as_bytes(), + ) + .await + .ok(); + } else if let Ok(ws) = tokio_tungstenite::accept_async(socket).await { + tokio::spawn(async move { + let _open = ws; + std::future::pending::<()>().await + }); + } + } + }); + (addr, attempts) +} + +fn trigger(url: String, trigger_mode: bool) -> ListeningTrigger { + ListeningTrigger { + path: "f/test/ws".to_string(), + is_flow: false, + workspace_id: "test".to_string(), + edited_by: "test".to_string(), + permissioned_as: "u/test".to_string(), + trigger_config: WebsocketConfig { + url, + filters: vec![], + filter_logic: "and".to_string(), + initial_messages: None, + url_runnable_args: None, + can_return_message: false, + can_return_error_result: false, + heartbeat: None, + }, + script_path: "f/test/script".to_string(), + trigger_mode, + error_handling: None, + suspended_mode: false, + } +} + +async fn get_consumer_result( + lt: &ListeningTrigger, + err_message: Arc>>, +) -> windmill_common::error::Result::Consumer>> { + // The static-URL path of `get_consumer` never touches the DB; a lazy pool + // satisfies the signature without a running postgres. + let db: windmill_common::DB = + sqlx::Pool::connect_lazy("postgres://unused:unused@127.0.0.1:1/unused").unwrap(); + let (_killpill_tx, killpill_rx) = broadcast::channel::<()>(1); + WebsocketTrigger + .get_consumer(&db, lt, err_message, killpill_rx) + .await +} + +#[tokio::test] +async fn transient_502s_are_retried_until_the_upgrade_succeeds() { + std::env::set_var(ALLOW_PRIVATE_WEBSOCKET_URLS_ENV, "true"); + let (addr, attempts) = mock_ws_server(2, "502 Bad Gateway").await; + let lt = trigger(format!("ws://{addr}"), true); + let err_message = Arc::new(RwLock::new(None)); + + let consumer = get_consumer_result(<, err_message.clone()) + .await + .expect("connect should succeed after retries"); + + assert!(consumer.is_some(), "expected an established connection"); + assert_eq!(attempts.load(Ordering::SeqCst), 3); + // Retry progress was reported through the shared status lock. + let status = err_message.read().await.clone().unwrap(); + assert!(status.contains("attempt 2/5"), "got status: {status}"); + assert!(status.contains("502"), "got status: {status}"); +} + +#[tokio::test] +async fn non_transient_http_errors_fail_on_the_first_attempt() { + std::env::set_var(ALLOW_PRIVATE_WEBSOCKET_URLS_ENV, "true"); + let (addr, attempts) = mock_ws_server(u32::MAX, "404 Not Found").await; + let lt = trigger(format!("ws://{addr}"), true); + + let err = get_consumer_result(<, Arc::new(RwLock::new(None))) + .await + .expect_err("a 404 upgrade response should not be retried"); + + assert!(err.to_string().contains("404"), "got error: {err}"); + assert_eq!(attempts.load(Ordering::SeqCst), 1); +} + +#[tokio::test] +async fn capture_mode_fails_fast_even_on_transient_errors() { + std::env::set_var(ALLOW_PRIVATE_WEBSOCKET_URLS_ENV, "true"); + let (addr, attempts) = mock_ws_server(u32::MAX, "502 Bad Gateway").await; + let lt = trigger(format!("ws://{addr}"), false); + + let err = get_consumer_result(<, Arc::new(RwLock::new(None))) + .await + .expect_err("capture mode should surface the first failure immediately"); + + assert!(err.to_string().contains("502"), "got error: {err}"); + assert_eq!(attempts.load(Ordering::SeqCst), 1); +} diff --git a/backend/windmill-trigger/src/global_handler.rs b/backend/windmill-trigger/src/global_handler.rs index 3fbbcddd0e..98b689dda9 100644 --- a/backend/windmill-trigger/src/global_handler.rs +++ b/backend/windmill-trigger/src/global_handler.rs @@ -14,7 +14,7 @@ use windmill_api_jobs::execution::cancel_jobs; use windmill_common::{ db::{UserDB, DB}, error::{self, Error, Result}, - jobs::JobTriggerKind, + jobs::{delete_jobs, JobTriggerKind}, triggers::TriggerMetadata, }; @@ -262,9 +262,7 @@ pub async fn resume_suspended_trigger_jobs( .execute(&mut *tx) .await?; - sqlx::query!("DELETE FROM v2_job WHERE id = $1", job.id) - .execute(&mut *tx) - .await?; + delete_jobs(&mut *tx, &[job.id]).await?; } } diff --git a/backend/windmill-trigger/src/handler.rs b/backend/windmill-trigger/src/handler.rs index 65072bc9ab..ed3b89fc65 100644 --- a/backend/windmill-trigger/src/handler.rs +++ b/backend/windmill-trigger/src/handler.rs @@ -905,6 +905,26 @@ async fn delete_trigger( tx.commit().await?; + // Reset the fork/parent workspace_diff tally for this path, exactly as + // create/update and every other kind's delete does. Without this a deleted + // trigger leaves its cached `has_changes=true` diff row behind: the compare + // trusts it (triggers aren't re-validated like scripts/flows), then drops it + // as it no longer exists in the table — a phantom "ahead" item that reads as + // "changes not visible to your user" and hides the deploy button, even for + // superadmins. Re-tallying sets has_changes=NULL so the next compare + // re-evaluates and corrects/removes the row. + handle_deployment_metadata( + &authed.email, + &authed.username, + &db, + &workspace_id, + T::get_deployed_object(path.to_string(), None), + Some(format!("{} '{}' deleted", T::DEPLOYMENT_NAME, path)), + true, + None, + ) + .await?; + // Trigger gone for everyone: wipe ALL users' drafts at this path; see scripts.rs. delete_all_drafts_for_path(&db, &workspace_id, T::user_draft_item_kind(), path).await?; diff --git a/backend/windmill-trigger/src/trigger_helpers.rs b/backend/windmill-trigger/src/trigger_helpers.rs index 44e5884420..1427f7f102 100644 --- a/backend/windmill-trigger/src/trigger_helpers.rs +++ b/backend/windmill-trigger/src/trigger_helpers.rs @@ -924,6 +924,8 @@ async fn trigger_script_with_retry_and_error_handler<'c>( path, hash: Some(hash), flow_version: None, + // Keep the flow path until native retry covers handler semantics. + language: None, args: HashMap::from(&push_args), retry, error_handler_path, diff --git a/backend/windmill-types/src/flows.rs b/backend/windmill-types/src/flows.rs index 2d1a538124..006185dbd9 100644 --- a/backend/windmill-types/src/flows.rs +++ b/backend/windmill-types/src/flows.rs @@ -155,6 +155,19 @@ fn validate_retry(retry: &Retry, module_id: &str) -> anyhow::Result<()> { Ok(()) } +/// Script/sub-flow step references must be workspace paths (`u/`, `f/`, `g/`) or a hub +/// reference (`hub/`). Empty is tolerated for intermediate/incomplete steps. This blocks +/// absolute or local filesystem paths (e.g. `/tmp/.../ops/scripts/...` baked in by a +/// `wmill sync push` from a feature-branch checkout) from being persisted into a flow, +/// where they silently mis-resolve to an unrelated script at runtime (#9751). +fn is_workspace_runnable_path(path: &str) -> bool { + path.is_empty() + || path.starts_with("u/") + || path.starts_with("f/") + || path.starts_with("g/") + || path.starts_with("hub/") +} + fn validate_flow_value<'de, D>(deserializer: D) -> Result, D::Error> where D: Deserializer<'de>, @@ -164,21 +177,38 @@ where let flow_value: FlowValue = serde_json::from_str(raw_value.get()) .map_err(|e| serde::de::Error::custom(format!("Invalid flow value: {}", e)))?; - FlowModule::traverse_modules(&flow_value.modules, &mut |module| { + let mut validate_module = |module: &FlowModule| -> anyhow::Result<()> { if let Some(ref retry) = module.retry { validate_retry(retry, &module.id)?; } - return Ok(()); - }) - .map_err(|e| serde::de::Error::custom(e.to_string()))?; + if let Ok(FlowModuleValue::Script { path, .. } | FlowModuleValue::Flow { path, .. }) = + module.get_value() + { + if !is_workspace_runnable_path(&path) { + return Err(anyhow::anyhow!( + "step '{}' references '{}', which is not a workspace path (expected u/, \ + f/, g/ or hub/). Absolute or local filesystem paths are not allowed in \ + flow steps.", + module.id, + path + )); + } + } + Ok(()) + }; - if let Some(ref _failure_module) = flow_value.failure_module { - //add validation logic here for failure module - } - - if let Some(ref _preprocessor_module) = flow_value.preprocessor_module { - //add validation logic here for preprocessor module - } + // The API is the authoritative guard (it can be called directly, bypassing the CLI), so + // it must cover every step that resolves a path: the main modules AND the failure / + // preprocessor modules (which can themselves be sub-flows/loops/branches). + let extra_modules: Vec = flow_value + .failure_module + .iter() + .chain(flow_value.preprocessor_module.iter()) + .map(|m| (**m).clone()) + .collect(); + FlowModule::traverse_modules(&flow_value.modules, &mut validate_module) + .and_then(|()| FlowModule::traverse_modules(&extra_modules, &mut validate_module)) + .map_err(|e| serde::de::Error::custom(e.to_string()))?; Ok(raw_value) } @@ -1228,6 +1258,108 @@ mod tests { assert_eq!(val.modules.len(), 1); } + #[test] + fn flow_rejects_absolute_step_path() { + // #9751: an absolute local path baked into a step must be rejected on deploy. + let bad = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "validate_onboard_target", + "value": { + "type": "script", + "path": "/tmp/tmp.X/f/ops/scripts/clean_device/pre_clean", + "input_transforms": {} + } + }]} + }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "unexpected error: {err}" + ); + assert!( + err.contains("validate_onboard_target"), + "error should name the step: {err}" + ); + } + + #[test] + fn flow_rejects_absolute_step_path_in_nested_module() { + let bad = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "loop", + "value": { + "type": "forloopflow", + "iterator": {"type": "javascript", "expr": "[1]"}, + "modules": [{ + "id": "inner", + "value": {"type": "script", "path": "/abs/path", "input_transforms": {}} + }] + } + }]} + }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "unexpected error: {err}" + ); + } + + #[test] + fn flow_rejects_absolute_path_in_failure_and_preprocessor_modules() { + for slot in ["failure_module", "preprocessor_module"] { + // Build the value with the slot as an explicit (interpolated) key. + let mut value = serde_json::Map::new(); + value.insert("modules".to_string(), json!([])); + value.insert( + slot.to_string(), + json!({ + "id": slot, + "value": {"type": "script", "path": "/abs/path", "input_transforms": {}} + }), + ); + let bad = json!({ "path": "f/test/flow", "summary": "", "value": value }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "{slot} should be validated, got: {err}" + ); + } + } + + #[test] + fn flow_accepts_workspace_step_paths() { + for p in [ + "f/ops/scripts/x", + "u/me/y", + "g/grp/z", + "hub/123/foo", + "", // tolerated for incomplete steps + ] { + let ok = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "a", + "value": {"type": "script", "path": p, "input_transforms": {}} + }]} + }); + assert!( + serde_json::from_value::(ok).is_ok(), + "path {p:?} should be accepted" + ); + } + } + #[test] fn ai_agent_omit_output_from_conversation_defaults_to_false() { let input = json!({ diff --git a/backend/windmill-types/src/jobs.rs b/backend/windmill-types/src/jobs.rs index ac15430759..f9372b949d 100644 --- a/backend/windmill-types/src/jobs.rs +++ b/backend/windmill-types/src/jobs.rs @@ -46,6 +46,9 @@ pub enum JobTriggerKind { // A run dispatched because an upstream pipeline script wrote an asset // this runnable subscribes to via `// on s3://...` annotations. Asset, + // A run pushed by the pipeline freshness watchdog (EE) because the + // script's `// freshness` window elapsed without a successful run. + Freshness, } impl std::fmt::Display for JobTriggerKind { @@ -68,6 +71,7 @@ impl std::fmt::Display for JobTriggerKind { JobTriggerKind::Github => "github", JobTriggerKind::CiTest => "ci_test", JobTriggerKind::Asset => "asset", + JobTriggerKind::Freshness => "freshness", }; write!(f, "{}", kind) } @@ -232,6 +236,14 @@ pub struct QueuedJob { pub runnable_settings_handle: Option, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + // True when this job is a native retry attempt (has a native_retry_attempt + // marker). Lets the run-page chain distinguish real retries from other + // same-script children (e.g. WAC inline children). The list and single-job + // GET endpoints select it; `#[sqlx(default)]` lets any other query omit the + // column and default to None. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_retry: Option, } impl QueuedJob { @@ -307,6 +319,7 @@ impl Default for QueuedJob { preprocessed: None, runnable_settings_handle: None, labels: None, + is_retry: None, } } } @@ -362,6 +375,14 @@ pub struct CompletedJob { pub labels: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub preprocessed: Option, + // True when this job is a native retry attempt (has a native_retry_attempt + // marker). Lets the run-page chain distinguish real retries from other + // same-script children (e.g. WAC inline children). The list and single-job + // GET endpoints select it; `#[sqlx(default)]` lets any other query omit the + // column and default to None. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_retry: Option, } impl CompletedJob { @@ -473,6 +494,10 @@ pub enum JobPayload { path: String, hash: Option, flow_version: Option, + // Set when wrapping a script (not a flow). Lets `push` materialize a + // bare-script-with-retry as a native retryable `Script` job instead of + // spawning a one-step flow. + language: Option, args: HashMap>, retry: Option, error_handler_path: Option, diff --git a/backend/windmill-types/src/runnable_settings.rs b/backend/windmill-types/src/runnable_settings.rs index dfff0df79e..ea0fd30dc0 100644 --- a/backend/windmill-types/src/runnable_settings.rs +++ b/backend/windmill-types/src/runnable_settings.rs @@ -1,9 +1,75 @@ use serde::{Deserialize, Serialize}; +use crate::flows::{ConstantDelay, ExponentialDelay, Retry, RetryIf}; + #[derive(Deserialize, Clone, Copy, Serialize, Default, Hash)] pub struct RunnableSettings { pub debouncing_settings: Option, pub concurrency_settings: Option, + pub retry_settings: Option, +} + +/// Flattened, dedup-friendly representation of a [`Retry`] policy. Native script +/// retry stores the policy here (via `runnable_settings_handle`) instead of +/// wrapping the script in a one-step flow. +#[derive( + Debug, Default, Clone, Serialize, Deserialize, Hash, PartialEq, sqlx::FromRow, sqlx::Decode, +)] +pub struct RetrySettings { + #[serde(skip_serializing_if = "Option::is_none")] + pub constant_attempts: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub constant_seconds: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_attempts: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_multiplier: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_seconds: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_random_factor: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub retry_if_expr: Option, +} + +impl From<&Retry> for RetrySettings { + fn from(r: &Retry) -> Self { + Self { + // attempts are u32; saturate the narrowing to i32 (the seconds/ + // multiplier/random_factor fields are u16/i8 and can't overflow i32). + constant_attempts: Some(r.constant.attempts.min(i32::MAX as u32) as i32), + constant_seconds: Some(r.constant.seconds as i32), + exponential_attempts: Some(r.exponential.attempts.min(i32::MAX as u32) as i32), + exponential_multiplier: Some(r.exponential.multiplier as i32), + exponential_seconds: Some(r.exponential.seconds as i32), + exponential_random_factor: r.exponential.random_factor.map(|x| x as i32), + retry_if_expr: r.retry_if.as_ref().map(|x| x.expr.clone()), + } + } +} + +impl From for Retry { + fn from(s: RetrySettings) -> Self { + Retry { + constant: ConstantDelay { + attempts: s.constant_attempts.unwrap_or(0).max(0) as u32, + seconds: s.constant_seconds.unwrap_or(0).clamp(0, u16::MAX as i32) as u16, + }, + exponential: ExponentialDelay { + attempts: s.exponential_attempts.unwrap_or(0).max(0) as u32, + // Mirror ExponentialDelay::default().multiplier (1) when absent. + multiplier: s + .exponential_multiplier + .unwrap_or(1) + .clamp(0, u16::MAX as i32) as u16, + seconds: s.exponential_seconds.unwrap_or(0).clamp(0, u16::MAX as i32) as u16, + random_factor: s + .exponential_random_factor + .map(|x| x.clamp(i8::MIN as i32, i8::MAX as i32) as i8), + }, + retry_if: s.retry_if_expr.map(|expr| RetryIf { expr }), + } + } } // TODO: Add validation logic. @@ -111,3 +177,56 @@ impl From for ConcurrencySettings { } } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::flows::{ConstantDelay, ExponentialDelay, RetryIf}; + + #[test] + fn retry_settings_roundtrips_retry() { + let cases = [ + // constant only + Retry { + constant: ConstantDelay { attempts: 3, seconds: 5 }, + exponential: ExponentialDelay::default(), + retry_if: None, + }, + // exponential with jitter + Retry { + constant: ConstantDelay::default(), + exponential: ExponentialDelay { + attempts: 4, + multiplier: 2, + seconds: 3, + random_factor: Some(20), + }, + retry_if: None, + }, + // mixed + retry_if + max-ish narrowings + Retry { + constant: ConstantDelay { attempts: 1, seconds: u16::MAX }, + exponential: ExponentialDelay { + attempts: 2, + multiplier: u16::MAX, + seconds: 7, + random_factor: Some(i8::MIN), + }, + retry_if: Some(RetryIf { expr: "result.error.code != 'fatal'".to_string() }), + }, + ]; + for r in cases { + let back: Retry = RetrySettings::from(&r).into(); + assert_eq!(back, r, "RetrySettings round-trip must preserve {r:?}"); + } + } + + #[test] + fn retry_settings_default_maps_to_default_exponential() { + // All-None settings must mirror ExponentialDelay::default() (multiplier 1), + // so a row with no exponential values doesn't decode to a 0 multiplier. + let r: Retry = RetrySettings::default().into(); + assert_eq!(r, Retry::default()); + assert_eq!(r.exponential.multiplier, 1); + } +} diff --git a/backend/windmill-worker/Cargo.toml b/backend/windmill-worker/Cargo.toml index 14c3698162..14aeb734ed 100644 --- a/backend/windmill-worker/Cargo.toml +++ b/backend/windmill-worker/Cargo.toml @@ -21,6 +21,7 @@ bigquery = ["dep:gcp_auth"] benchmark = ["windmill-queue/benchmark", "windmill-common/benchmark"] parquet = ["windmill-common/parquet", "windmill-object-store/parquet"] flow_testing = [] +failpoints = [] cloud = [] sqlx = [] deno_core = ["dep:windmill-runtime-nativets"] @@ -38,7 +39,7 @@ java = ["dep:windmill-parser-java"] ruby = ["dep:windmill-parser-ruby"] rlang = ["dep:windmill-parser-r"] duckdb = ["dep:libloading"] -quickjs = ["windmill-jseval/quickjs"] +quickjs = ["windmill-jseval/quickjs", "windmill-queue/quickjs"] bedrock = ["windmill-ai/bedrock"] [dependencies] @@ -114,7 +115,8 @@ hmac.workspace = true pem = { workspace = true, optional = true } rsa = { workspace = true, optional = true } urlencoding.workspace = true -nix.workspace = true +# `fs` adds flock(2) for the cross-process Python install lock (shared cache mounts) +nix = { workspace = true, features = ["fs"] } bytes.workspace = true reqwest.workspace = true reqwest-middleware.workspace = true diff --git a/backend/windmill-worker/loader.bun.windows.js b/backend/windmill-worker/loader.bun.windows.js index e89b70c26c..a877266662 100644 --- a/backend/windmill-worker/loader.bun.windows.js +++ b/backend/windmill-worker/loader.bun.windows.js @@ -10,7 +10,7 @@ const p = { name: "windmill-relative-resolver", async setup(build) { const { readFileSync } = await import("fs"); - const { resolve } = await import("node:path"); + const { resolve, dirname } = await import("node:path"); const base_internal_url = "BASE_INTERNAL_URL".replace( "localhost", @@ -26,13 +26,27 @@ const p = { // Normalize path to forward slashes to match Bun's resolver output on Windows const cdirFwd = cdir.replace(/\\/g, "/"); const cdirPosix = cdirFwd.replace(/^[a-zA-Z]:/, ""); + // Match either an already-normalized `.ts` specifier OR an *extensionless* + // windmill relative/workspace import (`./`, `../`, `f/`, `/f/`, `u/`, `/u/`). + // The extensionless branch is essential on Windows: the `main.ts` onLoad that + // would rewrite `./mid` → `./mid.ts` only fires when its path filter matches + // bun's resolver output, and the 8.3 short-name (`RUNNER~1`) vs canonical-path + // mismatch makes that unreliable — so bare imports must be resolvable here + // directly rather than depending on the rewrite. The `(?!.*\.[A-Za-z0-9]+$)` + // guard keeps extension-bearing relative imports (a package's internal + // `./foo.js`/`./x.json` requires) OUT of this resolver so they fall through + // to bun's default resolver — a windmill script import is always `.ts` or + // extensionless. Bare npm specifiers (no `./` prefix, not `f/`/`u/`) never + // match either branch. const filterResolve = new RegExp( - `^(?!\\.\/main\\.ts)(?!\\.\/_wm_)(?!${cdirFwd}\/main\\.ts)(?!${cdirFwd}\/_wm_)(?!${cdirPosix}\/main\\.ts)(?!${cdirPosix}\/_wm_)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$` + `^(?!\\.\/main\\.ts)(?!\\.\/_wm_)(?!${cdirFwd}\/main\\.ts)(?!${cdirFwd}\/_wm_)(?!${cdirPosix}\/main\\.ts)(?!${cdirPosix}\/_wm_)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs)(?:.*\\.ts|(?:\\.\\.?\/|\/?f\/|\/?u\/)(?!.*\\.[A-Za-z0-9]+$).*)$` ); - let cdirNodeModules = `${cdirFwd}/node_modules/`; - - const filterLoad = new RegExp(`^${cdir}\/main\\.ts$`); + // Match the entry main.ts against bun's forward-slash resolver output on + // Windows — raw `cdir` carries backslashes that corrupt the regex, so the + // onLoad below would never fire and extensionless relative imports in + // main.ts would slip past filterResolve unrewritten. + const filterLoad = new RegExp(`^(?:${cdirFwd}|${cdirPosix})\/main\\.ts$`); const transpiler = new Bun.Transpiler({ loader: "ts", }); @@ -117,16 +131,35 @@ const p = { // Resolve windmill script imports from the file namespace (e.g. from main.ts) build.onResolve({ filter: filterResolve }, (args) => { const importerFwd = args.importer?.replace(/\\/g, "/") ?? ""; - if (importerFwd.startsWith(cdirNodeModules)) { + // Let bun natively resolve any import originating INSIDE a dependency, so a + // package's own relative requires (`./string.js`, extensionless `./foo`) + // are never sent to the windmill resolver. Match `/node_modules/` anywhere + // rather than a `cdir`-anchored prefix: on Windows `cdir` (canonical, e.g. + // `runneradmin`) and the importer path (8.3 short name, e.g. `RUNNER~1`) + // disagree, so a prefix check silently fails and dependency imports 404. + if (importerFwd.includes("/node_modules/")) { return undefined; } - // Check if the import resolves to a local module file (written by write_module_files) + // Check if the import resolves to a local module file (written by + // write_module_files, which can nest files in subdirectories). Resolve + // candidates against the importer's own directory — not the job root — so + // a subdir module importing a sibling (`dir/a.ts` → `./b`) finds + // `dir/b.ts` rather than a phantom `job_dir/b.ts`. For imports from + // `main.ts` the importer dir IS the job root, so behavior is unchanged. + // Module files carry a `.ts` extension, so also try the `.ts` variant of + // a bare relative import before falling through to the remote resolver. if (args.path.startsWith(".")) { - const cwdPath = resolve(cdir, args.path); - try { - readFileSync(cwdPath); - return { path: cwdPath }; - } catch {} + const importerDir = args.importer ? dirname(args.importer) : cdir; + const candidates = args.path.endsWith(".ts") + ? [args.path] + : [args.path, args.path + ".ts"]; + for (const candidate of candidates) { + const cwdPath = resolve(importerDir, candidate); + try { + readFileSync(cwdPath); + return { path: cwdPath }; + } catch {} + } } const isMainTs = args.importer == "./main.ts" || importerFwd.endsWith("/main.ts"); diff --git a/backend/windmill-worker/src/agent_workers.rs b/backend/windmill-worker/src/agent_workers.rs index 9bf0ea1834..5320c8ef6c 100644 --- a/backend/windmill-worker/src/agent_workers.rs +++ b/backend/windmill-worker/src/agent_workers.rs @@ -78,4 +78,22 @@ pub async fn get_datatable_resource_from_agent_http( .await } +/// Record a materialization outcome from an agent worker (no direct DB) via the +/// API, so `materialized_partition` state lands the same as on a Sql worker. +// Only called from the duckdb executor, which is itself `#[cfg(feature = "duckdb")]`. +#[cfg(feature = "duckdb")] +pub async fn record_materialization_from_agent_http( + client: &HttpClient, + w_id: &str, + req: &windmill_common::materialization::RecordMaterializationRequest, +) -> anyhow::Result<()> { + client + .post( + &format!("/api/w/{}/agent_workers/record_materialization", w_id), + None, + req, + ) + .await +} + pub const UPDATE_PING_URL: &str = "/api/agent_workers/update_ping"; diff --git a/backend/windmill-worker/src/ai/tools.rs b/backend/windmill-worker/src/ai/tools.rs index 612b534578..1e965ccb3f 100644 --- a/backend/windmill-worker/src/ai/tools.rs +++ b/backend/windmill-worker/src/ai/tools.rs @@ -198,6 +198,10 @@ async fn execute_mcp_tool_call( arguments: arguments.clone(), }); + if let Some(parent_job) = ctx.parent_job { + update_flow_status_module_with_actions(ctx.db, parent_job, actions).await?; + } + match tool_result { Ok(result) => { let result_str = @@ -227,6 +231,10 @@ async fn execute_mcp_tool_call( stream_event_processor.send(event, final_events_str).await?; } + if let Some(parent_job) = ctx.parent_job { + update_flow_status_module_with_actions_success(ctx.db, parent_job, true).await?; + } + // Add tool message to conversation if chat_input_enabled let content = format!("Used {} tool", tool_call.function.name); add_tool_message_to_chat(ctx, None, &content, true).await; @@ -259,6 +267,10 @@ async fn execute_mcp_tool_call( stream_event_processor.send(event, final_events_str).await?; } + if let Some(parent_job) = ctx.parent_job { + update_flow_status_module_with_actions_success(ctx.db, parent_job, false).await?; + } + // Add tool message to conversation if chat_input_enabled add_tool_message_to_chat(ctx, None, &error_msg, false).await; } diff --git a/backend/windmill-worker/src/ai_executor.rs b/backend/windmill-worker/src/ai_executor.rs index 2591cc6495..8c8c94c7e6 100644 --- a/backend/windmill-worker/src/ai_executor.rs +++ b/backend/windmill-worker/src/ai_executor.rs @@ -591,7 +591,7 @@ pub async fn run_agent( let api_key = credentials.api_key.as_deref().unwrap_or(""); // Create the query builder for the provider - let query_builder = create_query_builder(&credentials); + let query_builder = create_query_builder(&credentials, args.provider.get_model()); // Initialize messages let mut messages = @@ -869,6 +869,7 @@ pub async fn run_agent( tool_defs.as_deref(), args.provider.get_model(), args.temperature, + args.provider.get_reasoning_effort(), args.max_completion_tokens, api_key, region, @@ -895,6 +896,7 @@ pub async fn run_agent( tools: tool_defs.as_deref(), model: args.provider.get_model(), temperature: args.temperature, + reasoning_effort: args.provider.get_reasoning_effort(), max_tokens: args.max_completion_tokens, output_schema: args.output_schema.as_ref(), output_type, @@ -1028,6 +1030,11 @@ pub async fn run_agent( // Add websearch tool message if websearch was used if used_websearch { actions.push(AgentAction::WebSearch {}); + if let Some(parent_job) = parent_job { + update_flow_status_module_with_actions(db, parent_job, &actions).await?; + update_flow_status_module_with_actions_success(db, parent_job, true) + .await?; + } messages.push(OpenAIMessage { role: "tool".to_string(), content: Some(OpenAIContent::Text( diff --git a/backend/windmill-worker/src/ansible_executor.rs b/backend/windmill-worker/src/ansible_executor.rs index dec0623e36..292246c62c 100644 --- a/backend/windmill-worker/src/ansible_executor.rs +++ b/backend/windmill-worker/src/ansible_executor.rs @@ -13,7 +13,7 @@ use tokio::process::Command; use uuid::Uuid; use windmill_common::{ error, - git_sync_oss::prepend_token_to_github_url, + git_sync_oss::{prepend_token_to_github_url, sanitize_git_url}, worker::{ is_allowed_file_location, split_python_requirements, to_raw_value, write_file, write_file_at_user_defined_location, Connection, PyVAlias, WORKER_CONFIG, @@ -847,7 +847,7 @@ pub async fn get_git_repo_full_head_commit_hash( .first() .ok_or(anyhow!( "The HEAD commit hash was not found for repo `{}`", - &repo.url + sanitize_git_url(&repo.url) ))? .split_whitespace() .next() @@ -938,6 +938,143 @@ remote_tmp={job_dir}/.ansible/tmp Ok(()) } +/// Read a colon-separated path list (e.g. `roles_path`, `collections_path`) from +/// the `[defaults]` section of an ansible.cfg. Returns the raw entries as written, +/// unresolved. Deliberately minimal: no inline-comment or continuation handling, +/// which ansible's configparser also does not apply to these values. `key` and `=` +/// or `:` as the delimiter are both matched (Python configparser accepts either), +/// with the value itself split on `:` (`os.pathsep`). +fn parse_ansible_cfg_path_list(content: &str, key: &str) -> Option> { + let mut in_defaults = false; + for line in content.lines() { + let trimmed = line.trim(); + if trimmed.starts_with('[') && trimmed.ends_with(']') { + in_defaults = trimmed[1..trimmed.len() - 1] + .trim() + .eq_ignore_ascii_case("defaults"); + continue; + } + if !in_defaults || trimmed.starts_with('#') || trimmed.starts_with(';') { + continue; + } + // configparser delimits key/value on the first `=` or `:`, whichever + // comes first; the remaining `:` in the value are path separators. + let sep = trimmed.find('=').into_iter().chain(trimmed.find(':')).min(); + if let Some(sep) = sep { + let (k, rest) = trimmed.split_at(sep); + let v = &rest[1..]; + if k.trim().eq_ignore_ascii_case(key) { + let entries: Vec = v + .split(':') + .map(|s| s.trim()) + .filter(|s| !s.is_empty()) + .map(|s| s.to_string()) + .collect(); + return (!entries.is_empty()).then_some(entries); + } + } + } + None +} + +/// Prepend Windmill's dependency install dir to the repo cfg's declared path list. +/// Relative entries from the repo cfg are resolved against `cfg_dir` to match how +/// ansible resolves them relative to the config file's own directory. +fn resolve_and_prepend_path( + base: String, + repo_entries: Option>, + cfg_dir: &str, +) -> String { + let mut paths = vec![base]; + if let Some(entries) = repo_entries { + for e in entries { + if e.starts_with('/') || e.starts_with('~') { + paths.push(e); + } else { + paths.push(format!("{cfg_dir}/{e}")); + } + } + } + paths.join(":") +} + +/// Build the environment overrides applied when delegating to a git repo that +/// ships its own ansible.cfg. See the call site for the layering rationale. +async fn build_ansible_cfg_override_envs( + cfg_path: &str, + job_dir: &str, + vault_password_file_exists: bool, + reqs: Option<&AnsibleRequirements>, +) -> error::Result> { + let mut envs = vec![ + ("ANSIBLE_CONFIG".to_string(), cfg_path.to_string()), + // Runtime-bound: reference the ephemeral job dir, cannot be set statically. + ("ANSIBLE_HOME".to_string(), format!("{job_dir}/.ansible")), + ( + "ANSIBLE_LOCAL_TEMP".to_string(), + format!("{job_dir}/.ansible/tmp"), + ), + ( + "ANSIBLE_REMOTE_TEMP".to_string(), + format!("{job_dir}/.ansible/tmp"), + ), + ]; + + // Vault: Windmill manages the secret, so its config wins over the repo cfg. + if vault_password_file_exists { + envs.push(( + "ANSIBLE_VAULT_PASSWORD_FILE".to_string(), + format!("{job_dir}/{WINDMILL_ANSIBLE_PASSWORD_FILENAME}"), + )); + } + if let Some(vault_ids) = reqs.map(|r| &r.vault_id).filter(|v| !v.is_empty()) { + // Defense in depth: entries are validated at parse time, but re-check + // here since they are interpolated raw into the env value. + for vault_id in vault_ids { + validate_vault_id(vault_id)?; + } + envs.push(( + "ANSIBLE_VAULT_IDENTITY_LIST".to_string(), + vault_ids.join(","), + )); + } + + // Dependency search paths: additive. Windmill installs galaxy roles into + // `{job_dir}/roles` and collections into `{job_dir}`; prepend those to the + // repo cfg's declared paths so both Windmill-installed and repo deps resolve. + let cfg_dir = std::path::Path::new(cfg_path) + .parent() + .and_then(|p| p.to_str()) + .unwrap_or(job_dir); + let cfg_content = tokio::fs::read_to_string(cfg_path).await.map_err(|e| { + windmill_common::error::Error::internal_err(format!( + "Failed to read delegated ansible.cfg at `{cfg_path}`: {e}" + )) + })?; + + envs.push(( + "ANSIBLE_ROLES_PATH".to_string(), + resolve_and_prepend_path( + format!("{job_dir}/roles"), + parse_ansible_cfg_path_list(&cfg_content, "roles_path"), + cfg_dir, + ), + )); + envs.push(( + "ANSIBLE_COLLECTIONS_PATH".to_string(), + resolve_and_prepend_path( + job_dir.to_string(), + // Also probe the deprecated plural ini alias; env vars replace (not + // merge) the cfg value, so a repo using it would otherwise be dropped. + parse_ansible_cfg_path_list(&cfg_content, "collections_path") + .or_else(|| parse_ansible_cfg_path_list(&cfg_content, "collections_paths")), + cfg_dir, + ), + )); + + Ok(envs) +} + pub async fn get_git_ssh_cmd( reqs: &AnsibleRequirements, job_dir: &str, @@ -1136,6 +1273,9 @@ pub async fn handle_ansible_job( let mut nsjail_extra_mounts = vec![]; let mut playbook_override = None; + // Absolute path of a repo-provided `ansible.cfg` to use as the effective + // config, set when `delegate_to_git_repo.ansible_cfg` is provided. + let mut ansible_config_override: Option = None; if let Some(r) = reqs.as_ref() { nsjail_extra_mounts = create_file_resources( @@ -1254,7 +1394,12 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } else { clone_repo( &repo, @@ -1269,7 +1414,12 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } append_logs( @@ -1291,6 +1441,23 @@ pub async fn handle_ansible_job( inventories.push(format!("{}/{}", &repo.target_path, inv)); } + if let Some(cfg_rel) = delegated_git_repo.ansible_cfg.as_ref() { + let cfg_rel = interpolate_template( + cfg_rel, + interpolated_args.as_ref(), + "delegate_to_git_repo.ansible_cfg", + )?; + validate_relative_path(&cfg_rel, "delegate_to_git_repo.ansible_cfg")?; + let cfg_path = format!("{}/{}/{}", job_dir, &repo.target_path, cfg_rel); + if !tokio::fs::try_exists(&cfg_path).await.unwrap_or(false) { + return Err(windmill_common::error::Error::BadRequest(format!( + "delegate_to_git_repo.ansible_cfg: no ansible.cfg found in the cloned repo at `{}/{}`", + &repo.target_path, cfg_rel + ))); + } + ansible_config_override = Some(cfg_path); + } + if delegated_git_repo.install_requirements { install_requirements_from_cloned_repo( &repo.target_path, @@ -1316,7 +1483,7 @@ pub async fn handle_ansible_job( append_logs( &job.id, &job.workspace_id, - format!("\nCloning {}...\n", &repo.url), + format!("\nCloning {}...\n", sanitize_git_url(&repo.url)), conn, ) .await; @@ -1338,13 +1505,18 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } else { if req_lockfiles.is_some() { append_logs( &job.id, &job.workspace_id, - format!("Warning: `{}` is using latest commit because the lockfile didn't store a commit hash for this repo. Updates to the repo could break the deployed playbook.\n", &repo.url), + format!("Warning: `{}` is using latest commit because the lockfile didn't store a commit hash for this repo. Updates to the repo could break the deployed playbook.\n", sanitize_git_url(&repo.url)), conn, ) .await; @@ -1362,13 +1534,22 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } append_logs( &job.id, &job.workspace_id, - format!("Cloned {} into {}\n", &repo.url, &repo.target_path), + format!( + "Cloned {} into {}\n", + sanitize_git_url(&repo.url), + &repo.target_path + ), conn, ) .await; @@ -1426,6 +1607,32 @@ pub async fn handle_ansible_job( create_ansible_cfg(reqs.as_ref(), job_dir, vault_password_file_exists)?; + // When the run delegates to a git repo that ships its own ansible.cfg, that + // file becomes the effective config (ansible loads exactly one config file and + // does not merge). These env vars layer Windmill's runtime-bound settings back + // on top — env vars outrank ansible.cfg. Only applied on the non-sandboxed + // path: git-repo delegation clones into `job_dir` which the nsjail profile does + // not mount, so it already requires DISABLE_NSJAIL. + let ansible_env_overrides = match ansible_config_override.as_ref() { + Some(cfg_path) => { + if is_sandboxing_enabled() { + tracing::warn!( + "delegate_to_git_repo.ansible_cfg is set but sandboxing is enabled; \ + git-repo delegation requires DISABLE_NSJAIL, the ansible.cfg override \ + will not take effect" + ); + } + build_ansible_cfg_override_envs( + cfg_path, + job_dir, + vault_password_file_exists, + reqs.as_ref(), + ) + .await? + } + None => vec![], + }; + let mut reserved_variables = get_reserved_variables(job, &client.token, conn, parent_runnable_path).await?; let additional_python_paths_folders = additional_python_paths.join(":"); @@ -1536,6 +1743,7 @@ fi .env("TZ", TZ_ENV.as_str()) .env("BASE_INTERNAL_URL", base_internal_url) .env("HOME", HOME_ENV.as_str()) + .envs(ansible_env_overrides) .stdin(Stdio::null()) .stdout(Stdio::piped()) .stderr(Stdio::piped()); @@ -1832,4 +2040,230 @@ mod tests { assert!(create_ansible_cfg(Some(&reqs), job_dir, false).is_err()); assert!(!dir.path().join("ansible.cfg").exists()); } + + #[test] + fn test_parse_ansible_cfg_path_list() { + let cfg = "\ +[defaults] +roles_path = roles:extra/roles +collections_path=/opt/collections +host_key_checking = False + +[inventory] +roles_path = ignored/section +"; + assert_eq!( + parse_ansible_cfg_path_list(cfg, "roles_path"), + Some(vec!["roles".to_string(), "extra/roles".to_string()]) + ); + assert_eq!( + parse_ansible_cfg_path_list(cfg, "collections_path"), + Some(vec!["/opt/collections".to_string()]) + ); + // Keys only in another section are not picked up. + assert_eq!(parse_ansible_cfg_path_list(cfg, "library"), None); + } + + #[test] + fn test_parse_ansible_cfg_path_list_ignores_comments() { + let cfg = "\ +[defaults] +# roles_path = commented +; roles_path = also_commented +"; + assert_eq!(parse_ansible_cfg_path_list(cfg, "roles_path"), None); + } + + #[test] + fn test_parse_ansible_cfg_path_list_colon_delimiter() { + // configparser accepts `:` as a key/value delimiter, and the value can + // itself be a `:`-separated list. + let cfg = "\ +[defaults] +roles_path: my_roles +collections_path : a/col:b/col +"; + assert_eq!( + parse_ansible_cfg_path_list(cfg, "roles_path"), + Some(vec!["my_roles".to_string()]) + ); + assert_eq!( + parse_ansible_cfg_path_list(cfg, "collections_path"), + Some(vec!["a/col".to_string(), "b/col".to_string()]) + ); + } + + #[test] + fn test_resolve_and_prepend_path() { + // No repo entries: only Windmill's install dir. + assert_eq!( + resolve_and_prepend_path("/job/roles".to_string(), None, "/job/repo"), + "/job/roles" + ); + // Relative repo entries resolve against the cfg dir; absolute/~ kept as-is. + assert_eq!( + resolve_and_prepend_path( + "/job/roles".to_string(), + Some(vec![ + "roles".to_string(), + "/abs/roles".to_string(), + "~/r".to_string() + ]), + "/job/repo/config" + ), + "/job/roles:/job/repo/config/roles:/abs/roles:~/r" + ); + } + + fn ansible_playbook_available() -> bool { + std::process::Command::new("ansible-playbook") + .arg("--version") + .output() + .is_ok() + } + + /// End-to-end: with a delegated repo that ships its own `ansible.cfg` pointing + /// `roles_path` at an in-repo directory, the override env vars must make the + /// real `ansible-playbook` resolve a role it otherwise cannot. Requires the + /// `ansible-playbook` binary; self-skips when absent (e.g. standard CI). Run on + /// a worker devbox with `cargo test -p windmill-worker --features python`. + #[tokio::test] + async fn test_ansible_cfg_override_resolves_repo_roles_e2e() { + if !ansible_playbook_available() { + eprintln!( + "SKIP test_ansible_cfg_override_resolves_repo_roles_e2e: ansible-playbook not found on PATH" + ); + return; + } + + let dir = tempfile::tempdir().unwrap(); + let job_dir = dir.path().to_str().unwrap(); + let repo = dir.path().join(DELEGATE_GIT_REPO_TARGET); + let role_tasks = repo.join("my_roles/greet/tasks"); + std::fs::create_dir_all(&role_tasks).unwrap(); + + std::fs::write( + repo.join("ansible.cfg"), + "[defaults]\nroles_path = my_roles\n", + ) + .unwrap(); + std::fs::write( + role_tasks.join("main.yml"), + "- debug:\n msg: \"hello from greet role\"\n", + ) + .unwrap(); + let play = "- hosts: localhost\n connection: local\n gather_facts: false\n roles:\n - greet\n"; + std::fs::write(repo.join("play.yml"), play).unwrap(); + + // Windmill's own generated cfg (the negative-control config that exists today). + create_ansible_cfg(None, job_dir, false).unwrap(); + + let playbook = format!("{DELEGATE_GIT_REPO_TARGET}/play.yml"); + let run = |envs: Vec<(String, String)>| { + std::process::Command::new("ansible-playbook") + .arg(&playbook) + .current_dir(job_dir) + .envs(envs) + .output() + .unwrap() + }; + + // Negative control: today's behavior (Windmill cfg via cwd, no override) — + // the role lives in the repo subdir and is not found. + let cfg_path = repo.join("ansible.cfg"); + let before = run(vec![]); + assert!( + !before.status.success(), + "without the override the repo role must NOT resolve; stdout={}", + String::from_utf8_lossy(&before.stdout) + ); + + // With the override: ANSIBLE_CONFIG points at the repo cfg and roles_path + // is honored, so the role runs. + let envs = + build_ansible_cfg_override_envs(cfg_path.to_str().unwrap(), job_dir, false, None) + .await + .unwrap(); + let after = run(envs); + let stdout = String::from_utf8_lossy(&after.stdout); + assert!( + after.status.success() && stdout.contains("hello from greet role"), + "with the override the repo role must resolve; status={:?} stdout={stdout} stderr={}", + after.status, + String::from_utf8_lossy(&after.stderr) + ); + } + + #[tokio::test] + async fn test_build_ansible_cfg_override_envs() { + let dir = tempfile::tempdir().unwrap(); + let job_dir = dir.path().to_str().unwrap(); + let repo_dir = dir.path().join("delegate_git_repository"); + std::fs::create_dir_all(&repo_dir).unwrap(); + let cfg_path = repo_dir.join("ansible.cfg"); + std::fs::write(&cfg_path, "[defaults]\nroles_path = my_roles\n").unwrap(); + let cfg_path = cfg_path.to_str().unwrap(); + + let reqs = AnsibleRequirements { + vault_id: vec!["dev@vault_pass.txt".to_string()], + ..Default::default() + }; + let envs = build_ansible_cfg_override_envs(cfg_path, job_dir, true, Some(&reqs)) + .await + .unwrap(); + let map: std::collections::HashMap<_, _> = envs.into_iter().collect(); + + assert_eq!( + map.get("ANSIBLE_CONFIG").map(|s| s.as_str()), + Some(cfg_path) + ); + assert_eq!( + map.get("ANSIBLE_HOME"), + Some(&format!("{job_dir}/.ansible")) + ); + assert_eq!( + map.get("ANSIBLE_VAULT_PASSWORD_FILE"), + Some(&format!("{job_dir}/{WINDMILL_ANSIBLE_PASSWORD_FILENAME}")) + ); + assert_eq!( + map.get("ANSIBLE_VAULT_IDENTITY_LIST").map(|s| s.as_str()), + Some("dev@vault_pass.txt") + ); + // Windmill's `{job_dir}/roles` is prepended to the repo cfg's own path, + // which is resolved against the cfg directory. + assert_eq!( + map.get("ANSIBLE_ROLES_PATH"), + Some(&format!( + "{job_dir}/roles:{}/my_roles", + repo_dir.to_str().unwrap() + )) + ); + // No collections_path in the repo cfg → only Windmill's job dir. + assert_eq!( + map.get("ANSIBLE_COLLECTIONS_PATH").map(|s| s.as_str()), + Some(job_dir) + ); + } + + #[tokio::test] + async fn test_build_ansible_cfg_override_envs_collections_paths_alias() { + let dir = tempfile::tempdir().unwrap(); + let job_dir = dir.path().to_str().unwrap(); + let repo_dir = dir.path().join("delegate_git_repository"); + std::fs::create_dir_all(&repo_dir).unwrap(); + let cfg_path = repo_dir.join("ansible.cfg"); + // Deprecated plural alias must still be picked up so the repo's collections + // are not silently dropped when the env override replaces the cfg value. + std::fs::write(&cfg_path, "[defaults]\ncollections_paths = my_cols\n").unwrap(); + + let envs = + build_ansible_cfg_override_envs(cfg_path.to_str().unwrap(), job_dir, false, None) + .await + .unwrap(); + let map: std::collections::HashMap<_, _> = envs.into_iter().collect(); + assert_eq!( + map.get("ANSIBLE_COLLECTIONS_PATH"), + Some(&format!("{job_dir}:{}/my_cols", repo_dir.to_str().unwrap())) + ); + } } diff --git a/backend/windmill-worker/src/bun_executor.rs b/backend/windmill-worker/src/bun_executor.rs index 1e4f4aafd6..f6b093f766 100644 --- a/backend/windmill-worker/src/bun_executor.rs +++ b/backend/windmill-worker/src/bun_executor.rs @@ -1,6 +1,11 @@ #[cfg(feature = "deno_core")] use std::time::Instant; -use std::{collections::HashMap, fs, process::Stdio}; +use std::{ + collections::{HashMap, HashSet}, + fs, + process::Stdio, + sync::Arc, +}; use base64::Engine; use itertools::Itertools; @@ -27,9 +32,10 @@ use crate::{ NSJAIL_AVAILABLE, NSJAIL_PATH, PATH_ENV, PROXY_ENVS, TRACING_PROXY_CA_CERT_PATH, TZ_ENV, }; use windmill_common::{ + cache, client::AuthedClient, jobs::JobKind, - scripts::{id_to_codebase_info, CodebaseInfo, ScriptLang}, + scripts::{id_to_codebase_info, CodebaseInfo, ScriptHash, ScriptLang}, utils::WarnAfterExt, workspace_dependencies::WorkspaceDependenciesPrefetched, }; @@ -44,7 +50,6 @@ use tokio::io::AsyncReadExt; use windmill_common::{ error::{self, Result}, - get_latest_hash_for_path, worker::{write_file, Connection, DISABLE_BUNDLING}, DB, }; @@ -1242,16 +1247,101 @@ pub fn ensure_bundle_output_exists(bundle_path: &str) -> Result<()> { pub const BUN_BUNDLE_OBJECT_STORE_PREFIX: &str = "bun_bundle/"; -async fn get_script_import_updated_at(db: &DB, w_id: &str, script_path: &str) -> Result { - let script_hash = get_latest_hash_for_path(db, w_id, script_path, false).await?; - let last_updated_at = sqlx::query_scalar!( - "SELECT created_at FROM script WHERE workspace_id = $1 AND hash = $2", - w_id, - script_hash.0 .0 +// A script version's relative-import list never changes (content is immutable +// per hash), so parses are memoized without any invalidation. +lazy_static::lazy_static! { + static ref RELATIVE_IMPORTS_PER_HASH: quick_cache::sync::Cache>> = + quick_cache::sync::Cache::new(1000); +} + +const MAX_TRANSITIVE_IMPORT_PATHS: usize = 256; + +/// `(path, latest hash)` for the whole transitive closure of relative imports +/// of `inner_content` — the set of scripts whose code gets inlined into the +/// bundle, so all of them must key the bundle cache. Resolution goes through +/// `IMPORTED_SCRIPT_HASH_CACHE` (notify-evicted, 60s TTL fallback, same +/// version-selection predicate as the loader's content endpoint) and the +/// per-hash script/parse caches, so steady state costs no DB queries. +async fn collect_transitive_import_versions( + db: &DB, + w_id: &str, + script_path: &str, + inner_content: &str, +) -> Vec<(String, i64)> { + let conn = Connection::from(db.clone()); + let mut queue = crate::worker_lockfiles::extract_relative_imports( + inner_content, + script_path, + &Some(ScriptLang::Bun), ) - .fetch_one(db) - .await?; - Ok(last_updated_at.to_string()) + .unwrap_or_default(); + let mut visited: HashSet = HashSet::new(); + let mut versions: Vec<(String, i64)> = vec![]; + while let Some(path) = queue.pop() { + if !visited.insert(path.clone()) { + continue; + } + if visited.len() > MAX_TRANSITIVE_IMPORT_PATHS { + tracing::warn!( + "transitive relative-import closure of {script_path} exceeds \ + {MAX_TRANSITIVE_IMPORT_PATHS} scripts; bundle cache key covers only the first \ + {MAX_TRANSITIVE_IMPORT_PATHS}" + ); + break; + } + let hash = match windmill_common::get_latest_script_hash_for_import_cached(db, w_id, &path) + .await + { + Ok(Some(hash)) => hash, + // Not a deployed script at this path (deleted, or not a script): + // excluded from the key, matching what the bundler can inline. + Ok(None) => continue, + Err(e) => { + tracing::warn!( + "could not resolve import {path} while computing bundle cache key for \ + {script_path}: {e:#}" + ); + continue; + } + }; + versions.push((path.clone(), hash)); + let imports = match RELATIVE_IMPORTS_PER_HASH.get(&hash) { + Some(imports) => imports, + None => match cache::script::fetch(&conn, ScriptHash(hash)).await { + Ok((data, meta)) => { + let imports = Arc::new(match meta.language { + Some(ScriptLang::Bun) + | Some(ScriptLang::Bunnative) + | Some(ScriptLang::Deno) => { + crate::worker_lockfiles::extract_relative_imports( + &data.code, + &path, + &meta.language, + ) + .unwrap_or_default() + } + _ => vec![], + }); + RELATIVE_IMPORTS_PER_HASH.insert(hash, imports.clone()); + imports + } + // A fetch error is transient, not a property of the (immutable) + // content — memoizing it would drop this subtree from the key + // until worker restart. Skip caching and retry next run. + Err(e) => { + tracing::warn!( + "could not fetch import {path} (hash {hash}) while computing bundle \ + cache key for {script_path}: {e:#}" + ); + Arc::new(vec![]) + } + }, + }; + queue.extend(imports.iter().cloned()); + } + // deterministic key regardless of traversal order + versions.sort(); + versions } pub async fn compute_bundle_local_and_remote_path( @@ -1265,16 +1355,13 @@ pub async fn compute_bundle_local_and_remote_path( let mut input_src = format!("{inner_content}{lock}",); if let Some(db) = db { - let relative_imports = crate::worker_lockfiles::extract_relative_imports( - &inner_content, - script_path, - &Some(ScriptLang::Bun), - ); - for path in relative_imports.unwrap_or_default() { - if let Ok(updated_at) = get_script_import_updated_at(&db, w_id, &path).await { - input_src.push_str(&path); - input_src.push_str(&updated_at.to_string()); - } + // The bundle inlines the whole transitive relative-import closure, so a + // new deployed version of ANY script in it must change the key. + for (path, hash) in + collect_transitive_import_versions(db, w_id, script_path, inner_content).await + { + input_src.push_str(&path); + input_src.push_str(&hash.to_string()); } }; diff --git a/backend/windmill-worker/src/deno_executor.rs b/backend/windmill-worker/src/deno_executor.rs index c39e9e999e..a446548e1f 100644 --- a/backend/windmill-worker/src/deno_executor.rs +++ b/backend/windmill-worker/src/deno_executor.rs @@ -434,7 +434,11 @@ try {{ if let Some(ref npmrc_content) = npmrc { if !npmrc_content.trim().is_empty() { write_file(job_dir, ".npmrc", npmrc_content)?; - write_file(job_dir, "deno.json", "{}")?; + // minimumDependencyAge=0 opts out of Deno's supply-chain guard that rejects + // npm packages published within the last ~24h. Private/internal registries + // routinely serve just-published versions, so the guard would break them. + // Older Deno ignores the unknown field, so this is safe across versions. + write_file(job_dir, "deno.json", r#"{"minimumDependencyAge":"0"}"#)?; } } diff --git a/backend/windmill-worker/src/duckdb_executor.rs b/backend/windmill-worker/src/duckdb_executor.rs index 1a474ee9a0..e4e24d6653 100644 --- a/backend/windmill-worker/src/duckdb_executor.rs +++ b/backend/windmill-worker/src/duckdb_executor.rs @@ -11,15 +11,15 @@ use serde_json::{json, Value}; use uuid::Uuid; use windmill_common::error::{to_anyhow, Error, Result}; use windmill_common::utils::sanitize_string_from_password; -use windmill_common::worker::{get_memory, Connection, SqlResultCollectionStrategy}; +use windmill_common::worker::{get_memory, to_raw_value, Connection, SqlResultCollectionStrategy}; use windmill_common::workspaces::{ get_datatable_resource_from_db_unchecked, get_ducklake_from_db_unchecked, - DucklakeCatalogResourceType, + strip_fork_reserved_attach_args, DucklakeCatalogResourceType, }; use windmill_common::PgDatabase; use windmill_object_store::S3_PROXY_LAST_ERRORS_CACHE; use windmill_parser_sql::{parse_duckdb_sig, parse_sql_blocks}; -use windmill_queue::{CanceledBy, MiniPulledJob}; +use windmill_queue::{append_logs, CanceledBy, MiniPulledJob}; use windmill_types::s3::S3Object; use crate::agent_workers::{get_datatable_resource_from_agent_http, get_ducklake_from_agent_http}; @@ -32,6 +32,1268 @@ use crate::sql_utils::remove_comments; use windmill_common::client::AuthedClient; use windmill_object_store::DEFAULT_STORAGE; +// What a `// materialize` run records into `materialized_partition` once it +// finishes. `asset_path` is the full `/
` (the asset identity); +// `partition` is "" for an unpartitioned (whole-table) materialization. +struct MaterializeExec { + asset_kind: windmill_common::assets::AssetKind, + asset_path: String, + partition: String, + // Number of `// data_test` checks the codegen embedded. Enforcement recovers + // the per-test outcomes from the summary row; if it recovers fewer than this + // (e.g. an FFI serialization change drops the column), we fail loud rather + // than silently pass declared-but-unverified tests. + n_data_tests: usize, + // `on_schema_change=warn` (default) on a positional persist-and-mutate write: + // the summary carries a `schema_drift` column the worker logs + returns. + on_schema_change: windmill_parser::asset_parser::OnSchemaChange, + // `on_schema_change=sync`: the pre-pass probe (setup + target ATTACH + the + // SELECT's/table's column reads) whose result drives the injected ALTER DDL. + // `None` for every other mode / non-persist-and-mutate strategy. + sync_prepass: Option, +} + +// Inputs for the `on_schema_change=sync` pre-pass: a probe query read against +// the live session and the target table it migrates. The worker runs the probe +// (same interpolation + ATTACH transform as the main query), diffs the SELECT's +// columns against the table's, and splices `ALTER TABLE … ADD/DROP COLUMN` DDL +// into the plan at the [`SYNC_ALTER_SENTINEL`] slot before the write. +struct SyncPrepass { + // Setup + synthetic target ATTACH + the two column reads. Carries `$args`/ + // `{partition}` verbatim (the worker interpolates it like the main query). + probe_query: String, + // Target table within `_wm_target` (e.g. `orders` or `schema.orders`), used + // to build the quoted `ALTER TABLE _wm_target.
` statements. + target_table: String, +} + +// Fetch and validate a custom data-test script's body. v1 custom tests are +// DuckDB scripts holding a single SELECT/CTE that returns the violating rows +// (dbt's singular-test convention); the worker embeds that query as a subquery +// check in the materialize connection (the single-statement constraint is +// enforced in sql_materialize.rs). Server workers only — agent (Http) workers +// have no script cache to read deployed content from. +async fn fetch_custom_test_body(conn: &Connection, w_id: &str, path: &str) -> Result { + let Connection::Sql(db) = conn else { + return Err(Error::ExecutionErr(format!( + "data_test custom `{path}`: custom tests require a server worker (not supported on \ + agent workers in v1)" + ))); + }; + let hash = windmill_common::get_latest_script_hash(db, path, w_id) + .await? + .ok_or_else(|| { + Error::ExecutionErr(format!( + "data_test custom `{path}`: no deployed script found at this path" + )) + })?; + let content = + crate::get_script_content_by_hash(&windmill_common::scripts::ScriptHash(hash), w_id, conn) + .await?; + if !matches!( + content.language, + Some(windmill_common::scripts::ScriptLang::DuckDb) + ) { + return Err(Error::ExecutionErr(format!( + "data_test custom `{path}`: must be a DuckDB script returning the violating rows \ + (got language {:?})", + content.language + ))); + } + Ok(content.content) +} + +// If `query` declares `// materialize `, return what to record plus, +// for the default managed mode, the rewritten managed-write SQL (in `manual` +// mode the script writes its own DDL, so the rewrite is `None`). The rewritten +// SQL contains a synthetic `ATTACH 'ducklake://' AS _wm_target` that the +// normal ATTACH-transform pass resolves to real credentials — the same path as +// the user's own ATTACH. `// data_test` lines append verifier probes that run +// against the freshly-materialized target and raise (failing the run) on +// violation. Returns `None` when there is no materialize annotation or the +// target isn't a ducklake (only ducklake is materialized in v1). +fn build_materialized_query( + query: &str, + partition_value: Option<&str>, + // Custom (`// data_test `) test bodies, pre-fetched by the caller + // (`fetch_custom_test_bodies`) so this stays pure/sync and unit-testable — + // the DB read is the only thing that needs a connection. Keyed by script path. + custom_test_bodies: &std::collections::HashMap, +) -> Result, MaterializeExec)>> { + use windmill_parser::asset_parser::{ + parse_pipeline_annotations, AssetKind as PAssetKind, DataTest, + }; + use windmill_parser::sql_materialize::{ + build_wrap_blocks, DataTestResolved, MaterializeStrategy, TARGET_ALIAS, + }; + + let ann = parse_pipeline_annotations(query); + let has_tests = !ann.data_tests.is_empty(); + let Some(m) = ann.materialize else { + // Data tests run *against the materialized asset*; without a + // `// materialize` target there is nothing to test. Fail loudly rather + // than silently skip the declared checks. + if has_tests { + return Err(Error::ExecutionErr( + "data_test: requires a `// materialize` target — data tests run against the \ + materialized asset" + .to_string(), + )); + } + return Ok(None); + }; + if m.target_kind != PAssetKind::Ducklake { + if has_tests { + return Err(Error::ExecutionErr( + "data_test: only `ducklake://` materialization targets support data tests in v1" + .to_string(), + )); + } + return Ok(None); + } + let partitioned = ann.partition.is_some(); + let partition = partition_value.unwrap_or("").to_string(); + // Partition *resolution* is enterprise; in its absence a partitioned + // materialize only runs with an explicit `partition` arg. Fail loudly rather + // than silently materialize the wrong (empty) slice. + if partitioned && partition.is_empty() { + return Err(Error::ExecutionErr( + "materialize: a `// partitioned` script ran with no resolved partition — pass an \ + explicit `partition` arg, or enable enterprise partition resolution" + .to_string(), + )); + } + // Convention: `ducklake:///
` — is the configured + // ducklake (resolved like a user ATTACH),
is the rest. + let (ducklake_name, table) = m + .target_path + .split_once('/') + .unwrap_or((m.target_path.as_str(), "")); + let mut meta = MaterializeExec { + asset_kind: windmill_common::assets::AssetKind::Ducklake, + asset_path: m.target_path.clone(), + partition: partition.clone(), + n_data_tests: ann.data_tests.len(), + on_schema_change: m.on_schema_change, + // Set below once the strategy is known (managed persist-and-mutate only). + sync_prepass: None, + }; + + // `{partition}` → escaped SQL literal substitution, applied to the managed + // SELECT, its setup, and any custom-test body so a partitioned test can + // filter by the active slice. Always a complete `'…'` literal (with `'` + // doubled) whether or not the author quoted it, so a run caller can't break + // out and alter statement boundaries. The pre-quoted `'{partition}'` form is + // matched first so it doesn't become `''…''`. No-op when unpartitioned. + let lit = format!("'{}'", partition.replace('\'', "''")); + let substitute = |s: &str| -> String { + if !partitioned { + return s.to_string(); + } + let tok = windmill_common::assets::PARTITION_TOKEN; + let quoted_tok = format!("'{tok}'"); + s.replace("ed_tok, &lit).replace(tok, &lit) + }; + + if m.manual { + // Escape hatch: the script owns its DDL. We can't reliably attach the + // managed target or know the partition column it wrote, so data tests + // are not generated for manual mode in v1. + if has_tests { + return Err(Error::ExecutionErr( + "data_test: not supported with `// materialize manual` in v1 — use managed \ + `// materialize`" + .to_string(), + )); + } + return Ok(Some((None, meta))); + } + if table.is_empty() { + return Err(Error::ExecutionErr(format!( + "materialize: target `ducklake://{}` has no table (use ducklake:///
)", + m.target_path + ))); + } + let mut plan = classify_wrap_or_err(query)?; + plan.output = substitute(&plan.output); + for s in plan.setup.iter_mut() { + *s = substitute(s); + } + // Prepend the `wm_partition(ts)` helper macro for a time-partitioned + // materialize so the SELECT can filter to the active slice with + // `WHERE wm_partition() = {partition}`. Runs first (setup precedes + // the wrapped SELECT and the write transaction), carries no `{partition}` + // token so it's inserted post-substitution, and reads its format from the + // same source the resolver stamped the identity with — so they can't drift. + if partitioned { + if let Some(macro_sql) = ann + .partition + .as_ref() + .and_then(windmill_parser::sql_materialize::wm_partition_macro) + { + plan.setup.insert(0, macro_sql); + } + } + // Deploy (`create_script_internal`) already rejects the invalid SCD2 combos, + // but preview/test runs reach the executor without a deploy — re-check so a + // bad combo fails with the same clear message instead of a raw DuckDB error. + m.validate(partitioned).map_err(Error::ExecutionErr)?; + let strategy = if m.scd2 { + // SCD2 needs a natural key to identify an entity across versions (checked + // by `validate` above, which guarantees a non-empty key here). + let key = m.unique_key.clone().ok_or_else(|| { + Error::ExecutionErr( + "materialize scd2: requires a natural key — add `key=`".to_string(), + ) + })?; + MaterializeStrategy::Scd2 { key, track: m.track.clone(), close_deleted: m.close_deleted } + } else if m.append { + MaterializeStrategy::Append + } else if let Some(uk) = m.unique_key { + MaterializeStrategy::Merge { unique_key: uk } + } else { + MaterializeStrategy::Replace + }; + // Inline the partition as an escaped SQL literal (DuckLake has no bind for + // the partition column in our generated DDL). + let pval = lit.clone(); + let synthetic_attach = format!("ATTACH 'ducklake://{ducklake_name}' AS {TARGET_ALIAS};"); + + // Resolve data tests (fetch + partition-substitute custom bodies) so codegen + // can embed every check's violating-row count in the materialize summary. + // The summary then carries the full per-test breakdown back to the worker, + // which runs them all and decides pass/fail (no abort-on-first). Empty when + // there are no `// data_test` lines. + let mut resolved = Vec::with_capacity(ann.data_tests.len()); + for test in &ann.data_tests { + match test { + DataTest::Custom { path } => { + let raw = custom_test_bodies.get(path).ok_or_else(|| { + Error::ExecutionErr(format!( + "data_test custom `{path}`: body not fetched before codegen (internal)" + )) + })?; + resolved + .push(DataTestResolved::Custom { path: path.clone(), body: substitute(raw) }); + } + other => resolved.push(DataTestResolved::BuiltIn(other.clone())), + } + } + + let cg_probe = windmill_parser::sql_materialize::MaterializeCodegen { + target_qualified: "", + select_sql: "", + partition_col: "_wm_partition", + partition_value_sql: "", + partitioned, + strategy: strategy.clone(), + on_schema_change: m.on_schema_change, + }; + // `sync` needs a host round-trip: a pre-pass probe reads the SELECT's and the + // table's columns so the ALTER DDL can be computed, then spliced into the + // plan at the sentinel slot. Build the probe query here (setup + target + // ATTACH + the two column reads) while we still have the substituted SELECT + + // setup; the executor runs it through the same interpolation/ATTACH transform + // as the main query. Only for the positional persist-and-mutate strategies + // (scd2 / whole-table replace emit no sentinel). + if m.on_schema_change == windmill_parser::asset_parser::OnSchemaChange::Sync + && cg_probe.is_persist_and_mutate() + { + meta.sync_prepass = Some(SyncPrepass { + probe_query: build_sync_probe_query( + &plan.setup, + &synthetic_attach, + &plan.output, + table, + ), + target_table: table.to_string(), + }); + } + + let mat_plan = build_wrap_blocks( + &plan, + &synthetic_attach, + table, + &m.target_path, + "_wm_partition", + &pval, + partitioned, + strategy, + m.on_schema_change, + &resolved, + ) + .map_err(Error::ExecutionErr)?; + // Enterprise seam: assembles the plan into the final statement list — + // verbatim on the public build (commit-then-test), restructured into + // write-audit-publish (guarded transaction, rollback on violation) on EE. + let blocks = + windmill_common::pipeline_advanced::finalize_materialize_query(mat_plan, &m.target_path); + + Ok(Some((Some(blocks.join("\n")), meta))) +} + +// Build the `on_schema_change=sync` pre-pass probe: the substituted setup + the +// synthetic target ATTACH, then two column reads unioned — the SELECT's columns +// (via `DESCRIBE`) tagged `sel`, and the target table's (via information_schema, +// which yields zero rows when the table doesn't exist yet, so a first +// materialize is a no-drift skip rather than an error) tagged `tbl`. The managed +// `_wm_partition` column is filtered out of the table side. Carries +// `$args`/`{partition}` verbatim; the caller interpolates + ATTACH-transforms it +// like the main query so the DESCRIBE binds run-time args. +fn build_sync_probe_query( + setup: &[String], + synthetic_attach: &str, + select_sql: &str, + table: &str, +) -> String { + let (schema, tname) = match table.split_once('.') { + Some((s, t)) => (Some(s), t), + None => (None, table), + }; + let esc = |s: &str| s.replace('\'', "''"); + let schema_filter = schema + .map(|s| format!(" AND table_schema = '{}'", esc(s))) + .unwrap_or_default(); + let mut out = String::new(); + let mut push_stmt = |s: &str| { + let t = s.trim_end(); + if t.is_empty() { + return; + } + out.push_str(t); + if !t.ends_with(';') { + out.push(';'); + } + out.push('\n'); + }; + for s in setup { + push_stmt(s); + } + push_stmt(synthetic_attach); + // Read the target's columns from `information_schema.columns` scoped to the + // attached target catalog via `table_catalog` (DuckDB's catalog-qualified + // `.information_schema` does NOT exist; the unqualified view spans + // attached catalogs and distinguishes them by `table_catalog`). It yields + // zero rows for a not-yet-created table — a no-drift skip, not an error + // (unlike `pragma_table_info`/`DESCRIBE`). + out.push_str(&format!( + "SELECT 'sel' AS _wm_which, column_name AS _wm_name, column_type AS _wm_type \ + FROM (DESCRIBE SELECT * FROM ({select_sql})) \ + UNION ALL \ + SELECT 'tbl' AS _wm_which, column_name AS _wm_name, data_type AS _wm_type \ + FROM information_schema.columns \ + WHERE table_catalog = '{alias}' AND table_name = '{tname}'{schema_filter} \ + AND column_name <> '_wm_partition';", + alias = windmill_parser::sql_materialize::TARGET_ALIAS, + tname = esc(tname), + )); + out +} + +// Run the sync pre-pass probe (already interpolated + ATTACH-transformed by the +// caller) and turn its column diff into the `ALTER TABLE … ADD/DROP COLUMN` DDL +// to splice at the sentinel. Empty string when the table is fresh (no rows) or +// the SELECT already matches. The probe failing is fatal (no silent fallback). +async fn compute_sync_alter_ddl( + probe_blocks: Vec, + job_args: Vec, + target_table: String, + token: String, + base_internal_url: String, + w_id: String, + job_dir: String, +) -> Result { + let n = probe_blocks.len(); + let (result, _) = tokio::task::spawn_blocking(move || { + run_duckdb_ffi_safe( + probe_blocks.iter().map(String::as_str), + n, + job_args, + &token, + &base_internal_url, + &w_id, + &job_dir, + SqlResultCollectionStrategy::LastStatementAllRows, + ) + }) + .await + .map_err(|e| Error::from(to_anyhow(e))) + .and_then(|r| r) + .map_err(|e| { + Error::ExecutionErr(format!( + "on_schema_change=sync: the schema pre-pass probe failed ({e}); refusing to write" + )) + })?; + + let (added, removed) = parse_sync_drift(&result)?; + let tq = format!( + "{}.{}", + windmill_parser::sql_materialize::TARGET_ALIAS, + quote_qualified_table(&target_table) + ); + let mut stmts = Vec::new(); + for (name, typ) in &added { + if typ.is_empty() { + return Err(Error::ExecutionErr(format!( + "on_schema_change=sync: could not resolve a type for new column `{name}`" + ))); + } + // `typ` is DuckDB's own DESCRIBE type name (e.g. BIGINT, VARCHAR, + // DECIMAL(10,2)) — a valid type expression, injected verbatim. + stmts.push(format!( + "ALTER TABLE {tq} ADD COLUMN \"{}\" {typ};", + name.replace('"', "\"\"") + )); + } + for name in &removed { + stmts.push(format!( + "ALTER TABLE {tq} DROP COLUMN \"{}\";", + name.replace('"', "\"\"") + )); + } + Ok(stmts.join("\n")) +} + +// Parse the sync probe result rows (`_wm_which`/`_wm_name`/`_wm_type`) into the +// column diff: `added` = SELECT columns (with their types) absent from the +// table, `removed` = table columns absent from the SELECT. `_wm_partition` and +// empty names are ignored. An empty table side means the table doesn't exist +// yet — a no-op first materialize, so both lists are empty. +fn parse_sync_drift(result: &RawValue) -> Result<(Vec<(String, String)>, Vec)> { + let root: Value = serde_json::from_str(result.get()).map_err(|e| { + Error::ExecutionErr(format!("on_schema_change=sync: bad probe result json: {e}")) + })?; + // The FFI may surface the rows as an array, a single object, or a JSON + // string — normalize to a list of row objects. + let owned; + let rows: Vec<&Value> = match &root { + Value::Array(a) => a.iter().collect(), + Value::Object(_) => vec![&root], + Value::String(s) => { + owned = serde_json::from_str::(s).unwrap_or(Value::Null); + match &owned { + Value::Array(a) => a.iter().collect(), + Value::Object(_) => vec![&owned], + _ => vec![], + } + } + _ => vec![], + }; + let mut sel: Vec<(String, String)> = Vec::new(); + let mut tbl: std::collections::HashSet = std::collections::HashSet::new(); + for row in rows { + let Some(o) = row.as_object() else { continue }; + let name = o + .get("_wm_name") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + if name.is_empty() || name == "_wm_partition" { + continue; + } + let typ = o + .get("_wm_type") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + match o.get("_wm_which").and_then(|v| v.as_str()).unwrap_or("") { + "sel" => sel.push((name, typ)), + "tbl" => { + tbl.insert(name); + } + _ => {} + } + } + if tbl.is_empty() { + return Ok((Vec::new(), Vec::new())); + } + let sel_names: std::collections::HashSet<&String> = sel.iter().map(|(n, _)| n).collect(); + let added = sel + .iter() + .filter(|(n, _)| !tbl.contains(n)) + .cloned() + .collect(); + let removed = tbl.into_iter().filter(|n| !sel_names.contains(n)).collect(); + Ok((added, removed)) +} + +// Fetch the deployed body of every `// data_test ` custom test declared in +// `query`, keyed by path, so the sync `build_materialized_query` can splice them +// in. The DB read is the only part of materialize codegen that needs a +// connection; isolating it here keeps the codegen pure and unit-testable. +// Server workers only (`fetch_custom_test_body` errors on agent workers). Empty +// when there are no custom tests. +async fn fetch_custom_test_bodies( + query: &str, + conn: &Connection, + w_id: &str, +) -> Result> { + use windmill_parser::asset_parser::{parse_pipeline_annotations, DataTest}; + let ann = parse_pipeline_annotations(query); + let mut bodies = std::collections::HashMap::new(); + for test in &ann.data_tests { + if let DataTest::Custom { path } = test { + if !bodies.contains_key(path) { + let body = fetch_custom_test_body(conn, w_id, path).await?; + bodies.insert(path.clone(), body); + } + } + } + Ok(bodies) +} + +// classify_wrap with the spec's actionable message turned into an executor error. +fn classify_wrap_or_err(query: &str) -> Result { + windmill_parser::sql_materialize::classify_wrap(query) + .map_err(|e| Error::ExecutionErr(e.message())) +} + +// One workspace-macro registry row (`macro_definition`, written at deploy of a +// `// macros` library script). Shared with windmill-common so the registry +// cache can live there (main.rs evicts it on `notify_macro_registry_change`). +use windmill_common::assets::MacroRegistryEntry as MacroRow; + +// The registry read runs on every DuckDB job (including the common macro-free +// case), so it's cached per workspace. Invalidation is primarily the +// transactional `notify_macro_registry_change` event (registry mutations → +// notify_event table → the poller in main.rs evicts); the TTL bounds +// staleness for anything that doesn't emit. +async fn fetch_macro_registry( + db: &windmill_common::DB, + w_id: &str, +) -> Result>> { + use windmill_common::assets::{ + ExpiringMacroRegistry, MACRO_REGISTRY_CACHE, MACRO_REGISTRY_CACHE_DISABLED, + MACRO_REGISTRY_TTL, + }; + // Cloud: unbounded workspace count makes the 1000-entry per-workspace + // cache churn instead of hit, and its per-entry memory (full registry + // text) is tenant-controlled — skip it there; self-hosted keeps the fast + // path. + let use_cache = !*windmill_common::worker::CLOUD_HOSTED + && !MACRO_REGISTRY_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); + if use_cache { + if let Some(e) = MACRO_REGISTRY_CACHE.get(w_id) { + if e.expires_at > std::time::Instant::now() { + return Ok(e.rows); + } + } + } + let rows = sqlx::query_as!( + MacroRow, + "SELECT name, params, body, is_table_macro, provider_path FROM macro_definition WHERE workspace_id = $1", + w_id + ) + .fetch_all(db) + .await?; + let rows = std::sync::Arc::new(rows); + if use_cache { + MACRO_REGISTRY_CACHE.insert( + w_id.to_string(), + ExpiringMacroRegistry { + rows: rows.clone(), + expires_at: std::time::Instant::now() + MACRO_REGISTRY_TTL, + }, + ); + } + Ok(rows) +} + +// Selection pass: seed = macros the (comment-stripped, transformed) blocks +// call, plus every macro of each `// use` library; then the transitive +// closure over macro bodies. Local definitions win — their names are removed +// from the injectable set entirely, so a later workspace-library deploy can +// never silently replace a script's own macro. Pure and separate from +// `plan_macro_injection` so the shell knows which provider libraries to +// fetch (their setup statements are injected too) before planning. +fn select_workspace_macros( + blocks: &[String], + registry: &[MacroRow], + use_libs: &[String], +) -> Result> { + use std::collections::{BTreeMap, HashSet}; + use windmill_parser::duckdb_macros::{detect_macro_calls, locally_defined_macro_names}; + + let local = locally_defined_macro_names(blocks); + let all_names: HashSet = registry + .iter() + .map(|r| r.name.clone()) + .filter(|n| !local.contains(n)) + .collect(); + let by_name: BTreeMap<&str, &MacroRow> = + registry.iter().map(|r| (r.name.as_str(), r)).collect(); + + let mut selected: HashSet = HashSet::new(); + for path in use_libs { + let lib_names: Vec<&str> = registry + .iter() + .filter(|r| &r.provider_path == path) + .map(|r| r.name.as_str()) + .collect(); + if lib_names.is_empty() { + return Err(Error::ExecutionErr(format!( + "`// use {path}`: no deployed macro library at this path" + ))); + } + selected.extend( + lib_names + .into_iter() + .filter(|n| all_names.contains(*n)) + .map(String::from), + ); + } + selected.extend(detect_macro_calls(&blocks.join("\n"), &all_names)); + + // Transitive closure over macro bodies (a selected macro may call others). + let mut frontier: Vec = selected.iter().cloned().collect(); + while let Some(name) = frontier.pop() { + if let Some(row) = by_name.get(name.as_str()) { + for dep in detect_macro_calls(&row.body, &all_names) { + if selected.insert(dep.clone()) { + frontier.push(dep); + } + } + } + } + Ok(selected) +} + +// Fixpoint over library-level `// use`: a library may declare `// use` for +// dynamic calls its macro bodies make (string-hidden from lexical detection, +// e.g. inside `query('…')`). Those declarations are honored transitively — +// consuming a macro from lib B pulls in whatever B `// use`s, so the dynamic +// dependency stays encapsulated in the library instead of leaking to every +// consumer. `lib_uses` maps provider path → its parsed `// use` list; the +// shell grows it lazily and re-resolves until no new library appears. +// Returns the selected macro names plus the effective `// use` list +// (consumer's own first, in annotation order, then discovered libs). +fn resolve_macro_selection( + blocks: &[String], + registry: &[MacroRow], + consumer_use_libs: &[String], + lib_uses: &std::collections::BTreeMap>, +) -> Result<(std::collections::HashSet, Vec)> { + let mut effective: Vec = Vec::new(); + for p in consumer_use_libs { + if !effective.contains(p) { + effective.push(p.clone()); + } + } + loop { + let selected = select_workspace_macros(blocks, registry, &effective)?; + let mut relevant: Vec = effective.clone(); + for name in &selected { + if let Some(r) = registry.iter().find(|r| &r.name == name) { + if !relevant.contains(&r.provider_path) { + relevant.push(r.provider_path.clone()); + } + } + } + let mut grew = false; + for lib in &relevant { + for u in lib_uses.get(lib).map(Vec::as_slice).unwrap_or(&[]) { + if !effective.contains(u) { + effective.push(u.clone()); + grew = true; + } + } + } + if !grew { + return Ok((selected, effective)); + } + } +} + +// Emit the blocks to inject: first every relevant library's setup statements +// (a macro body may reference its own lib's ATTACH, and DuckDB bind-checks +// at CREATE — so setup is injected for the provider of every selected macro, +// not just `// use` libs), then the selected definitions in dependency order. +// `lib_bodies` carries the parsed deployed content per provider path; setup +// order = `// use` libs in annotation order, then remaining providers sorted. +// Exact-duplicate setup statements across libraries are deduped (two libs +// attaching the same catalog the same way must not double-ATTACH). Pure — +// fetches happen in `inject_workspace_macros` — so this is unit-testable. +fn plan_macro_injection( + selected: &std::collections::HashSet, + registry: &[MacroRow], + use_libs: &[String], + lib_bodies: &std::collections::BTreeMap< + String, + Vec, + >, +) -> Result> { + use std::collections::{BTreeMap, BTreeSet, HashSet}; + use windmill_parser::duckdb_macros::{macro_create_statement, topo_order_macros, LibStatement}; + + let by_name: BTreeMap<&str, &MacroRow> = + registry.iter().map(|r| (r.name.as_str(), r)).collect(); + + // Providers whose setup must run: every `// use` lib (annotation order — + // even with zero selected macros, the explicit `use` carries its ATTACH + // side effects), then the provider of each selected macro (sorted). + let mut providers: Vec<&str> = use_libs.iter().map(String::as_str).collect(); + let selected_providers: BTreeSet<&str> = selected + .iter() + .filter_map(|n| by_name.get(n.as_str()).map(|r| r.provider_path.as_str())) + .collect(); + for p in selected_providers { + if !providers.contains(&p) { + providers.push(p); + } + } + + let mut injected: Vec = Vec::new(); + let mut seen_setup: HashSet = HashSet::new(); + for provider in providers { + let Some(statements) = lib_bodies.get(provider) else { + return Err(Error::ExecutionErr(format!( + "macro library `{provider}` has registry entries but its deployed script could \ + not be loaded" + ))); + }; + for s in statements { + if let LibStatement::Setup(stmt) = s { + let stmt = format!("{};", stmt.trim_end_matches(';').trim_end()); + if seen_setup.insert(stmt.clone()) { + injected.push(stmt); + } + } + } + } + + if selected.is_empty() { + return Ok(injected); + } + let defs: BTreeMap = selected + .iter() + .filter_map(|n| by_name.get(n.as_str()).map(|r| (n.clone(), r.body.clone()))) + .collect(); + let order = topo_order_macros(selected, &defs).map_err(Error::ExecutionErr)?; + for name in order { + let r = by_name.get(name.as_str()).ok_or_else(|| { + Error::ExecutionErr(format!("workspace macro `{name}` has no registry row")) + })?; + injected.push(macro_create_statement( + &r.name, + &r.params, + r.is_table_macro, + &r.body, + )); + } + Ok(injected) +} + +// Weave the injected blocks into the user's statement list. DuckDB +// bind-checks macro bodies at CREATE, in both directions: +// - an injected definition that a LOCAL macro calls must land *before* +// that local definition (pulling its own injected dependencies with it); +// - an injected definition that references a local macro must stay *after* +// it — a conflict between the two requirements is an error, not a +// silent mis-ordering. +// The default slot (no local interplay) is after the leading prefix of setup +// statements and local definitions, as before. Injected setup statements +// (library ATTACHes) go at the earliest slot any injected definition landed +// on: injected bodies are self-contained w.r.t. their own library's setup +// and never depend on the consumer's setup. +fn weave_macro_blocks(blocks: Vec, injected: Vec) -> Result> { + if injected.is_empty() { + return Ok(blocks); + } + use std::collections::HashSet; + use windmill_parser::duckdb_macros::{ + detect_macro_calls, is_macro_definition, parse_macro_definition, + }; + use windmill_parser::sql_materialize::{classify_block, BlockClass}; + + // Default slot: insert before the first block that is neither setup nor + // a local macro definition (i.e. after the whole leading prefix). + let default_slot = blocks + .iter() + .position(|b| !matches!(classify_block(b), BlockClass::Setup) && !is_macro_definition(b)) + .unwrap_or(blocks.len()); + + // The user's own macro definitions, as placement anchors. + let locals: Vec<(usize, windmill_parser::duckdb_macros::ParsedMacro)> = blocks + .iter() + .enumerate() + .filter_map(|(i, b)| parse_macro_definition(b).map(|m| (i, m))) + .collect(); + let local_names: HashSet = locals.iter().map(|(_, m)| m.name.clone()).collect(); + + // Plan emits [lib setup…, definitions in topo order…]. + let (setup, defs): (Vec, Vec) = + injected.into_iter().partition(|s| !is_macro_definition(s)); + let def_metas: Vec = defs + .iter() + .map(|s| { + parse_macro_definition(s).ok_or_else(|| { + Error::ExecutionErr(format!( + "internal: generated macro statement failed to re-parse: {}", + s.chars().take(80).collect::() + )) + }) + }) + .collect::>()?; + let def_names: HashSet = def_metas.iter().map(|m| m.name.clone()).collect(); + + // Per-injected-definition slot bounds from the local anchors. `slot = i` + // means "insert before block i". + let mut max_slot: Vec = vec![default_slot; defs.len()]; + let mut min_slot: Vec = vec![0; defs.len()]; + for (li, lm) in &locals { + let local_calls = detect_macro_calls(&lm.body, &def_names); + let referenced_locals_of: Vec = def_metas + .iter() + .enumerate() + .filter(|(_, dm)| detect_macro_calls(&dm.body, &local_names).contains(&lm.name)) + .map(|(di, _)| di) + .collect(); + for (di, dm) in def_metas.iter().enumerate() { + if local_calls.contains(&dm.name) { + max_slot[di] = max_slot[di].min(*li); + } + } + for di in referenced_locals_of { + min_slot[di] = min_slot[di].max(*li + 1); + } + } + // An injected definition must not land after any injected definition + // that depends on it: propagate upper bounds backwards through the topo + // order (dependents come later in `defs`). + let mut eff_slot = max_slot.clone(); + for i in (0..defs.len()).rev() { + for j in (i + 1)..defs.len() { + if detect_macro_calls(&def_metas[j].body, &def_names).contains(&def_metas[i].name) { + eff_slot[i] = eff_slot[i].min(eff_slot[j]); + } + } + if min_slot[i] > eff_slot[i] { + return Err(Error::ExecutionErr(format!( + "workspace macro `{}` and this script's own macro definitions have conflicting \ + order requirements (a local macro calls it while it references a local macro \ + defined later); reorder the local definitions", + def_metas[i].name + ))); + } + } + // Library setup runs before everything: the statements are self-contained + // (validated plain/non-managed at lib deploy, within-lib order preserved) + // and ANY consumer statement may depend on them at bind time — including + // a leading local macro definition that shadows a lib macro (so no def is + // injected) but still references the lib's ATTACHed catalog. + let mut out: Vec = Vec::with_capacity(blocks.len() + defs.len() + setup.len()); + out.extend(setup); + for slot in 0..=blocks.len() { + for (di, d) in defs.iter().enumerate() { + if eff_slot[di] == slot { + out.push(d.clone()); + } + } + if let Some(b) = blocks.get(slot) { + out.push(b.clone()); + } + } + Ok(out) +} + +// Workspace-macro injection (`// macros` libraries): fetch the registry, plan +// the needed `CREATE OR REPLACE TEMP MACRO` blocks and splice them into the +// job's statement list. Late-bound: every run reads the current registry, so a +// lib redeploy applies to the next run. On agent (Http) workers — no DB — the +// implicit path silently degrades (a called macro then fails with DuckDB's +// clear Catalog Error) but an explicit `// use` hard-errors like custom tests. +async fn inject_workspace_macros( + conn: &Connection, + w_id: &str, + is_macro_lib: bool, + use_libs: &[String], + blocks: Vec, +) -> Result> { + if is_macro_lib { + // A library run executes its own definitions; nothing to inject. + return Ok(blocks); + } + let db = match conn { + Connection::Sql(db) => db, + Connection::Http(_) => { + if !use_libs.is_empty() { + return Err(Error::ExecutionErr( + "`// use` requires a server worker (not supported on agent workers in v1)" + .to_string(), + )); + } + return Ok(blocks); + } + }; + let registry = fetch_macro_registry(db, w_id).await?; + if registry.is_empty() && use_libs.is_empty() { + return Ok(blocks); + } + if select_workspace_macros(&blocks, ®istry, use_libs)?.is_empty() && use_libs.is_empty() { + return Ok(blocks); + } + // Every relevant provider's deployed body is fetched: its setup + // statements are injected ahead of the definitions (a macro body may + // reference its own lib's ATTACH, which must run before the injected + // CREATE binds), and its own `// use` declarations are honored + // transitively — so the loop fetches lazily and re-resolves until no new + // library appears. Content fetches are cached by hash. + let mut lib_bodies: std::collections::BTreeMap< + String, + Vec, + > = Default::default(); + let mut lib_uses: std::collections::BTreeMap> = Default::default(); + let (selected, effective_use) = loop { + let (selected, effective) = + resolve_macro_selection(&blocks, ®istry, use_libs, &lib_uses)?; + let mut relevant: Vec = effective.clone(); + for name in &selected { + if let Some(r) = registry.iter().find(|r| &r.name == name) { + if !relevant.contains(&r.provider_path) { + relevant.push(r.provider_path.clone()); + } + } + } + let missing: Vec = relevant + .into_iter() + .filter(|l| !lib_bodies.contains_key(l)) + .collect(); + if missing.is_empty() { + break (selected, effective); + } + for path in missing { + let content = fetch_macro_lib_body(conn, w_id, &path).await?; + let statements = windmill_parser::duckdb_macros::parse_macro_library(&content) + .map_err(|e| { + Error::ExecutionErr(format!("macro library `{path}`: invalid content: {e}")) + })?; + lib_uses.insert( + path.clone(), + windmill_parser::asset_parser::parse_pipeline_annotations(&content).use_libs, + ); + lib_bodies.insert(path, statements); + } + }; + let injected = plan_macro_injection(&selected, ®istry, &effective_use, &lib_bodies)?; + weave_macro_blocks(blocks, injected) +} + +// Fetch a macro library's deployed body — for its setup statements; the +// macro definitions themselves come from the registry. Used both for `// use` +// libs and for the provider of every selected macro. Same server-worker +// fetch path as custom data tests. +async fn fetch_macro_lib_body(conn: &Connection, w_id: &str, path: &str) -> Result { + let Connection::Sql(db) = conn else { + return Err(Error::ExecutionErr( + "workspace macros require a server worker (not supported on agent workers in v1)" + .to_string(), + )); + }; + let hash = windmill_common::get_latest_script_hash(db, path, w_id) + .await? + .ok_or_else(|| { + Error::ExecutionErr(format!( + "macro library `{path}`: no deployed script found at this path" + )) + })?; + let content = + crate::get_script_content_by_hash(&windmill_common::scripts::ScriptHash(hash), w_id, conn) + .await?; + if !matches!( + content.language, + Some(windmill_common::scripts::ScriptLang::DuckDb) + ) { + return Err(Error::ExecutionErr(format!( + "macro library `{path}`: must be a DuckDB `// macros` library (got language {:?})", + content.language + ))); + } + Ok(content.content) +} + +// Pull a named i64 field (`snapshot_id` / `rows`) out of the trailing summary +// read — which in wrap mode is the job result. Shape-tolerant (object / array / +// nested), returns None if absent (literal mode, or capture failed). +fn extract_i64(result: &RawValue, field: &str) -> Option { + fn find(v: &Value, field: &str) -> Option { + match v { + Value::Number(n) => n.as_i64(), + Value::Object(m) => m.get(field).and_then(|x| find(x, field)), + Value::Array(a) => a.iter().find_map(|x| find(x, field)), + _ => None, + } + } + find(&serde_json::from_str::(result.get()).ok()?, field) +} + +// One data test's outcome as carried by the materialize summary's `data_tests` +// column: its display name, how many rows violated it (0 = pass), and an +// optional bounded sample of the violating rows. The sample is decoration +// only — enforcement reads `violating`, never `sample` — so a NULL, dropped +// (over the size cap) or unparseable sample must never affect pass/fail. +struct DataTestOutcome { + name: String, + violating: i64, + sample: Option, +} + +// Pull the per-test breakdown out of the materialize summary result. The +// `data_tests` column is a DuckLake list-of-struct `[{test, violating}, …]`; +// the FFI may surface it as a nested JSON array or as a JSON string, so accept +// both. Returns empty when there are no tests (the column is absent). +fn extract_data_tests(result: &RawValue) -> Vec { + fn collect(v: &Value, out: &mut Vec) { + if let Value::Array(arr) = v { + for item in arr { + if let Value::Object(o) = item { + if let Some(Value::String(name)) = o.get("test") { + let violating = o + .get("violating") + .and_then(|x| x.as_i64().or_else(|| x.as_f64().map(|f| f as i64))) + .unwrap_or(0); + // The probe serializes the sample as a JSON *string* + // (a VARCHAR through the FFI), deliberately — parse it + // only here, so sampled user columns named `rows` / + // `snapshot_id` stay invisible to the key-recursive + // `extract_i64` scans over the summary result. Accept + // a native array too (FFI JSON-column quirk); anything + // else (NULL, size-capped, garbage) degrades to None. + let sample = o.get("sample").and_then(|s| match s { + arr @ Value::Array(_) => Some(arr.clone()), + Value::String(txt) => serde_json::from_str::(txt) + .ok() + .filter(|v| v.is_array()), + _ => None, + }); + out.push(DataTestOutcome { name: name.clone(), violating, sample }); + } + } + } + } + } + fn find_field(v: &Value) -> Option<&Value> { + match v { + Value::Object(o) => o.get("data_tests"), + Value::Array(a) => a.iter().find_map(find_field), + _ => None, + } + } + let mut out = Vec::new(); + let Ok(root) = serde_json::from_str::(result.get()) else { + return out; + }; + match find_field(&root) { + Some(arr @ Value::Array(_)) => collect(arr, &mut out), + // FFI serialized the list-of-struct as a JSON string — parse it. + Some(Value::String(s)) => { + if let Ok(parsed) = serde_json::from_str::(s) { + collect(&parsed, &mut out); + } + } + _ => {} + } + out +} + +// Pull the captured output schema out of the materialize summary's +// `output_schema` column (gap #2a): a list-of-struct `[{name, type}, …]` the +// codegen built from a `DESCRIBE`. Like `data_tests`, the FFI may surface it as +// a nested JSON array or a JSON string — accept both. Returns `None` when the +// column is absent (literal mode, manual mode, or capture failed) so the worker +// records the run without a schema rather than an empty one. +fn extract_schema( + result: &RawValue, +) -> Option> { + use windmill_common::materialization::SchemaColumn; + fn collect(v: &Value) -> Option> { + let Value::Array(arr) = v else { return None }; + let mut out = Vec::with_capacity(arr.len()); + for item in arr { + let o = item.as_object()?; + let name = o.get("name")?.as_str()?.to_string(); + let data_type = o.get("type")?.as_str()?.to_string(); + out.push(SchemaColumn { name, data_type }); + } + Some(out) + } + fn find_field(v: &Value) -> Option<&Value> { + match v { + Value::Object(o) => o.get("output_schema"), + Value::Array(a) => a.iter().find_map(find_field), + _ => None, + } + } + let root = serde_json::from_str::(result.get()).ok()?; + match find_field(&root)? { + arr @ Value::Array(_) => collect(arr), + // FFI serialized the list-of-struct as a JSON string — parse it. + Value::String(s) => collect(&serde_json::from_str::(s).ok()?), + _ => None, + } +} + +// Pull the `on_schema_change=warn` drift out of the materialize summary's +// `schema_drift` column: a `{added: [..], removed: [..]}` struct (or NULL when +// the SELECT matched the table). Like `output_schema`, the FFI may surface it +// nested or as a JSON string — accept both. `None` when there is no drift. +fn extract_schema_drift(result: &RawValue) -> Option<(Vec, Vec)> { + fn str_list(v: Option<&Value>) -> Vec { + match v { + Some(Value::Array(a)) => a + .iter() + .filter_map(|x| x.as_str().map(str::to_string)) + .collect(), + _ => Vec::new(), + } + } + fn find_field(v: &Value) -> Option<&Value> { + match v { + Value::Object(o) => o.get("schema_drift"), + Value::Array(a) => a.iter().find_map(find_field), + _ => None, + } + } + let root = serde_json::from_str::(result.get()).ok()?; + let field = find_field(&root)?; + let owned; + let obj = match field { + Value::Object(_) => field, + // FFI serialized the struct as a JSON string — parse it. + Value::String(s) => { + owned = serde_json::from_str::(s).ok()?; + &owned + } + // NULL / absent ⇒ no drift. + _ => return None, + }; + let o = obj.as_object()?; + let added = str_list(o.get("added")); + let removed = str_list(o.get("removed")); + if added.is_empty() && removed.is_empty() { + None + } else { + Some((added, removed)) + } +} + +// Render the full pass/fail breakdown for a failed data-test run — every test, +// not just the first failure, so the user sees the whole picture in one place. +fn format_data_test_breakdown(asset_path: &str, tests: &[DataTestOutcome]) -> String { + let failed = tests.iter().filter(|t| t.violating > 0).count(); + let mut lines = vec![format!( + "data tests failed on {asset_path} ({failed}/{} failed):", + tests.len() + )]; + for t in tests { + if t.violating > 0 { + lines.push(format!(" ✗ {} — {} violating row(s)", t.name, t.violating)); + } else { + lines.push(format!(" ✓ {}", t.name)); + } + } + lines.join("\n") +} + +// Best-effort record of a materialization outcome. On a Sql connection it writes +// the row directly; on an agent worker (Http, no direct DB) it posts to the API +// so state lands the same way. Never fails the job — a lost row degrades the +// grid, not the run. +async fn record_mat( + conn: &Connection, + w_id: &str, + job_id: Uuid, + meta: &MaterializeExec, + status: windmill_common::materialization::MaterializationStatus, + snapshot_id: Option, + row_count: Option, + // Captured output schema (gap #2a). Only set on a successful materialize; + // when present, also upserts a `materialized_asset_schema` version. + schema: Option>, + error: Option<&str>, +) { + let req = windmill_common::materialization::RecordMaterializationRequest { + asset_kind: meta.asset_kind, + asset_path: meta.asset_path.clone(), + partition: meta.partition.clone(), + status, + snapshot_id, + row_count, + job_id: Some(job_id), + error: error.map(|e| e.to_string()), + schema: schema.clone(), + }; + let res: anyhow::Result<()> = match conn { + Connection::Sql(db) => { + let partition_res = windmill_common::materialization::record_materialization( + db, + w_id, + req.asset_kind, + &req.asset_path, + &req.partition, + req.status, + req.snapshot_id, + req.row_count, + req.job_id, + req.error.as_deref(), + ) + .await + .map_err(|e| anyhow::anyhow!("{e:#}")); + // Schema capture is a separate, independently best-effort write (its + // own transaction for the per-asset advisory lock); a failure here + // must not lose the partition row above. + if let Some(cols) = schema.as_ref() { + if let Err(e) = record_asset_schema_best_effort( + db, + w_id, + meta.asset_kind, + &meta.asset_path, + cols, + snapshot_id, + job_id, + ) + .await + { + tracing::warn!("failed to record captured asset schema: {e:#}"); + } + } + partition_res + } + Connection::Http(client) => { + crate::agent_workers::record_materialization_from_agent_http(client, w_id, &req).await + } + }; + if let Err(e) = res { + tracing::warn!("failed to record materialization state: {e:#}"); + } +} + +// Open a short transaction (needed for the per-asset advisory lock) and upsert +// the captured schema version. Isolated so its tx lifetime doesn't entangle the +// partition write. +async fn record_asset_schema_best_effort( + db: &windmill_common::DB, + w_id: &str, + asset_kind: windmill_common::assets::AssetKind, + asset_path: &str, + columns: &[windmill_common::materialization::SchemaColumn], + snapshot_id: Option, + job_id: Uuid, +) -> anyhow::Result<()> { + let mut tx = db.begin().await?; + windmill_common::materialization::record_asset_schema( + &mut tx, + w_id, + asset_kind, + asset_path, + columns, + snapshot_id, + Some(job_id), + ) + .await?; + tx.commit().await?; + Ok(()) +} + pub async fn do_duckdb( job: &MiniPulledJob, client: &AuthedClient, @@ -68,7 +1330,59 @@ pub async fn do_duckdb( let mut hidden_passwords = hidden_passwords.clone(); let mut bigquery_credentials = None; + // Materialization (`// materialize`): rewrite a wrap script into managed + // DDL (its synthetic target ATTACH is resolved by the transform pass + // below, like the user's own ATTACH); a literal script is left as-is. + // `materialize` also carries what to record once the run finishes. + let partition_value: Option = job + .args + .as_ref() + .and_then(|a| a.0.get(windmill_common::partition::PARTITION_ARG)) + .and_then(|rv| serde_json::from_str::(rv.get()).ok()) + .filter(|s| !s.is_empty()); + let materialize = if query.contains("materialize") || query.contains("data_test") { + // Custom-test bodies need a DB read; fetch them first so the codegen + // itself stays pure/sync. + let custom_test_bodies = + fetch_custom_test_bodies(query, conn, &job.workspace_id).await?; + build_materialized_query(query, partition_value.as_deref(), &custom_test_bodies)? + } else { + None + }; + // Parse the signature from the ORIGINAL script: managed materialize wraps + // the trailing SELECT and strips line comments, which drops the + // `-- $name (type)` arg declarations while their `$name` references + // survive in the embedded SELECT. Parsing args here (pre-wrap) keeps them + // declared so they are still bound — and s3object args translated to + // `s3://` URIs — at run time. let sig = parse_duckdb_sig(query)?.args; + + // `// macros` / `// use` also come off the ORIGINAL script text (the + // materialize rewrite strips the annotation comments). Drives the + // workspace-macro injection after the ATTACH-transform pass below. + let macro_ann = windmill_parser::asset_parser::parse_pipeline_annotations(query); + + let materialized_query; + let query: &str = match &materialize { + Some((Some(rewritten), _)) => { + materialized_query = rewritten.clone(); + &materialized_query + } + _ => query, + }; + + // Managed materialize generates its own trailing summary row (asset / + // rows / snapshot_id / data_tests), and data-test enforcement below reads + // the `data_tests` column off that row. The row shape is ours, not the + // user's — so force the full-last-row strategy regardless of any + // `// result_collection` annotation, which would otherwise reshape it + // (e.g. a scalar mode drops every column but the first) and silently + // bypass test enforcement. + let collection_strategy = if matches!(&materialize, Some((Some(_), _))) { + SqlResultCollectionStrategy::LastStatementAllRows + } else { + collection_strategy + }; let mut job_args = build_args_values(job, client, conn).await?; let reserved_variables = @@ -76,7 +1390,27 @@ pub async fn do_duckdb( let (query, _) = &sanitize_and_interpolate_unsafe_sql_args(query, &sig, &job_args, &reserved_variables)?; - let query = transform_s3_uris(query).await?; + let mut query = transform_s3_uris(query).await?; + + // `on_schema_change=sync`: interpolate the pre-pass probe with the same + // sig / args / reserved vars as the main query (before `sig` is consumed + // and the arg map drained just below) so its `DESCRIBE` of the SELECT + // binds `$args`/`$partition` the same way. + let sync_probe_interpolated = match materialize + .as_ref() + .and_then(|(_, m)| m.sync_prepass.as_ref()) + { + Some(pp) => { + let (interp, _) = sanitize_and_interpolate_unsafe_sql_args( + &pp.probe_query, + &sig, + &job_args, + &reserved_variables, + )?; + Some(transform_s3_uris(&interp).await?) + } + None => None, + }; let job_args = { let mut m = Vec::new(); @@ -111,6 +1445,96 @@ pub async fn do_duckdb( m }; + // `on_schema_change=sync` pre-pass: run the probe (through the same + // ATTACH transform as the main query), diff the SELECT's columns against + // the table's, and splice the `ALTER TABLE … ADD/DROP COLUMN` DDL into + // the plan at the sentinel slot (removing the sentinel when there is no + // drift). Runs before the main query so the ALTERs are in place when the + // BY NAME insert executes. + if let (Some(probe_sql), Some(pp)) = ( + &sync_probe_interpolated, + materialize + .as_ref() + .and_then(|(_, m)| m.sync_prepass.as_ref()), + ) { + let mut probe_blocks = vec![]; + // Held for the whole probe (through compute_sync_alter_ddl): a + // `TYPE bigquery` ATTACH needs GOOGLE_APPLICATION_CREDENTIALS + the + // temp creds file set before DuckDB binds the extension, exactly as + // the main query path does — otherwise the probe DESCRIBE fails. + let mut probe_bigquery_credentials = None; + for query_block in parse_sql_blocks(probe_sql, true).iter() { + let query_block = remove_comments(query_block); + if let Some(parsed) = parse_attach_db_resource(query_block) { + probe_blocks.extend( + transform_attach_db_resource_query( + &parsed, + &job.id, + client, + &mut hidden_passwords, + ) + .await?, + ); + if parsed.db_type == "bigquery" { + probe_bigquery_credentials = Some(UseBigQueryCredentialsFile::new( + job.id, + parsed.resource_path, + )?); + } + } else if let Some(q) = transform_attach_ducklake( + &query_block, + conn, + &mut hidden_passwords, + &job.workspace_id, + materialize.as_ref().map(|(_, m)| m.asset_path.as_str()), + ) + .await? + { + probe_blocks.extend(q); + } else if let Some(q) = transform_attach_datatable( + &query_block, + conn, + &mut hidden_passwords, + &job.workspace_id, + ) + .await? + { + probe_blocks.extend(q); + } else { + probe_blocks.push(query_block.to_string()); + } + } + // The probe `DESCRIBE`s the same SELECT as the main query, so it + // needs the same workspace-macro / `// use` library definitions + // injected (post-ATTACH), or a macro-calling SELECT fails to bind + // here and the fatal probe makes `sync` unusable with macros. + let probe_blocks = inject_workspace_macros( + conn, + &job.workspace_id, + macro_ann.macros, + ¯o_ann.use_libs, + probe_blocks, + ) + .await?; + let alter_ddl = compute_sync_alter_ddl( + probe_blocks, + job_args.clone(), + pp.target_table.clone(), + token.clone(), + client.base_internal_url.clone(), + job.workspace_id.clone(), + job_dir.to_string(), + ) + .await?; + query = query.replace( + windmill_parser::sql_materialize::SYNC_ALTER_SENTINEL, + &alter_ddl, + ); + // Kept alive until the probe finished; the main query path recreates + // its own credentials for its own ATTACH pass below. + drop(probe_bigquery_credentials); + } + let query_block_list = parse_sql_blocks(&query, true); // Replace custom ATTACH statements with the real instructions @@ -139,6 +1563,7 @@ pub async fn do_duckdb( conn, &mut hidden_passwords, &job.workspace_id, + materialize.as_ref().map(|(_, m)| m.asset_path.as_str()), ) .await? { @@ -159,6 +1584,19 @@ pub async fn do_duckdb( v }; + // Workspace macros: splice `CREATE OR REPLACE TEMP MACRO` blocks for + // registry macros this script calls (plus whole `// use` libraries) + // after the setup/ATTACH prefix — post-transform, so macro bodies can + // reference the attached catalogs when DuckDB bind-checks the CREATE. + let query_block_list = inject_workspace_macros( + conn, + &job.workspace_id, + macro_ann.macros, + ¯o_ann.use_libs, + query_block_list, + ) + .await?; + let base_internal_url = client.base_internal_url.clone(); let w_id = job.workspace_id.clone(); let job_dir = job_dir.to_string(); @@ -199,6 +1637,20 @@ pub async fn do_duckdb( let (result, column_order) = match result { Ok(r) => r, Err(e) => { + if let Some((_, meta)) = &materialize { + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + None, + None, + None, + Some(&e.to_string()), + ) + .await; + } if let Some(s3_proxy_err) = S3_PROXY_LAST_ERRORS_CACHE.get(&client.token) { return Err(Error::ExecutionErr(format!( "{}\n\nS3 Related Error: {}", @@ -210,6 +1662,157 @@ pub async fn do_duckdb( } }; + if let Some((_, meta)) = &materialize { + // In wrap mode the job result is the summary read (snapshot_id + + // rows + the per-test breakdown); in literal mode there is none. + let snapshot_id = extract_i64(&result, "snapshot_id"); + let row_count = extract_i64(&result, "rows"); + // Data tests all ran (every check counted in one query); decide + // pass/fail here. Any violation fails the run — the write is already + // committed (like dbt), so the slice is recorded `Failed` and the + // cascade stops. The error lists *every* test so the user sees the + // whole picture, not just the first failure. + // Under enterprise write-audit-publish an in-transaction guard + // (the enterprise `finalize_materialize_query` restructure) already aborted a failing run before + // COMMIT — that surfaces on the Err path above with the same + // breakdown in the error string, and nothing was published; this + // post-commit path then only ever sees passing counts. + let tests = extract_data_tests(&result); + // Captured output schema (gap #2a) — recorded only on the successful + // path below, not on the failure paths (a failed run shouldn't + // advance the asset's recorded schema version). Managed mode ONLY: + // in `// materialize manual` the result is the user's own query + // output (we generate no summary), so an `output_schema` field there + // is caller-shaped and must not be trusted — `materialize` is + // `Some((Some(_), _))` for managed, `Some((None, _))` for manual. + let is_managed = matches!(&materialize, Some((Some(_), _))); + let schema = if is_managed { + extract_schema(&result) + } else { + None + }; + // `on_schema_change=warn` (default): the summary carries a + // `schema_drift` column when the SELECT's columns diverged from the + // fixed table schema. The write already happened positionally (data + // may have landed in the wrong/old columns) — log it LOUDLY so the + // drift isn't silent. The drift also rides back in the job result + // (it's a column of the returned summary row). + if is_managed + && meta.on_schema_change == windmill_parser::asset_parser::OnSchemaChange::Warn + { + if let Some((added, removed)) = extract_schema_drift(&result) { + let fmt = |cols: &[String]| { + if cols.is_empty() { + "(none)".to_string() + } else { + cols.join(", ") + } + }; + let warning = format!( + "\n⚠️ SCHEMA DRIFT on {asset}: the SELECT's columns no longer match the \ + existing table.\n added (in SELECT, not in table): {added}\n removed \ + (in table, not in SELECT): {removed}\n The write proceeded POSITIONALLY \ + against the existing table schema — data may have landed in the wrong \ + columns. Set `on_schema_change=fail` to block this, or \ + `on_schema_change=sync` to migrate the table automatically.\n", + asset = meta.asset_path, + added = fmt(&added), + removed = fmt(&removed), + ); + append_logs(&job.id, &job.workspace_id, warning, conn).await; + } + } + // Defense-in-depth: codegen embedded `n_data_tests` checks, so the + // summary row must carry that many outcomes. Recovering fewer means + // the `data_tests` column was dropped/reshaped before we read it — + // fail loud rather than silently pass unverified tests. + if tests.len() < meta.n_data_tests { + let msg = format!( + "data tests on {}: expected {} test outcome(s) but recovered {} from the \ + result — aborting to avoid a silent pass", + meta.asset_path, + meta.n_data_tests, + tests.len() + ); + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + snapshot_id, + row_count, + None, + Some(&msg), + ) + .await; + return Err(Error::ExecutionErr(msg)); + } + if tests.iter().any(|t| t.violating > 0) { + let breakdown = format_data_test_breakdown(&meta.asset_path, &tests); + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + snapshot_id, + row_count, + None, + Some(&breakdown), + ) + .await; + // Structured failure payload: the queue wraps it as + // `{"error": {...}}` (`WrappedError`) and result_processor + // derives the run description from the top-level `message`, + // so keep `message` at the top and add no `error` nesting of + // our own. Samples ride only on failed tests, only in this + // structured result — the message text stays counts-only. + let has_samples = tests.iter().any(|t| t.violating > 0 && t.sample.is_some()); + let message = if has_samples { + // Wording must not match the UI's breakdown-line parsing + // (no ✓/✗, no `— N violating`), which older-result + // rendering still relies on. + format!( + "{breakdown}\n\nSamples of the violating rows are \ + attached to this run's result (error.data_tests)." + ) + } else { + breakdown + }; + let data_tests = tests + .iter() + .map(|t| { + let mut o = json!({ "test": t.name, "violating": t.violating }); + if t.violating > 0 { + if let Some(s) = &t.sample { + o["sample"] = s.clone(); + } + } + o + }) + .collect::>(); + return Err(Error::ExecutionRawError(to_raw_value(&json!({ + "message": message, + "name": "ExecutionErr", + "step_id": job.flow_step_id, + "data_tests": data_tests, + })))); + } + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Materialized, + snapshot_id, + row_count, + schema, + None, + ) + .await; + } + drop(bigquery_credentials); *column_order_ref = column_order; @@ -237,14 +1840,46 @@ pub async fn do_duckdb( match result { Ok(result) => Ok(result), Err(e) => { - // Passwords might appear in the error message - let mut err_str = e.to_string(); - for pwd in hidden_passwords.lock().unwrap().iter() { - if let Some(sanitized) = sanitize_string_from_password(&err_str, &pwd.clone()) { - err_str = sanitized; + // Passwords might appear in the error message — and, for the + // structured data-test failure, in sampled row data read from an + // attached database — so every outgoing error is sanitized here. + let sanitize = |mut s: String| { + for pwd in hidden_passwords.lock().unwrap().iter() { + if let Some(sanitized) = sanitize_string_from_password(&s, &pwd.clone()) { + s = sanitized; + } } + s + }; + match e { + // The structured payload must keep its variant: flattening it + // to a string (the arm below) would strip the data-test + // samples result_processor places verbatim into the failed + // job's result. Sanitize its string *leaves*, not the + // serialized text: a secret containing `"` or `\` is + // JSON-escaped in the text, so a plain-substring pass would + // miss it — and samples carry raw row data from attached + // databases. + Error::ExecutionRawError(raw) => { + fn walk(v: &mut Value, f: &dyn Fn(String) -> String) { + match v { + Value::String(s) => *s = f(std::mem::take(s)), + Value::Array(a) => a.iter_mut().for_each(|x| walk(x, f)), + Value::Object(o) => o.values_mut().for_each(|x| walk(x, f)), + _ => {} + } + } + Err(match serde_json::from_str::(raw.get()) { + Ok(mut v) => { + walk(&mut v, &sanitize); + Error::ExecutionRawError(to_raw_value(&v)) + } + // Not valid JSON (shouldn't happen) — redact as text. + Err(_) => Error::ExecutionErr(sanitize(raw.get().to_string())), + }) + } + e => Err(Error::ExecutionErr(sanitize(e.to_string()))), } - Err(Error::ExecutionErr(err_str)) } } } @@ -383,6 +2018,15 @@ fn cgroup_bytes_to_duckdb_memory_limit(bytes: i64) -> Option { } // Read backend/windmill-duckdb-ffi-internal/README_DEV.md for details about why we use FFI +// The FFI returns errors as `ERROR ` (the message is +// serde_json::to_string'd). Decode the JSON string back so multi-line errors +// render with real newlines, not escaped `\n` inside wrapping quotes; fall back +// to the raw slice if it is somehow not a JSON string. +fn decode_ffi_error(result_str: &str) -> String { + let raw = result_str.strip_prefix("ERROR ").unwrap_or(result_str); + serde_json::from_str::(raw).unwrap_or_else(|_| raw.to_string()) +} + fn run_duckdb_ffi_safe<'a>( query_block_list: impl Iterator, query_block_list_count: usize, @@ -448,7 +2092,7 @@ fn run_duckdb_ffi_safe<'a>( }; if result_str.starts_with("ERROR") { - Err(Error::ExecutionErr(result_str[6..].to_string())) + Err(Error::ExecutionErr(decode_ffi_error(&result_str))) } else { let result = if collection_strategy == SqlResultCollectionStrategy::AllStatementsAllRows { // Avoid parsing JSON @@ -512,7 +2156,7 @@ fn prepare_duckdb_ffi_safe<'a>( }; if result_str.starts_with("ERROR") { - Err(Error::ExecutionErr(result_str[6..].to_string())) + Err(Error::ExecutionErr(decode_ffi_error(&result_str))) } else { Ok(serde_json::value::RawValue::from_string(result_str).map_err(to_anyhow)?) } @@ -663,6 +2307,7 @@ async fn transform_attach_ducklake( conn: &Connection, hidden_passwords: &mut Arc>>, w_id: &str, + materialize_target: Option<&str>, ) -> Result>> { lazy_static::lazy_static! { static ref RE: regex::Regex = regex::Regex::new(r"(?i)ATTACH\s*'ducklake(://[^':]+)?'\s*AS\s+([^ ;]+)\s*(\([^)]*\))?").unwrap(); @@ -672,15 +2317,28 @@ async fn transform_attach_ducklake( }; let name = cap.get(1).map(|m| &m.as_str()[3..]).unwrap_or("main"); let alias_name = cap.get(2).map(|m| m.as_str()).unwrap_or(""); - let extra_args = cap + let user_extra_args = cap .get(3) - .map(|m| format!(", {}", &m.as_str()[1..m.as_str().len() - 1])) - .unwrap_or("".to_string()); + .map(|m| m.as_str()[1..m.as_str().len() - 1].to_string()) + .unwrap_or_default(); let ducklake = match conn { Connection::Http(client) => get_ducklake_from_agent_http(client, name, w_id).await?, Connection::Sql(db) => get_ducklake_from_db_unchecked(name, w_id, db).await?, }; + // In a fork, METADATA_SCHEMA / DATA_PATH / OVERRIDE_DATA_PATH are owned by the fork + // resolution (DuckDB silently keeps the last occurrence of a duplicated option, so a + // user-supplied one would escape the fork namespace back to the parent's). + let user_extra_args = if ducklake.fork_defer.is_some() { + strip_fork_reserved_attach_args(&user_extra_args) + } else { + user_extra_args + }; + let extra_args = if user_extra_args.is_empty() { + String::new() + } else { + format!(", {}", user_extra_args) + }; let db_type = match ducklake.catalog.resource_type { DucklakeCatalogResourceType::Instance => "postgres", _ => ducklake.catalog.resource_type.as_ref(), @@ -726,17 +2384,170 @@ async fn transform_attach_ducklake( } else { format!(", AUTOMATIC_MIGRATION TRUE{extra_args}") }; + // In a fork, re-emit the fork-owned DATA_PATH as the LAST option: DuckDB keeps the last + // occurrence of a duplicated option, so this wins over anything the arg stripping might + // not recognize (e.g. a dollar-quoted literal), regardless of literal syntax. The + // METADATA_SCHEMA injected by the fork resolution is already last within `extra_args`. + let extra_args = if ducklake.fork_defer.is_some() { + format!("{extra_args}, DATA_PATH 's3://{storage}/{data_path}'") + } else { + extra_args + }; let attach_str = format!( "ATTACH 'ducklake:{db_type}:{db_conn_str}' AS {alias_name} (DATA_PATH 's3://{storage}/{data_path}'{extra_args});", ); let install_db_ext_str = get_attach_db_install_str(db_type)?; - Ok(Some(vec![ + let mut statements = vec![ "INSTALL ducklake;".to_string(), install_db_ext_str.to_string(), attach_str, - ])) + ]; + if let Some(defer) = ducklake.fork_defer.as_ref() { + statements.extend(fork_defer_statements( + name, + alias_name, + defer, + materialize_target, + hidden_passwords, + )?); + } + Ok(Some(statements)) +} + +// Double-quote a possibly schema-qualified table reference, each dotted segment +// independently (local copy of sql_materialize's private helper). +fn quote_qualified_table(name: &str) -> String { + name.split('.') + .map(|id| format!("\"{}\"", id.replace('"', "\"\""))) + .collect::>() + .join(".") +} + +// `schema.table` → `schema.table_current`: the SCD2 companion view lives next to its table. +fn current_companion(table: &str) -> String { + match table.rsplit_once('.') { + Some((s, t)) => format!("{s}.{t}_current"), + None => format!("{table}_current"), + } +} + +/// Statements appended after a fork workspace's lake ATTACH: read-only attaches of the +/// ancestor namespaces plus `CREATE VIEW IF NOT EXISTS` defer views over the direct parent for +/// every table the fork has not materialized. When this job's managed materialize targets one +/// of the deferred tables, its defer view is dropped instead of created — the write must hit a +/// real fork table (`CREATE [OR REPLACE] TABLE` refuses to replace a view). +fn fork_defer_statements( + lake_name: &str, + alias_name: &str, + defer: &windmill_common::workspaces::DucklakeForkDefer, + materialize_target: Option<&str>, + hidden_passwords: &mut Arc>>, +) -> Result> { + let mut stmts = vec![]; + if defer.ancestors.is_empty() { + // Defer unavailable (an ancestor no longer defines this lake); the fork namespace + // still isolates writes. + return Ok(stmts); + } + for a in &defer.ancestors { + if let Some(pwd) = a.catalog_resource.get("password").and_then(|p| p.as_str()) { + hidden_passwords.lock().unwrap().push(pwd.to_string()); + } + let db_type = match a.catalog.resource_type { + DucklakeCatalogResourceType::Instance => "postgres", + _ => a.catalog.resource_type.as_ref(), + }; + stmts.push(get_attach_db_install_str(db_type)?.to_string()); + let conn_str = + format_attach_db_conn_str(a.catalog_resource.clone(), db_type)?.replace('\'', "''"); + let storage = a + .storage + .storage + .as_deref() + .unwrap_or(DEFAULT_STORAGE) + .replace('\'', "''"); + let data_path = a.storage.path.replace('\'', "''"); + let metadata_schema = a + .metadata_schema + .as_ref() + .map(|s| format!(", METADATA_SCHEMA '{}'", s.replace('\'', "''"))) + .unwrap_or_default(); + // The ancestor config's own non-reserved args (e.g. `ENCRYPTED true`) — an + // option-dependent lake wouldn't attach without them. Emitted FIRST: DuckDB keeps the + // last occurrence of a duplicated option, so the fork-owned DATA_PATH / READ_ONLY / + // METADATA_SCHEMA after them always win. + let extra_args = a + .extra_args + .as_ref() + .map(|e| format!("{e}, ")) + .unwrap_or_default(); + // READ_ONLY: a fork job must never write an ancestor namespace. No AUTOMATIC_MIGRATION + // / CREATE_IF_NOT_EXISTS: an ancestor lake that would need creating or migrating fails + // loudly rather than being mutated from a fork. IF NOT EXISTS: several ATTACH blocks in + // one script (e.g. the user's + the materialize synthetic one) emit the same ancestors. + stmts.push(format!( + "ATTACH IF NOT EXISTS 'ducklake:{db_type}:{conn_str}' AS {} ({extra_args}DATA_PATH 's3://{storage}/{data_path}', OVERRIDE_DATA_PATH TRUE, READ_ONLY{metadata_schema});", + a.alias + )); + } + let parent_alias = &defer.ancestors[0].alias; + let target_table = materialize_target.and_then(|ap| { + let (l, t) = ap.split_once('/')?; + (l == lake_name).then_some(t) + }); + let mut created_schemas = std::collections::HashSet::new(); + for dt in &defer.defer_tables { + if Some(dt.table.as_str()) == target_table { + continue; + } + // Each view targets the NEAREST ancestor that physically owns the table (a direct + // parent that itself defers has no copy to bind against). Out-of-range index (never + // produced by the resolver, but the field crosses the agent wire) falls back to the + // direct parent rather than panicking. + let owner_alias = defer + .ancestors + .get(dt.ancestor_idx as usize) + .map(|a| a.alias.as_str()) + .unwrap_or(parent_alias); + if let Some((schema, _)) = dt.table.rsplit_once('.') { + if created_schemas.insert(schema) { + stmts.push(format!( + "CREATE SCHEMA IF NOT EXISTS {alias_name}.{};", + quote_qualified_table(schema) + )); + } + } + let q = quote_qualified_table(&dt.table); + stmts.push(format!( + "CREATE VIEW IF NOT EXISTS {alias_name}.{q} AS SELECT * FROM {owner_alias}.{q};" + )); + if dt.with_current_view { + let qc = quote_qualified_table(¤t_companion(&dt.table)); + stmts.push(format!( + "CREATE VIEW IF NOT EXISTS {alias_name}.{qc} AS SELECT * FROM {owner_alias}.{qc};" + )); + } + } + if let Some(t) = target_table { + // View→table transition: replace the target's defer view(s) with the real table this + // job writes (`CREATE [OR REPLACE] TABLE` refuses to replace a view). Keyed on the + // catalog's ACTUAL live views — not on recorded materialization status, which after a + // failed run can't tell a defer view from a real table, and a mismatched DROP VIEW + // would wedge the asset. `_current` is dropped too when it is a view — SCD2 codegen + // recreates it with `IF NOT EXISTS`, which would otherwise silently keep a view over + // the parent. + for name in [t.to_string(), current_companion(t)] { + if defer.fork_views.iter().any(|v| v == &name) { + stmts.push(format!( + "DROP VIEW IF EXISTS {alias_name}.{};", + quote_qualified_table(&name) + )); + } + } + } + Ok(stmts) } async fn transform_attach_datatable( @@ -841,6 +2652,591 @@ pub struct Arg { mod tests { use super::*; + #[test] + fn decode_ffi_error_unescapes_multiline_and_strips_quotes() { + // Mirror the FFI: JSON-encode the raw DuckDB message, prefix "ERROR ". + let raw_msg = "Invalid Input Error: data tests failed on main/raw_orders \ + (1/3 failed) — write rolled back, previous version left live:\n \ + ✓ not_null(order_id)\n ✗ accepted_values(status) — 12 violating row(s)"; + let ffi = format!("ERROR {}", serde_json::to_string(raw_msg).unwrap()); + let decoded = decode_ffi_error(&ffi); + assert_eq!(decoded, raw_msg); + // real newlines, no literal `\n`, no wrapping quotes + assert!(decoded.contains('\n') && !decoded.contains("\\n")); + assert!(!decoded.starts_with('"')); + // single-line error with inner quotes round-trips unescaped + let single = format!( + "ERROR {}", + serde_json::to_string("Binder Error: column \"x\" not found").unwrap() + ); + assert_eq!( + decode_ffi_error(&single), + "Binder Error: column \"x\" not found" + ); + // non-JSON payload falls back to the raw slice + assert_eq!(decode_ffi_error("ERROR not json"), "not json"); + } + + fn test_ancestor(wid: &str) -> windmill_common::workspaces::DucklakeAncestorAttach { + windmill_common::workspaces::DucklakeAncestorAttach { + workspace_id: wid.to_string(), + alias: format!("__wm_dl_lake_{}_abcd1234", wid.replace('-', "_")), + catalog: windmill_common::workspaces::DucklakeCatalog { + resource_type: windmill_common::workspaces::DucklakeCatalogResourceType::Instance, + resource_path: "cat_db".to_string(), + }, + catalog_resource: serde_json::json!({ + "host": "h", "dbname": "cat_db", "user": "u", "password": "pw" + }), + storage: windmill_common::workspaces::DucklakeStorage { + storage: None, + path: "lake".to_string(), + }, + metadata_schema: None, + extra_args: None, + } + } + + fn test_fork_defer_chain( + ancestor_wids: Vec<&str>, + defer_tables: Vec<(&str, bool, u32)>, + fork_views: Vec<&str>, + ) -> windmill_common::workspaces::DucklakeForkDefer { + windmill_common::workspaces::DucklakeForkDefer { + ancestors: ancestor_wids.into_iter().map(test_ancestor).collect(), + defer_tables: defer_tables + .into_iter() + .map( + |(t, cur, idx)| windmill_common::materialization::ForkDeferTable { + table: t.to_string(), + with_current_view: cur, + ancestor_idx: idx, + }, + ) + .collect(), + fork_views: fork_views.into_iter().map(str::to_string).collect(), + } + } + + fn test_fork_defer( + defer_tables: Vec<(&str, bool)>, + fork_views: Vec<&str>, + ) -> windmill_common::workspaces::DucklakeForkDefer { + test_fork_defer_chain( + vec!["parent-ws"], + defer_tables.into_iter().map(|(t, c)| (t, c, 0)).collect(), + fork_views, + ) + } + + #[test] + fn test_fork_defer_statements_ancestor_extra_args() { + // The ancestor config's own args (e.g. ENCRYPTED) must ride on its read-only attach — + // BEFORE the fork-owned options, so DuckDB's last-occurrence-wins keeps DATA_PATH / + // READ_ONLY / METADATA_SCHEMA authoritative even if the args tried to override them. + let mut defer = test_fork_defer(vec![("orders", false)], vec![]); + defer.ancestors[0].extra_args = Some("ENCRYPTED true".to_string()); + let mut hp = Arc::new(Mutex::new(vec![])); + let stmts = fork_defer_statements("lake", "dl", &defer, None, &mut hp).unwrap(); + let attach = stmts + .iter() + .find(|s| s.starts_with("ATTACH IF NOT EXISTS")) + .unwrap(); + assert!(attach.contains("(ENCRYPTED true, DATA_PATH "), "{attach}"); + assert!( + attach.find("ENCRYPTED true").unwrap() < attach.find("READ_ONLY").unwrap(), + "{attach}" + ); + } + + #[test] + fn test_fork_defer_statements_chained_ancestors() { + // fork → parent → root: `orders` was only materialized in the root (idx 1) — its view + // must target the ROOT alias (the parent has no physical copy); `daily` owned by the + // direct parent (idx 0) targets the parent alias. Out-of-range idx falls back to the + // direct parent instead of panicking. + let defer = test_fork_defer_chain( + vec!["parent-ws", "root-ws"], + vec![("orders", false, 1), ("daily", false, 0), ("oob", false, 9)], + vec![], + ); + let mut hp = Arc::new(Mutex::new(vec![])); + let stmts = fork_defer_statements("lake", "dl", &defer, None, &mut hp).unwrap(); + let joined = stmts.join("\n"); + assert!( + joined.contains( + "CREATE VIEW IF NOT EXISTS dl.\"orders\" AS SELECT * FROM __wm_dl_lake_root_ws_abcd1234.\"orders\"" + ), + "{joined}" + ); + assert!( + joined.contains( + "CREATE VIEW IF NOT EXISTS dl.\"daily\" AS SELECT * FROM __wm_dl_lake_parent_ws_abcd1234.\"daily\"" + ), + "{joined}" + ); + assert!( + joined.contains( + "CREATE VIEW IF NOT EXISTS dl.\"oob\" AS SELECT * FROM __wm_dl_lake_parent_ws_abcd1234.\"oob\"" + ), + "{joined}" + ); + // Both ancestors attached read-only. + assert_eq!( + joined.matches("ATTACH IF NOT EXISTS").count(), + 2, + "{joined}" + ); + } + + #[test] + fn test_fork_defer_statements_shape() { + let defer = test_fork_defer(vec![("orders", false), ("dim", true)], vec![]); + let mut hp = Arc::new(Mutex::new(vec![])); + let stmts = fork_defer_statements("lake", "dl", &defer, None, &mut hp).unwrap(); + let joined = stmts.join("\n"); + // Ancestor attach: read-only, idempotent, never auto-migrating or auto-creating. + assert!(joined.contains("ATTACH IF NOT EXISTS"), "{joined}"); + assert!(joined.contains("READ_ONLY"), "{joined}"); + assert!(!joined.contains("AUTOMATIC_MIGRATION"), "{joined}"); + assert!(!joined.contains("CREATE_IF_NOT_EXISTS"), "{joined}"); + // Ancestor catalog password is hidden from logs. + assert_eq!(hp.lock().unwrap().as_slice(), ["pw"]); + // Defer views over the parent alias, plus the SCD2 `_current` companion. + assert!( + joined.contains( + "CREATE VIEW IF NOT EXISTS dl.\"orders\" AS SELECT * FROM __wm_dl_lake_parent_ws_abcd1234.\"orders\"" + ), + "{joined}" + ); + assert!(joined.contains("dl.\"dim_current\""), "{joined}"); + // No target → no transition drops. + assert!(!joined.contains("DROP VIEW"), "{joined}"); + } + + #[test] + fn test_fork_defer_statements_target_transition() { + // Target currently a defer view → skip its CREATE, drop the view (+ companion). + let defer = test_fork_defer(vec![("orders", false)], vec!["orders", "orders_current"]); + let mut hp = Arc::new(Mutex::new(vec![])); + let stmts = + fork_defer_statements("lake", "_wm_target", &defer, Some("lake/orders"), &mut hp) + .unwrap(); + let joined = stmts.join("\n"); + assert!(!joined.contains("CREATE VIEW"), "{joined}"); + assert!( + joined.contains("DROP VIEW IF EXISTS _wm_target.\"orders\";"), + "{joined}" + ); + assert!( + joined.contains("DROP VIEW IF EXISTS _wm_target.\"orders_current\";"), + "{joined}" + ); + + // Target already a real table (NOT in fork_views, e.g. after a failed re-run whose + // status can't be trusted) → no DROP VIEW, or the job would wedge on a type mismatch. + let defer = test_fork_defer(vec![("orders", false)], vec![]); + let stmts = + fork_defer_statements("lake", "_wm_target", &defer, Some("lake/orders"), &mut hp) + .unwrap(); + assert!(!stmts.join("\n").contains("DROP VIEW"), "{stmts:?}"); + + // Target in a different lake → this lake's defer views are untouched. + let defer = test_fork_defer(vec![("orders", false)], vec!["orders"]); + let stmts = + fork_defer_statements("lake", "dl", &defer, Some("other/orders"), &mut hp).unwrap(); + let joined = stmts.join("\n"); + assert!( + joined.contains("CREATE VIEW IF NOT EXISTS dl.\"orders\""), + "{joined}" + ); + assert!(!joined.contains("DROP VIEW"), "{joined}"); + } + + #[test] + fn test_fork_defer_statements_schema_qualified() { + let defer = test_fork_defer(vec![("staging.raw", false)], vec![]); + let mut hp = Arc::new(Mutex::new(vec![])); + let stmts = fork_defer_statements("lake", "dl", &defer, None, &mut hp).unwrap(); + let joined = stmts.join("\n"); + assert!( + joined.contains("CREATE SCHEMA IF NOT EXISTS dl.\"staging\";"), + "{joined}" + ); + assert!( + joined.contains("CREATE VIEW IF NOT EXISTS dl.\"staging\".\"raw\""), + "{joined}" + ); + } + + fn mrow(name: &str, params: &str, body: &str, is_table: bool, provider: &str) -> MacroRow { + MacroRow { + name: name.to_string(), + params: params.to_string(), + body: body.to_string(), + is_table_macro: is_table, + provider_path: provider.to_string(), + } + } + + fn blocks(list: &[&str]) -> Vec { + list.iter().map(|s| s.to_string()).collect() + } + + // Mimics the shell: resolve (incl. transitive library `// use`), parse the + // given lib sources, plan. `libs` maps provider path → deployed source + // (must cover every relevant provider). + fn select_and_plan( + b: &[String], + registry: &[MacroRow], + use_libs: &[&str], + libs: &[(&str, &str)], + ) -> Result> { + use windmill_parser::asset_parser::parse_pipeline_annotations; + use windmill_parser::duckdb_macros::parse_macro_library; + let use_libs: Vec = use_libs.iter().map(|s| s.to_string()).collect(); + let lib_uses = libs + .iter() + .map(|(p, src)| (p.to_string(), parse_pipeline_annotations(src).use_libs)) + .collect(); + let (selected, effective) = resolve_macro_selection(b, registry, &use_libs, &lib_uses)?; + let lib_bodies = libs + .iter() + .map(|(p, src)| (p.to_string(), parse_macro_library(src).unwrap())) + .collect(); + plan_macro_injection(&selected, registry, &effective, &lib_bodies) + } + + #[test] + fn macro_injection_detects_and_orders_transitively() { + // consumer calls `outer`; `outer` calls `inner` — both injected, inner first. + let registry = vec![ + mrow("outer", "a", "inner(a) + 1", false, "f/lib/m"), + mrow("inner", "a", "a * 2", false, "f/lib/m"), + mrow("unused", "a", "a", false, "f/lib/m"), + ]; + let b = blocks(&["ATTACH 'x.duckdb' AS ext;", "SELECT outer(1);"]); + let injected = select_and_plan(&b, ®istry, &[], &[("f/lib/m", "")]).unwrap(); + assert_eq!( + injected, + vec![ + "CREATE OR REPLACE TEMP MACRO inner(a) AS a * 2;".to_string(), + "CREATE OR REPLACE TEMP MACRO outer(a) AS inner(a) + 1;".to_string(), + ] + ); + } + + #[test] + fn implicit_call_injects_provider_setup() { + // A macro whose body references its own lib's ATTACH must carry that + // setup even on the implicit (detection) path — DuckDB bind-checks the + // body at CREATE, so `ext` must be attached first. + let registry = vec![mrow( + "lookup", + "k", + "(SELECT v FROM ext.kv WHERE key = k)", + false, + "f/lib/m", + )]; + let b = blocks(&["SELECT lookup('a');"]); + let injected = select_and_plan( + &b, + ®istry, + &[], + &[( + "f/lib/m", + "ATTACH 'ext.duckdb' AS ext;\nCREATE MACRO lookup(k) AS (SELECT v FROM ext.kv WHERE key = k);", + )], + ) + .unwrap(); + assert_eq!(injected[0], "ATTACH 'ext.duckdb' AS ext;"); + assert!(injected[1].contains("TEMP MACRO lookup(k)")); + } + + #[test] + fn provider_lib_use_is_honored_transitively() { + // Lib B's macro calls `base_macro` inside a string (invisible to + // lexical detection), so B declares `// use f/lib/base`. A consumer + // that merely calls B's macro must still get base's whole library — + // the dynamic dependency is encapsulated in B, not leaked to every + // consumer. + let registry = vec![ + mrow( + "str_macro", + "", + "(SELECT v FROM query('SELECT base_macro() AS v'))", + false, + "f/lib/b", + ), + mrow("base_macro", "", "42", false, "f/lib/base"), + ]; + let b = blocks(&["SELECT str_macro();"]); + let injected = select_and_plan( + &b, + ®istry, + &[], + &[ + ( + "f/lib/b", + "-- macros\n-- use f/lib/base\nCREATE MACRO str_macro() AS (SELECT v FROM query('SELECT base_macro() AS v'));", + ), + ("f/lib/base", "-- macros\nCREATE MACRO base_macro() AS 42;"), + ], + ) + .unwrap(); + assert!( + injected + .iter() + .any(|s| s.contains("TEMP MACRO base_macro()")), + "{injected:?}" + ); + assert!(injected + .iter() + .any(|s| s.contains("TEMP MACRO str_macro()"))); + // Both defs are injected; string-hidden deps carry no topo edge, so + // their relative order falls back to name-sorted ties (deterministic + // for this input — the assertion pins the current behavior). + let base_idx = injected + .iter() + .position(|s| s.contains("TEMP MACRO base_macro()")) + .unwrap(); + let str_idx = injected + .iter() + .position(|s| s.contains("TEMP MACRO str_macro()")) + .unwrap(); + assert!(base_idx < str_idx, "{injected:?}"); + } + + #[test] + fn duplicate_setup_across_libs_is_deduped() { + // Two libs attaching the same catalog identically must not double-ATTACH. + let registry = vec![ + mrow("m1", "a", "a", false, "f/lib/one"), + mrow("m2", "a", "a", false, "f/lib/two"), + ]; + let b = blocks(&["SELECT m1(1), m2(2);"]); + let src = "ATTACH 'ext.duckdb' AS ext;\nCREATE MACRO m1(a) AS a;"; + let src2 = "ATTACH 'ext.duckdb' AS ext;\nCREATE MACRO m2(a) AS a;"; + let injected = select_and_plan( + &b, + ®istry, + &[], + &[("f/lib/one", src), ("f/lib/two", src2)], + ) + .unwrap(); + assert_eq!( + injected + .iter() + .filter(|s| s.starts_with("ATTACH 'ext.duckdb'")) + .count(), + 1 + ); + } + + #[test] + fn macro_injection_empty_when_nothing_called() { + let registry = vec![mrow("m", "a", "a", false, "f/lib/m")]; + let b = blocks(&["SELECT 1;"]); + assert!(select_workspace_macros(&b, ®istry, &[]) + .unwrap() + .is_empty()); + } + + #[test] + fn local_definition_wins_over_registry() { + // A consumer defining its own `dbl` must never get the registry's + // version injected — a library deploy can't change this job. + let registry = vec![mrow("dbl", "a", "a * 10", false, "f/lib/m")]; + let b = blocks(&["CREATE TEMP MACRO dbl(a) AS a * 2;", "SELECT dbl(4);"]); + assert!(select_workspace_macros(&b, ®istry, &[]) + .unwrap() + .is_empty()); + } + + #[test] + fn use_lib_setup_survives_when_no_macros_selected() { + // Every lib macro shadowed locally — the explicit `// use` must still + // carry the lib's setup statements (its ATTACH side effects). + let registry = vec![mrow("dbl", "a", "a * 10", false, "f/lib/m")]; + let b = blocks(&["CREATE TEMP MACRO dbl(a) AS a * 2;", "SELECT dbl(4);"]); + let injected = select_and_plan( + &b, + ®istry, + &["f/lib/m"], + &[( + "f/lib/m", + "ATTACH 'ext.duckdb' AS ext;\nCREATE MACRO dbl(a) AS a * 10;", + )], + ) + .unwrap(); + assert_eq!(injected, vec!["ATTACH 'ext.duckdb' AS ext;".to_string()]); + } + + #[test] + fn weave_lands_after_local_definition_it_references() { + // Injected bodies may call a local macro (local-wins excludes it from + // the registry set), so the leading local CREATE must run first. + let b = blocks(&[ + "ATTACH 'x' AS a;", + "CREATE MACRO local_dbl(a) AS a * 2;", + "SELECT registry_m(1);", + ]); + let out = weave_macro_blocks( + b, + vec!["CREATE OR REPLACE TEMP MACRO registry_m(a) AS local_dbl(a) + 1;".into()], + ) + .unwrap(); + assert_eq!( + out[2], + "CREATE OR REPLACE TEMP MACRO registry_m(a) AS local_dbl(a) + 1;" + ); + } + + #[test] + fn weave_lands_before_local_definition_that_calls_it() { + // The inverse direction: a LOCAL macro whose body calls a registry + // macro bind-checks at its own CREATE, so the injected definition + // (and its library setup) must come first. + let b = blocks(&[ + "CREATE MACRO outer(x) AS shared_inner(x) + 1;", + "SELECT outer(1);", + ]); + let out = weave_macro_blocks( + b, + vec![ + "ATTACH 'ext.duckdb' AS ext;".into(), + "CREATE OR REPLACE TEMP MACRO shared_inner(x) AS x * 2;".into(), + ], + ) + .unwrap(); + assert_eq!( + out, + blocks(&[ + "ATTACH 'ext.duckdb' AS ext;", + "CREATE OR REPLACE TEMP MACRO shared_inner(x) AS x * 2;", + "CREATE MACRO outer(x) AS shared_inner(x) + 1;", + "SELECT outer(1);", + ]) + ); + } + + #[test] + fn weave_pulls_injected_dependencies_before_the_calling_local() { + // outer(local) calls injected `mid`, whose body calls injected `base`: + // both must precede the local definition, base before mid. + let b = blocks(&["CREATE MACRO outer(x) AS mid(x) + 1;", "SELECT outer(1);"]); + let out = weave_macro_blocks( + b, + vec![ + "CREATE OR REPLACE TEMP MACRO base(x) AS x * 2;".into(), + "CREATE OR REPLACE TEMP MACRO mid(x) AS base(x) + 1;".into(), + ], + ) + .unwrap(); + let pos = |needle: &str| out.iter().position(|s| s.contains(needle)).unwrap(); + assert!(pos("MACRO base(x)") < pos("MACRO mid(x)")); + assert!(pos("MACRO mid(x)") < pos("MACRO outer(x)")); + } + + #[test] + fn weave_conflicting_local_order_errors() { + // local_a calls injected X; X references local_b, defined after + // local_a — unsatisfiable, must error rather than silently mis-order. + let b = blocks(&[ + "CREATE MACRO local_a(x) AS conflicted(x);", + "CREATE MACRO local_b(x) AS x * 3;", + "SELECT local_a(1);", + ]); + let err = weave_macro_blocks( + b, + vec!["CREATE OR REPLACE TEMP MACRO conflicted(x) AS local_b(x) + 1;".into()], + ) + .unwrap_err() + .to_string(); + assert!(err.contains("conflicting"), "{err}"); + } + + #[test] + fn use_lib_injects_all_macros_and_setup() { + let registry = vec![ + mrow("m1", "a", "a", false, "f/lib/m"), + mrow("m2", "", "SELECT 1", true, "f/lib/m"), + ]; + let b = blocks(&["SELECT 'no calls here';"]); + let injected = select_and_plan( + &b, + ®istry, + &["f/lib/m"], + &[( + "f/lib/m", + "ATTACH 'ext.duckdb' AS ext;\nCREATE MACRO m1(a) AS a;\nCREATE MACRO m2() AS TABLE SELECT 1;", + )], + ) + .unwrap(); + assert_eq!(injected[0], "ATTACH 'ext.duckdb' AS ext;"); + assert!(injected[1..].iter().any(|s| s.contains("TEMP MACRO m1(a)"))); + assert!(injected[1..] + .iter() + .any(|s| s.contains("TEMP MACRO m2() AS TABLE"))); + } + + #[test] + fn use_lib_without_registry_rows_errors() { + let b = blocks(&["SELECT 1;"]); + let err = select_workspace_macros(&b, &[], &["f/lib/gone".to_string()]) + .unwrap_err() + .to_string(); + assert!(err.contains("no deployed macro library"), "{err}"); + } + + #[test] + fn weave_lands_after_setup_prefix() { + let b = blocks(&[ + "INSTALL ducklake;", + "ATTACH 'ducklake:postgres:...' AS lake (DATA_PATH 's3://x');", + "CREATE TABLE IF NOT EXISTS lake.t AS SELECT 1;", + "SELECT dbl(1);", + ]); + let out = weave_macro_blocks( + b, + vec!["CREATE OR REPLACE TEMP MACRO dbl(a) AS a * 2;".into()], + ) + .unwrap(); + assert_eq!(out[2], "CREATE OR REPLACE TEMP MACRO dbl(a) AS a * 2;"); + assert_eq!(out.len(), 5); + } + + #[test] + fn weave_setup_runs_before_all_user_blocks() { + // Injected library setup is self-contained and anything in the + // consumer may depend on it at bind time — it always goes first. + let out = weave_macro_blocks(blocks(&["ATTACH 'x' AS a;"]), blocks(&["m;"])).unwrap(); + assert_eq!(out, blocks(&["m;", "ATTACH 'x' AS a;"])); + let out = weave_macro_blocks(vec![], blocks(&["m;"])).unwrap(); + assert_eq!(out, blocks(&["m;"])); + } + + #[test] + fn weave_use_setup_precedes_local_shadow_that_references_it() { + // `// use` with every lib macro shadowed locally: no defs are + // injected, but the local shadow's body references the lib's + // ATTACHed catalog — the injected setup must still run first or the + // local CREATE bind-fails. + let b = blocks(&[ + "CREATE TEMP MACRO kv_lookup(k) AS (SELECT v FROM ext.kv WHERE key = k);", + "SELECT kv_lookup('a');", + ]); + let out = weave_macro_blocks(b, vec!["ATTACH 'ext.duckdb' AS ext;".into()]).unwrap(); + assert_eq!( + out, + blocks(&[ + "ATTACH 'ext.duckdb' AS ext;", + "CREATE TEMP MACRO kv_lookup(k) AS (SELECT v FROM ext.kv WHERE key = k);", + "SELECT kv_lookup('a');", + ]) + ); + } + #[test] fn cgroup_bytes_unlimited_or_invalid_returns_none() { assert_eq!(cgroup_bytes_to_duckdb_memory_limit(0), None); @@ -879,6 +3275,319 @@ mod tests { ); } + // Managed `// materialize` may take SQL args (e.g. an s3object uploaded on + // the run form). The wrap strips line comments — including the + // `-- $name (type)` declarations — so the executor parses the signature from + // the original script (done above, before the rewrite) while the `$name` + // references survive inside the wrapped SELECT. This pins both halves of that + // contract so a regression that drops either is caught. + #[test] + fn materialize_preserves_sql_args() { + let script = "-- materialize ducklake://main/rows\n\ + -- $file (s3object)\n\ + SELECT * FROM read_json_auto($file)"; + + // The signature is recoverable from the original (un-wrapped) script. + let sig = parse_duckdb_sig(script).expect("sig parses").args; + let file_arg = sig + .iter() + .find(|a| a.name == "file") + .expect("`$file` declared"); + assert_eq!(file_arg.otyp.as_deref(), Some("s3object")); + + // The wrapped query still references `$file`, so the parsed sig binds it. + // No custom data tests here, so no fetched bodies are needed. + let (rewritten, _) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("$file"), + "wrapped query must keep the `$file` reference, got:\n{rewritten}" + ); + // The declaration comment is gone (wrap strips line comments) — which is + // exactly why the sig must come from the original, not the rewrite. + assert!(!rewritten.contains("-- $file")); + } + + // A `// partitioned` script referencing `$partition` needs no manual + // `-- $partition (text)` declaration: the parser auto-declares the arg, so + // the executor binds the injected `partition` job arg instead of failing + // with duckdb's "Wrong number of parameters" at prepare time. + #[test] + fn partitioned_auto_declares_partition_arg() { + let script = "// partitioned daily\n\ + // materialize ducklake://main/sales_daily\n\ + SELECT $partition AS day, count(*) AS n FROM dl.sales WHERE day = $partition"; + + let sig = parse_duckdb_sig(script).expect("sig parses").args; + let partition_arg = sig + .iter() + .find(|a| a.name == "partition") + .expect("`partition` auto-declared"); + assert_eq!(partition_arg.otyp.as_deref(), Some("text")); + + // The wrapped query keeps the `$partition` references so the parsed sig + // binds them at run time. + let (rewritten, _) = build_materialized_query( + script, + Some("2026-07-02"), + &std::collections::HashMap::new(), + ) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("$partition"), + "wrapped query must keep the `$partition` reference, got:\n{rewritten}" + ); + } + + // SCD2 managed mode wraps the SELECT into the diff → close-old → open-new + // shape (unit-covered in the parser's codegen tests); here we pin the + // executor-level wiring: the natural key flows through and the wrap is + // generated (not the manual track-only path). + #[test] + fn materialize_scd2_wraps_with_history() { + // Primary spelling: `key=history` on a merge. + let script = "-- materialize ducklake://main/dim key=id history track=name\n\ + SELECT id, name FROM dl.src"; + let (rewritten, _) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("scd2 is managed — must rewrite"); + assert!( + rewritten.contains("valid_from"), + "adds SCD2 columns:\n{rewritten}" + ); + assert!(rewritten.contains("is_current")); + assert!( + rewritten.contains("_wm_scd2_changed"), + "captures changed keys" + ); + assert!( + rewritten.contains("UPDATE _wm_target.dim SET valid_to"), + "closes prior version" + ); + assert!( + rewritten.contains("CREATE VIEW IF NOT EXISTS _wm_target.dim_current"), + "emits the consumer-convenience current view" + ); + // default is soft-delete — no deleted-key set without `deletes=close` + assert!(!rewritten.contains("_wm_scd2_deleted")); + assert!(!rewritten.contains("MERGE INTO")); + } + + #[test] + fn materialize_scd2_deletes_close_wires_through() { + let script = "-- materialize ducklake://main/dim key=id history deletes=close\n\ + SELECT id, name FROM dl.src"; + let (rewritten, _) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("scd2 is managed — must rewrite"); + assert!( + rewritten.contains("_wm_scd2_deleted"), + "deletes=close adds the vanished-key set + close:\n{rewritten}" + ); + } + + #[test] + fn materialize_scd2_requires_key() { + let script = "-- materialize scd2 ducklake://main/dim\nSELECT id, name FROM dl.src"; + let err = match build_materialized_query(script, None, &std::collections::HashMap::new()) { + Err(e) => e, + Ok(_) => panic!("scd2 without key must error"), + }; + assert!( + format!("{err}").contains("requires a natural key"), + "got: {err}" + ); + } + + #[test] + fn materialize_scd2_rejects_partitioned() { + let script = "-- pipeline\n-- partitioned daily\n\ + -- materialize scd2 ducklake://main/dim key=id\nSELECT id, name FROM dl.src"; + let err = match build_materialized_query( + script, + Some("2026-07-01"), + &std::collections::HashMap::new(), + ) { + Err(e) => e, + Ok(_) => panic!("partitioned + scd2 must error"), + }; + assert!( + format!("{err}").contains("not supported with scd2"), + "got: {err}" + ); + } + + // The rewritten SQL is the plan assembled by + // `pipeline_advanced::finalize_materialize_query`, so what it contains is + // build-dependent: the public assembly runs the plan verbatim (tests only + // in the post-commit summary breakdown), the enterprise assembly adds the + // in-transaction write-audit-publish guard (whose shape/placement is + // tested next to its implementation in windmill-common's + // `pipeline_advanced_ee`). + #[test] + fn materialize_rewrite_carries_data_test_summary() { + let script = "-- materialize ducklake://main/orders\n\ + -- data_test not_null id\n\ + SELECT id FROM dl.src"; + let (rewritten, meta) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!(rewritten.contains("AS data_tests")); + assert_eq!(meta.n_data_tests, 1); + #[cfg(not(feature = "private"))] + assert!( + !rewritten.contains("error("), + "public assembly is commit-then-test (no guard)" + ); + #[cfg(all(feature = "private", feature = "enterprise"))] + assert!( + rewritten.contains("error("), + "enterprise assembly places the WAP guard" + ); + } + + // on_schema_change=fail on a persist-and-mutate strategy (merge) emits the + // drift guard inside the write, and no sync artifacts. + #[test] + fn materialize_fail_emits_drift_guard() { + let script = "-- materialize ducklake://main/dim key=id on_schema_change=fail\n\ + SELECT id, name FROM dl.src"; + let (rewritten, meta) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("CAST(error(") && rewritten.contains("on_schema_change=fail"), + "fail emits the drift guard: {rewritten}" + ); + assert!(!rewritten.contains("BY NAME"), "fail writes positionally"); + assert!(meta.sync_prepass.is_none(), "fail needs no pre-pass probe"); + } + + // on_schema_change=sync writes BY NAME, emits the ALTER sentinel, and builds + // the pre-pass probe (DESCRIBE of the SELECT + information_schema read). + #[test] + fn materialize_sync_by_name_and_prepass() { + let script = "-- materialize ducklake://main/dim key=id on_schema_change=sync\n\ + SELECT id, name FROM dl.src"; + let (rewritten, meta) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("INSERT INTO _wm_target.dim BY NAME"), + "sync insert is name-mapped: {rewritten}" + ); + assert!( + rewritten.contains(windmill_parser::sql_materialize::SYNC_ALTER_SENTINEL), + "sync emits the ALTER injection sentinel" + ); + let pp = meta.sync_prepass.expect("sync builds a pre-pass probe"); + assert_eq!(pp.target_table, "dim"); + assert!(pp + .probe_query + .contains("DESCRIBE SELECT * FROM (SELECT id, name FROM dl.src)")); + assert!(pp.probe_query.contains("FROM information_schema.columns")); + assert!(pp + .probe_query + .contains("table_catalog = '_wm_target' AND table_name = 'dim'")); + assert!(pp + .probe_query + .contains("ATTACH 'ducklake://main' AS _wm_target;")); + } + + // on_schema_change=warn (default) on a persist-and-mutate strategy folds the + // drift into the summary; no guard, no sync artifacts. + #[test] + fn materialize_warn_summary_carries_schema_drift() { + let script = "-- materialize ducklake://main/dim key=id\n\ + SELECT id, name FROM dl.src"; + let (rewritten, meta) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("AS schema_drift"), + "warn folds drift into the summary: {rewritten}" + ); + // No schema-drift *fail* guard in warn mode (the keyed merge still emits + // its own duplicate-source-key guard, which is a different `error(...)`). + assert!(!rewritten.contains("on_schema_change=fail blocked")); + assert!(rewritten.contains("keyed merge on `id` blocked")); + assert!(!rewritten.contains("BY NAME")); + assert!(meta.sync_prepass.is_none()); + } + + // Whole-table replace (unpartitioned, no key/append) is not persist-and- + // mutate: even with on_schema_change=fail there is no guard / sentinel — the + // CREATE OR REPLACE already rebuilds the schema each run. + #[test] + fn materialize_whole_table_replace_ignores_guardrail() { + let script = "-- materialize ducklake://main/t on_schema_change=fail\n\ + SELECT a, b FROM dl.src"; + let (rewritten, meta) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!(rewritten.contains("CREATE OR REPLACE TABLE _wm_target.t")); + assert!(!rewritten.contains("on_schema_change=fail")); + assert!(!rewritten.contains(windmill_parser::sql_materialize::SYNC_ALTER_SENTINEL)); + assert!(meta.sync_prepass.is_none()); + } + + #[test] + fn extract_schema_drift_parses_struct_and_string_and_null() { + // nested struct form + let r = raw( + r#"{"materialized":"ducklake://main/dim","schema_drift":{"added":["c"],"removed":["b"]}}"#, + ); + let (added, removed) = extract_schema_drift(&r).expect("drift present"); + assert_eq!(added, vec!["c".to_string()]); + assert_eq!(removed, vec!["b".to_string()]); + // FFI string-encoded form + let r = raw(r#"{"schema_drift":"{\"added\":[\"x\"],\"removed\":[]}"}"#); + let (added, removed) = extract_schema_drift(&r).expect("drift present"); + assert_eq!(added, vec!["x".to_string()]); + assert!(removed.is_empty()); + // NULL / both-empty ⇒ no drift + assert!(extract_schema_drift(&raw(r#"{"schema_drift":null}"#)).is_none()); + assert!( + extract_schema_drift(&raw(r#"{"schema_drift":{"added":[],"removed":[]}}"#)).is_none() + ); + assert!(extract_schema_drift(&raw(r#"{"rows":3}"#)).is_none()); + } + + #[test] + fn parse_sync_drift_computes_added_removed_and_skips_fresh_table() { + // table (a, b) vs SELECT (a, c): add c, drop b; _wm_partition ignored + let r = raw(r#"[{"_wm_which":"sel","_wm_name":"a","_wm_type":"BIGINT"}, + {"_wm_which":"sel","_wm_name":"c","_wm_type":"VARCHAR"}, + {"_wm_which":"tbl","_wm_name":"a","_wm_type":"BIGINT"}, + {"_wm_which":"tbl","_wm_name":"b","_wm_type":"BIGINT"}]"#); + let (added, removed) = parse_sync_drift(&r).unwrap(); + assert_eq!(added, vec![("c".to_string(), "VARCHAR".to_string())]); + assert_eq!(removed, vec!["b".to_string()]); + // no `tbl` rows ⇒ table doesn't exist yet ⇒ no migration + let r = raw(r#"[{"_wm_which":"sel","_wm_name":"a","_wm_type":"BIGINT"}]"#); + let (added, removed) = parse_sync_drift(&r).unwrap(); + assert!(added.is_empty() && removed.is_empty()); + } + // Tests for parse_attach_db_resource function #[test] fn test_parse_attach_db_resource_postgres_res_prefix() { @@ -1237,4 +3946,112 @@ mod tests { let serialized = serde_json::to_string(&arg).unwrap(); assert!(serialized.contains("\"json_value\":{\"key\":\"value\"}")); } + + fn raw(s: &str) -> Box { + serde_json::from_str(s).unwrap() + } + + #[test] + fn extract_data_tests_parses_nested_array() { + // The real result shape: an array of one summary row carrying a nested + // `data_tests` array (how the FFI serialises the list-of-struct). + let r = raw( + r#"[{"rows":3,"snapshot_id":17,"materialized":"ducklake://a/b", + "data_tests":[{"test":"unique(order_id)","violating":0,"sample":null}, + {"test":"accepted_values(status)","violating":2, + "sample":"[{\"id\":1,\"status\":\"bad\"}]"}]}]"#, + ); + let out = extract_data_tests(&r); + assert_eq!(out.len(), 2); + assert_eq!(out[0].name, "unique(order_id)"); + assert_eq!(out[0].violating, 0); + assert!(out[0].sample.is_none()); + assert_eq!(out[1].name, "accepted_values(status)"); + assert_eq!(out[1].violating, 2); + // The probe emits the sample as a JSON string; it parses to rows here. + assert_eq!( + out[1].sample, + Some(serde_json::json!([{"id": 1, "status": "bad"}])) + ); + } + + #[test] + fn extract_data_tests_handles_string_encoded_and_absent() { + // Fallback: some serialisations surface the list-of-struct as a JSON string. + let s = raw(r#"{"data_tests":"[{\"test\":\"not_null(x)\",\"violating\":1}]"}"#); + let out = extract_data_tests(&s); + assert_eq!(out.len(), 1); + assert_eq!(out[0].name, "not_null(x)"); + assert_eq!(out[0].violating, 1); + assert!(out[0].sample.is_none()); + // Absent column (no tests) -> empty, no panic. + assert!(extract_data_tests(&raw(r#"[{"rows":3}]"#)).is_empty()); + } + + #[test] + fn extract_data_tests_sample_degrades_to_none_never_flips_outcome() { + // sample is optional by contract: native array accepted; non-array + // JSON, garbage text, and absence all degrade to None without + // touching `violating`. + let r = raw(r#"[{"data_tests":[ + {"test":"a","violating":1,"sample":[{"id":9}]}, + {"test":"b","violating":2,"sample":"{\"not\":\"an array\"}"}, + {"test":"c","violating":3,"sample":"not json at all"}, + {"test":"d","violating":4}]}]"#); + let out = extract_data_tests(&r); + assert_eq!(out.len(), 4); + assert_eq!(out[0].sample, Some(serde_json::json!([{"id": 9}]))); + for (i, t) in out.iter().enumerate().skip(1) { + assert!(t.sample.is_none(), "test {} should have no sample", t.name); + assert_eq!(t.violating, i as i64 + 1); + } + } + + #[test] + fn extract_schema_parses_nested_and_string_encoded() { + // Real shape: the summary row carries a nested `output_schema` + // list-of-struct from the DESCRIBE fold. + let r = raw( + r#"[{"materialized":"ducklake://a/b","rows":3,"snapshot_id":17, + "output_schema":[{"name":"order_id","type":"BIGINT"}, + {"name":"status","type":"VARCHAR"}]}]"#, + ); + let cols = extract_schema(&r).expect("schema present"); + assert_eq!(cols.len(), 2); + assert_eq!(cols[0].name, "order_id"); + assert_eq!(cols[0].data_type, "BIGINT"); + assert_eq!(cols[1].name, "status"); + assert_eq!(cols[1].data_type, "VARCHAR"); + // Fallback: FFI serialised the list-of-struct as a JSON string. + let s = raw(r#"{"output_schema":"[{\"name\":\"x\",\"type\":\"INTEGER\"}]"}"#); + let cols = extract_schema(&s).expect("schema present"); + assert_eq!(cols.len(), 1); + assert_eq!(cols[0].name, "x"); + assert_eq!(cols[0].data_type, "INTEGER"); + // Absent column (literal/manual mode) -> None, no panic. + assert!(extract_schema(&raw(r#"[{"rows":3}]"#)).is_none()); + } + + #[test] + fn format_data_test_breakdown_lists_all_with_marks() { + let tests = vec![ + DataTestOutcome { name: "unique(order_id)".into(), violating: 1, sample: None }, + DataTestOutcome { name: "not_null(user_id)".into(), violating: 0, sample: None }, + DataTestOutcome { + name: "accepted_values(status)".into(), + violating: 2, + // The breakdown is counts-only by design — samples never + // appear in the error text. + sample: Some(serde_json::json!([{"status": "bad"}])), + }, + ]; + let msg = format_data_test_breakdown("analytics/orders", &tests); + assert_eq!( + msg, + "data tests failed on analytics/orders (2/3 failed):\n \ + ✗ unique(order_id) — 1 violating row(s)\n \ + ✓ not_null(user_id)\n \ + ✗ accepted_values(status) — 2 violating row(s)" + ); + } } diff --git a/backend/windmill-worker/src/pg_executor.rs b/backend/windmill-worker/src/pg_executor.rs index a1a26d0630..98c1b5bba7 100644 --- a/backend/windmill-worker/src/pg_executor.rs +++ b/backend/windmill-worker/src/pg_executor.rs @@ -615,6 +615,25 @@ pub async fn do_postgresql( return Err(Error::BadRequest("Missing database argument".to_string())); }; + // Surface in the job logs (not just the worker logs) when a verify-ca/verify-full + // resource is connecting without actually verifying the server certificate, so the + // person running the query can see and fix the misconfiguration. + if database.verify_mode_skips_verification() { + windmill_queue::append_logs( + &job.id, + &job.workspace_id, + format!( + "warning: sslmode={} but the server's TLS certificate is not being verified \ + (accept_invalid_certs is enabled, or no root certificate is configured). Set \ + accept_invalid_certs to false or provide root_certificate_pem to verify the \ + server identity.\n", + database.sslmode.as_deref().unwrap_or("") + ), + conn, + ) + .await; + } + let annotations = windmill_common::worker::SqlAnnotations::parse(query); let collection_strategy = if annotations.raw_output || annotations.return_last_result { // raw_output emits a single envelope from the last statement, so the @@ -629,10 +648,26 @@ pub async fn do_postgresql( // Include use_iam_auth in cache key to distinguish IAM vs non-IAM connections to the same host. // The cache key is static (doesn't include the token), which is correct because PostgreSQL // connections remain valid after initial auth — fresh tokens are generated on cache miss. + // + // to_uri() collapses require/verify-ca/verify-full to the same string, so the TLS verification + // inputs are folded into the key separately. Without this a connection established under a + // weaker sslmode (or a different root cert) could be reused for a stricter request, undoing + // the verification configured in PgDatabase::configure_pg_tls_verification. + let tls_disc = { + use std::hash::{Hash, Hasher}; + let mut h = std::collections::hash_map::DefaultHasher::new(); + database.sslmode.hash(&mut h); + database.root_certificate_pem.hash(&mut h); + database.accept_invalid_certs.hash(&mut h); + h.finish() + }; + // to_uri() already ends with `?sslmode=...`, so append further key segments + // with `&` to keep database_string a well-formed URI (it is only ever a cache + // key, but a malformed one would mislead anyone who later logs or parses it). let database_string = if use_iam_auth { - format!("{}?iam=true", database.to_uri()) + format!("{}&iam=true&tls={tls_disc:x}", database.to_uri()) } else { - database.to_uri() + format!("{}&tls={tls_disc:x}", database.to_uri()) }; let database_string_clone = database_string.clone(); diff --git a/backend/windmill-worker/src/python_executor.rs b/backend/windmill-worker/src/python_executor.rs index 409dc362af..9ca79c6852 100644 --- a/backend/windmill-worker/src/python_executor.rs +++ b/backend/windmill-worker/src/python_executor.rs @@ -37,8 +37,8 @@ use windmill_common::{ scripts::ScriptLang, utils::calculate_hash, worker::{ - copy_dir_recursively, pad_string, split_python_requirements, write_file, Connection, - PyVAlias, PythonAnnotations, WORKER_CONFIG, + copy_dir_recursively, is_allowed_file_location, pad_string, split_python_requirements, + write_file, Connection, PyVAlias, PythonAnnotations, WORKER_CONFIG, }, }; @@ -62,6 +62,13 @@ lazy_static::lazy_static! { static ref PY_CONCURRENT_DOWNLOADS: usize = var("PY_CONCURRENT_DOWNLOADS").ok().map(|flag| flag.parse().unwrap_or(20)).unwrap_or(20); + // uv's HTTP request timeout (seconds). spawn_uv_install uses env_clear(), so a + // UV_HTTP_TIMEOUT set on the worker is dropped unless forwarded explicitly. + // Only forwarded when set; otherwise uv keeps its own default. Lets operators + // raise it for slow/contended private registries ("operation timed out"). + static ref UV_HTTP_TIMEOUT: Option = + var("UV_HTTP_TIMEOUT").ok().filter(|v| !v.is_empty()); + static ref NON_ALPHANUM_CHAR: Regex = regex::Regex::new(r"[^0-9A-Za-z=.-]").unwrap(); @@ -664,10 +671,16 @@ pub fn compute_python_module_dir(script_path: &str) -> String { .replace("-", "_") .replace("@", "."); if dirs_full.len() > 0 { - dirs_full - .strip_prefix("/") - .unwrap_or(&dirs_full) - .to_string() + let dirs = dirs_full.strip_prefix("/").unwrap_or(&dirs_full); + // This directory is appended to job_dir and written to. Neutralize any + // `.`/`..` segment so the result stays a relative path inside job_dir: a + // Preview path is request-supplied and skips the DB `proper_id` CHECK that + // deployed runnables get, and the `@`->`.` rewrite above can also turn a + // segment like `@.` into `..`. + dirs.split('/') + .map(|seg| if seg == "." || seg == ".." { "_" } else { seg }) + .collect::>() + .join("/") } else { "tmp".to_string() } @@ -1668,6 +1681,10 @@ async fn prepare_wrapper( last }; let module_dir = format!("{}/{}", job_dir, dirs); + // Defense-in-depth: `dirs`/`last` derive from the (request-supplied for + // previews) script path. compute_python_module_dir already neutralizes `..`, + // but assert containment here too so the write can never escape job_dir. + is_allowed_file_location(job_dir, &format!("{dirs}/{last}.py"))?; tokio::fs::create_dir_all(format!("{module_dir}/")).await?; let _ = write_file(&module_dir, &format!("{last}.py"), inner_content)?; @@ -2040,6 +2057,33 @@ Returned from server: py_version - {:?}, py_version_v2 - {:?} lazy_static::lazy_static! { static ref PIP_SECRET_VARIABLE: Regex = Regex::new(r"\$\{PIP_SECRET:([^\s\}]+)\}").unwrap(); + + /// venv paths whose wheel RECORD this process has already verified against + /// disk. A cache entry is only damaged out-of-band (disk-pressure eviction, + /// an interrupted extraction on a shared cache volume, or a corrupt entry + /// that predates this worker), never spontaneously while we keep running, so + /// re-verifying it once per process is enough — every later reuse trusts the + /// in-memory marker and pays only the original single stat. + static ref VERIFIED_VENVS: tokio::sync::Mutex> = + tokio::sync::Mutex::new(HashSet::new()); + + // In-process locks serializing concurrent installs into the same shared + // `venv_p` cache dir; `uv --reinstall` removes a package's .dist-info/RECORD + // before rewriting it, so a sibling install/verify racing it corrupts the + // dir. Keyed by venv_p so distinct deps still install in parallel. + static ref PY_INSTALL_LOCKS: tokio::sync::Mutex>>> = + tokio::sync::Mutex::new(std::collections::HashMap::new()); +} + +/// Returns the in-process install lock for a given target cache dir, creating it +/// on first use. Idle entries (only the map holds a reference) are pruned each +/// call so the map stays bounded by the number of in-flight installs. +async fn get_venv_install_lock(venv_p: &str) -> Arc> { + let mut map = PY_INSTALL_LOCKS.lock().await; + map.retain(|_, v| Arc::strong_count(v) > 1); + map.entry(venv_p.to_string()) + .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(()))) + .clone() } /// Spawn process of uv install @@ -2085,6 +2129,9 @@ async fn spawn_uv_install( if *NATIVE_CERT { vars.push(("UV_NATIVE_TLS", "true")); } + if let Some(timeout) = UV_HTTP_TIMEOUT.as_ref() { + vars.push(("UV_HTTP_TIMEOUT", timeout.as_str())); + } let _owner; if let Some(py_path) = py_path.as_ref() { @@ -2188,6 +2235,9 @@ async fn spawn_uv_install( let mut envs = vec![("PATH", PATH_ENV.as_str())]; envs.push(("HOME", HOME_ENV.as_str())); envs.push(("UV_INDEX_STRATEGY", uv_index_strategy)); + if let Some(timeout) = UV_HTTP_TIMEOUT.as_ref() { + envs.push(("UV_HTTP_TIMEOUT", timeout.as_str())); + } if let Some(mirror) = uv_python_install_mirror.as_ref() { envs.push(("UV_PYTHON_INSTALL_MIRROR", mirror)); } @@ -2479,8 +2529,53 @@ pub async fn handle_python_reqs( req.replace(' ', "").replace('/', "").replace(':', "") ); if metadata(venv_p.clone() + "/.valid.windmill").await.is_ok() { - req_paths.push(venv_p); - in_cache.push(req.to_string()); + // The .valid.windmill marker is written once at creation time, after + // verify_wheel_record passes on the install/pull paths. It is an empty + // file with no binding to the directory contents, so a file dropped + // out-of-band afterwards (disk-pressure eviction, interrupted tar + // extraction on a shared cache volume, or a corrupt entry that + // predates this worker) leaves the marker intact while the wheel is + // incomplete. Re-verify the RECORD once per process so such an entry + // is repaired rather than trusted; VERIFIED_VENVS makes every later + // reuse skip the scan and pay only the single stat above. + let already_verified = VERIFIED_VENVS.lock().await.contains(&venv_p); + let verify_res = if already_verified { + Ok(()) + } else { + verify_wheel_record(&venv_p).await + }; + match verify_res { + Ok(()) => { + if !already_verified { + VERIFIED_VENVS.lock().await.insert(venv_p.clone()); + } + req_paths.push(venv_p); + in_cache.push(req.to_string()); + } + Err(verify_err) => { + tracing::warn!( + workspace_id = %w_id, + job_id = %job_id, + "Local cache for {venv_p} failed wheel RECORD verification, will reinstall: {verify_err}" + ); + append_logs( + &job_id, + w_id, + format!( + "\n[!] cached wheel for {req} failed integrity check, reinstalling: {verify_err}\n" + ), + conn, + ) + .await; + if let Err(rm_err) = tokio::fs::remove_dir_all(&venv_p).await { + tracing::warn!( + workspace_id = %w_id, + "could not remove broken cache dir {venv_p}: {rm_err}" + ); + } + req_with_penv.push((req.to_string(), venv_p)); + } + } } else { // There is no valid or no wheel at all. Regardless of if there is content or not, we will overwrite it with --reinstall flag req_with_penv.push((req.to_string(), venv_p)); @@ -2716,6 +2811,96 @@ pub async fn handle_python_reqs( ); let start = std::time::Instant::now(); + + // Lock the shared target dir (see PY_INSTALL_LOCKS). In-process lock + // first; only one task per dir then contends the cross-process file + // lock below. Both guards drop on every return path. + let venv_lock = get_venv_install_lock(&venv_p).await; + let _venv_guard = tokio::select! { + _ = kill_rx.recv() => { + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Err(Error::from(anyhow::anyhow!( + "install of {venv_p} canceled while waiting for venv lock" + ))); + } + guard = venv_lock.lock_owned() => guard, + }; + + // Cross-process advisory lock. Best-effort: if the filesystem doesn't + // support flock we log and proceed — verify_wheel_record + job retry + // still guard correctness, just without the dedup. + #[cfg(unix)] + let _venv_file_lock: Option = { + use std::os::unix::io::AsRawFd; + let lock_path = format!("{venv_p}.lock"); + if let Some(parent) = std::path::Path::new(&lock_path).parent() { + let _ = std::fs::create_dir_all(parent); + } + match std::fs::OpenOptions::new().create(true).write(true).open(&lock_path) { + Ok(f) => { + // Bounded wait: a holder that crashes releases the lock (the + // kernel drops it on fd close), but a live-but-stuck holder + // (e.g. uv wedged on a hung mount) would otherwise block us + // forever. After the cap, proceed degraded rather than hang — + // verify_wheel_record + retry still guard correctness. + const MAX_WAIT: std::time::Duration = std::time::Duration::from_secs(300); + let waited_since = std::time::Instant::now(); + loop { + match nix::fcntl::flock(f.as_raw_fd(), nix::fcntl::FlockArg::LockExclusiveNonblock) { + Ok(()) => break Some(f), + // EWOULDBLOCK == EAGAIN on Linux: another holder has the lock. + Err(nix::errno::Errno::EWOULDBLOCK) => { + if waited_since.elapsed() >= MAX_WAIT { + tracing::warn!( + workspace_id = %w_id, + "venv install lock {lock_path} still held after {}s, proceeding without cross-process install lock", + MAX_WAIT.as_secs() + ); + break Some(f); + } + tokio::select! { + _ = kill_rx.recv() => { + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Err(Error::from(anyhow::anyhow!( + "install of {venv_p} canceled while waiting for venv file lock" + ))); + } + _ = tokio::time::sleep(std::time::Duration::from_millis(200)) => {} + } + } + Err(e) => { + tracing::warn!( + workspace_id = %w_id, + "could not flock {lock_path}, proceeding without cross-process install lock: {e}" + ); + break Some(f); + } + } + } + } + Err(e) => { + tracing::warn!( + workspace_id = %w_id, + "could not open install lock file {lock_path}, proceeding without cross-process install lock: {e}" + ); + None + } + } + }; + + // Double-checked: another job (this process or another sharing the + // mount) may have installed this exact dep while we waited on the + // locks. Reuse it instead of reinstalling. + if metadata(format!("{venv_p}/.valid.windmill")).await.is_ok() { + print_success( + false, false, &job_id, &w_id, &req, req_tl, counter_arc, + total_to_install, start, &conn, + ) + .await; + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Ok(()); + } + #[cfg(all(feature = "enterprise", feature = "parquet"))] if is_not_pro { if let Some(os) = windmill_object_store::get_object_store().await { @@ -3357,6 +3542,17 @@ mod tests { assert_eq!(compute_python_module_dir("f/in/script"), "f/_in"); } + #[test] + fn test_compute_python_module_dir_neutralizes_traversal() { + // A Preview path skips the DB `proper_id` CHECK, so it can carry `..`. + // `..`/`.` segments must be neutralized so the dir stays inside job_dir. + let dirs = compute_python_module_dir("u/x/../../../../tmp/evil/payload"); + assert!(!dirs.split('/').any(|s| s == ".." || s == ".")); + assert_eq!(dirs, "u/x/_/_/_/_/tmp/evil"); + // The `@`->`.` rewrite must not be able to synthesize a `..` segment. + assert_eq!(compute_python_module_dir("u/@./script"), "u/_"); + } + #[test] fn test_compute_py_codegen_basic_args() { let code = "def main(x: str, y: int):\n return x\n"; @@ -3446,4 +3642,179 @@ mod tests { assert_eq!(kept, lines(&["# py: 3.11", "requests==2.0"])); assert_eq!(ignored, lines(&["pyyaml==6.0"])); } + + /// Materialize a fake installed wheel: every file in `files` is created, and + /// `record_entries` is written verbatim as the RECORD (so a test can list a + /// path in RECORD without creating it, to simulate out-of-band loss). + fn write_fake_wheel(root: &std::path::Path, files: &[&str], record_entries: &[&str]) { + for f in files { + let full = root.join(f); + std::fs::create_dir_all(full.parent().unwrap()).unwrap(); + std::fs::write(full, b"x").unwrap(); + } + let dist_info = root.join("pkg-1.0.0.dist-info"); + std::fs::create_dir_all(&dist_info).unwrap(); + std::fs::write(dist_info.join("RECORD"), record_entries.join("\n") + "\n").unwrap(); + } + + #[tokio::test] + async fn test_verify_wheel_record_ok_when_all_present() { + let dir = tempfile::tempdir().unwrap(); + write_fake_wheel( + dir.path(), + &["pkg/__init__.py", "pkg/mod.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "pkg/mod.py,sha256=bbb,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_ok()); + } + + #[tokio::test] + async fn test_verify_wheel_record_err_when_file_missing() { + let dir = tempfile::tempdir().unwrap(); + // RECORD lists pkg/mod.py but we never create it: the exact failure mode + // the customer hit (wmill/s3_reader.py present in RECORD, gone on disk). + write_fake_wheel( + dir.path(), + &["pkg/__init__.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "pkg/mod.py,sha256=bbb,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + let err = verify_wheel_record(dir.path().to_str().unwrap()) + .await + .unwrap_err(); + assert!(err.contains("pkg/mod.py"), "unexpected error: {err}"); + } + + #[tokio::test] + async fn test_verify_wheel_record_err_when_no_dist_info() { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join("loose.py"), b"x").unwrap(); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_err()); + } + + #[tokio::test] + async fn test_verify_wheel_record_skips_absolute_and_escaping_entries() { + let dir = tempfile::tempdir().unwrap(); + // Absolute and `..` RECORD entries are not package-relative and must be + // skipped rather than reported missing. + write_fake_wheel( + dir.path(), + &["pkg/__init__.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "/etc/passwd,sha256=ccc,1", + "../outside.py,sha256=ddd,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_ok()); + } + + // Regression tests for the concurrent-install guard. Two jobs installing the + // same uncached dep into the shared `venv_p` used to race uv's `--reinstall`, + // corrupting the on-disk wheel and failing with "Env installation did not + // succeed". The guard serializes those installs. + + #[tokio::test] + async fn test_venv_install_lock_serializes_same_path() { + use std::sync::atomic::{AtomicUsize, Ordering}; + // Same target path => one shared lock => no two tasks install at once. + let active = Arc::new(AtomicUsize::new(0)); + let max_seen = Arc::new(AtomicUsize::new(0)); + let mut handles = vec![]; + for _ in 0..8 { + let active = active.clone(); + let max_seen = max_seen.clone(); + handles.push(tokio::spawn(async move { + let lock = get_venv_install_lock("/cache/py/3.11/samedep==1.0").await; + let _g = lock.lock_owned().await; + let cur = active.fetch_add(1, Ordering::SeqCst) + 1; + max_seen.fetch_max(cur, Ordering::SeqCst); + // Yield so any concurrency would be observed by another task. + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + active.fetch_sub(1, Ordering::SeqCst); + })); + } + for h in handles { + h.await.unwrap(); + } + assert_eq!( + max_seen.load(Ordering::SeqCst), + 1, + "installs into the same target dir must be serialized" + ); + } + + #[tokio::test] + async fn test_venv_install_lock_distinct_paths_are_independent() { + // Different target paths get different locks and never block each other. + let a = get_venv_install_lock("/cache/py/3.11/depA==1.0").await; + let b = get_venv_install_lock("/cache/py/3.11/depB==1.0").await; + let _ga = a.lock_owned().await; + // Holding depA's lock must not prevent acquiring depB's. + assert!( + b.try_lock().is_ok(), + "distinct deps must install in parallel" + ); + // Same path returns the same underlying lock. + let a2 = get_venv_install_lock("/cache/py/3.11/depA==1.0").await; + assert!( + a2.try_lock().is_err(), + "same target dir must map to the same lock" + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn test_venv_file_lock_excludes_across_descriptions() { + // The cross-process layer: flock on a sibling `.lock` excludes a second + // independent open file description (i.e. another worker process) while + // held, and frees it on close. Mirrors the loop in handle_python_reqs. + use nix::fcntl::{flock, FlockArg}; + use std::os::unix::io::AsRawFd; + + let dir = std::env::temp_dir().join("wm_venv_lock_test"); + std::fs::create_dir_all(&dir).unwrap(); + let lock_path = dir.join("dep==1.0.lock"); + + let f1 = std::fs::OpenOptions::new() + .create(true) + .write(true) + .open(&lock_path) + .unwrap(); + flock(f1.as_raw_fd(), FlockArg::LockExclusiveNonblock).unwrap(); + + // A second descriptor (stand-in for another process) cannot take it. + let f2 = std::fs::OpenOptions::new() + .create(true) + .write(true) + .open(&lock_path) + .unwrap(); + assert_eq!( + flock(f2.as_raw_fd(), FlockArg::LockExclusiveNonblock), + Err(nix::errno::Errno::EWOULDBLOCK), + "a second holder must be blocked while the lock is held" + ); + + // Releasing the first lets the second acquire it. + drop(f1); + flock(f2.as_raw_fd(), FlockArg::LockExclusiveNonblock) + .expect("lock must be acquirable once the holder releases it"); + + drop(f2); + let _ = std::fs::remove_file(&lock_path); + } } diff --git a/backend/windmill-worker/src/python_versions.rs b/backend/windmill-worker/src/python_versions.rs index 0ae6b3462a..3a1f35a302 100644 --- a/backend/windmill-worker/src/python_versions.rs +++ b/backend/windmill-worker/src/python_versions.rs @@ -551,6 +551,13 @@ impl PyV { .stdout(Stdio::piped()) .stderr(Stdio::piped()); + if let Some(cert_path) = INDEX_CERT.as_ref() { + child_cmd.env("SSL_CERT_FILE", cert_path); + } + if *NATIVE_CERT { + child_cmd.env("UV_NATIVE_TLS", "true"); + } + if let Some(mirror) = UV_PYTHON_INSTALL_MIRROR.read().await.as_ref() { child_cmd.env("UV_PYTHON_INSTALL_MIRROR", mirror); } diff --git a/backend/windmill-worker/src/result_processor.rs b/backend/windmill-worker/src/result_processor.rs index ab2cad4da9..26fbdb21af 100644 --- a/backend/windmill-worker/src/result_processor.rs +++ b/backend/windmill-worker/src/result_processor.rs @@ -1295,7 +1295,7 @@ pub async fn handle_job_error( db, &parent_job, mem_peak, - canceled_by, + canceled_by.clone(), e, worker_name, false, diff --git a/backend/windmill-worker/src/worker.rs b/backend/windmill-worker/src/worker.rs index 1ca77310ff..2335a78453 100644 --- a/backend/windmill-worker/src/worker.rs +++ b/backend/windmill-worker/src/worker.rs @@ -44,7 +44,7 @@ use windmill_common::{ schema::{should_validate_schema, SchemaValidator}, utils::{create_directory_async, WarnAfterExt}, worker::{ - make_pull_query, write_file, Connection, HttpClient, MAX_TIMEOUT, + is_allowed_file_location, make_pull_query, write_file, Connection, HttpClient, MAX_TIMEOUT, MIN_PERIODIC_SCRIPT_INTERVAL_SECONDS, ROOT_CACHE_DIR, ROOT_CACHE_NOMOUNT_DIR, WINDMILL_DIR, }, worker_group_job_stats::JobStatsMap, @@ -4324,20 +4324,25 @@ async fn resolve_partition_for_job( job: &MiniPulledJob, code: &str, conn: &Connection, -) -> error::Result> { +) -> error::Result<(Option, bool)> { use windmill_common::partition::{resolve_partition, PARTITION_ARG}; use windmill_parser::asset_parser::PartitionKind; - // Only deployed scripts participate in asset pipelines. Cheap - // substring guard so the overwhelming majority of script jobs (no - // `// partitioned` line) skip the full annotation scan on the hot - // path; a false positive only costs one extra parse, never wrong. - if !matches!(job.kind, JobKind::Script) || !code.contains("partitioned") { - return Ok(None); + // Only deployed scripts participate in asset pipelines. Cheap substring + // guard so the overwhelming majority of script jobs skip the annotation + // scan; when one might be present we parse *once* here and reuse the result + // for both `in_pipeline` (→ WM_PIPELINE env, read by the wmll.ducklake SDK to + // record state) and `partition` resolution — no second parse downstream. The + // bool is whether the script is a `// pipeline` member. + if !matches!(job.kind, JobKind::Script) + || !(code.contains("pipeline") || code.contains("partitioned")) + { + return Ok((None, false)); } - let Some(spec) = windmill_parser::asset_parser::parse_pipeline_annotations(code).partition - else { - return Ok(None); + let ann = windmill_parser::asset_parser::parse_pipeline_annotations(code); + let in_pipeline = ann.in_pipeline; + let Some(spec) = ann.partition else { + return Ok((None, in_pipeline)); }; // Already resolved upstream — explicit run arg, backfill, or @@ -4349,7 +4354,7 @@ async fn resolve_partition_for_job( .is_some_and(|s| !s.is_empty()) }); if already_set { - return Ok(None); + return Ok((None, in_pipeline)); } // `dynamic` extracts from the triggering payload (the `trigger` object @@ -4382,7 +4387,7 @@ async fn resolve_partition_for_job( job_id = %job.id, "partitioned script resolved to no partition (before start anchor); running without one" ); - return Ok(None); + return Ok((None, in_pipeline)); }; // Persist back so dispatch_asset_triggers (which reads the producer's @@ -4404,7 +4409,7 @@ async fn resolve_partition_for_job( windmill_common::worker::to_raw_value(&value), ); updated.args = Some(Json(map)); - Ok(Some(updated)) + Ok((Some(updated), in_pipeline)) } #[tracing::instrument(level = "trace", skip_all)] @@ -4566,7 +4571,8 @@ async fn handle_code_execution_job( // `// partitioned` (if any) and shadow `job` with a clone whose args // carry the resolved `partition` for the rest of execution. let _job_with_partition; - let job = match resolve_partition_for_job(job, code, conn).await? { + let (resolved_job, in_pipeline) = resolve_partition_for_job(job, code, conn).await?; + let job = match resolved_job { Some(j) => { _job_with_partition = j; &_job_with_partition @@ -4619,48 +4625,153 @@ async fn handle_code_execution_job( lock, &modules, false, + in_pipeline, ) .await } +/// True when `path` contains only `Normal`/`CurDir` components, i.e. it cannot +/// escape the directory it is joined onto (no `..`, no absolute root, no Windows +/// drive prefix). +fn is_contained_relative_path(path: &str) -> bool { + use std::path::Component; + std::path::Path::new(path) + .components() + .all(|c| matches!(c, Component::Normal(_) | Component::CurDir)) +} + pub async fn write_module_files( job_dir: &str, modules: &std::collections::HashMap, base_dir: Option<&str>, ) -> error::Result<()> { + // base_dir is derived from the runnable path, which on a preview run can + // carry `..` traversal (it is not the validated module-map key). Reject it + // before it is used to build any write target, otherwise a module could + // escape job_dir and write arbitrary files. + if let Some(dir) = base_dir { + if !is_contained_relative_path(dir) { + return Err(error::Error::BadRequest(format!( + "Invalid module base directory (path traversal): {dir}" + ))); + } + } for (relpath, module) in modules { - // Reject path traversal attempts in module paths - if relpath.contains("..") { + // Reject path traversal attempts in module paths (the module-map key). + if !is_contained_relative_path(relpath) { tracing::warn!("Skipping module with path traversal: {relpath}"); continue; } - let full_path = match base_dir { - Some(dir) => format!("{}/{}/{}", job_dir, dir, relpath), - None => format!("{}/{}", job_dir, relpath), + let relpath_from_job_dir = match base_dir { + Some(dir) => format!("{}/{}", dir, relpath), + None => relpath.to_string(), }; - if let Some(parent) = std::path::Path::new(&full_path).parent() { + // Authoritative guard: resolve the path and assert it stays inside job_dir. + let full_path = is_allowed_file_location(job_dir, &relpath_from_job_dir)?; + if let Some(parent) = full_path.parent() { tokio::fs::create_dir_all(parent).await?; } // For Python modules, create __init__.py in each intermediate directory // between base_dir and the module's parent so that relative imports work. if let Some(dir) = base_dir { - let rel = std::path::Path::new(relpath); - let base = std::path::Path::new(job_dir).join(dir); - let mut current = base.clone(); - for component in rel.parent().into_iter().flat_map(|p| p.components()) { + let mut current = std::path::PathBuf::from(dir); + for component in std::path::Path::new(relpath) + .parent() + .into_iter() + .flat_map(|p| p.components()) + { current = current.join(component); - let init_py = current.join("__init__.py"); + let init_py = is_allowed_file_location( + job_dir, + ¤t.join("__init__.py").to_string_lossy(), + )?; if !init_py.exists() { tokio::fs::write(&init_py, "").await?; } } } - tracing::debug!("Writing module file: {full_path}"); + tracing::debug!("Writing module file: {}", full_path.display()); tokio::fs::write(&full_path, &module.content).await?; } Ok(()) } +#[cfg(test)] +mod write_module_files_tests { + use super::*; + use std::collections::HashMap; + use windmill_common::scripts::ScriptLang; + + fn module(content: &str) -> ScriptModule { + ScriptModule { content: content.to_string(), language: ScriptLang::Python3, lock: None } + } + + #[test] + fn contained_relative_path_rejects_traversal_and_absolute() { + assert!(is_contained_relative_path("u/admin/pkg")); + assert!(is_contained_relative_path("./pkg/sub")); + // A `..` in a filename is a valid name, not a traversal. + assert!(is_contained_relative_path("weird..name")); + + assert!(!is_contained_relative_path("u/x/../../../etc")); + assert!(!is_contained_relative_path("../escape")); + assert!(!is_contained_relative_path("/etc/cron.d/wm")); + } + + #[tokio::test] + async fn base_dir_traversal_is_rejected_and_writes_nothing() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + // Sentinel just outside job_dir that a successful traversal would create. + let outside = job.path().parent().unwrap().join("wm_escaped_marker"); + + let mut modules = HashMap::new(); + modules.insert( + "wm_escaped_marker".to_string(), + module("* * * * * root id\n"), + ); + + // base_dir derived from a preview path carrying `..` traversal. + let res = write_module_files(job_dir, &modules, Some("u/x/../../../../../..")).await; + assert!(res.is_err(), "traversal base_dir must be rejected"); + assert!(!outside.exists(), "no file may be written outside job_dir"); + } + + #[tokio::test] + async fn relpath_traversal_is_skipped() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + let outside = job.path().parent().unwrap().join("wm_relpath_escape.py"); + + let mut modules = HashMap::new(); + modules.insert("../wm_relpath_escape.py".to_string(), module("x = 1")); + + write_module_files(job_dir, &modules, None).await.unwrap(); + assert!(!outside.exists()); + } + + #[tokio::test] + async fn legitimate_modules_are_written_with_init_py() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + + let mut modules = HashMap::new(); + modules.insert("pkg/sub/mod.py".to_string(), module("VALUE = 42")); + + write_module_files(job_dir, &modules, Some("u/admin")) + .await + .unwrap(); + + let base = job.path().join("u/admin"); + assert_eq!( + std::fs::read_to_string(base.join("pkg/sub/mod.py")).unwrap(), + "VALUE = 42" + ); + assert!(base.join("pkg/__init__.py").exists()); + assert!(base.join("pkg/sub/__init__.py").exists()); + } +} + pub async fn run_language_executor( job: &MiniPulledJob, conn: &Connection, @@ -4685,6 +4796,9 @@ pub async fn run_language_executor( lock: &Option, modules: &Option>, run_inline: bool, + // Whether the script is a `// pipeline` member (parsed once upstream) — sets + // WM_PIPELINE so the wmll.ducklake SDK helpers record materialization state. + in_pipeline: bool, ) -> error::Result> { // Defense-in-depth (GHSA-wxjq-w5pj-jqhx): the entrypoint override is // interpolated verbatim into a code position of the generated language @@ -5047,6 +5161,11 @@ mount {{ #[allow(unused_mut)] let mut envs = build_envs(envs.as_ref())?; + // Signal pipeline context to the script so the wmll.ducklake SDK helpers + // record materialization state (the grid/backfill) and skip it otherwise. + if in_pipeline { + envs.insert("WM_PIPELINE".to_string(), "true".to_string()); + } let Some(language) = language else { return Err(Error::ExecutionErr( @@ -5832,6 +5951,7 @@ pub fn init_worker_internal_server_inline_utils( &None, &None, true, + false, ) .await }) @@ -5913,6 +6033,7 @@ pub fn init_worker_internal_server_inline_utils( &content_info.lockfile, &content_info.modules, true, + false, ) .await }) diff --git a/backend/windmill-worker/src/worker_flow.rs b/backend/windmill-worker/src/worker_flow.rs index 7f90f9240e..10e8218a77 100644 --- a/backend/windmill-worker/src/worker_flow.rs +++ b/backend/windmill-worker/src/worker_flow.rs @@ -28,7 +28,7 @@ use serde::{Deserialize, Serialize}; use serde_json::value::RawValue; use serde_json::{json, Value}; use sqlx::types::Json; -use sqlx::{FromRow, Postgres, Transaction}; +use sqlx::{Acquire, FromRow, Postgres, Transaction}; use tracing::instrument; use uuid::Uuid; use windmill_common::auth::get_job_perms; @@ -332,14 +332,14 @@ fn get_stop_after_if_data(stop_after_if: Option<&StopAfterIf>) -> (bool, Option< return (false, None, false); } -async fn get_id_ctx_for_expr( +async fn get_id_ctx_for_expr<'c>( expr: &str, flow: uuid::Uuid, - db: &DB, + e: impl sqlx::PgExecutor<'c>, status: &FlowStatus, ) -> error::Result> { if expr.contains("results.") || expr.contains("results[") || expr.contains("results?.") { - let flow_job = get_mini_pulled_job(db, &flow).await?; + let flow_job = get_mini_pulled_job(e, &flow).await?; if let Some(flow_job) = flow_job { Ok(Some(get_transform_context(&flow_job, "", &status))) } else { @@ -351,7 +351,7 @@ async fn get_id_ctx_for_expr( } async fn evaluate_stop_after_all_iters_if( - db: &DB, + tx: &mut Transaction<'_, Postgres>, stop_after_all_iters_if: &StopAfterIf, module_status: &FlowStatusModule, w_id: &str, @@ -366,9 +366,20 @@ async fn evaluate_stop_after_all_iters_if( flow: uuid::Uuid, status: &FlowStatus, ) -> error::Result<()> { + // Test hook (see test_stop_after_all_iters_if_db_error_isolated_by_savepoint): + // run a query that aborts this (savepoint) transaction so the test can verify the + // caller's savepoint keeps the outer status-update transaction committable. + #[cfg(feature = "failpoints")] + if stop_after_all_iters_if.expr == "__wm_failpoint_abort_tx__" { + sqlx::query("SELECT 1/0") + .execute(&mut **tx) + .await + .map_err(|e| Error::internal_err(format!("failpoint abort_tx: {e:#}")))?; + } + let iters_result = match &module_status { FlowStatusModule::InProgress { flow_jobs: Some(flow_jobs), .. } => { - Arc::new(retrieve_flow_jobs_results(db, w_id, flow_jobs).await?) + Arc::new(retrieve_flow_jobs_results(&mut **tx, w_id, flow_jobs).await?) } _ => { return Err(Error::internal_err(format!( @@ -379,7 +390,8 @@ async fn evaluate_stop_after_all_iters_if( *nresult = Some(iters_result.clone()); // as an optimization, we store the result of all jobs as when stop_early_after_all_iters evaluates to false, it would have to be computed (finished loop/branchall) - let id_ctx = get_id_ctx_for_expr(&stop_after_all_iters_if.expr, flow, db, status).await?; + let id_ctx = + get_id_ctx_for_expr(&stop_after_all_iters_if.expr, flow, &mut **tx, status).await?; let stop_early_after_all_iters = compute_bool_from_expr( &stop_after_all_iters_if.expr, @@ -976,8 +988,15 @@ pub async fn update_flow_status_after_job_completion_internal( .and_then(|x| x.stop_after_all_iters_if.as_ref()) { let args = from_result_to_args(args.as_ref().await.get_ref())?; - if let Err(e) = evaluate_stop_after_all_iters_if( - db, + // Isolate the reads in a savepoint on the same connection: the + // caller below swallows our error and keeps using `tx`, so a DB + // read failure must not leave the outer transaction aborted (that + // would fail the later commit). On error we roll back to the + // savepoint, matching the previous pool-read behaviour where a + // failed read left `tx` usable and the iteration was marked failed. + let mut sp = tx.begin().await?; + let eval_res = evaluate_stop_after_all_iters_if( + &mut sp, stop_after_all_iters_if, module_status, w_id, @@ -992,15 +1011,19 @@ pub async fn update_flow_status_after_job_completion_internal( flow, &old_status, ) - .await - { - tracing::error!("error evaluating stop_after_all_iters_if: {e:#}"); - stop_early = true; - skip_if_stop_early = false; - stop_early_err_msg = Some(format!( - "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", - stop_after_all_iters_if.expr - )); + .await; + match eval_res { + Ok(()) => sp.commit().await?, + Err(e) => { + let _ = sp.rollback().await; + tracing::error!("error evaluating stop_after_all_iters_if: {e:#}"); + stop_early = true; + skip_if_stop_early = false; + stop_early_err_msg = Some(format!( + "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", + stop_after_all_iters_if.expr + )); + } } } @@ -1054,7 +1077,7 @@ pub async fn update_flow_status_after_job_completion_internal( let r = sqlx::query_scalar!( "DELETE FROM parallel_monitor_lock WHERE parent_flow_id = $1 RETURNING last_ping", flow, - ).fetch_optional(db).await.map_err(|e| { + ).fetch_optional(&mut *tx).await.map_err(|e| { Error::internal_err(format!( "error while deleting parallel_monitor_lock: {e:#}" )) @@ -1198,8 +1221,12 @@ pub async fn update_flow_status_after_job_completion_internal( { let args = from_result_to_args(args.as_ref().await.get_ref())?; - if let Err(e) = evaluate_stop_after_all_iters_if( - db, + // See the matching savepoint comment above: isolate the reads so a + // DB read failure (whose error the caller swallows) cannot abort + // the outer transaction and break the later commit. + let mut sp = tx.begin().await?; + let eval_res = evaluate_stop_after_all_iters_if( + &mut sp, stop_after_all_iters_if, module_status, w_id, @@ -1214,14 +1241,18 @@ pub async fn update_flow_status_after_job_completion_internal( flow, &old_status, ) - .await - { - stop_early = true; - skip_if_stop_early = false; - stop_early_err_msg = Some(format!( - "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", - stop_after_all_iters_if.expr - )); + .await; + match eval_res { + Ok(()) => sp.commit().await?, + Err(e) => { + let _ = sp.rollback().await; + stop_early = true; + skip_if_stop_early = false; + stop_early_err_msg = Some(format!( + "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", + stop_after_all_iters_if.expr + )); + } } } } @@ -1462,7 +1493,7 @@ pub async fn update_flow_status_after_job_completion_internal( match &new_status { Some(FlowStatusModule::Success { flow_jobs: Some(jobs), .. }) | Some(FlowStatusModule::Failure { flow_jobs: Some(jobs), .. }) => { - Arc::new(retrieve_flow_jobs_results(db, w_id, jobs).await?) + Arc::new(retrieve_flow_jobs_results(&mut *tx, w_id, jobs).await?) } _ => result.clone(), } @@ -1604,6 +1635,7 @@ pub async fn update_flow_status_after_job_completion_internal( windmill_common::runnable_settings::RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: None, + retry_settings: None, }, db, ) @@ -2273,8 +2305,8 @@ async fn set_success_and_duration_in_flow_job_success<'c>( Ok(()) } -async fn retrieve_flow_jobs_results( - db: &DB, +async fn retrieve_flow_jobs_results<'c>( + e: impl sqlx::PgExecutor<'c>, w_id: &str, job_uuids: &Vec, ) -> error::Result> { @@ -2285,7 +2317,7 @@ async fn retrieve_flow_jobs_results( job_uuids.as_slice(), w_id ) - .fetch_all(db) + .fetch_all(e) .await? .into_iter() .map(|br| (br.id, br.result)) @@ -4326,7 +4358,7 @@ async fn push_next_flow_job( .filter(|t| !t.is_empty() && *t != flow_job.tag.as_str()) { check_tag_available_for_workspace_internal( - &db, + db, &flow_job.workspace_id, tag_str, email, @@ -6179,8 +6211,15 @@ fn needs_resume(flow: &FlowValue, status: &FlowStatus) -> Option<(Suspend, Uuid) return None; } - if let &FlowStatusModule::Success { job, .. } = status.modules.get(prev)? { - Some((suspend.unwrap(), job)) + if let &FlowStatusModule::Success { job, skipped, .. } = status.modules.get(prev)? { + // A step skipped via skip_if never ran, so its suspend/approval was never + // armed and no resume event will ever arrive. Gating the next step on it + // would park the flow forever. + if skipped { + None + } else { + Some((suspend.unwrap(), job)) + } } else { None } diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index cf66a11663..b7a8d0c397 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.733.1"; +export const VERSION = "v1.753.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/build-npm.ts b/cli/build-npm.ts index 5c9353dd21..1dbc9261de 100644 --- a/cli/build-npm.ts +++ b/cli/build-npm.ts @@ -4,24 +4,20 @@ import { join } from "node:path"; const outDir = "./npm"; -// Parser npm packages — used as externals and added to generated package.json -const parserPackages = [ - "windmill-parser-wasm-py", "windmill-parser-wasm-ts", - "windmill-parser-wasm-regex", "windmill-parser-wasm-go", - "windmill-parser-wasm-php", "windmill-parser-wasm-rust", - "windmill-parser-wasm-yaml", "windmill-parser-wasm-csharp", - "windmill-parser-wasm-nu", "windmill-parser-wasm-java", - "windmill-parser-wasm-ruby", - "windmill-parser-wasm-py-imports", -]; -const parserExternals = parserPackages.flatMap(p => ["--external", p]); - // Forward parser specs from the dev package.json so the published CLI pins -// to the same versions devs install/test against. Falls back to "*" if not -// listed locally. +// to the same versions devs install/test against. const cliDeps: Record = JSON.parse(readFileSync("./package.json", "utf-8")).dependencies ?? {}; +// Parser packages: bundle externals + published dependencies. Derived from +// package.json (not hand-listed) — a parser missing from this set ships a CLI +// whose loadParser() silently falls back. package.json itself is kept in sync +// with loadParser() call sites by test/parser_packages_unit.test.ts. +const parserPackages = Object.keys(cliDeps).filter((p) => + p.startsWith("windmill-parser-wasm") +); +const parserExternals = parserPackages.flatMap(p => ["--external", p]); + // Clean output directory rmSync(outDir, { recursive: true, force: true }); diff --git a/cli/bun.lock b/cli/bun.lock index c94551beb3..a745acbcba 100644 --- a/cli/bun.lock +++ b/cli/bun.lock @@ -19,6 +19,7 @@ "pg-gateway": "0.3.0-beta.4", "svelte": "^5.45.2", "tar-stream": "^3.1.7", + "windmill-parser-wasm-asset": "1.749.0", "windmill-parser-wasm-csharp": "1.510.1", "windmill-parser-wasm-go": "1.510.1", "windmill-parser-wasm-java": "1.510.1", @@ -26,6 +27,7 @@ "windmill-parser-wasm-php": "1.647.1", "windmill-parser-wasm-py": "1.693.1", "windmill-parser-wasm-py-imports": "1.693.1", + "windmill-parser-wasm-r": "1.668.1", "windmill-parser-wasm-regex": "1.692.0", "windmill-parser-wasm-ruby": "1.526.1", "windmill-parser-wasm-rust": "1.647.1", @@ -288,6 +290,8 @@ "utility-types": ["utility-types@3.11.0", "", {}, "sha512-6Z7Ma2aVEWisaL6TvBCy7P8rm2LQoPv6dJ7ecIaIixHcwfbJ0x7mWdbcwlIM5IGQxPZSFYeqRCqlOOeKoJYMkw=="], + "windmill-parser-wasm-asset": ["windmill-parser-wasm-asset@1.749.0", "", {}, "sha512-gj8g9sWQ0tXKfXso7xJxR56sS8Loe/RsnFy+0af5R8siZeCag9ikquGbQ8d8kOqIK2U8eCNA0LqsU/xAFDJIOg=="], + "windmill-parser-wasm-csharp": ["windmill-parser-wasm-csharp@1.510.1", "", {}, "sha512-qm09YmnbeYHLwYn1jUnObVzPhYO9NZKMlIO7nlo7zPJBXqksgG5fK/KCtwGw9rChrnz+DsvM9wP5FhrwRLMtwQ=="], "windmill-parser-wasm-go": ["windmill-parser-wasm-go@1.510.1", "", {}, "sha512-HOkk6LXK0wrwvkn+zjm3Gxo90HmyL6TYqmLo2yp8fZuppy7GOngT27zwYeBtwONiPyvDKskzoqPQoEfd8VuUsQ=="], @@ -302,6 +306,8 @@ "windmill-parser-wasm-py-imports": ["windmill-parser-wasm-py-imports@1.693.1", "", {}, "sha512-FC0KbREe2G/sa/9kYIR930wmWw+VL6PvEIqg12J3dsJes3A+0x5JIUPT/jeD+c24DrG0ko/Ub7yDnYs56Bem7g=="], + "windmill-parser-wasm-r": ["windmill-parser-wasm-r@1.668.1", "", {}, "sha512-5YNeUibxpNBvYrxCgQcz1PxGhTFx2CyEpg2udtIhq7bx0d4gF/KDZVupMeQmAObmrEtTSFGUWNRJ4zXSWNrSpQ=="], + "windmill-parser-wasm-regex": ["windmill-parser-wasm-regex@1.692.0", "", {}, "sha512-BHGTxrinZJ9ef6hFxbKiBqBEr5uqgG/QySOgMA5r1LswO9n/8fyGswr8JcPT2kGaoeoweV6/RQ+RHVaOhosnKw=="], "windmill-parser-wasm-ruby": ["windmill-parser-wasm-ruby@1.526.1", "", {}, "sha512-rMBQA8s21wmL2kA5ztRs/ZgVA3ckxe9/NLjxl3iQPL0CX6DlvfaUH0O+AnhpXXDMyBs1Y1SZIhcnbnvsHZ3R8g=="], diff --git a/cli/package-lock.json b/cli/package-lock.json index c502b3b25e..a9d72696b2 100644 --- a/cli/package-lock.json +++ b/cli/package-lock.json @@ -20,6 +20,7 @@ "pg-gateway": "0.3.0-beta.4", "svelte": "^5.45.2", "tar-stream": "^3.1.7", + "windmill-parser-wasm-asset": "1.749.0", "windmill-parser-wasm-csharp": "1.510.1", "windmill-parser-wasm-go": "1.510.1", "windmill-parser-wasm-java": "1.510.1", @@ -27,6 +28,7 @@ "windmill-parser-wasm-php": "1.647.1", "windmill-parser-wasm-py": "1.693.1", "windmill-parser-wasm-py-imports": "1.693.1", + "windmill-parser-wasm-r": "1.668.1", "windmill-parser-wasm-regex": "1.692.0", "windmill-parser-wasm-ruby": "1.526.1", "windmill-parser-wasm-rust": "1.647.1", @@ -1409,6 +1411,11 @@ "node": ">= 4" } }, + "node_modules/windmill-parser-wasm-asset": { + "version": "1.749.0", + "resolved": "https://registry.npmjs.org/windmill-parser-wasm-asset/-/windmill-parser-wasm-asset-1.749.0.tgz", + "integrity": "sha512-gj8g9sWQ0tXKfXso7xJxR56sS8Loe/RsnFy+0af5R8siZeCag9ikquGbQ8d8kOqIK2U8eCNA0LqsU/xAFDJIOg==" + }, "node_modules/windmill-parser-wasm-csharp": { "version": "1.510.1", "resolved": "https://registry.npmjs.org/windmill-parser-wasm-csharp/-/windmill-parser-wasm-csharp-1.510.1.tgz", @@ -1444,6 +1451,11 @@ "resolved": "https://registry.npmjs.org/windmill-parser-wasm-py-imports/-/windmill-parser-wasm-py-imports-1.693.1.tgz", "integrity": "sha512-FC0KbREe2G/sa/9kYIR930wmWw+VL6PvEIqg12J3dsJes3A+0x5JIUPT/jeD+c24DrG0ko/Ub7yDnYs56Bem7g==" }, + "node_modules/windmill-parser-wasm-r": { + "version": "1.668.1", + "resolved": "https://registry.npmjs.org/windmill-parser-wasm-r/-/windmill-parser-wasm-r-1.668.1.tgz", + "integrity": "sha512-5YNeUibxpNBvYrxCgQcz1PxGhTFx2CyEpg2udtIhq7bx0d4gF/KDZVupMeQmAObmrEtTSFGUWNRJ4zXSWNrSpQ==" + }, "node_modules/windmill-parser-wasm-regex": { "version": "1.692.0", "resolved": "https://registry.npmjs.org/windmill-parser-wasm-regex/-/windmill-parser-wasm-regex-1.692.0.tgz", diff --git a/cli/package.json b/cli/package.json index 2f46e3829b..589bc52271 100644 --- a/cli/package.json +++ b/cli/package.json @@ -28,6 +28,7 @@ "pg-gateway": "0.3.0-beta.4", "svelte": "^5.45.2", "tar-stream": "^3.1.7", + "windmill-parser-wasm-asset": "1.749.0", "windmill-parser-wasm-csharp": "1.510.1", "windmill-parser-wasm-go": "1.510.1", "windmill-parser-wasm-java": "1.510.1", @@ -35,6 +36,7 @@ "windmill-parser-wasm-php": "1.647.1", "windmill-parser-wasm-py": "1.693.1", "windmill-parser-wasm-py-imports": "1.693.1", + "windmill-parser-wasm-r": "1.668.1", "windmill-parser-wasm-regex": "1.692.0", "windmill-parser-wasm-ruby": "1.526.1", "windmill-parser-wasm-rust": "1.647.1", diff --git a/cli/src/commands/datatable/datatable.ts b/cli/src/commands/datatable/datatable.ts index 1291d4cc13..6609d08c50 100644 --- a/cli/src/commands/datatable/datatable.ts +++ b/cli/src/commands/datatable/datatable.ts @@ -9,6 +9,13 @@ import { GlobalOptions } from "../../types.ts"; import { runCatalogQuery } from "../../utils/catalog.ts"; import { psql as psqlDatatable } from "./psql.ts"; import { serve as serveDatatable } from "./serve.ts"; +import { + createMigration, + pushLocalMigrations, + rollbackMigrations, + runMigrations, + validateLocalMigrations, +} from "../datatable_migrations.ts"; const DEFAULT_DATATABLE_NAME = "main"; @@ -41,6 +48,69 @@ async function run( await runCatalogQuery(opts, "datatable", name, sql); } +function migrateNew( + opts: GlobalOptions & { datatable?: string }, + name: string, +) { + createMigration(opts.datatable ?? DEFAULT_DATATABLE_NAME, name); +} + +async function migrateUp(opts: GlobalOptions & { datatable?: string }) { + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + const dt = opts.datatable ?? DEFAULT_DATATABLE_NAME; + // Reject malformed local migrations (duplicate timestamps, orphan downs) before + // pushing — the same check `wmill sync push` runs — so a duplicate timestamp + // can't silently overwrite one migration on upsert. + const errors = validateLocalMigrations(new Set([dt])); + if (errors.length > 0) { + log.error( + "Invalid datatable migrations, aborting:\n" + + errors.map((e) => ` - ${e}`).join("\n"), + ); + process.exit(1); + } + // Push any locally-created/edited migration files first (without running + // them), so `migrate up` works even before a `wmill sync push`. + await pushLocalMigrations(workspace.workspaceId, dt); + await runMigrations(workspace.workspaceId, dt); +} + +async function migrateDown(opts: GlobalOptions & { datatable?: string }) { + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + const dt = opts.datatable ?? DEFAULT_DATATABLE_NAME; + await rollbackMigrations(workspace.workspaceId, dt); +} + +const migrateCommand = new Command() + .description("manage datatable migrations") + .command("new", "scaffold a new migration (.up.sql / .down.sql files)") + .arguments("") + .option( + "-d --datatable ", + "Target datatable (default: main)", + ) + .action(migrateNew as any) + .command( + "up", + "apply all pending migrations to the main datatable (or one via --datatable)", + ) + .option( + "-d --datatable ", + "Target datatable (default: main)", + ) + .action(migrateUp as any) + .command( + "down", + "roll back the most recent migration on the main datatable (or one via --datatable)", + ) + .option( + "-d --datatable ", + "Target datatable (default: main)", + ) + .action(migrateDown as any); + async function create( opts: GlobalOptions & { resource?: string; force?: boolean }, name?: string, @@ -124,6 +194,7 @@ const command = new Command() "Output only the final result as JSON. Useful for scripting.", ) .action(run as any) + .command("migrate", migrateCommand) .command( "create", "register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable://", diff --git a/cli/src/commands/datatable_migrations.ts b/cli/src/commands/datatable_migrations.ts new file mode 100644 index 0000000000..4fcaffd4fa --- /dev/null +++ b/cli/src/commands/datatable_migrations.ts @@ -0,0 +1,340 @@ +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as log from "../core/log.ts"; +import { colors } from "@cliffy/ansi/colors"; +import * as wmill from "../../gen/services.gen.ts"; +import { readTextFile } from "../utils/utils.ts"; +import { Confirm } from "@cliffy/prompt/confirm"; + +// Migrations live under /migrations/datatable//, one folder per +// target data table, as `_.up.sql` (and optional `.down.sql`). +// They are synced as ordinary workspace files (see the workspace tarball export +// and the `datatable_migration` handling in sync.ts); this module only holds the +// `wmill datatable migrate` command helpers and the per-file push primitive. +const MIGRATIONS_DIR = path.join("migrations", "datatable"); + +// Migration names map directly onto file names and the DB `name` column. +const MIGRATION_NAME_RE = /^[a-zA-Z0-9_-]+$/; + +/** Current UTC time as a YYYYMMDDHHMMSS migration version. */ +function migrationTimestamp(): string { + const d = new Date(); + const p = (n: number) => String(n).padStart(2, "0"); + return ( + `${d.getUTCFullYear()}${p(d.getUTCMonth() + 1)}${p(d.getUTCDate())}` + + `${p(d.getUTCHours())}${p(d.getUTCMinutes())}${p(d.getUTCSeconds())}` + ); +} + +/** + * A migration version unique within a data table folder: the current UTC + * timestamp bumped past any existing version, so two migrations scaffolded in + * the same second don't collide on the `(datatable, timestamp)` identity used to + * upsert them. + */ +function nextMigrationTimestamp(dir: string): string { + const now = Number(migrationTimestamp()); + let max = 0; + if (fs.existsSync(dir)) { + for (const file of fs.readdirSync(dir)) { + const m = file.match(/^(\d+)_.*\.(up|down)\.sql$/); + if (m) max = Math.max(max, Number(m[1])); + } + } + return String(max >= now ? max + 1 : now); +} + +/** + * Scaffold a new migration under migrations/datatable// as empty + * `_.up.sql` and `.down.sql` files. Purely local — no network. + */ +export function createMigration(datatable: string, name: string): void { + if (!MIGRATION_NAME_RE.test(name)) { + throw new Error( + `Invalid migration name '${name}': use only letters, digits, '_' and '-'`, + ); + } + const dir = path.join(process.cwd(), MIGRATIONS_DIR, datatable); + fs.mkdirSync(dir, { recursive: true }); + + const timestamp = nextMigrationTimestamp(dir); + const base = `${timestamp}_${name}`; + const up = path.join(dir, `${base}.up.sql`); + const down = path.join(dir, `${base}.down.sql`); + // Frame the body in an explicit transaction so it applies atomically, matching + // the template the UI's "New migration" modal seeds. + const template = (direction: string) => + `-- ${direction} migration: ${name}\nBEGIN;\n\n-- Add your migration here\n\nEND;\n`; + fs.writeFileSync(up, template("up"), "utf-8"); + fs.writeFileSync(down, template("down"), "utf-8"); + + log.info( + colors.green(`Created migration ${base} in ${MIGRATIONS_DIR}/${datatable}/`), + ); + for (const f of [up, down]) { + log.info(colors.gray(` ${path.relative(process.cwd(), f)}`)); + } +} + +/** + * Apply the workspace's pending migrations to a data table (forwards migrations + * recorded in `_wm_migrations`). Mirrors `wmill datatable migrate up`. + */ +export async function runMigrations( + workspace: string, + datatableName: string, +): Promise { + const result = await wmill.runDatatableMigrations({ + workspace, + datatableName, + }); + const applied = result.applied ?? []; + if (applied.length === 0) { + log.info(colors.gray(`No pending migrations to run on '${datatableName}'`)); + return; + } + log.info( + colors.green(`Applied ${applied.length} migration(s) to '${datatableName}':`), + ); + for (const m of applied) { + log.info(colors.gray(` ${m.version} ${m.name}`)); + } +} + +/** + * Roll back the most recently applied migration on a data table (one step). + * Mirrors `wmill datatable migrate down`. + */ +export async function rollbackMigrations( + workspace: string, + datatableName: string, +): Promise { + const result = await wmill.rollbackDatatableMigrations({ + workspace, + datatableName, + }); + const rolledBack = result.rolled_back ?? []; + if (rolledBack.length === 0) { + log.info( + colors.gray(`No applied migrations to roll back on '${datatableName}'`), + ); + return; + } + for (const m of rolledBack) { + log.info( + colors.green(`Rolled back migration ${m.version} ${m.name} on '${datatableName}'`), + ); + } +} + +/** + * Validate the on-disk migration files for the given data tables (or all of + * them when `datatables` is omitted). Returns a list of human-readable problems; + * an empty list means the migrations are well-formed. Two states are invalid: + * - two up (or two down) files sharing the same timestamp, which collide on the + * `(datatable, timestamp)` identity used to upsert; and + * - a `.down.sql` with no matching `.up.sql` (an up file is mandatory). + */ +export function validateLocalMigrations(datatables?: Set): string[] { + const errors: string[] = []; + const root = path.join(process.cwd(), MIGRATIONS_DIR); + if (!fs.existsSync(root)) return errors; + + for (const datatable of fs.readdirSync(root)) { + if (datatables && !datatables.has(datatable)) continue; + const dtDir = path.join(root, datatable); + if (!fs.statSync(dtDir).isDirectory()) continue; + + const upNamesByTs = new Map(); + const downNamesByTs = new Map(); + const upBases = new Set(); + const downBases: { ts: number; name: string }[] = []; + + for (const file of fs.readdirSync(dtDir)) { + const m = file.match(/^(\d+)_(.*)\.(up|down)\.sql$/); + if (!m) continue; + const ts = Number(m[1]); + const name = m[2]; + if (m[3] === "up") { + (upNamesByTs.get(ts) ?? upNamesByTs.set(ts, []).get(ts)!).push(name); + upBases.add(`${ts}_${name}`); + } else { + (downNamesByTs.get(ts) ?? downNamesByTs.set(ts, []).get(ts)!).push(name); + downBases.push({ ts, name }); + } + } + + for (const [ts, names] of upNamesByTs) { + if (names.length > 1) { + errors.push( + `${datatable}: ${names.length} up migrations share timestamp ${ts} (${names.join(", ")})`, + ); + } + } + for (const [ts, names] of downNamesByTs) { + if (names.length > 1) { + errors.push( + `${datatable}: ${names.length} down migrations share timestamp ${ts} (${names.join(", ")})`, + ); + } + } + for (const d of downBases) { + if (!upBases.has(`${d.ts}_${d.name}`)) { + errors.push( + `${datatable}: ${d.ts}_${d.name}.down.sql has no matching ${d.ts}_${d.name}.up.sql`, + ); + } + } + } + + return errors; +} + +/** + * Sync a single migration to the workspace based on the current on-disk state of + * its `/_.up.sql` file: upsert it when the up file + * exists, otherwise delete it. Called by `wmill sync push` for each changed + * `datatable_migration` file. + */ +export async function pushMigrationFromDisk( + workspace: string, + m: { datatable: string; timestamp: number }, +): Promise { + const dir = path.join(process.cwd(), MIGRATIONS_DIR, m.datatable); + // Find the up file for this timestamp regardless of its name segment. A rename + // (`123_old.up.sql` -> `123_new.up.sql`) keeps the (datatable, timestamp) + // identity but changes the name; the diff sorter may process the deleted old + // path before the added new one, so keying off the passed name would delete + // the record. Scanning by timestamp upserts the surviving file instead. + const upFile = fs.existsSync(dir) + ? fs.readdirSync(dir).find((f) => { + const parsed = f.match(/^(\d+)_(.*)\.up\.sql$/); + return parsed !== null && Number(parsed[1]) === m.timestamp; + }) + : undefined; + + if (upFile === undefined) { + log.info(colors.red(`Deleting datatable_migration ${m.datatable}/${m.timestamp}`)); + await wmill.deleteDatatableMigration({ + workspace, + datatableName: m.datatable, + timestamp: m.timestamp, + }); + return; + } + + const name = upFile.match(/^(\d+)_(.*)\.up\.sql$/)![2]; + const base = `${m.timestamp}_${name}`; + const code_up = await readTextFile(path.join(dir, upFile)); + const downPath = path.join(dir, `${base}.down.sql`); + const code_down = fs.existsSync(downPath) ? await readTextFile(downPath) : undefined; + + log.info(colors.green(`Pushing datatable_migration ${m.datatable}/${base}`)); + await wmill.upsertDatatableMigration({ + workspace, + datatableName: m.datatable, + requestBody: { + timestamp: m.timestamp, + name, + code_up, + ...(code_down !== undefined ? { code_down } : {}), + }, + }); +} + +/** + * Upsert the on-disk migrations of a data table to the workspace, so a freshly + * created migration file works with `wmill datatable migrate up` even without a + * prior `wmill sync push`. Pushes only migrations that are new or edited + * (compared against the workspace's current definitions); it never deletes + * remote migrations absent on disk and never touches other item kinds. + */ +export async function pushLocalMigrations( + workspace: string, + datatableName: string, +): Promise { + const dir = path.join(process.cwd(), MIGRATIONS_DIR, datatableName); + if (!fs.existsSync(dir)) return; + + // Local migrations are identified by their `.up.sql` file (the up file is + // mandatory); this deliberately ignores files that were only deleted locally. + const local: { timestamp: number; name: string }[] = []; + for (const file of fs.readdirSync(dir)) { + const m = file.match(/^(\d+)_(.*)\.up\.sql$/); + if (m) local.push({ timestamp: Number(m[1]), name: m[2] }); + } + if (local.length === 0) return; + + const remote = await wmill.listDatatableMigrations({ workspace }); + const remoteByTs = new Map( + remote + .filter((r) => r.datatable === datatableName) + .map((r) => [r.timestamp, r] as const), + ); + + for (const { timestamp, name } of local) { + const base = `${timestamp}_${name}`; + const code_up = await readTextFile(path.join(dir, `${base}.up.sql`)); + const downPath = path.join(dir, `${base}.down.sql`); + const code_down = fs.existsSync(downPath) + ? await readTextFile(downPath) + : undefined; + + const r = remoteByTs.get(timestamp); + const unchanged = + r !== undefined && + r.name === name && + r.code_up === code_up && + (r.code_down ?? undefined) === code_down; + if (unchanged) continue; + + log.info(colors.green(`Pushing datatable_migration ${datatableName}/${base}`)); + await wmill.upsertDatatableMigration({ + workspace, + datatableName, + requestBody: { + timestamp, + name, + code_up, + ...(code_down !== undefined ? { code_down } : {}), + }, + }); + } +} + +/** + * After a push that introduced new migrations, list them and (interactively) + * offer to run them, equivalent to `wmill datatable migrate up` on each affected + * data table. + */ +export async function offerToRunNewMigrations( + workspace: string, + newMigrations: { datatable: string; timestamp: number; name: string }[], + opts?: { yes?: boolean; jsonOutput?: boolean }, +): Promise { + if (newMigrations.length === 0) return; + + log.info(colors.green("New migrations were pushed:")); + for (const m of newMigrations) { + log.info(colors.gray(` ${m.datatable}: ${m.timestamp} ${m.name}`)); + } + + // Running migrations mutates the data tables, so skip the prompt in + // non-interactive contexts (--yes, --json, no TTY). + const interactive = !opts?.jsonOutput && !opts?.yes && !!process.stdin.isTTY; + if (!interactive) { + return; + } + + const shouldRun = await Confirm.prompt({ + message: "New migrations were pushed, run them?", + default: false, + }); + if (!shouldRun) { + return; + } + + for (const datatable of new Set(newMigrations.map((m) => m.datatable))) { + await runMigrations(workspace, datatable); + } +} diff --git a/cli/src/commands/docs/docs.ts b/cli/src/commands/docs/docs.ts index fed91201e0..4ba7ae95d5 100644 --- a/cli/src/commands/docs/docs.ts +++ b/cli/src/commands/docs/docs.ts @@ -7,24 +7,16 @@ import { GlobalOptions } from "../../types.ts"; import { getHeaders } from "../../utils/utils.ts"; import { detectAuthGatewayChallenge } from "../../utils/http_guards.ts"; -interface DocContentItem { - title: string; +interface DocsSearchResult { url: string; - source?: { - content?: Array<{ text: string }>; - }; + title: string; + score: number; + snippets: string[]; } -interface InkeepResponse { - choices?: Array<{ - message?: { - content?: string; - }; - }>; -} - -interface ParsedContent { - content?: DocContentItem[]; +interface DocsSearchResponse { + text: string; + results: DocsSearchResult[]; } async function docs( @@ -34,7 +26,9 @@ async function docs( await requireLogin(opts); const workspace = await resolveWorkspace(opts); - const url = `${workspace.remote}api/inkeep`; + // The backend self-hosts the docs corpus and does the search, so this works + // against any instance (no windmill.dev egress required). + const url = `${workspace.remote}api/docs/search?query=${encodeURIComponent(query)}`; console.log(colors.bold(`\nSearching Windmill docs...\n`)); @@ -42,13 +36,11 @@ async function docs( let res: Response; try { res = await fetch(url, { - method: "POST", + method: "GET", headers: { - "Content-Type": "application/json", Authorization: `Bearer ${workspace.token}`, ...extraHeaders, }, - body: JSON.stringify({ query }), }); } catch (e) { throw new Error(`Network error connecting to ${workspace.remote}: ${e}`); @@ -56,54 +48,31 @@ async function docs( await detectAuthGatewayChallenge(res, url); - if (res.status === 403) { - log.info( - "Windmill documentation search is an Enterprise Edition feature. Please upgrade to use this command." - ); - return; - } - if (!res.ok) { throw new Error( `Documentation search failed: ${res.status} ${res.statusText}\n${await res.text()}` ); } - const data = (await res.json()) as InkeepResponse; - const raw = data.choices?.[0]?.message?.content; - - if (!raw) { - log.info("No documentation found for this query."); - return; - } - - let parsed: ParsedContent; - try { - parsed = JSON.parse(raw); - } catch { - throw new Error("Failed to parse documentation response."); - } - - const items = parsed.content ?? []; - - if (items.length === 0) { - log.info("No documentation found for this query."); - return; - } + const data = (await res.json()) as DocsSearchResponse; + const items = data.results ?? []; if (opts.json) { console.log(JSON.stringify(items, null, 2)); return; } + if (items.length === 0) { + log.info("No documentation found for this query."); + return; + } + for (const item of items) { console.log(colors.bold(colors.cyan(`📄 ${item.title}`))); if (item.url) { console.log(` ${colors.underline(item.url)}`); } - const text = item.source?.content?.[0]?.text; - if (text) { - const snippet = text.length > 500 ? text.slice(0, 500) + "..." : text; + for (const snippet of item.snippets ?? []) { console.log(` ${snippet}`); } console.log(); diff --git a/cli/src/commands/flow/flow.ts b/cli/src/commands/flow/flow.ts index c0e94c478d..71d90af6fe 100644 --- a/cli/src/commands/flow/flow.ts +++ b/cli/src/commands/flow/flow.ts @@ -135,6 +135,31 @@ function warnAboutLocalPathScriptDivergence( const alreadySynced: string[] = []; +// Collect every script/sub-flow step path in a flow value — recursively through loops, +// branches, and the failure/preprocessor modules — for workspace-path validation. Unlike +// `collectPathScriptPaths` this also includes `type: "flow"` sub-flow steps. +function collectStepPaths(flowValue: any): string[] { + const paths: string[] = []; + const walk = (modules: any[] | undefined) => { + for (const m of modules ?? []) { + const v = m?.value; + if (!v) continue; + if ((v.type === "script" || v.type === "flow") && typeof v.path === "string") { + paths.push(v.path); + } + walk(v.modules); + walk(v.default); + for (const b of v.branches ?? []) walk(b?.modules); + // AI-agent tools are step-like and can carry script paths too. + walk(v.tools); + } + }; + walk(flowValue?.modules); + if (flowValue?.failure_module) walk([flowValue.failure_module]); + if (flowValue?.preprocessor_module) walk([flowValue.preprocessor_module]); + return paths; +} + export async function pushFlow( workspace: string, remotePath: string, @@ -190,6 +215,21 @@ export async function pushFlow( ); } + // Reject script/sub-flow steps whose path is not a workspace path (u/, f/, g/ or hub/). + // A flow.yaml generated from a feature-branch checkout can carry absolute local paths + // (e.g. /tmp/.../ops/scripts/...); pushed, they silently mis-resolve at runtime (#9751). + // The backend re-validates the same rule for every step type, so this is a fail-fast. + const badStepPaths = collectStepPaths(localFlow.value).filter( + (p) => p !== "" && !/^(u|f|g|hub)\//.test(p) + ); + if (badStepPaths.length > 0) { + throw new Error( + `Cannot push flow ${remotePath}: step(s) reference non-workspace path(s): ${badStepPaths.join(", ")}. ` + + `Flow step paths must be workspace paths (u/, f/, g/ or hub/), not absolute or local filesystem paths. ` + + `This usually means flow.yaml was generated with paths from a checkout directory.` + ); + } + const hasOnBehalfOf = (localFlow as any).has_on_behalf_of ?? !!localFlow.on_behalf_of_email; delete (localFlow as any).has_on_behalf_of; diff --git a/cli/src/commands/generate-metadata/generate-metadata.ts b/cli/src/commands/generate-metadata/generate-metadata.ts index 8517ac0912..7eab79b52d 100644 --- a/cli/src/commands/generate-metadata/generate-metadata.ts +++ b/cli/src/commands/generate-metadata/generate-metadata.ts @@ -2,7 +2,7 @@ import { Command } from "@cliffy/command"; import { Confirm } from "@cliffy/prompt/confirm"; import { colors } from "@cliffy/ansi/colors"; import { sep as SEP } from "node:path"; -import { GlobalOptions } from "../../types.ts"; +import { GlobalOptions, isDatatableMigrationPath } from "../../types.ts"; import { SyncOptions, mergeConfigWithConfigFile } from "../../core/conf.ts"; import { resolveWorkspace } from "../../core/context.ts"; import { requireLogin } from "../../core/auth.ts"; @@ -54,6 +54,8 @@ async function walkLocalScripts( (!isD && !exts.some((ext) => p.endsWith(ext))) || ignore(p, isD) || isFolderResourcePathAnyFormat(p) || + // Datatable migration `.sql` files aren't Windmill scripts. + isDatatableMigrationPath(p) || (isScriptModulePath(p) && !isModuleEntryPoint(p)), false, {}, @@ -221,6 +223,8 @@ function categorizeLocalFiles( } else if ( exts.some((ext) => p.endsWith(ext)) && !isFolderResourcePathAnyFormat(p) && + // Datatable migration `.sql` files aren't Windmill scripts. + !isDatatableMigrationPath(p) && !(isScriptModulePath(p) && !isModuleEntryPoint(p)) ) { scripts.push(p); diff --git a/cli/src/commands/init/init.ts b/cli/src/commands/init/init.ts index f1a10d7cc9..fc6e70c5b2 100644 --- a/cli/src/commands/init/init.ts +++ b/cli/src/commands/init/init.ts @@ -251,7 +251,7 @@ async function initAction(opts: InitOptions) { ); } - // Generate resource type namespace (only if a workspace was bound) + // Generate resource type namespace (needs a bound workspace) if (didBindWorkspace && boundProfile) { try { // Cache the bound profile so resolveWorkspace doesn't re-resolve and prompt again @@ -266,9 +266,51 @@ async function initAction(opts: InitOptions) { ); } } else { - log.info( - colors.gray("Skipped resource type namespace generation (no workspace bound). Run 'wmill workspace bind' then 'wmill init' to generate it.") + // generateRTNamespace resolves a workspace; its non-interactive + // multiple-workspaces path process.exit(-1)s (uncatchable), aborting init. + // So generate only for a single resolvable workspace (baseUrl + matching + // profile), passed via __secret_workspace to skip resolution; else skip. + const { readConfigFile, getWorkspaceNames, getEffectiveWorkspaceId } = + await import("../../core/conf.ts"); + const config = await readConfigFile({ warnIfMissing: false }); + const resolvable = getWorkspaceNames(config.workspaces).filter( + (n) => !!(config.workspaces as any)?.[n]?.baseUrl ); + let boundProfileForGen: Workspace | undefined; + if (resolvable.length === 1) { + const name = resolvable[0]; + const entry = (config.workspaces as any)[name]; + let normalizedBaseUrl: string | undefined; + try { + normalizedBaseUrl = new URL(entry.baseUrl).toString(); + } catch { + normalizedBaseUrl = undefined; + } + if (normalizedBaseUrl) { + const workspaceId = getEffectiveWorkspaceId(name, entry); + const profiles = await allWorkspaces(opts.configDir); + boundProfileForGen = profiles.find( + (p) => p.remote === normalizedBaseUrl && p.workspaceId === workspaceId + ); + } + } + if (boundProfileForGen) { + try { + const rtOpts = { ...opts } as GlobalOptions; + (rtOpts as any).__secret_workspace = boundProfileForGen; + await generateRTNamespace(rtOpts); + } catch (error) { + log.warn( + `Could not pull resource types and generate TypeScript namespace: ${ + error instanceof Error ? error.message : error + }` + ); + } + } else { + log.info( + colors.gray("Skipped resource type namespace generation (no workspace bound). Run 'wmill workspace bind' then 'wmill init' to generate it.") + ); + } } } diff --git a/cli/src/commands/init/template.ts b/cli/src/commands/init/template.ts index 210aae5aca..b0c83d7039 100644 --- a/cli/src/commands/init/template.ts +++ b/cli/src/commands/init/template.ts @@ -180,8 +180,12 @@ export const CONFIG_REFERENCE: ConfigOption[] = [ additionalProperties: WORKSPACE_CONFIG_SCHEMA, section: "Workspace bindings", sectionNote: "Map workspace names to Windmill instances and override settings per workspace.\nThe key is a human-friendly workspace name. gitBranch and workspaceId default to the key name.", - templateValue: "\n {{BRANCH}}: {}", + // Empty ` {}`, not a live `: {}` stub: a stub has no baseUrl yet + // counts as a configured workspace, breaking auto-selection and making a + // later `workspace bind` ambiguous. The example stays commented. + templateValue: " {}", example: [ + " # {{BRANCH}}:", "{{BASEURL_LINE}}", "{{WORKSPACE_ID_LINE}}", " # gitBranch: main # git branch (defaults to workspace name)", diff --git a/cli/src/commands/pipeline/boundedCascade.ts b/cli/src/commands/pipeline/boundedCascade.ts new file mode 100644 index 0000000000..69166726c5 --- /dev/null +++ b/cli/src/commands/pipeline/boundedCascade.ts @@ -0,0 +1,355 @@ +// Bounded-cascade graph engine for `wmill pipeline run --to`. +// +// MIRROR of frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts — +// the two have no shared import path (frontend is a separate package), so keep +// them in sync. The grammar is intentionally tiny: there is no dbt-style +// `--select` string. The user names a start (a schedule / manual root) and one +// or more end nodes; the run is the "path between" them: +// +// descendants(start) ∩ (ancestors(ends) ∪ ends) ∪ {start} +// +// Node ids: assets `${kind}:${path}` (e.g. `datatable:main/raw`); runnables +// `script:${path}`. Operates on the asset-graph payload shape used by +// pipeline.ts. + +export type BCGraph = { + runnables: { + path: string; + usage_kind: "script" | "flow" | "job"; + // `// partitioned ` on the script (daily/hourly/weekly/monthly/dynamic). + // Emitted by both the deployed graph endpoint and the local builder. + partition_kind?: string; + // `// macros` library: non-empty on a workspace macro library (deployed + // graph only). Definition-only — never a runnable cascade step. + macros?: { name: string }[]; + }[]; + assets: { kind: string; path: string }[]; + edges: { + runnable_kind: string; + runnable_path: string; + asset_kind: string; + asset_path: string; + access_type?: "r" | "w" | "rw"; + }[]; + triggers: ( + | { + trigger_kind: "asset"; + asset_kind: string; + asset_path: string; + runnable_kind: string; + runnable_path: string; + } + | { trigger_kind: string; runnable_kind: string; runnable_path: string } + )[]; + // `// data_test` ordering edges (HD-1): the referenced asset's producer must + // materialize before the tested script runs. Fed into the lineage DAG so a + // bounded/cold cascade orders the referenced dimension first. Optional — the + // deployed graph omits it when empty, and older local graphs never emit it. + test_edges?: { + producer_kind: string; + producer_path: string; + runnable_kind: string; + runnable_path: string; + asset_kind: string; + asset_path: string; + }[]; +}; + +export const SCRIPT_PREFIX = "script:"; +export const scriptNodeId = (path: string): string => `${SCRIPT_PREFIX}${path}`; +export const isScriptNode = (id: string): boolean => id.startsWith(SCRIPT_PREFIX); +export const scriptPathOf = (id: string): string => id.slice(SCRIPT_PREFIX.length); +const assetNodeId = (kind: string, path: string): string => `${kind}:${path}`; + +// Native trigger kinds whose scripts must NOT be auto-run by the CLI cascade: +// event triggers fan out per external event, and `webhook`/`data_upload` are UI +// entrypoints that need caller-supplied input (a request body / an uploaded +// S3Object) — previewing any of them with empty args runs the wrong thing. +// The deployed graph omits `webhook`/`data_upload` rows, but the local graph +// emits them, so a `// on data_upload` script would otherwise read as a manual +// root and get auto-run without its upload argument. +const NON_AUTORUN_TRIGGER_KINDS = new Set([ + "kafka", + "mqtt", + "nats", + "postgres", + "sqs", + "gcp", + "email", + "webhook", + "data_upload", +]); + +/** Resolve an asset URI (`datatable://x`, `s3://b/k`, …) to its node id. */ +export function assetUriToNodeId(uri: string): string | undefined { + const m = uri.match(/^([a-z0-9_]+):\/\/(.+)$/i); + if (!m) return undefined; + const prefix = m[1].toLowerCase(); + const kind = prefix === "s3" ? "s3object" : prefix; + // Mirror Rust `parse_asset_syntax`: strip all leading slashes from S3 keys so a + // `--to s3:///exports/x` token resolves to the canonical graph node + // `s3object:exports/x` (default storage), same as `s3://exports/x`, and a + // canonical key never starts with `/`. + const path = kind === "s3object" ? m[2].replace(/^\/+/, "") : m[2]; + return `${kind}:${path}`; +} + +export type LineageDag = { + down: Map>; + up: Map>; + nodes: Set; +}; + +function addEdge(dag: LineageDag, a: string, b: string) { + if (a === b) return; + dag.nodes.add(a); + dag.nodes.add(b); + (dag.down.get(a) ?? dag.down.set(a, new Set()).get(a)!).add(b); + (dag.up.get(b) ?? dag.up.set(b, new Set()).get(b)!).add(a); +} + +/** Unified upstream→downstream lineage DAG over scripts ∪ assets. */ +export function buildLineageDag(g: BCGraph): LineageDag { + const dag: LineageDag = { down: new Map(), up: new Map(), nodes: new Set() }; + for (const r of g.runnables ?? []) { + if (r.usage_kind === "script") dag.nodes.add(scriptNodeId(r.path)); + } + for (const a of g.assets ?? []) dag.nodes.add(assetNodeId(a.kind, a.path)); + for (const e of g.edges ?? []) { + if (e.runnable_kind !== "script") continue; + const aid = assetNodeId(e.asset_kind, e.asset_path); + const access = e.access_type ?? "r"; + if (access === "w" || access === "rw") { + addEdge(dag, scriptNodeId(e.runnable_path), aid); // producer + } else if (access === "r") { + addEdge(dag, aid, scriptNodeId(e.runnable_path)); // pure reader + } + } + for (const t of g.triggers ?? []) { + if (t.trigger_kind !== "asset" || t.runnable_kind !== "script") continue; + const at = t as Extract; + addEdge(dag, assetNodeId(at.asset_kind, at.asset_path), scriptNodeId(at.runnable_path)); + } + // Data-test ordering edges: route through the referenced asset node so the + // existing producer → asset write edge extends into producer → asset → testing + // script (mirrors frontend boundedCascade.ts) — the tested script runs after + // the referenced dimension materializes. + for (const t of g.test_edges ?? []) { + if (t.runnable_kind !== "script") continue; + addEdge(dag, assetNodeId(t.asset_kind, t.asset_path), scriptNodeId(t.runnable_path)); + } + return dag; +} + +function closure(adj: Map>, start: string): Set { + const seen = new Set(); + const queue = [start]; + while (queue.length > 0) { + const cur = queue.shift()!; + for (const n of adj.get(cur) ?? []) { + if (seen.has(n)) continue; + seen.add(n); + queue.push(n); + } + } + // A cycle back to `start` would have re-added it; the contract excludes + // the node itself. + seen.delete(start); + return seen; +} + +export const descendants = (dag: LineageDag, n: string): Set => closure(dag.down, n); +export const ancestors = (dag: LineageDag, n: string): Set => closure(dag.up, n); + +/** + * Nodes reachable from `starts` over the lineage DAG, treating `barriers` as cut + * points: a barrier node is neither included NOR traversed through. So a node + * reachable ONLY via a barrier is excluded, while one also reachable via another + * path stays. Used for whole-pipeline runs to keep event handlers AND their + * event-only downstream closure out (subtracting only the handler would leave a + * consumer whose producer was skipped, which topoOrder would then run with + * missing/stale inputs). + */ +export function reachableCutting( + dag: LineageDag, + starts: Iterable, + barriers: Set, +): Set { + const seen = new Set(); + const queue: string[] = []; + for (const s of starts) { + if (barriers.has(s) || seen.has(s)) continue; + seen.add(s); + queue.push(s); + } + while (queue.length > 0) { + const n = queue.shift()!; + for (const next of dag.down.get(n) ?? []) { + if (barriers.has(next) || seen.has(next)) continue; + seen.add(next); + queue.push(next); + } + } + return seen; +} + +export type BoundedResult = { + nodes: Set; + reachableEnds: string[]; + droppedEnds: string[]; +}; + +/** Path-between node set for `start` and `ends` (inclusive). */ +export function boundedSet(dag: LineageDag, start: string, ends: string[]): BoundedResult { + const downSet = new Set(descendants(dag, start)); + downSet.add(start); + const reachableEnds = ends.filter((e) => downSet.has(e)); + const droppedEnds = ends.filter((e) => !downSet.has(e)); + if (reachableEnds.length === 0) { + return { nodes: new Set([start]), reachableEnds, droppedEnds }; + } + const upClosure = new Set(); + for (const e of reachableEnds) { + upClosure.add(e); + for (const a of ancestors(dag, e)) upClosure.add(a); + } + const nodes = new Set(); + for (const n of downSet) if (upClosure.has(n)) nodes.add(n); + nodes.add(start); + return { nodes, reachableEnds, droppedEnds }; +} + +/** Script node ids eligible to start a bounded run. */ +export function validStarts(g: BCGraph): Set { + const subscribers = new Set(); + const scheduleScripts = new Set(); + const nonAutorunScripts = new Set(); + for (const t of g.triggers ?? []) { + if (t.runnable_kind !== "script") continue; + if (t.trigger_kind === "asset") subscribers.add(t.runnable_path); + else if (t.trigger_kind === "schedule") scheduleScripts.add(t.runnable_path); + else if (NON_AUTORUN_TRIGGER_KINDS.has(t.trigger_kind)) nonAutorunScripts.add(t.runnable_path); + } + const out = new Set(); + for (const r of g.runnables ?? []) { + if (r.usage_kind !== "script") continue; + const p = r.path; + if (scheduleScripts.has(p)) out.add(scriptNodeId(p)); + else if (!subscribers.has(p) && !nonAutorunScripts.has(p)) out.add(scriptNodeId(p)); + } + return out; +} + +/** + * Script node ids eligible as an EXPLICIT bounded-run start from *anywhere* in + * the DAG (dbt's `--select model+`): every script that can run with empty args — + * i.e. all scripts except non-autorun-trigger ones (kafka/mqtt/…/webhook/ + * data_upload, which fan out per event or need caller-supplied input). Unlike + * `validStarts` (schedule/manual roots only), this INCLUDES mid-DAG asset + * subscribers and pure readers, so `--from ` runs that node plus its + * transitive downstream WITHOUT re-running upstream. Roots stay a subset of this + * set. A non-autorun handler still qualifies once it's `--upload`-bound (handled + * by the caller, which unions in the bound node ids). + */ +export function validFromStarts(g: BCGraph): Set { + const nonAutorun = nonAutorunTriggerScripts(g); + // Seed with the schedule/manual roots: `validStarts` lets a schedule identity + // win over a secondary non-autorun trigger (a `// on schedule` + `// on + // data_upload` script IS a scheduled root), so a root must stay `--from`- + // eligible even though it's also in `nonAutorunTriggerScripts`. + const out = new Set(validStarts(g)); + for (const r of g.runnables ?? []) { + if (r.usage_kind !== "script") continue; + const id = scriptNodeId(r.path); + if (!nonAutorun.has(id)) out.add(id); + } + return out; +} + +/** + * Script node ids that carry a trigger requiring caller-supplied input or + * per-event fanout (kafka/mqtt/nats/postgres/sqs/gcp/email/webhook/data_upload). + * These can't be run with empty args, so a whole-pipeline run must exclude them + * even when they're a lineage descendant of a valid start (not just a root). + */ +export function nonAutorunTriggerScripts(g: BCGraph): Set { + const out = new Set(); + for (const t of g.triggers ?? []) { + if (t.runnable_kind === "script" && NON_AUTORUN_TRIGGER_KINDS.has(t.trigger_kind)) { + out.add(scriptNodeId(t.runnable_path)); + } + } + return out; +} + +/** Project a node-id set to the script paths it contains. */ +export function scriptsOf(nodes: Iterable): string[] { + const out: string[] = []; + for (const id of nodes) if (isScriptNode(id)) out.push(scriptPathOf(id)); + return out; +} + +/** + * Resolve a CLI `--to` / `--from` token to a node id, or undefined if it + * matches nothing. Asset URIs (`kind://path`) resolve to the asset node; a bare + * token matches a runnable by exact path or by short (last-segment) name. + */ +export function resolveToken(g: BCGraph, token: string): string | undefined { + if (token.includes("://")) { + const id = assetUriToNodeId(token); + return id && g.assets.some((a) => `${a.kind}:${a.path}` === id) ? id : undefined; + } + const scripts = (g.runnables ?? []).filter((r) => r.usage_kind === "script"); + const exact = scripts.find((r) => r.path === token); + if (exact) return scriptNodeId(exact.path); + const byShort = scripts.filter((r) => (r.path.split("/").pop() ?? r.path) === token); + return byShort.length === 1 ? scriptNodeId(byShort[0].path) : undefined; +} + +/** + * Topological order of `scripts` over the in-set producer→subscriber edges + * (assets collapsed). Scripts on a cycle are returned in `cyclic` and excluded + * from `order`. Serial-run friendly: every script comes after its in-set + * upstreams. + */ +export function topoOrder( + g: BCGraph, + scripts: Set, +): { order: string[]; cyclic: string[] } { + const dag = buildLineageDag(g); + const down = new Map>(); + const indegree = new Map(); + for (const s of scripts) indegree.set(s, 0); + // One-hop (through a single asset) script→script edges, restricted to the set. + for (const s of scripts) { + const sid = scriptNodeId(s); + const oneHop = new Set(); + for (const asset of dag.down.get(sid) ?? []) { + for (const sub of dag.down.get(asset) ?? []) { + if (isScriptNode(sub)) { + const p = scriptPathOf(sub); + if (p !== s && scripts.has(p)) oneHop.add(p); + } + } + } + if (oneHop.size > 0) { + down.set(s, oneHop); + for (const p of oneHop) indegree.set(p, (indegree.get(p) ?? 0) + 1); + } + } + const ready = [...scripts].filter((s) => (indegree.get(s) ?? 0) === 0); + const remaining = new Map(indegree); + const order: string[] = []; + while (ready.length > 0) { + const n = ready.shift()!; + order.push(n); + for (const p of down.get(n) ?? []) { + const d = (remaining.get(p) ?? 0) - 1; + remaining.set(p, d); + if (d === 0) ready.push(p); + } + } + const orderedSet = new Set(order); + const cyclic = [...scripts].filter((s) => !orderedSet.has(s)); + return { order, cyclic }; +} diff --git a/cli/src/commands/pipeline/dev.ts b/cli/src/commands/pipeline/dev.ts new file mode 100644 index 0000000000..3d11dfc2bd --- /dev/null +++ b/cli/src/commands/pipeline/dev.ts @@ -0,0 +1,290 @@ +// `wmill pipeline dev [folder]` — live-preview a data pipeline from local files. +// +// The pipeline analog of `wmill dev` / `wmill app dev`: watch a folder of +// `// pipeline` scripts, rebuild the asset graph from the working tree on every +// save, and push it over a WebSocket to the `/pipeline_dev` page, which renders +// the same graph editor the UI uses and runs the cascade via preview (no deploy). +// Editing stays in the user's own editor; the page live-reloads. + +import { Command } from "@cliffy/command"; +import { colors } from "@cliffy/ansi/colors"; +import * as http from "node:http"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import { createHash, randomBytes } from "node:crypto"; +import process from "node:process"; +import * as open from "open"; +import { WebSocket, WebSocketServer } from "ws"; +import * as log from "../../core/log.ts"; +import { GlobalOptions } from "../../types.ts"; +import { requireLogin } from "../../core/auth.ts"; +import { resolveWorkspace } from "../../core/context.ts"; +import { + mergeConfigWithConfigFile, + type SyncOptions, +} from "../../core/conf.ts"; +import { listSyncCodebases } from "../../utils/codebase.ts"; +import { resolveBindPort } from "../../utils/port-probe.ts"; +import { getConfigDirPath } from "../../../windmill-utils-internal/src/config/config.ts"; +import { buildLocalPipelineGraph, workspaceRoot } from "./localGraph.ts"; + +const PORT = 3201; +// Bind loopback only: each WS frame ships the folder's full script source +// (`scripts[].content` + `temp_script_refs`) with no auth, so it must not be +// reachable from the LAN. The webview connects via `ws://localhost`, and an SSH +// `-L` / devbox port-forward targets 127.0.0.1 on the host, so both still work. +const LISTEN_HOST = "127.0.0.1"; + +interface PipelineDevOpts extends GlobalOptions, SyncOptions { + port?: number; + open?: boolean; + defaultTs?: "bun" | "deno"; + frontend?: string; +} + +// The WS token gates access to local source. Persist it under the user-private +// config dir (0600) so a `pipeline dev` restart reuses it — an already-open +// `/pipeline_dev` page auto-reconnecting with the token from its URL then +// survives the restart, instead of every upgrade being rejected by +// `verifyClient` until the freshly printed URL is reopened. Scoped by +// remote+workspace+root+folder+port (NOT port alone): a same-session restart +// reconnects, but any *different* session — another folder, workspace, remote, or +// local checkout on the same port — gets a different token, so a stale tab can't +// reconnect and receive another session's source. Falls back to an ephemeral +// token if the config dir can't be read/written. +async function stableWsToken( + remote: string, + workspaceId: string, + root: string, + folder: string, + port: number, +): Promise { + // Hash the tuple (NUL-delimited so no value can spoof the boundary) → a + // collision-resistant, filesystem-safe key. A plain sanitized join would let + // different values collide (`a/b` and `a_b` → same name), which would reuse a + // token across sessions and reintroduce the cross-session leak. + const key = createHash("sha256") + .update(`${remote}\0${workspaceId}\0${root}\0${folder}\0${port}`) + .digest("hex") + .slice(0, 32); + let tokenFile: string | undefined; + try { + tokenFile = path.join(await getConfigDirPath(), `pipeline-dev-${key}.token`); + const existing = fs.readFileSync(tokenFile, "utf-8").trim(); + if (existing) return existing; + } catch { + // no reusable token file yet (or config dir unavailable) — mint a fresh one + } + const token = randomBytes(24).toString("base64url"); + if (tokenFile) { + try { + fs.writeFileSync(tokenFile, token, { mode: 0o600 }); + } catch { + // best-effort persistence; fall back to the in-memory token + } + } + return token; +} + +// Resolve the target folder: explicit arg, else auto-detect when cwd sits inside +// `f//…` (the supported `cd f/my_pipeline && wmill pipeline dev` form). +function resolveFolder(root: string, folderArg?: string): string | undefined { + if (folderArg) return folderArg.replace(/^f\//, "").replace(/\/$/, ""); + const rel = path.relative(root, process.cwd()).replaceAll("\\", "/"); + if (rel === "" || rel.startsWith("..")) return undefined; + const segs = rel.split("/"); + if (segs[0] === "f" && segs[1]) return segs[1]; + return undefined; +} + +async function dev(opts: PipelineDevOpts, folderArg?: string) { + const root = workspaceRoot(); + const folder = resolveFolder(root, folderArg); + if (!folder) { + log.error( + colors.red( + "Could not determine the pipeline folder. Pass it explicitly " + + "(`wmill pipeline dev `) or run from inside an `f/` directory.", + ), + ); + process.exit(1); + } + const folderDir = path.join(root, "f", folder); + if (!fs.existsSync(folderDir)) { + log.error(colors.red(`Folder not found on disk: ${folderDir}`)); + process.exit(1); + } + + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + const merged = await mergeConfigWithConfigFile(opts); + const codebases = await listSyncCodebases(merged); + + // Resolve relative imports from local (not-yet-deployed) content for previews. + // Snapshot at startup like `wmill dev`; restart to refresh. Degrades to + // undefined on older backends. + let tempScriptRefs: Record | undefined; + try { + const { buildPreviewTempScriptRefs } = await import( + "../generate-metadata/generate-metadata.ts" + ); + tempScriptRefs = await buildPreviewTempScriptRefs( + workspace, + merged, + codebases, + { kind: "all" }, + ); + } catch { + // best-effort + } + + const EMPTY_GRAPH = { runnables: [], assets: [], edges: [], triggers: [] }; + async function buildBundle() { + const { graph, scripts } = await buildLocalPipelineGraph({ + root, + folder: folder!, + defaultTs: merged.defaultTs, + }); + return { + type: "pipeline" as const, + folder, + graph, + scripts, + temp_script_refs: tempScriptRefs, + }; + } + + // Don't let a transient build error (e.g. a half-written file) abort startup — + // serve an empty graph and recover on the next save. + let current: Awaited>; + try { + current = await buildBundle(); + } catch (e: any) { + log.error(colors.red(`Initial graph build failed: ${e.message}`)); + current = { type: "pipeline", folder, graph: EMPTY_GRAPH, scripts: [], temp_script_refs: tempScriptRefs }; + } + log.info( + colors.blue( + `Watching f/${folder} — ${current.scripts.length} pipeline script(s)`, + ), + ); + + const clients = new Set(); + function broadcast() { + const msg = JSON.stringify(current); + for (const ws of clients) { + if (ws.readyState === WebSocket.OPEN) ws.send(msg); + } + } + + // Debounced rebuild on any change under the folder. + let timer: ReturnType | undefined; + const watcher = fs.watch(folderDir, { recursive: true }); + watcher.on("change", (_ev, filename) => { + if (filename && filename.toString().endsWith(".lock")) return; + if (timer) clearTimeout(timer); + timer = setTimeout(async () => { + timer = undefined; + try { + current = await buildBundle(); + log.info(colors.cyan(`↻ rebuilt graph (${current.scripts.length} scripts)`)); + broadcast(); + } catch (e: any) { + log.error(colors.red(`Failed to rebuild pipeline graph: ${e.message}`)); + } + }, 150); + }); + watcher.on("error", (e) => log.error(colors.red(`Watcher error: ${e.message}`))); + + const port = await resolveBindPort(opts.port ?? PORT, "wmill pipeline dev", { + info: (m) => log.info(m), + warn: (m) => log.warn(m), + }); + + const server = http.createServer((_req, res) => { + res.writeHead(200); + res.end(); + }); + // Loopback bind keeps the LAN out, but any browser tab can still open a + // `ws://localhost:` connection — and each frame ships the folder's full + // script source. Gate the upgrade on an unguessable per-session token (carried + // in the dev-page URL) so a stray page on the predictable dev port can't + // exfiltrate the source. base64url → safe as a query value. + // Stable across restarts (scoped to remote+workspace+root+folder+port) so an + // already-open page reconnects after a CLI restart (see stableWsToken), not + // just after a transient WS drop. + const wsToken = await stableWsToken( + workspace.remote, + workspace.workspaceId, + root, + folder, + port, + ); + const wss = new WebSocketServer({ + server, + verifyClient: (info) => { + try { + const u = new URL(info.req.url ?? "", "http://localhost"); + return u.searchParams.get("token") === wsToken; + } catch { + return false; + } + }, + }); + wss.on("connection", (ws: WebSocket) => { + clients.add(ws); + // Push the current bundle immediately so the page renders without waiting + // for the first file change. + try { + ws.send(JSON.stringify(current)); + } catch { + // ignore + } + ws.on("close", () => clients.delete(ws)); + ws.on("error", () => clients.delete(ws)); + }); + + // The `/pipeline_dev` page is served by the frontend, not the backend. By + // default we open it on the workspace remote, but that 404s on a remote whose + // deployed frontend predates this route — `--frontend` points the page at a + // locally-run frontend (`REMOTE= npm run dev`) while the API/token + // still target the remote. Normalize to a single trailing slash either way. + const pageBase = (opts.frontend ?? workspace.remote).replace(/\/?$/, "/"); + const url = + `${pageBase}pipeline_dev?workspace=${workspace.workspaceId}` + + `&wm_token=${workspace.token}&folder=${encodeURIComponent(folder)}&port=${port}` + + `&ws_token=${encodeURIComponent(wsToken)}`; + + server.listen(port, LISTEN_HOST, () => { + log.info(colors.green.bold(`🚀 Pipeline dev server on ws://localhost:${port}/ws`)); + log.info(colors.gray(`Open: ${url}`)); + if (opts.open !== false) { + open.default(url).catch((e: any) => + log.warn(colors.yellow(`Failed to open browser: ${e.message}`)), + ); + } + }); + + process.on("SIGINT", () => { + log.info(colors.yellow("\n🛑 Shutting down…")); + watcher.close(); + for (const ws of clients) ws.close(); + server.close(); + process.exit(0); + }); +} + +const command = new Command() + .description( + "Live-preview a data pipeline from local files: watch an `f/` of `// pipeline` scripts, push the working-tree graph to the dev page, and run the cascade via preview (no deploy).", + ) + .arguments("[folder:string]") + .option("--port ", "Port for the dev WebSocket server.") + .option("--no-open", "Do not open the browser automatically.") + .option( + "--frontend ", + "Origin serving the /pipeline_dev page (e.g. http://localhost:3000 for a locally-run frontend). Defaults to the workspace remote; use it when the remote's deployed frontend predates the dev page.", + ) + .action(dev as any); + +export default command; diff --git a/cli/src/commands/pipeline/docs.ts b/cli/src/commands/pipeline/docs.ts new file mode 100644 index 0000000000..abee64cbf3 --- /dev/null +++ b/cli/src/commands/pipeline/docs.ts @@ -0,0 +1,296 @@ +// `wmill pipeline docs ` — generate a PIPELINE.md (+ AGENTS.md / CLAUDE.md +// pointer) describing a folder's pipeline so an editor or agentic loop has the +// same context the UI surfaces: the asset DAG, per-script triggers/IO, and the +// schemas of the datatables the pipeline touches. Mirrors the app docs pattern +// (`app/generate_agents.ts:regenerateAgentDocs`) but scoped to a pipeline folder. + +import { writeFile } from "node:fs/promises"; +import { existsSync, readFileSync } from "node:fs"; +import * as path from "node:path"; +import { OpenAPI } from "../../../gen/index.ts"; +import * as wmill from "../../../gen/services.gen.ts"; +import { requireLogin } from "../../core/auth.ts"; +import { resolveWorkspace } from "../../core/context.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; +import * as log from "../../core/log.ts"; +import { colors } from "@cliffy/ansi/colors"; +import { GlobalOptions } from "../../types.ts"; +import { + type AssetGraph, + buildLocalPipelineGraph, + workspaceRoot, +} from "./localGraph.ts"; + +const ASSET_KINDS = "s3object,ducklake,datatable,volume"; + +function assetUri(kind: string, p: string): string { + const prefix = kind === "s3object" ? "s3" : kind; + return `${prefix}://${p}`; +} + +async function fetchDeployedGraph( + workspaceId: string, + folder: string, +): Promise { + const res = await fetch( + `${OpenAPI.BASE}/w/${workspaceId}/assets/graph?folder=${encodeURIComponent(folder)}&asset_kinds=${ASSET_KINDS}`, + { headers: { Authorization: `Bearer ${OpenAPI.TOKEN}` } }, + ); + if (!res.ok) { + throw new Error(`GET assets/graph -> ${res.status}: ${await res.text()}`); + } + return (await res.json()) as AssetGraph; +} + +// Render the pipeline graph as a markdown document. +export function generatePipelineMarkdown( + folder: string, + graph: AssetGraph, + datatableSchemas: any[], + local: boolean, +): string { + const writesByScript = new Map(); + const readsByScript = new Map(); + for (const e of graph.edges) { + if (e.runnable_kind !== "script") continue; + const uri = assetUri(e.asset_kind, e.asset_path); + if (e.access_type === "w" || e.access_type === "rw") { + (writesByScript.get(e.runnable_path) ?? writesByScript.set(e.runnable_path, []).get(e.runnable_path)!).push(uri); + } + if (e.access_type === "r" || e.access_type === "rw" || e.access_type === undefined) { + (readsByScript.get(e.runnable_path) ?? readsByScript.set(e.runnable_path, []).get(e.runnable_path)!).push(uri); + } + } + const onByScript = new Map(); + const nativeByScript = new Map(); + for (const t of graph.triggers) { + if (t.runnable_kind !== "script") continue; + if (t.trigger_kind === "asset") { + const at = t as Extract; + (onByScript.get(at.runnable_path) ?? onByScript.set(at.runnable_path, []).get(at.runnable_path)!).push(assetUri(at.asset_kind, at.asset_path)); + } else { + (nativeByScript.get(t.runnable_path) ?? nativeByScript.set(t.runnable_path, []).get(t.runnable_path)!).push(t.trigger_kind); + } + } + + // `// macros` libraries are definition-only nodes (not runnable pipeline + // steps), so they get their own section below and are excluded from the + // per-script listing + the script count. + const macroLibs = graph.runnables + .filter((r) => r.usage_kind === "script" && (r.macros?.length ?? 0) > 0) + .sort((a, b) => a.path.localeCompare(b.path)); + const macroLibPaths = new Set(macroLibs.map((r) => r.path)); + const macroConsumersByLib = new Map(); + for (const me of graph.macro_edges ?? []) { + (macroConsumersByLib.get(me.lib_path) ?? macroConsumersByLib.set(me.lib_path, []).get(me.lib_path)!).push({ + consumer: me.consumer_path, + names: me.macro_names, + viaUse: me.via_use, + }); + } + + const scripts = graph.runnables + .filter((r) => r.usage_kind === "script" && !macroLibPaths.has(r.path)) + .map((r) => r.path) + .sort(); + + let md = `# Pipeline \`f/${folder}\` + +${local ? "_Built from local working-tree files (`// pipeline` scripts)._" : "_Built from the deployed workspace asset graph._"} + +A data pipeline is a folder of scripts marked \`// pipeline\` and wired by asset +annotations. A script subscribes to upstream data with \`// on \` and +produces data by reading/writing assets in its body (\`datatable://\`, +\`ducklake://\`, \`s3://\`, \`volume://\`). The cascade runs a producer, then every +downstream subscriber, in topological order. + +- **${scripts.length}** script${scripts.length === 1 ? "" : "s"} · **${graph.assets.length}** asset${graph.assets.length === 1 ? "" : "s"}${macroLibs.length > 0 ? ` · **${macroLibs.length}** macro librar${macroLibs.length === 1 ? "y" : "ies"}` : ""} + +## Scripts + +`; + + for (const s of scripts) { + const on = onByScript.get(s) ?? []; + const native = nativeByScript.get(s) ?? []; + const writes = [...new Set(writesByScript.get(s) ?? [])].sort(); + const reads = [...new Set(readsByScript.get(s) ?? [])].sort(); + md += `### \`${s}\`\n\n`; + if (native.length) md += `- **Triggers:** ${native.map((n) => `\`${n}\``).join(", ")}\n`; + if (on.length) md += `- **On (subscribes to):** ${on.map((u) => `\`${u}\``).join(", ")}\n`; + if (reads.length) md += `- **Reads:** ${reads.map((u) => `\`${u}\``).join(", ")}\n`; + if (writes.length) md += `- **Writes:** ${writes.map((u) => `\`${u}\``).join(", ")}\n`; + if (!native.length && !on.length && !reads.length && !writes.length) { + md += `- _No declared triggers or asset IO._\n`; + } + md += `\n`; + } + + // Macro libraries: `// macros` scripts whose macros are injected into consuming + // DuckDB scripts at run time. List each library's signatures and its callers so + // an agent discovers the reuse layer instead of re-inlining the logic. + if (macroLibs.length > 0) { + md += `## Macro libraries\n\n`; + md += `\`// macros\` DuckDB libraries. Their \`CREATE MACRO\` definitions are injected as\nTEMP macros into consuming scripts at run time — call a macro by name, or force the\nwhole library in with \`// use \` (needed for macros only reached via dynamic SQL).\n\n`; + for (const lib of macroLibs) { + md += `### \`${lib.path}\`\n\n`; + for (const m of lib.macros ?? []) { + md += `- \`${m.name}(${m.params ?? ""})\`${m.is_table ? " → TABLE" : ""}\n`; + } + const consumers = [...(macroConsumersByLib.get(lib.path) ?? [])].sort((a, b) => + a.consumer.localeCompare(b.consumer), + ); + if (consumers.length > 0) { + md += `- **Used by:**\n`; + for (const c of consumers) { + md += ` - \`${c.consumer}\` (${c.viaUse ? "via \`// use\`" : `calls ${c.names.map((n) => `\`${n}\``).join(", ")}`})\n`; + } + } + md += `\n`; + } + } + + // Datatable schemas, restricted to datatables this pipeline references. + const referencedDatatables = new Set( + graph.assets.filter((a) => a.kind === "datatable").map((a) => a.path.split("/")[0]), + ); + const relevant = (datatableSchemas ?? []).filter((dt: any) => referencedDatatables.has(dt.datatable_name)); + if (relevant.length > 0) { + md += `## Datatable schemas\n\n`; + for (const dt of relevant) { + md += `### Datatable \`${dt.datatable_name}\`\n\n`; + if (dt.error) { + md += `> ⚠️ ${dt.error}\n\n`; + continue; + } + for (const [schemaName, tables] of Object.entries(dt.schemas ?? {})) { + for (const [tableName, columns] of Object.entries(tables as Record)) { + const ref = schemaName === "public" + ? `${dt.datatable_name}/${tableName}` + : `${dt.datatable_name}/${schemaName}:${tableName}`; + md += `- \`datatable://${ref}\`\n`; + for (const [col, type] of Object.entries(columns as Record)) { + md += ` - \`${col}\`: ${type}\n`; + } + } + } + md += `\n`; + } + } + + md += `## Working with this pipeline + +- **Inspect the graph:** \`wmill pipeline show ${folder} --local\` +- **Run the cascade locally (no deploy):** \`wmill pipeline run ${folder} --local\` + (optionally \`--from %sveltekit.head% @@ -56,7 +156,7 @@ /> (undefined) export async function loadCopilot(workspace: string) { + const token = ++loadCopilotToken workspaceAIClients.init(workspace) try { const info = await WorkspaceService.getCopilotInfo({ workspace }) + if (token !== loadCopilotToken) return setCopilotInfo(info) + copilotWorkspace.set(workspace) } catch (err) { + if (token !== loadCopilotToken) return setCopilotInfo({}) + copilotWorkspace.set(workspace) console.error('Could not get copilot info', err) } } @@ -204,3 +219,12 @@ export function getCombinedCustomPrompt(mode: string): string | undefined { return prompts.join('\n\n') } + +// Like getCombinedCustomPrompt but keeps the workspace and user slices separate so the +// Global system prompt can label them distinctly — only the user slice is editable by the +// update_user_instructions tool. +export function getCustomPromptParts(mode: string): { workspace?: string; user?: string } { + const workspace = get(copilotInfo).customPrompts?.[mode]?.trim() || undefined + const user = getUserCustomPrompts()[mode]?.trim() || undefined + return { workspace, user } +} diff --git a/frontend/src/lib/assets/app.css b/frontend/src/lib/assets/app.css index ae7efaaf60..23bf45c7fa 100644 --- a/frontend/src/lib/assets/app.css +++ b/frontend/src/lib/assets/app.css @@ -129,6 +129,36 @@ scrollbar-width: none; /* Firefox */ } } + + /* Subtle scrollbar: a thin, rounded thumb that only appears on hover, on both + axes. Shared by ScrollableX (tab strips, code blocks) and the AI chat. Size + via the `--wm-scrollbar-size` var (default 6px). Higher specificity than the + app-wide `*::-webkit-scrollbar`, so it overrides it. */ + .scrollbar-subtle { + scrollbar-width: thin; + scrollbar-color: transparent transparent; + } + .scrollbar-subtle:hover { + scrollbar-color: rgb(var(--color-text-hint) / 0.4) transparent; + } + .scrollbar-subtle::-webkit-scrollbar { + width: var(--wm-scrollbar-size, 6px); + height: var(--wm-scrollbar-size, 6px); + } + .scrollbar-subtle::-webkit-scrollbar-track { + background: transparent; + } + .scrollbar-subtle::-webkit-scrollbar-thumb { + background: transparent; + border-radius: 9999px; + transition: background-color 0.15s; + } + .scrollbar-subtle:hover::-webkit-scrollbar-thumb { + background: rgb(var(--color-text-hint) / 0.35); + } + .scrollbar-subtle:hover::-webkit-scrollbar-thumb:hover { + background: rgb(var(--color-text-secondary) / 0.55); + } } .driver-popover-title { diff --git a/frontend/src/lib/components/AIProviderPicker.svelte b/frontend/src/lib/components/AIProviderPicker.svelte index f6f2a4f43c..4392a2d513 100644 --- a/frontend/src/lib/components/AIProviderPicker.svelte +++ b/frontend/src/lib/components/AIProviderPicker.svelte @@ -11,6 +11,7 @@ import ToggleButtonMore from './common/toggleButton-v2/ToggleButtonMore.svelte' import Toggle from './Toggle.svelte' import { saveConfig, removeConfig, isSameAsStoredConfig } from './aiProviderStorage' + import AIReasoningEffortPicker from './AIReasoningEffortPicker.svelte' interface Props { value: ProviderConfig | undefined @@ -106,6 +107,8 @@ value.kind = selectedProvider value.resource = '' value.model = '' + // Reasoning effort is model-specific; reset it with the model. + value.reasoning_effort = undefined } } @@ -225,6 +228,18 @@ /> + + {#if value?.model} +
+

reasoning effort

+ value?.reasoning_effort, (v) => value && (value.reasoning_effort = v)} + providerConfig={value} + {disabled} + /> +
+ {/if} +
+ import { Brain, ChevronDown } from 'lucide-svelte' + import DropdownV2 from './DropdownV2.svelte' + import Button from './common/button/Button.svelte' + import type { Item } from '$lib/utils' + import type { AIProvider } from '$lib/gen' + import { getReasoningCapability, explicitOffToken } from './copilot/reasoningRegistry' + + interface Props { + // The provider-native reasoning token sent to the model (e.g. `high`, + // `none`), or undefined to leave the provider default untouched. + value: string | undefined + // The selected provider config; only `kind` and `model` are read. + providerConfig: { kind?: AIProvider; model?: string } | string | undefined + disabled?: boolean + } + + let { value = $bindable(), providerConfig, disabled = false }: Props = $props() + + let provider = $derived( + typeof providerConfig === 'object' + ? (providerConfig?.kind as AIProvider | undefined) + : undefined + ) + let model = $derived(typeof providerConfig === 'object' ? providerConfig?.model : undefined) + + let capability = $derived( + provider && model + ? getReasoningCapability(provider, model) + : { supported: false, levels: [], canDisable: false } + ) + + // The token that turns reasoning off on a model that reasons by default + // (e.g. Gemini/OpenAI 'none'). Undefined means omission already disables it, + // so leaving the effort unset is itself off. + let offToken = $derived(provider && model ? explicitOffToken(provider, model) : undefined) + + // When the model reasons only on request (Anthropic/Bedrock), an unset effort + // already means off, so the default choice is labelled "off". Models that + // reason by default keep a distinct "Model default" plus an explicit off. + let offViaOmission = $derived(capability.canDisable && offToken === undefined) + + // Label shown on the trigger for the current selection. + let currentLabel = $derived( + !value ? (offViaOmission ? 'off' : 'Model default') : value === offToken ? 'off' : value + ) + + let items = $derived.by((): Item[] => { + const opts: Item[] = [ + { + displayName: offViaOmission ? 'off' : 'Model default', + selected: !value, + action: () => (value = undefined) + } + ] + if (capability.canDisable && offToken !== undefined) { + opts.push({ + displayName: 'off', + selected: value === offToken, + action: () => (value = offToken) + }) + } + for (const level of capability.levels) { + opts.push({ displayName: level, selected: value === level, action: () => (value = level) }) + } + return opts + }) + + // Clear a stale selection once the model no longer accepts it — either it + // can't reason at all, or the token isn't a valid level/off for this model + // (e.g. carrying `xhigh` from Opus onto a model that tops out at `high`). + // Sending an unsupported token would be rejected by the provider. + $effect(() => { + if (!provider || !model || value === undefined) return + // Valid values are exactly the selectable options: the levels, plus the + // explicit off token only when this model can actually disable reasoning. + const valid = + capability.supported && + (capability.levels.includes(value) || + (capability.canDisable && offToken !== undefined && value === offToken)) + if (!valid) value = undefined + }) + + +{#if !provider || !model} +
Select a model to configure reasoning effort.
+{:else if !capability.supported} +
The selected model does not support reasoning effort.
+{:else} + + {#snippet buttonReplacement()} + + {/snippet} + +{/if} diff --git a/frontend/src/lib/components/AppConnectInner.svelte b/frontend/src/lib/components/AppConnectInner.svelte index 1046ee69b0..ef632f5738 100644 --- a/frontend/src/lib/components/AppConnectInner.svelte +++ b/frontend/src/lib/components/AppConnectInner.svelte @@ -162,6 +162,9 @@ let clientId = $state('') let clientSecret = $state('') let ccInstance = $state('') + /** Bring-your-own resource-level token endpoint override (optional). Only sent + * for non-instance-templated providers, where it isn't host-pinned. */ + let tokenUrl = $state('') let resourceTypeInfo: ResourceType | undefined = $state(undefined) let resourceTypeNotFound = $state(false) @@ -222,6 +225,7 @@ clientId = '' clientSecret = '' ccInstance = '' + tokenUrl = '' scopes = [] } @@ -602,7 +606,13 @@ scopes: scopes, cc_client_id: trimmedClientId, cc_client_secret: trimmedClientSecret, - ...(needsInstance ? { cc_instance: trimmedInstance } : {}) + // Instance-templated providers are host-pinned via the instance + // name; only other providers accept a free-form token URL override. + ...(needsInstance + ? { cc_instance: trimmedInstance } + : tokenUrl.trim() + ? { cc_token_url: tokenUrl.trim() } + : {}) } }) @@ -734,10 +744,13 @@ accountData.cc_client_id = clientId.trim() accountData.cc_client_secret = clientSecret.trim() // Instance-templated providers send an instance name; the backend - // resolves and stores the host-pinned token URL. Other registry - // providers need nothing more (token URL comes from the registry). + // resolves and stores the host-pinned token URL. Other providers may + // send an optional token URL override (stored for refresh); without + // it the token URL comes from the registry/instance config. if (ccInstanceMeta) { accountData.cc_instance = ccInstance.trim() + } else if (tokenUrl.trim()) { + accountData.cc_token_url = tokenUrl.trim() } } @@ -978,13 +991,15 @@
{:else if step == 2 && manual}
- + {#if deployTo} + {:else} + {/if} {/if} @@ -1217,13 +1249,15 @@ Finish connection in popup window {/if} {:else} - + {#if deployTo}
+ + (runMigrationsModalOpen = false)} + > +
+

+ Run the deployed migrations in {runMigrationsTargetWorkspaceName} now? These data tables + use a separate database, so the schema changes won't apply until the migrations are run. +

+
    + {#each runMigrationsDatatables as dt (dt)} +
  • {dt}
  • + {/each} +
+
+
+ + { + const it = createConfirm + createConfirm = undefined + if (it) createOnRemote(it) + }} + onCanceled={() => (createConfirm = undefined)} + > +

+ This copies the current value of {createConfirm?.path} + (including any secret value) from + {createConfirm?.onCurrent ? currentWorkspaceId : parentWorkspaceId} + into {createConfirm?.onCurrent ? parentWorkspaceId : currentWorkspaceId}. It stays + workspace-specific afterward, so later promotes won't overwrite it. If it already exists + there, it's left untouched and just marked workspace-specific. +

+
{:else}
No comparison data available
diff --git a/frontend/src/lib/components/CronInput.svelte b/frontend/src/lib/components/CronInput.svelte index b7febcdaca..d2b629d33e 100644 --- a/frontend/src/lib/components/CronInput.svelte +++ b/frontend/src/lib/components/CronInput.svelte @@ -204,18 +204,20 @@
-
- +
+
+ +
{#if !disabled}
{@render cronBuilder()} diff --git a/frontend/src/lib/components/DBManager.svelte b/frontend/src/lib/components/DBManager.svelte index c1fdf475de..35bd88abec 100644 --- a/frontend/src/lib/components/DBManager.svelte +++ b/frontend/src/lib/components/DBManager.svelte @@ -370,7 +370,7 @@ refresh?.() sendUserToast(`Schema '${schemaKey}' deleted successfully`) } catch (e) { - let msg: string | undefined = (e as Error).message + let msg: string | undefined = (e as any).body ?? (e as Error).message if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : undefined sendUserToast(msg ?? 'Action failed!', true) } @@ -436,7 +436,7 @@ refresh?.() sendUserToast(`Table '${tableKey}' deleted successfully`) } catch (e) { - let msg: string | undefined = (e as Error).message + let msg: string | undefined = (e as any).body ?? (e as Error).message if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : undefined sendUserToast(msg ?? 'Action failed!', true) } @@ -502,7 +502,7 @@ refresh?.() sendUserToast(`Table '${tableKey}' deleted successfully`) } catch (e) { - let msg: string | undefined = (e as Error).message + let msg: string | undefined = (e as any).body ?? (e as Error).message if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : undefined sendUserToast(msg ?? 'Action failed!', true) } @@ -605,7 +605,9 @@ onConfirm={async ({ values }) => { if (dbTableEditorState.alterTableKey && dbTableEditorAlterTableData.current) { let diff = diffTableEditorValues(dbTableEditorAlterTableData.current, values) - await dbSchemaOps.onAlter({ schema: selected.schemaKey, values: diff }) + // Reverse diff (new → old) so the migration's down undoes the alter. + let reverse = diffTableEditorValues(values, dbTableEditorAlterTableData.current) + await dbSchemaOps.onAlter({ schema: selected.schemaKey, values: diff, reverse }) } else { await dbSchemaOps.onCreate({ values, schema: selected.schemaKey }) } diff --git a/frontend/src/lib/components/DBManagerContent.svelte b/frontend/src/lib/components/DBManagerContent.svelte index d84bdb862e..9332d7ad26 100644 --- a/frontend/src/lib/components/DBManagerContent.svelte +++ b/frontend/src/lib/components/DBManagerContent.svelte @@ -20,6 +20,10 @@ import type { SelectedTable } from './DBManager.svelte' import { getDbFeatures } from './apps/components/display/dbtable/dbFeatures' import { resource } from 'runed' + import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte' + import { createAsyncConfirmationModal } from './common/confirmationModal/asyncConfirmationModal.svelte' + import Portal from '$lib/components/Portal.svelte' + import { outOfOrderRunMessage } from './workspaceSettings/datatableMigrationUtils' interface Props { input?: DbInput @@ -35,6 +39,10 @@ /** Tables that are already added and should show as disabled */ disabledTables?: SelectedTable[] onImport?: (mode: 'schema_and_data' | 'schema_only') => void + /** Workspace the datatable/schema lookups run against. Defaults to the + * navigation `$workspaceStore`; pass the acting workspace when embedded in + * a session preview whose workspace differs from the top nav. */ + workspace?: string } let { @@ -47,10 +55,15 @@ multiSelectMode = false, selectedTables = $bindable([]), disabledTables = [], - onImport + onImport, + workspace = undefined }: Props = $props() - let dbSchema: DBSchema | undefined = $derived(input && $dbSchemas[getDbSchemasPath(input)]) + let ws = $derived(workspace ?? $workspaceStore) + + let dbSchema: DBSchema | undefined = $derived(input && $dbSchemas[schemaCacheKey(input)]) + + const outOfOrderModal = createAsyncConfirmationModal() function getDbSchemasPath(input: DbInput): string { switch (input.type) { @@ -61,28 +74,36 @@ } } + // Scope the shared `dbSchemas` cache by the acting workspace: a datatable of + // the same name can exist in both the nav and the acting workspace, so the + // bare resource path alone would let one workspace's schema be reused for the + // other while DB operations target the acting one. + function schemaCacheKey(input: DbInput): string { + return `${ws}:${getDbSchemasPath(input)}` + } + let colDefs = resource( - () => [input], + () => [input, ws], async () => { if (!input) return - return await loadAllTablesMetaData($workspaceStore, input) + return await loadAllTablesMetaData(ws, input) } ) let dbSchemasPromise = resource( - () => [input], + () => [input, ws], async () => { if (!input) return - const dbSchemasPath = getDbSchemasPath(input) + const dbSchemasPath = schemaCacheKey(input) if (input.type == 'database') { $dbSchemas[dbSchemasPath] = await getDbSchemas( input.resourceType, input.resourcePath, - $workspaceStore, + ws, (message: string) => sendUserToast(message, true) ) } else if (input.type == 'ducklake') { $dbSchemas[dbSchemasPath] = await getDucklakeSchema({ - workspace: $workspaceStore!, + workspace: ws!, ducklake: input.ducklake }) } @@ -124,7 +145,7 @@ }} /> -{#if dbSchema && $workspaceStore && input} +{#if dbSchema && ws && input} {@const _input = input} {@const dbType = getDbType(_input)} @@ -159,11 +180,17 @@ colDefs, tableKey, input: _input, - workspace: $workspaceStore + workspace: ws })} dbSchemaOps={dbSchemaOpsWithPreviewScripts({ input: _input, - workspace: $workspaceStore + workspace: ws, + confirmRunOutOfOrder: (pending) => + outOfOrderModal.ask({ + title: 'Run migration out of order', + confirmationText: 'Run anyway', + children: outOfOrderRunMessage(pending) + }) })} initialTableKey={input.specificTable} initialSchemaKey={input.specificSchema} @@ -189,9 +216,11 @@ { replResultData = data }} + onSchemaChange={() => refresh()} placeholderTableName={sortArray( Object.keys( dbSchema?.schema[ @@ -214,3 +243,9 @@ {/if} + + + + + diff --git a/frontend/src/lib/components/DBManagerDrawer.svelte b/frontend/src/lib/components/DBManagerDrawer.svelte index 6598103751..712c4fbccf 100644 --- a/frontend/src/lib/components/DBManagerDrawer.svelte +++ b/frontend/src/lib/components/DBManagerDrawer.svelte @@ -16,6 +16,7 @@ Upload } from 'lucide-svelte' import DBManagerContent from './DBManagerContent.svelte' + import DataTableMigrationsButton from './workspaceSettings/DataTableMigrationsButton.svelte' import { resource } from 'runed' import { untrack } from 'svelte' import type { DbManagerUriState } from './dbManagerDrawerModel.svelte' @@ -34,13 +35,15 @@ let open = $derived(uriState.open) + // The workspace the drawer's DB operations run against — the acting workspace of + // the editor that opened it (set via openDrawer), else the nav workspace. + let ws = $derived(uriState.workspace ?? $workspaceStore) + // Load available datatables when drawer opens with datatable input const datatables = resource([], async () => { - if (!$workspaceStore) return [] + if (!ws) return [] try { - return (await WorkspaceService.listDataTables({ workspace: $workspaceStore })).map( - (d) => d.name - ) + return (await WorkspaceService.listDataTables({ workspace: ws })).map((d) => d.name) } catch (e) { console.error('Failed to load datatables:', e) return [] @@ -105,12 +108,17 @@ return toSourceIdentifier(input.resourcePath) } + function refreshManager() { + dbManagerContent?.refresh() + dbManagerContent?.dbManager()?.dbTable()?.refresh() + } + async function handleExportSchema() { const source = currentSourceIdentifier() - if (!source || !$workspaceStore) return + if (!source || !ws) return try { exportResult = await WorkspaceService.exportPgSchema({ - workspace: $workspaceStore, + workspace: ws, requestBody: { source } }) exportDrawerOpen = true @@ -120,13 +128,13 @@ } async function handleImportDatabase() { - if (!importSource || !$workspaceStore) return + if (!importSource || !ws) return const target = currentSourceIdentifier() if (!target) return importLoading = true try { await WorkspaceService.importPgDatabase({ - workspace: $workspaceStore, + workspace: ws, requestBody: { source: toSourceIdentifier(importSource), target, @@ -167,11 +175,12 @@ noPadding id="db-manager-drawer" > - {#if uriState.effectiveInput && $workspaceStore} + {#if uriState.effectiveInput && ws} {#key uriState.selectedDatatable} + {/if} {#if enableImportExport} + /> + {/if} + {:else} + + passed: + {t.test} + {/if} +
+ {#if t.violating > 0 && t.sample && t.sample.length > 0 && expanded.has(t.test)} +
+
+ sample of the violating rows ({t.sample.length} of {t.violating}, unordered) +
+ +
+ {/if} + + {/each} + +
diff --git a/frontend/src/lib/components/DatatableSchemaDiff.svelte b/frontend/src/lib/components/DatatableSchemaDiff.svelte index 037dc0acd5..9eccda8195 100644 --- a/frontend/src/lib/components/DatatableSchemaDiff.svelte +++ b/frontend/src/lib/components/DatatableSchemaDiff.svelte @@ -196,10 +196,17 @@ import Drawer from '$lib/components/common/drawer/Drawer.svelte' import SimpleEditor from '$lib/components/SimpleEditor.svelte' import { sendUserToast } from '$lib/toast' + import { userWorkspaces } from '$lib/stores' import { runScriptAndPollResult } from '$lib/components/jobs/utils' import YAML from 'yaml' import DrawerContent from './common/drawer/DrawerContent.svelte' import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte' + import { createAsyncConfirmationModal } from './common/confirmationModal/asyncConfirmationModal.svelte' + import Portal from '$lib/components/Portal.svelte' + import { + pendingMigrations, + outOfOrderRunMessage + } from './workspaceSettings/datatableMigrationUtils' import Alert from './common/alert/Alert.svelte' import ResizeTransitionWrapper from './common/ResizeTransitionWrapper.svelte' @@ -213,6 +220,11 @@ let loading = $state(true) let error: string | undefined = $state(undefined) let diffs: DatatableDiff[] = $state([]) + // Number of forked datatables this schema-diff section applies to: those that + // have NOT opted in to the migrations feature. When a datatable enables + // migrations, its changes flow through the normal item diff instead, so it is + // excluded here and the whole section hides once none remain. + let applicableCount = $state(0) let expandedDatatables: Set = $state(new Set()) // Drawer state @@ -223,6 +235,7 @@ let migrationSql = $state('') let migrationRunning = $state(false) let confirmDeployOpen = $state(false) + const outOfOrderModal = createAsyncConfirmationModal() async function loadDiffs() { loading = true @@ -233,7 +246,10 @@ workspace: currentWorkspaceId }) const datatables = forkSettings.datatable?.datatables ?? {} - const forkedEntries = Object.entries(datatables).filter(([_, dt]) => dt.forked_from != null) + const forkedEntries = Object.entries(datatables).filter( + ([_, dt]) => dt.forked_from != null && dt.migrations_enabled !== true + ) + applicableCount = forkedEntries.length if (forkedEntries.length === 0) { loading = false return @@ -343,14 +359,68 @@ const dtName = drawerDiff.datatableName try { - await runScriptAndPollResult({ + // If the target data table opted in to migrations, record this merge as a + // tracked migration (named after the fork) and run it, instead of applying + // raw SQL that would bypass the target's migration history. + // Don't swallow a status-check failure by defaulting to raw apply: that + // would apply the DDL untracked (schema drift) — exactly what this feature + // prevents. Let the error propagate (fail closed, handled by the outer + // catch); only fall back to raw apply when the API explicitly returns + // enabled === false. + const status = await WorkspaceService.getDatatableMigrationsStatus({ workspace: targetWorkspace, - requestBody: { - args: { database: `datatable://${dtName}` }, - language: 'postgresql', - content: migrationSql - } + datatableName: dtName }) + + if (status.enabled) { + // The merge migration gets the highest timestamp, so any still-pending + // migration on the target is earlier: running only the merge applies it + // out of order. Warn like the row-level Run action does. + const pending = pendingMigrations(status.migrations) + if (pending.length > 0) { + const confirmed = await outOfOrderModal.ask({ + title: 'Run migration out of order', + confirmationText: 'Run anyway', + children: outOfOrderRunMessage(pending.length) + }) + if (!confirmed) { + migrationRunning = false + return + } + } + const forkName = + $userWorkspaces.find((w) => w.id === currentWorkspaceId)?.name || currentWorkspaceId + const migName = `merge_${forkName}`.replace(/[^a-zA-Z0-9_-]+/g, '_') + const created = await WorkspaceService.createDatatableMigration({ + workspace: targetWorkspace, + datatableName: dtName, + requestBody: { name: migName, code_up: migrationSql } + }) + try { + await WorkspaceService.runDatatableMigrations({ + workspace: targetWorkspace, + datatableName: dtName, + only: created.timestamp + }) + } catch (runErr: any) { + // Undo the insertion so the user can fix and retry from a clean state. + await WorkspaceService.deleteDatatableMigration({ + workspace: targetWorkspace, + datatableName: dtName, + timestamp: created.timestamp + }).catch(() => {}) + throw runErr + } + } else { + await runScriptAndPollResult({ + workspace: targetWorkspace, + requestBody: { + args: { database: `datatable://${dtName}` }, + language: 'postgresql', + content: migrationSql + } + }) + } } catch (e: any) { sendUserToast(e?.body ?? e?.message ?? String(e), true) migrationRunning = false @@ -394,102 +464,107 @@ } -

Datatable schema changes

-{#if loading} -
- Loading datatable diffs... -
-{:else if error} -
Failed to load datatable diffs: {error}
-{:else if diffs.length > 0} -
- {#each diffs as diff} - - +{#if applicableCount > 0} +
+

Datatable schema changes

+ {#if loading} +
+ Loading datatable diffs... +
+ {:else if error} +
Failed to load datatable diffs: {error}
+ {:else if diffs.length > 0} +
+ {#each diffs as diff} + + - {#if expandedDatatables.has(diff.datatableName)} -
- {#if diff.aheadChanges.length > 0} -
-
Fork changes (ahead)
- {#each diff.aheadChanges as change} -
- {#if change.kind === 'added'} - - {:else if change.kind === 'removed'} - - {:else} - - {/if} - {change.schemaName}. - {change.tableName} - {operationSummary(change)} - + {#each diff.aheadChanges as change} +
+ {#if change.kind === 'added'} + + {:else if change.kind === 'removed'} + + {:else} + + {/if} + {change.schemaName}. + {change.tableName} + {operationSummary(change)} + +
+ {/each}
- {/each} + {/if} + {#if diff.behindChanges.length > 0} +
+
+ Parent changes (behind) +
+ {#each diff.behindChanges as change} +
+ {#if change.kind === 'added'} + + {:else if change.kind === 'removed'} + + {:else} + + {/if} + {change.schemaName}. + {change.tableName} + {operationSummary(change)} + +
+ {/each} +
+ {/if}
{/if} - {#if diff.behindChanges.length > 0} -
-
- Parent changes (behind) -
- {#each diff.behindChanges as change} -
- {#if change.kind === 'added'} - - {:else if change.kind === 'removed'} - - {:else} - - {/if} - {change.schemaName}. - {change.tableName} - {operationSummary(change)} - -
- {/each} -
- {/if} -
- {/if} -
- {/each} + + {/each} +
+ {:else} + No changes detected + {/if}
-{:else} - No changes detected {/if} @@ -580,3 +655,7 @@ >{migrationSql} + + + + diff --git a/frontend/src/lib/components/DdlMigrationGuard.svelte b/frontend/src/lib/components/DdlMigrationGuard.svelte new file mode 100644 index 0000000000..b6833521f8 --- /dev/null +++ b/frontend/src/lib/components/DdlMigrationGuard.svelte @@ -0,0 +1,159 @@ + + + + + +
+

+ This looks like a schema-changing (DDL) statement. Schema changes are best tracked as + migrations rather than run ad-hoc. Create a migration for it instead? +

+
{promptStatement ?? ''}
+
+ + +
+
+
+ + migrationsModal?.openMigration(m.timestamp)} +/> + + diff --git a/frontend/src/lib/components/DeployButton.svelte b/frontend/src/lib/components/DeployButton.svelte index d8828d515a..da1ba5dd42 100644 --- a/frontend/src/lib/components/DeployButton.svelte +++ b/frontend/src/lib/components/DeployButton.svelte @@ -6,7 +6,8 @@ const { loading = false, loadingSave = false, - dropdownItems = [] + dropdownItems = [], + unifiedSize = 'md' }: { loading?: boolean loadingSave?: boolean @@ -14,6 +15,7 @@ label: string onClick: () => void }> + unifiedSize?: 'sm' | 'md' | 'lg' } = $props() const dispatch = createEventDispatcher() @@ -28,7 +30,7 @@ disabled={loading} loading={loadingSave} variant="accent" - unifiedSize="md" + {unifiedSize} startIcon={{ icon: Save }} on:click={() => dispatch('save')} {dropdownItems} diff --git a/frontend/src/lib/components/DevWorkspaceSetting.svelte b/frontend/src/lib/components/DevWorkspaceSetting.svelte new file mode 100644 index 0000000000..e6295d24cd --- /dev/null +++ b/frontend/src/lib/components/DevWorkspaceSetting.svelte @@ -0,0 +1,240 @@ + + +{#if isDev && parentId} +
+

+ This is a {devLabelNoun(currentWs?.dev_workspace_label)} paired with root workspace + {parentId}. Promote changes from the home page banner or the Compare & Deploy page. +

+
+ Cosmetic label: {devBadgeText(currentLabel)} + +
+
+ +
+
+{:else if pairedDev} +
+

+ This workspace's {devLabelNoun(pairedDev.label)} is {pairedDev.name} ({pairedDev.id}). + Edits to this workspace are redirected there. +

+
+ {#if pairedDev.isMember} + + {/if} + +
+
+{:else if parentId} +

+ Dev workspace pairing is only available for root workspaces. This workspace is a fork of + {parentId}. +

+{:else} +
+

+ Pair this workspace with a dev workspace: the same code with a different environment (resource + and variable values). Edits are made in the dev workspace and promoted here. +

+
+ Attach an existing workspace as dev + diff --git a/frontend/src/lib/components/ObjectResourceInput.svelte b/frontend/src/lib/components/ObjectResourceInput.svelte index e9f6e4d6e3..e76a19c6da 100644 --- a/frontend/src/lib/components/ObjectResourceInput.svelte +++ b/frontend/src/lib/components/ObjectResourceInput.svelte @@ -20,6 +20,8 @@ disabled?: boolean datatableAsPgResource?: boolean onClear?: () => void + /** Workspace the resource picker lists from; defaults to the nav workspace. */ + workspace?: string } let { @@ -32,7 +34,8 @@ editor = $bindable(undefined), disabled = false, datatableAsPgResource = false, - onClear = undefined + onClear = undefined, + workspace = undefined }: Props = $props() function isResource() { @@ -55,7 +58,7 @@
{#if format === 'resource-s3_object'} - + {:else if value == undefined || typeof value === 'string'} valueToPath(), (v) => { diff --git a/frontend/src/lib/components/ObjectStoreConfigSettings.svelte b/frontend/src/lib/components/ObjectStoreConfigSettings.svelte index 5c96007d33..c7f7065565 100644 --- a/frontend/src/lib/components/ObjectStoreConfigSettings.svelte +++ b/frontend/src/lib/components/ObjectStoreConfigSettings.svelte @@ -1,5 +1,5 @@
@@ -31,6 +35,7 @@ {tooltip} {/if} + {@render titleActions?.()} {:else} @@ -40,6 +45,7 @@ {tooltip} {/if} + {@render titleActions?.()} {/if} diff --git a/frontend/src/lib/components/ParentWorkspaceProtectionAlert.svelte b/frontend/src/lib/components/ParentWorkspaceProtectionAlert.svelte index 9712062efb..994817f807 100644 --- a/frontend/src/lib/components/ParentWorkspaceProtectionAlert.svelte +++ b/frontend/src/lib/components/ParentWorkspaceProtectionAlert.svelte @@ -1,14 +1,15 @@ - - - + +
+ + +
diff --git a/frontend/src/lib/components/ResourceEditor.svelte b/frontend/src/lib/components/ResourceEditor.svelte index d503763a06..7acc4e485b 100644 --- a/frontend/src/lib/components/ResourceEditor.svelte +++ b/frontend/src/lib/components/ResourceEditor.svelte @@ -374,7 +374,7 @@
-
+
{#if otherDirty.length > 0} You are going to edit the value in: {otherDirty.join(', ')} @@ -402,6 +402,7 @@ {loadingSchema} {resourceToEdit} onLoadResourceType={() => resourceTypeResource.refetch()} + workspace={selected} /> {/key} {/if} diff --git a/frontend/src/lib/components/ResourceEditorDrawer.svelte b/frontend/src/lib/components/ResourceEditorDrawer.svelte index 919f8edeff..6c0da69675 100644 --- a/frontend/src/lib/components/ResourceEditorDrawer.svelte +++ b/frontend/src/lib/components/ResourceEditorDrawer.svelte @@ -58,6 +58,7 @@ {#await import('./ResourceEditor.svelte')} @@ -79,6 +80,7 @@ {#snippet banner()} resourceEditor?.localDraftDeployed()} getCurrent={() => resourceEditor?.localDraftCurrent()} onDiscard={() => resourceEditor?.discardLocalDraft()} diff --git a/frontend/src/lib/components/ResourceForm.svelte b/frontend/src/lib/components/ResourceForm.svelte index ced60bf4dc..ef226501b7 100644 --- a/frontend/src/lib/components/ResourceForm.svelte +++ b/frontend/src/lib/components/ResourceForm.svelte @@ -42,6 +42,9 @@ loadingSchema: boolean resourceToEdit: Resource | undefined onLoadResourceType?: () => void + /** Workspace the path is validated against and the connection is tested in; + * defaults to the nav workspace. */ + workspace?: string | undefined } let { @@ -62,9 +65,12 @@ resourceSchema, loadingSchema, resourceToEdit, - onLoadResourceType + onLoadResourceType, + workspace = undefined }: Props = $props() + let ws = $derived(workspace ?? $workspaceStore) + let editDescription = $state(false) let rawCode: string | undefined = $state(undefined) let textFileContent: string = $state('') @@ -131,11 +137,12 @@ {/if}
@@ -218,7 +225,11 @@ {#if resourceToEdit?.resource_type === 'nats' || resourceToEdit?.resource_type === 'kafka'} {:else} - + {/if} {#if resource_type === 'git_repository' && $workspaceStore && ($userStore?.is_admin || $userStore?.is_super_admin)} {/if} {:else if !can_write} diff --git a/frontend/src/lib/components/ResourcePicker.svelte b/frontend/src/lib/components/ResourcePicker.svelte index 4df153cefa..e74e6f7285 100644 --- a/frontend/src/lib/components/ResourcePicker.svelte +++ b/frontend/src/lib/components/ResourcePicker.svelte @@ -26,6 +26,7 @@ placeholder?: string | undefined selectInputClass?: string class?: string + error?: boolean onClear?: () => void excludedValues?: string[] datatableAsPgResource?: boolean @@ -47,6 +48,7 @@ placeholder = undefined, selectInputClass = '', class: className = '', + error = false, onClear = undefined, excludedValues = undefined, datatableAsPgResource = false, @@ -236,6 +238,7 @@ }} items={collection} clearable + {error} class="text-clip grow min-w-0" inputClass={selectInputClass} placeholder={placeholder ?? `${resourceType ?? 'any'} resource`} diff --git a/frontend/src/lib/components/RunForm.svelte b/frontend/src/lib/components/RunForm.svelte index b1b562b04d..89eaf8f7ab 100644 --- a/frontend/src/lib/components/RunForm.svelte +++ b/frontend/src/lib/components/RunForm.svelte @@ -114,6 +114,7 @@ scheduledForStr: string | undefined invisible_to_owner: boolean | undefined overrideTag: string | undefined + overrideTagNote?: string args?: Record jsonView?: boolean isValid?: boolean @@ -132,6 +133,7 @@ scheduledForStr = $bindable(), invisible_to_owner = $bindable(), overrideTag = $bindable(), + overrideTagNote = undefined, args = $bindable(), jsonView = false, isValid = $bindable(true) @@ -160,7 +162,7 @@ debounced && clearTimeout(debounced) debounced = setTimeout(() => { const nurl = new URL(window.location.href) - nurl.hash = computeSharableHash(args) + nurl.hash = computeSharableHash(args, overrideTag) try { replaceState(nurl.toString(), page.state) @@ -201,6 +203,7 @@ jsonEditor?.setCode(code) } $effect(() => { + overrideTag Object.keys(args ?? {}).forEach((key) => { args?.[key] }) @@ -387,6 +390,10 @@
tag override: {overrideTag}
+ {:else if overrideTagNote} +
+ {overrideTagNote} +
{/if} {#if invisible_to_owner}
diff --git a/frontend/src/lib/components/S3FilePicker.svelte b/frontend/src/lib/components/S3FilePicker.svelte index 55bb25c5ef..8829660ea3 100644 --- a/frontend/src/lib/components/S3FilePicker.svelte +++ b/frontend/src/lib/components/S3FilePicker.svelte @@ -18,6 +18,9 @@ selectedFileKey?: { s3: string; storage?: string } | undefined folderOnly?: boolean regexFilter?: RegExp | undefined + /** Workspace to browse S3 storage in — the acting workspace of the editor that + * opened the picker, else the nav workspace. */ + workspace?: string | undefined onClose?: () => void onSelectAndClose?: (selected: { s3: string; storage: string | undefined }) => void } @@ -30,10 +33,13 @@ selectedFileKey = $bindable(undefined), folderOnly = false, regexFilter = undefined, + workspace = undefined, onClose, onSelectAndClose }: Props = $props() + let ws = $derived(workspace ?? $workspaceStore) + let drawer: Drawer | undefined = $state() let s3FilePickerInner: S3FilePickerInner | undefined = $state() @@ -55,8 +61,8 @@ > = $state({}) let secondaryStorageNames = resource( - () => $workspaceStore, - () => SettingService.getSecondaryStorageNames({ workspace: $workspaceStore! }), + () => ws, + () => SettingService.getSecondaryStorageNames({ workspace: ws! }), { lazy: true } ) @@ -105,6 +111,7 @@ bind:uploadModalOpen {folderOnly} {regexFilter} + {workspace} /> {#snippet actions()}
diff --git a/frontend/src/lib/components/S3FilePickerInner.svelte b/frontend/src/lib/components/S3FilePickerInner.svelte index 333857b7eb..9475c0041a 100644 --- a/frontend/src/lib/components/S3FilePickerInner.svelte +++ b/frontend/src/lib/components/S3FilePickerInner.svelte @@ -65,6 +65,8 @@ regexFilter?: RegExp | undefined hideS3SpecificDetails?: boolean rootPath?: string + /** Workspace to browse S3 storage in — defaults to the nav workspace. */ + workspace?: string | undefined workspaceSettingsInitialized?: boolean storage?: string | undefined uploadModalOpen?: boolean @@ -103,6 +105,7 @@ regexFilter = undefined, hideS3SpecificDetails = false, rootPath: initialRootPath = '', + workspace = undefined, workspaceSettingsInitialized = $bindable(true), storage = $bindable(undefined), uploadModalOpen = $bindable(false), @@ -117,6 +120,8 @@ testConnectionRequest = HelpersService.datasetStorageTestConnection }: Props = $props() + let ws = $derived(workspace ?? $workspaceStore) + let rootPath = $state(initialRootPath) let rootPathNestingLevel = $derived(1 * (rootPath.split('/').length - 1)) @@ -183,7 +188,7 @@ async function loadFiles() { fileListLoading = true let availableFiles = await listStoredFilesRequest({ - workspace: $workspaceStore!, + workspace: ws!, maxKeys: maxKeys, // fixed pages of 1000 files for now marker: page == 0 ? undefined : listMarkers[page - 1], prefix: rootPath ?? (filter.trim() != '' ? filter : undefined), @@ -280,7 +285,7 @@ } fileInfoLoading = true let fileMetadataRaw = await loadFileMetadataRequest({ - workspace: $workspaceStore!, + workspace: ws!, fileKey: fileKey, storage: storage }) @@ -300,7 +305,7 @@ async function loadFilePreview(fileKey: string, fileSizeInBytes?: number, fileMimeType?: string) { let filePreviewRaw = await loadFilePreviewRequest({ - workspace: $workspaceStore!, + workspace: ws!, fileKey: fileKey, fileSizeInBytes: fileSizeInBytes, fileMimeType: fileMimeType, @@ -349,7 +354,7 @@ } try { await deleteS3FileRequest({ - workspace: $workspaceStore!, + workspace: ws!, fileKey: fileKey, storage: storage }) @@ -409,7 +414,7 @@ } try { await moveS3FileRequest({ - workspace: $workspaceStore!, + workspace: ws!, srcFileKey: srcFileKey, destFileKey: destFileKey!, storage: storage @@ -457,7 +462,7 @@ fileListLoading = true try { await testConnectionRequest({ - workspace: $workspaceStore!, + workspace: ws!, storage: storage }) workspaceSettingsInitialized = true @@ -716,7 +721,7 @@ {#if filePreview !== undefined && (!hideS3SpecificDetails || !readOnlyMode || allowDelete)}
{#if !hideS3SpecificDetails} - {@const downloadApiPath = `/w/${$workspaceStore}/job_helpers/download_s3_file?file_key=${encodeURIComponent(fileMetadata?.fileKey ?? '')}${storage ? `&storage=${storage}` : ''}`} + {@const downloadApiPath = `/w/${ws}/job_helpers/download_s3_file?file_key=${encodeURIComponent(fileMetadata?.fileKey ?? '')}${storage ? `&storage=${storage}` : ''}`} {@const downloadName = fileMetadata?.fileKey.split('/').pop() ?? 'unnamed_download.file'} {#if shouldDownloadViaClient()} diff --git a/frontend/src/lib/components/S3ObjectPicker.svelte b/frontend/src/lib/components/S3ObjectPicker.svelte index 1c461708cd..bb3e19599e 100644 --- a/frontend/src/lib/components/S3ObjectPicker.svelte +++ b/frontend/src/lib/components/S3ObjectPicker.svelte @@ -14,9 +14,15 @@ interface Props { value: any editor?: SimpleEditor | undefined + /** Workspace to browse/upload S3 objects in; defaults to the nav workspace. */ + workspace?: string | undefined } - let { value = $bindable(), editor = $bindable(undefined) }: Props = $props() + let { + value = $bindable(), + editor = $bindable(undefined), + workspace = undefined + }: Props = $props() const dispatch = createEventDispatcher() @@ -48,6 +54,7 @@ editor?.setCode(rawValue) }} readOnlyMode={false} + {workspace} />
@@ -85,6 +92,7 @@ } }} defaultValue={value?.s3} + {workspace} /> {/if} {#if customUi?.topBar?.path != false}
@@ -1883,13 +1928,15 @@ kind="script" summaryEditable={customUi?.topBar?.editableSummary != false} pathEditable={customUi?.topBar?.editablePath != false} + hidePath={condensedHeader} + workspaceId={autosaveWorkspace} onNavigate={(item) => onNavigate?.(item)} />
{/if} - {#if indicatorWorkspace} + {#if opWorkspace} (metadataOpen = true)} startIcon={{ icon: Settings }} iconOnly={compactTopbar} @@ -1940,7 +1987,7 @@
+ {#if showPipelineHint} +
+ + + This script can become a data pipeline step: annotate it with + -- pipeline + and + + -- materialize + + to materialize its result, or build it in the + pipeline editor. + + Learn more + + +
+
+
+ {/if} + + // Emitted with the test-form's full-schema validity whenever it changes, so + // a host (the pipeline editor) can gate a data-upload entry's readiness on + // whether every required field is filled, not just the S3 file. A callback + // rather than a bindable so we don't hit the `$bindable(default)` ban. + onIsValidChange?: (isValid: boolean) => void + // Custom timeout (in seconds) from the script settings. Forwarded to the + // preview run so "Test" honors the same timeout a deployed run would, + // instead of silently falling back to the instance default. + timeout?: number selectedTab?: 'main' | 'preprocessor' | 'diagram' hasPreprocessor?: boolean captureTable?: CaptureTable | undefined @@ -165,6 +185,11 @@ lastDeployedCode?: string | undefined disableAi?: boolean assets?: AssetWithAltAccessType[] + // Body-inferred column lineage (DuckDB SQL AST), surfaced alongside + // `assets` so the pipeline editor can render inferred column lineage on + // the live graph. Empty/undefined for non-DuckDB or when the parser + // build predates the inference. + inferredColumnLineage?: ColumnLineage[] modules?: { [key: string]: ScriptModule } | null editorBarRight?: import('svelte').Snippet enablePreprocessorSnippet?: boolean @@ -190,6 +215,19 @@ // regular /scripts/edit route keeps its current open-by-default UX; // the session preview opts in to save vertical real estate. initialTestPanelCollapsed?: boolean + // Lets the AI toolbar button open the script in a fresh AI session + // instead of the inline chat panel (see OpenInSessionButton for gating). + sessionOpen?: OpenInSessionSource + // Producer-side facts for the live schema-contract diagnostics + // (`on_schema_change=ignore` suppression + scd2 `_current` fallback), + // built by the pipeline page from the resolved graph. Absent outside the + // pipeline editor — the check still runs, just without suppression. + schemaContractContext?: SchemaContractGraphContext + // Workspace to scope this editor's calls to. Defaults to the nav + // `$workspaceStore`; an AI-session live editor passes the session's + // acting workspace (a fork) so tests, captures and toolbar lookups hit + // the right workspace instead of the nav one. + workspaceOverride?: string } let { @@ -213,6 +251,8 @@ customUi = undefined, requireValidAssets = false, args = $bindable(), + onIsValidChange, + timeout = undefined, selectedTab = $bindable('main'), hasPreprocessor = $bindable(false), captureTable = $bindable(undefined), @@ -222,15 +262,21 @@ lastDeployedCode = undefined, disableAi = false, assets = $bindable(), + inferredColumnLineage = $bindable(), modules = $bindable(undefined), editorBarRight, enablePreprocessorSnippet = false, previewLayout = 'right', onTestStateChange, onTestJob, - initialTestPanelCollapsed = false + initialTestPanelCollapsed = false, + sessionOpen = undefined, + schemaContractContext = undefined, + workspaceOverride = undefined }: Props = $props() + let opWs = $derived(workspaceOverride ?? $workspaceStore) + $effect(() => { onTestStateChange?.(testIsLoading) }) @@ -267,6 +313,7 @@ if (activeModuleTab === null && code !== lastSyncedCode) { editorCode = code lastSyncedCode = code + editor?.setCode(editorCode) // immediate sync, don't wait for the 800ms debounce untrack(() => inferSchema(code)) } }) @@ -557,7 +604,7 @@ let inferAssetsRes = resource([() => lang, () => code, () => code], () => inferAssets(lang, code)) let preparedSqlQueries = usePreparedAssetSqlQueries( () => inferAssetsRes.current?.sql_queries, - () => $workspaceStore + () => opWs ) // Asset-parse validity for the editor badge. `undefined` while loading (so // the badge doesn't flicker red); only an explicit parser error counts as @@ -577,7 +624,13 @@ watch( () => inferAssetsRes.current, () => { - if (!inferAssetsRes.current || inferAssetsRes.current?.status === 'error') return + if (!inferAssetsRes.current || inferAssetsRes.current?.status === 'error') { + // Clear stale lineage on parse error / unset, so a script switch + // whose new body fails to parse can't leave the previous script's + // inferred column lineage bound to the new path. + if (inferredColumnLineage !== undefined) inferredColumnLineage = undefined + return + } let newAssets = inferAssetsRes.current.assets as AssetWithAltAccessType[] for (const asset of newAssets) { const old = assets?.find((a) => assetEq(a, asset)) @@ -585,9 +638,43 @@ } const normalizedAssets = newAssets.length > 0 ? newAssets : undefined if (!deepEqual(assets, normalizedAssets)) assets = normalizedAssets + + const newLineage = inferAssetsRes.current.column_lineage + const normalizedLineage = newLineage && newLineage.length > 0 ? newLineage : undefined + if (!deepEqual(inferredColumnLineage, normalizedLineage)) + inferredColumnLineage = normalizedLineage } ) + // Live schema-contract diagnostics (pipelines gap #2b): diff the buffer's + // asset refs against the captured producer schemas and surface mismatches + // as Monaco warning squiggles — the as-you-type mirror of the authoritative + // save-time check. The result is a prop on Editor (not an imperative call) + // because this can resolve before Monaco initializes on mount. Sequenced so + // a slow schema fetch can't overwrite the markers of a newer keystroke. + let contractMarkers: ContractMarker[] = $state([]) + let contractCheckSeq = 0 + watch([() => inferAssetsRes.current, () => schemaContractContext], () => { + const res = inferAssetsRes.current + const workspace = opWs + const seq = ++contractCheckSeq + if (!workspace || !res || res.status === 'error') { + contractMarkers = [] + return + } + const bufferCode = code + computeContractMarkers( + workspace, + bufferCode, + (res.assets ?? []) as AssetWithAltAccessType[], + schemaContractContext + ) + .then((markers) => { + if (seq === contractCheckSeq) contractMarkers = markers + }) + .catch((e) => console.error('schema-contract diagnostics failed', e)) + }) + watch([() => code, () => lang], () => { if (lang !== 'ansible') return inferAnsibleExecutionMode(code).then((v) => { @@ -612,6 +699,10 @@ let jobLoader: JobLoader | undefined = $state(undefined) let isValid: boolean = $state(true) + // Mirror the test-form validity out to an optional host callback. + $effect(() => { + onIsValidChange?.(isValid) + }) let scriptProgress = $state(undefined) let logPanel: LogPanel | undefined = $state(undefined) @@ -717,7 +808,17 @@ args = nargs } - export async function runTest(opts?: { cascade?: boolean }) { + export async function runTest(opts?: { cascade?: boolean; skipDdlGuard?: boolean }) { + // Intercept DDL statements (offer to turn them into data table migrations) + // on every run path, not just the editor's Cmd+Enter. `skipDdlGuard` is set + // by the Cmd+Enter action, which already guarded before calling us. + if (!opts?.skipDdlGuard) { + if ((await editor?.guardDdlBeforeRun()) === false) return + // The guard may have rewritten the code (migrated statements stripped); + // `editorCode` is kept in sync by the editor binding, so mirror the + // on:change handler and pull it into `code` before we run. + if (activeModuleTab === null) code = editorCode + } // When the caller forces a cascade choice (e.g. the canvas runnable // menu's "Run + trigger N downstream"), also flip the persistent // `cascadeDownstream` state so the split button's label/icon reflect @@ -744,7 +845,7 @@ ? { _ENTRYPOINT_OVERRIDE: 'preprocessor', ...(args ?? {}) } : (args ?? {}) const testSchema = activeModuleTab !== null ? testPanelSchema : schema - const testArgs = await processSecretArgs(rawTestArgs, testSchema) + const testArgs = await processSecretArgs(rawTestArgs, testSchema, opWs) if (showPsCommonParams) { for (const [k, v] of Object.entries(psCommonParams)) { if (v !== undefined && v !== false && v !== '') { @@ -788,7 +889,9 @@ } }, undefined, - activeModuleTab !== null ? undefined : modules + activeModuleTab !== null ? undefined : modules, + undefined, + timeout ) if (job) { onTestJob?.({ jobId: job }) @@ -813,7 +916,7 @@ async function loadPastTests(): Promise { pastPreviewsRequest?.cancel() const req = JobService.listCompletedJobs({ - workspace: $workspaceStore!, + workspace: opWs!, jobKinds: 'preview', createdBy: $userStore?.username, scriptPathExact: path, @@ -878,16 +981,67 @@ // we reapply initial args as the schema form might have cleared them between mount and the schema inference args = initialArgs } + injectPartitionArg(nschema, args, nlang ?? lang, code) schema = nschema } catch (e) { validCode = false } } + // A `// partitioned` pipeline script is materialized one slice at a time and + // receives the slice as a runtime `partition` arg (the cascade injects it in + // production). It isn't a code parameter, so schema inference doesn't see it — + // surface it in the test form so a partitioned script can be run manually. + function injectPartitionArg( + s: any, + a: Record | undefined, + l: string | undefined, + c: string + ) { + try { + if (l !== 'duckdb' || !s?.properties) return + const part = parsePipelineAnnotations(c).partition + if (!part) return + // Date-based partition kinds render a date / datetime picker; a dynamic + // key is a free-form string. + const format = + part.kind === 'hourly' + ? 'date-time' + : part.kind === 'daily' || part.kind === 'weekly' || part.kind === 'monthly' + ? 'date' + : undefined + if (!s.properties['partition']) { + s.properties['partition'] = { + type: 'string', + ...(format ? { format } : {}), + // ISO output so partition keys sort lexicographically (the date + // picker defaults to dd-MM-yyyy otherwise). + ...(format === 'date' ? { dateFormat: 'yyyy-MM-dd' } : {}), + description: + part.kind === 'dynamic' + ? 'Partition key value to materialize.' + : `Partition (${part.kind}) to materialize.` + } + if (Array.isArray(s.order) && !s.order.includes('partition')) { + s.order = ['partition', ...s.order] + } + } + // Pre-fill the *test* arg with the current slice for date kinds — a + // convenience default, kept on the args (not baked into the schema, + // where it would persist to the deployed script and go stale). + if (format && a && (a['partition'] == null || a['partition'] === '')) { + const now = new Date() + a['partition'] = + format === 'date' ? now.toISOString().slice(0, 10) : now.toISOString().slice(0, 16) + } + } catch (e) {} + } + async function inferModuleSchema() { if (activeModuleTab === null) return try { await inferArgs(effectiveLang, editorCode, testPanelSchema) + injectPartitionArg(testPanelSchema, testPanelArgs, effectiveLang, editorCode) moduleTestState[activeModuleTab] = { args: testPanelArgs, schema: testPanelSchema } } catch (e) { // Module code may be in-progress; silently ignore @@ -1064,12 +1218,12 @@ dapClient = getDAPClient(dapServerUrl) // Fetch contextual variables (WM_WORKSPACE, WM_TOKEN, etc.) from backend - const env = await fetchContextualVariables($workspaceStore ?? '') + const env = await fetchContextualVariables(opWs ?? '') // Sign the debug request (creates audit log entry) let signedPayload try { - signedPayload = await signDebugRequest($workspaceStore ?? '', code ?? '', lang ?? 'python3') + signedPayload = await signDebugRequest(opWs ?? '', code ?? '', lang ?? 'python3') debugSessionJobId = signedPayload.job_id } catch (signError) { sendUserToast(getDebugErrorMessage(signError), true) @@ -1289,11 +1443,11 @@ // what's there" affordance, not a user action. Skipped when a test is // already running so a live job's stream is never clobbered. async function loadLastRunIntoTestPanel(): Promise { - if (!path || !$workspaceStore) return + if (!path || !opWs) return if (testIsLoading || testJob !== undefined) return try { const jobs = await JobService.listCompletedJobs({ - workspace: $workspaceStore, + workspace: opWs, scriptPathExact: path, hasNullParent: true, perPage: 1, @@ -1325,7 +1479,7 @@ let token: string | undefined try { - token = await signMultiplayerRequest($workspaceStore ?? '') + token = await signMultiplayerRequest(opWs ?? '') } catch (e) { console.error('Failed to sign multiplayer request:', e) sendUserToast('Failed to authorize multiplayer session', true) @@ -1340,7 +1494,7 @@ wsProvider = new WebsocketProvider( buildWsUrl('/ws_mp/'), - $workspaceStore + '/' + (path ?? 'no-room-name'), + opWs + '/' + (path ?? 'no-room-name'), ydoc, { connect: false, params: { token } } ) @@ -1408,7 +1562,7 @@ let url = new URL(window.location.toString().split('#')[0]) url.search = '' return ( - `${url}?collab=1&workspace=${encodeURIComponent($workspaceStore ?? '')}&lang=${encodeURIComponent(lang ?? '')}` + + `${url}?collab=1&workspace=${encodeURIComponent(opWs ?? '')}&lang=${encodeURIComponent(lang ?? '')}` + (edit ? '' : `&path=${path}`) ) } @@ -1539,18 +1693,29 @@ let error = $derived(getError(testJob)) $effect(() => { - const options: ScriptOptions = { - code, - lang: lang as ScriptLang, - error, - args: args ?? {}, - path, + ;[ + editor, lastSavedCode, lastDeployedCode, diffMode, - workflowAsCode: workflowAsCodeAiContext - } + workflowAsCodeAiContext, + args, + error, + lang, + path + ] untrack(() => { + const options: ScriptOptions = { + getCode: () => code, + lang: lang as ScriptLang, + error, + args: args ?? {}, + path, + lastSavedCode, + lastDeployedCode, + diffMode, + workflowAsCode: workflowAsCodeAiContext + } aiChatManager.scriptEditorOptions = options aiChatManager.scriptEditorApplyCode = async (code: string, opts?: ReviewChangesOpts) => { hideDiffMode() @@ -1568,6 +1733,7 @@ {#if args} { if (wsProvider?.shouldConnect) { @@ -1821,7 +1988,7 @@ it visually pinned to the top edge without relying on cross-browser overflow behaviour. -->
- {#if testJob?.id && testJob.type === 'CompletedJob' && $workspaceStore} + {#if testJob?.id && testJob.type === 'CompletedJob' && opWs} - {@const downstream = customUi!.previewPanel!.downstreamSubscribers!} + knows whether the next run will fan out. A + pure-reader-only root has no subscriber downstream + but still gets `onBoundedRun`, so the split button + also opens for it (with the cascade item hidden). --> + {@const downstream = customUi?.previewPanel?.downstreamSubscribers ?? 0}
- + {#if downstream > 0} + + {/if} + {#if customUi?.previewPanel?.onBoundedRun} + + {/if}
{/snippet} @@ -1958,6 +2152,7 @@ > {#key argsRender} {:else}
@@ -2461,7 +2663,7 @@ client={dapClient} currentFrameId={currentDebugFrameId} onClose={() => (showDebugConsole = false)} - workspace={$workspaceStore} + workspace={opWs} jobId={debugSessionJobId ?? undefined} /> @@ -2485,6 +2687,7 @@ bind:code={editorCode} bind:websocketAlive bind:this={editor} + schemaContractMarkers={contractMarkers} {yContent} awareness={wsProvider?.awareness} on:change={(e) => { @@ -2509,7 +2712,8 @@ } else { await inferModuleSchema() } - runTest() + // The Editor already ran the DDL guard before invoking this action. + runTest({ skipDdlGuard: true }) }} formatAction={async () => { if (activeModuleTab === null) { diff --git a/frontend/src/lib/components/Section.svelte b/frontend/src/lib/components/Section.svelte index 218beec7fa..f1425c51b7 100644 --- a/frontend/src/lib/components/Section.svelte +++ b/frontend/src/lib/components/Section.svelte @@ -104,7 +104,7 @@ {#if description}
{@html description}
{/if} -
+
{@render children?.()}
diff --git a/frontend/src/lib/components/SimpleEditor.svelte b/frontend/src/lib/components/SimpleEditor.svelte index 741d874f81..f1c4daa962 100644 --- a/frontend/src/lib/components/SimpleEditor.svelte +++ b/frontend/src/lib/components/SimpleEditor.svelte @@ -95,6 +95,7 @@ loadAsync = false, key, disabled = false, + readOnly = false, minHeight = 1000, renderLineHighlight = 'none', suggestion @@ -123,6 +124,9 @@ initialCursorPos?: IPosition key?: string disabled?: boolean + /** Read-only Monaco mode: not editable, but still scrollable/selectable + * (unlike `disabled`, which makes the editor non-interactive). */ + readOnly?: boolean minHeight?: number renderLineHighlight?: 'all' | 'line' | 'gutter' | 'none' suggestion?: string @@ -239,6 +243,9 @@ lineNumbers: $relativeLineNumbers ? 'relative' : 'on' }) }) + $effect(() => { + editor?.updateOptions({ readOnly }) + }) function onVimDisable() { vimDisposable?.dispose() @@ -342,6 +349,7 @@ ), model, ...(yPadding !== undefined ? { padding: { bottom: yPadding, top: yPadding } } : {}), + readOnly, renderLineHighlight, lineDecorationsWidth: 0, lineNumbersMinChars: 2, diff --git a/frontend/src/lib/components/SqlRepl.svelte b/frontend/src/lib/components/SqlRepl.svelte index 661b0ccd2c..5ca069b5be 100644 --- a/frontend/src/lib/components/SqlRepl.svelte +++ b/frontend/src/lib/components/SqlRepl.svelte @@ -1,49 +1,3 @@ - - @@ -37,20 +43,26 @@ {/if} {#snippet text()} - {id} + {#if tooltip} + {tooltip} + {:else} + {id} + {/if} {/snippet} {#if len > 0} {@const narrow = len / total < 0.09} diff --git a/frontend/src/lib/components/UserOffboardingModal.svelte b/frontend/src/lib/components/UserOffboardingModal.svelte index 8f6d375ca0..26d4e83256 100644 --- a/frontend/src/lib/components/UserOffboardingModal.svelte +++ b/frontend/src/lib/components/UserOffboardingModal.svelte @@ -43,6 +43,7 @@ let ownedCount = $derived(preview ? countPaths(preview.owned) : 0) let onBehalfCount = $derived(preview ? countPaths(preview.executing_on_behalf) : 0) let hasItems = $derived(ownedCount > 0 || onBehalfCount > 0) + let canReassignHere = $derived(hasItems && users.length > 0) let reassignTo = $derived( targetKind === 'user' @@ -83,6 +84,11 @@ preview = previewResult users = usernamesList.filter((u) => u !== username).map((u) => ({ label: u, value: u })) folders = foldersList.map((f) => ({ label: f.name, value: f.name })) + // Reassignment needs another workspace user as target/operator; with none + // (sole-member workspace) it is impossible, so fall back to plain removal. + if (!reassignOnly && users.length === 0) { + doReassign = false + } } catch (e) { sendUserToast('Failed to load offboard preview', true) onClose() @@ -173,33 +179,40 @@ {:else if preview}
{#if hasItems} - {#if !reassignOnly} - - {/if} - - {#if doReassign} - + {#if users.length === 0} +

+ No other users in this workspace. Items will be left as-is. +

{:else} - -

- All items owned by {username} ({ownedCount} owned, {onBehalfCount} running on behalf) - will be left as-is. Triggers and runnables may stop working if the user is removed. -

-
+ {#if !reassignOnly} + + {/if} + + {#if doReassign} + + {:else} + +

+ All items owned by {username} ({ownedCount} owned, {onBehalfCount} running on + behalf) will be left as-is. Triggers and runnables may stop working if the user + is removed. +

+
+ {/if} {/if} {:else}

@@ -214,7 +227,7 @@ different user/folder.

    - {#each conflicts as conflict} + {#each conflicts as conflict, i (i)}
  • {conflict}
  • {/each}
@@ -224,7 +237,7 @@ {/if}
- {#if hasItems || deleteUser} + {#if deleteUser || canReassignHere} {/if} diff --git a/frontend/src/lib/components/WorkspaceScopeTrigger.svelte b/frontend/src/lib/components/WorkspaceScopeTrigger.svelte new file mode 100644 index 0000000000..ac132bdca2 --- /dev/null +++ b/frontend/src/lib/components/WorkspaceScopeTrigger.svelte @@ -0,0 +1,205 @@ + + +{#snippet chipButton(grouped: boolean)} + +{/snippet} + +{#if menuItems?.length && !isCollapsed} +
+ {@render chipButton(true)} + + {#snippet buttonReplacement()} + + + + {/snippet} + +
+{:else} + {@render chipButton(false)} +{/if} diff --git a/frontend/src/lib/components/aiProviderStorage.ts b/frontend/src/lib/components/aiProviderStorage.ts index ce6af230bb..7f6d705d3a 100644 --- a/frontend/src/lib/components/aiProviderStorage.ts +++ b/frontend/src/lib/components/aiProviderStorage.ts @@ -50,6 +50,7 @@ export function isSameAsStoredConfig(config: ProviderConfig | undefined): boolea storedConfig !== undefined && storedConfig.kind === config?.kind && storedConfig.resource === config?.resource && - storedConfig.model === config?.model + storedConfig.model === config?.model && + storedConfig.reasoning_effort === config?.reasoning_effort ) } diff --git a/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte b/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte index 85f0e3193e..14832214b0 100644 --- a/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte +++ b/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte @@ -13,7 +13,7 @@ import { twMerge } from 'tailwind-merge' import type { Output } from '../../rx' import ResolveNavbarItemPath from './ResolveNavbarItemPath.svelte' - import { urlParamsToObject } from '$lib/utils' + import { urlParamsToObject, WINDMILL_RESERVED_QUERY_PARAMS } from '$lib/utils' interface Props { navbarItem: NavbarItem @@ -54,8 +54,22 @@ let resolvedHidden: boolean | undefined = $state(undefined) function extractPathDetails() { - const url = window.location.pathname + window.location.search + window.location.hash - const processedUrl = url.replace('/apps/edit/', '').replace('/apps/get/', '') + // Drop Windmill transport params (wm_embed, …) so they don't poison the + // comparison against the item's resolved path. + const params = new URLSearchParams(window.location.search) + const reserved: string[] = [] + params.forEach((_v, k) => { + if (WINDMILL_RESERVED_QUERY_PARAMS.has(k)) reserved.push(k) + }) + reserved.forEach((k) => params.delete(k)) + const qs = params.toString() + const url = window.location.pathname + (qs ? `?${qs}` : '') + window.location.hash + // `/app_embed/{workspace}/` is the opaque in-workspace viewer route + // (WIN-2006) — same app-path suffix as `/apps/get/`. + const processedUrl = url + .replace('/apps/edit/', '') + .replace('/apps/get/', '') + .replace(/^\/app_embed\/[^/]+\//, '') return processedUrl } diff --git a/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte b/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte index ba17f6c7c3..8e068274b8 100644 --- a/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte +++ b/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte @@ -86,7 +86,10 @@ } const resolvedConfig = $state( - initConfig(components['dbexplorercomponent'].initialData.configuration, untrack(() => configuration)) + initConfig( + components['dbexplorercomponent'].initialData.configuration, + untrack(() => configuration) + ) ) let timeoutInput: number | undefined = undefined @@ -180,18 +183,22 @@ ) } - let outputs = initOutput($worldStore, untrack(() => id), { - selectedRowIndex: 0, - selectedRow: {}, - selectedRows: [] as any[], - result: [] as any[], - inputs: {}, - loading: false, - page: 0, - newChange: { row: 0, column: '', value: undefined }, - ready: undefined as boolean | undefined, - openedModalRow: {} - }) + let outputs = initOutput( + $worldStore, + untrack(() => id), + { + selectedRowIndex: 0, + selectedRow: {}, + selectedRows: [] as any[], + result: [] as any[], + inputs: {}, + loading: false, + page: 0, + newChange: { row: 0, column: '', value: undefined }, + ready: undefined as boolean | undefined, + openedModalRow: {} + } + ) let lastResource: string | undefined = undefined @@ -260,9 +267,7 @@ resolvedConfig.type, { table: { - selectOptions: dbSchemas - ? await getTablesByResource(dbSchemas, dbtype, dbPath, $workspaceStore!) - : [], + selectOptions: dbSchemas ? await getTablesByResource(dbSchemas, dbtype) : [], loading: false } } diff --git a/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts b/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts index 3ec21a3801..1d0fa63342 100644 --- a/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts +++ b/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts @@ -1,4 +1,4 @@ -import { JobService, ResourceService } from '$lib/gen' +import { JobService } from '$lib/gen' import { runScriptAndPollResult } from '$lib/components/jobs/utils' import type { DbInput } from '$lib/components/dbTypes' @@ -39,17 +39,9 @@ export async function loadTableMetaData( const ducklake = input.type === 'ducklake' ? input.ducklake : undefined const dbArg = getDatabaseArg(input) - // MySQL needs the database name for metadata queries - let databaseName: string | undefined - if (input.type === 'database' && input.resourceType === 'mysql') { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: input.resourcePath - })) as any - databaseName = resourceObj?.database - } - - const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table, databaseName }, ducklake) + // MySQL: the metadata query resolves the database name server-side (it falls + // back to `DATABASE()`), so we don't read the resource value client-side for it. + const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table }, ducklake) const job = await JobService.runScriptPreview({ workspace, @@ -106,22 +98,10 @@ export async function loadAllTablesMetaData( const dbArg = getDatabaseArg(input) const ducklake = input.type === 'ducklake' ? input.ducklake : undefined - // MySQL needs the database name for metadata queries - let databaseName: string | undefined - if (input.type === 'database' && input.resourceType === 'mysql') { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: input.resourcePath - })) as any - databaseName = resourceObj?.database - } - const language = getLanguageByResourceType(dbType) - const content = makeMetadataMarker( - 'LOAD_TABLE_METADATA', - { table: undefined, databaseName }, - ducklake - ) + // MySQL db name is resolved server-side via `DATABASE()` (see loadTableMetaData); + // no client-side resource-value read. + const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table: undefined }, ducklake) let result = (await runScriptAndPollResult({ workspace, @@ -259,7 +239,11 @@ export async function getDbSchemas( const dbSchema = { lang: resourceTypeToLang(resourceType) as SQLSchema['lang'], schema, - publicOnly: !!schema.public || !!schema.PUBLIC || !!schema.dbo + publicOnly: !!schema.public || !!schema.PUBLIC || !!schema.dbo, + // MySQL introspection selects `DATABASE() AS default_db_name`; carry it + // so the table picker can tell the default db apart from other visible + // schemas. Other dbs don't return it (stays undefined). + defaultDb: Array.isArray(result) ? (result[0] as any)?.default_db_name : undefined } return { ...dbSchema, stringified: stringifySchema(dbSchema) } } else { @@ -283,9 +267,7 @@ export async function getDbSchemas( export async function getTablesByResource( schema: Partial>, - dbType: DbType | undefined, - dbPath: string, - workspace: string + dbType: DbType | undefined ): Promise { const s = Object.values(schema)?.[0] switch (dbType) { @@ -301,14 +283,15 @@ export async function getTablesByResource( return paths } case 'mysql': { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: dbPath.split('$res:')[1] - })) as any + // MySQL introspection lists DATABASE() plus any other visible non-system + // schemas. Show the default db's tables unprefixed and the rest as + // `db.table` — matching the pre-removal behavior (which matched the + // resource's `database`); `defaultDb` is the connection's DATABASE(). + const defaultDb = s && 'defaultDb' in s ? s.defaultDb : undefined const paths: string[] = [] for (const key in s?.schema) { for (const subKey in s.schema[key]) { - if (key === resourceObj?.database) { + if (key === defaultDb) { paths.push(`${subKey}`) } else { paths.push(`${key}.${subKey}`) diff --git a/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte b/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte index dd294be6af..c7c4e7d160 100644 --- a/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte +++ b/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte @@ -3,7 +3,7 @@ import type { AppInput } from '../../inputType' import type { Output } from '../../rx' import type { AppViewerContext, ListContext } from '../../types' - import { isScriptByNameDefined, isScriptByPathDefined } from '../../utils' + import { appNavigateSameWindow, isScriptByNameDefined, isScriptByPathDefined } from '../../utils' import NonRunnableComponent from './NonRunnableComponent.svelte' import RunnableComponent from './RunnableComponent.svelte' import { sendUserToast } from '$lib/toast' @@ -261,7 +261,9 @@ if (newTab) { window.open(gotoUrl, '_blank') } else { - window.location.href = gotoUrl + // Top-level load; inside the opaque viewer iframe this targets the + // top page (pre-sandbox behavior) instead of the cookieless frame. + appNavigateSameWindow(gotoUrl) } break diff --git a/frontend/src/lib/components/apps/components/helpers/eval.ts b/frontend/src/lib/components/apps/components/helpers/eval.ts index 86523b2959..bacfb65b2d 100644 --- a/frontend/src/lib/components/apps/components/helpers/eval.ts +++ b/frontend/src/lib/components/apps/components/helpers/eval.ts @@ -2,6 +2,8 @@ import type { World } from '../../rx' import { sendUserToast } from '$lib/toast' import { waitJob } from '$lib/components/waitJob' import { base } from '$lib/base' +import { appNavigateSameWindow } from '../../utils' +import { OpenAPI } from '$lib/gen/core/OpenAPI' export function computeGlobalContext( world: World | undefined, @@ -200,7 +202,9 @@ export async function eval_like( } window.open(x, '_blank') } else { - window.location.href = x + // Top-level load; inside the opaque viewer iframe this targets the + // top page (pre-sandbox behavior) instead of the cookieless frame. + appNavigateSameWindow(x) } }, (id, index) => { @@ -292,10 +296,30 @@ export async function eval_like( if (typeof input === 'object' && input.s3) { const workspaceId = ((context ?? {}) as any).ctx?.workspace - const s3href = `${base}/api/w/${workspaceId}/job_helpers/download_s3_file?file_key=${encodeURIComponent( - input?.s3 ?? '' - )}${input?.storage ? `&storage=${input.storage}` : ''}` - downloadFile(s3href, filename || input.s3) + const appPath = ((context ?? {}) as any).ctx?.app_path + let inSandbox = false + try { + inSandbox = + window.parent !== window && + new URLSearchParams(window.location.search).get('wm_embed') === '1' + } catch (_) {} + if (inSandbox && appPath && typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + // Sandboxed viewer: the opaque iframe carries no cookie, so the + // cookie-authed job_helpers download fails. Route through the + // app-policy-confined apps_u endpoint with the embed token in the + // query (like the image/file components), scoped to this app's path. + const params = new URLSearchParams() + params.append('s3', input.s3 ?? '') + if (input.storage) params.append('storage', input.storage) + params.append('token', OpenAPI.TOKEN) + const s3href = `${base}/api/w/${workspaceId}/apps_u/download_s3_file/${appPath}?${params.toString()}` + downloadFile(s3href, filename || input.s3) + } else { + const s3href = `${base}/api/w/${workspaceId}/job_helpers/download_s3_file?file_key=${encodeURIComponent( + input?.s3 ?? '' + )}${input?.storage ? `&storage=${input.storage}` : ''}` + downloadFile(s3href, filename || input.s3) + } } else if (typeof input === 'string') { if (input.startsWith('data:')) { downloadFile(input, filename) diff --git a/frontend/src/lib/components/apps/editor/AppEditor.svelte b/frontend/src/lib/components/apps/editor/AppEditor.svelte index 0cffdc2400..64cc94f551 100644 --- a/frontend/src/lib/components/apps/editor/AppEditor.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditor.svelte @@ -69,6 +69,7 @@ path, policy, summary, + labels, deployedBaseline = undefined, fromHub = false, diffDrawer = undefined, @@ -83,7 +84,8 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore }: AppEditorProps = $props() migrateApp(untrack(() => app)) @@ -885,12 +887,13 @@ void @@ -112,6 +120,10 @@ loadedFromDraft?: boolean othersDraftsCount?: number onOpenOthersDrafts?: () => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void } let { @@ -125,6 +137,7 @@ bottomPanelHidden = false, newApp, newPath = '', + labels: initialLabels = undefined, userDraftPath = '', onSavedNewAppPath, onShowLeftPanel, @@ -137,7 +150,8 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore }: Props = $props() /** Mirror of the path the user is editing in the pen popover. Initialized @@ -256,7 +270,8 @@ policy, deployment_message: deploymentMsg, custom_path: customPath, - preserve_on_behalf_of: preserveOnBehalfOf || undefined + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels } }) // New path now exists server-side — drop the autocomplete cache so @@ -267,7 +282,8 @@ value: structuredClone($state.snapshot($app)), path: path, policy: policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } closeSaveDrawer() sendUserToast('App deployed successfully') @@ -310,7 +326,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels }) ) ) { @@ -361,7 +378,8 @@ // it also means that customPath needs to be set to '' instead of undefined to unset it (when admin) custom_path: $userStore?.is_admin || $userStore?.is_super_admin ? (customPath ?? '') : undefined, - preserve_on_behalf_of: preserveOnBehalfOf || undefined + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels } }) invalidateWorkspacePaths($workspaceStore!) @@ -370,13 +388,19 @@ value: structuredClone($state.snapshot($app)), path: npath, policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } const appHistory = await AppService.getAppHistoryByPath({ workspace: $workspaceStore!, path: npath }) version = appHistory[0]?.version + // Re-pin the fork base to the just-deployed head: the editor stays open, so a + // follow-up deploy (or a new edit) would otherwise compare against the now- + // superseded base and falsely warn. parent_version is in + // DRAFT_COMPARE_IGNORED_FIELDS, so this write can't spawn a spurious draft. + if ($app) $app.parent_version = version closeSaveDrawer() sendUserToast('App deployed successfully') @@ -395,14 +419,16 @@ } } - async function setPublishState() { + async function setPublishState(message?: string) { policy = await updatePolicy($app, policy) await AppService.updateApp({ workspace: $workspaceStore!, path: $appPath, requestBody: { policy } }) - if (policy.execution_mode == 'anonymous') { + if (message) { + sendUserToast(message) + } else if (policy.execution_mode == 'anonymous') { sendUserToast('App require no login to be accessed') } else { sendUserToast('App require login and read-access') @@ -419,7 +445,12 @@ let onLatest = $state(true) async function compareVersions() { - if (version === undefined) { + // Compare the draft's pinned fork base (`$app.parent_version`) against the + // current head when editing a draft, else the load-time head. Catches both a + // concurrent deploy (head moved since open) AND a stale draft reopened after a + // deploy (head == load-time head, but the draft was forked from an older one). + const base = $app?.parent_version ?? version + if (base === undefined) { return } try { @@ -427,7 +458,7 @@ workspace: $workspaceStore!, path: $appPath }) - onLatest = appVersion?.version === undefined || version === appVersion?.version + onLatest = appVersion?.version === undefined || base === appVersion?.version } catch (e) { console.error('Error comparing versions', e) onLatest = true @@ -612,7 +643,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels } }) }, @@ -711,6 +743,7 @@ }) let customPath = $state(savedApp?.custom_path) + let labels = $state(untrack(() => initialLabels)) $effect(() => { if ($openDebugRun == undefined) { @@ -740,7 +773,8 @@ value: $app, path: newEditedPath || savedApp?.path, policy, - custom_path: customPath + custom_path: customPath, + labels }} /> @@ -784,7 +818,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels }, button: { text: 'Looks good, deploy', @@ -837,6 +872,7 @@ bind:pathError bind:newEditedPath bind:preserveOnBehalfOf + bind:labels hideSecretUrl={false} /> @@ -850,7 +886,7 @@ (historyBrowserDrawerOpen = false)}> - + onRestore?.(e.detail)} appPath={$appPath} /> @@ -872,15 +908,22 @@ bind:clientWidth={topbarWidth} class="flex flex-row justify-between gap-2 gap-y-2 px-2 items-center overflow-y-visible overflow-x-auto max-h-12 h-12 shrink-0" > -
- (onNavigate ? onNavigate(item) : goto(editPathFor(item)))} - /> -
+ +
+
+ (onNavigate ? onNavigate(item) : goto(editPathFor(item)))} + /> +
+
{#if $app} {#if $mode !== 'preview'} -
+
diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte index c9cc92f3b8..64bd2790cb 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte @@ -13,10 +13,14 @@ import RunnableNode from './RunnableNode.svelte' import TriggerNode, { type TriggerNodeKind } from './TriggerNode.svelte' import AddNode from './AddNode.svelte' + import DataTestNode from './DataTestNode.svelte' import AssetGraphEdge from './AssetGraphEdge.svelte' import PanToNode from './PanToNode.svelte' + import InitialFitView from './InitialFitView.svelte' import { layoutAssetGraph } from './assetGraphLayout' + import { computeMutedReadKeys } from './resolveGraph' import { buildDownstreamMap } from './graphTraversal' + import { buildLineageDownstreamMap } from './boundedCascade' import type { AssetGraphResponse, AssetGraphSelection, NativeTriggerKind } from './types' import type { RunnableRunState } from './activeRunnables.svelte' import type { AssetKind } from '$lib/gen' @@ -126,6 +130,9 @@ // form (with the auto-generated S3 picker) for the given script. // data_upload has no trigger row; it's a UI-first entry point. onOpenDataUpload?: (scriptPath: string) => void + // Data-upload entry scripts whose file has been staged in the run form — + // their source node renders green (ready) instead of the neutral prompt. + readyDataUploadPaths?: ReadonlySet // Script paths the cursor is over in the Activity panel — a thin neutral // ring each. One entry for a single run row; the whole cascade's runs // when hovering a group header. Distinct from `selectedRunPaths`. @@ -139,6 +146,31 @@ // editor passes it, so the asset-graph page is unaffected. The page // clears it once the pan has had time to settle. panToNodeId?: string | undefined + // Script paths eligible to *start* a bounded-cascade run — roots AND + // mid-DAG models (see boundedCascade.validFromStarts). When a path is + // in this set and `onStartBoundedRun` is wired, its node's cascade menu + // gains a "Run downstream up to…" entry. + validStartPaths?: ReadonlySet + onStartBoundedRun?: (startPath: string) => void + // Active bounded-run pick mode. Ids are in *canvas* space (`script:path` + // for runnables, `asset:${kind}:${path}` for assets). When set the canvas + // dims nodes outside `eligible ∪ {start}`, rings the `bounded` set, marks + // `ends`, and routes clicks on eligible nodes to `onPickEnd` instead of + // selecting. + boundPick?: { + start: string + eligible: ReadonlySet + ends: ReadonlySet + bounded: ReadonlySet + } + onPickEnd?: (canvasNodeId: string) => void + /** Hide the minimap when the canvas is too narrow for it to be worth the + * space (e.g. stacked layout in a side panel). Defaults to shown. */ + showMinimap?: boolean + /** Identity of the displayed graph (e.g. the pipeline folder). The + * initial viewport fit re-arms when it changes, so switching folders + * in-place gets a fresh fit. */ + viewportFitKey?: string } let { graph, @@ -158,9 +190,16 @@ onDeleteTrigger, onOpenWebhook, onOpenDataUpload, + readyDataUploadPaths, hoveredPaths, selectedRunPaths, - panToNodeId + panToNodeId, + validStartPaths, + onStartBoundedRun, + boundPick, + onPickEnd, + showMinimap = true, + viewportFitKey = '' }: Props = $props() // `${kind}:${path}` ids for the hovered / pinned runs (both script and flow @@ -180,12 +219,35 @@ // the DAG. They force sugiyama to put + at layer 0 (top) and center // it horizontally over the roots — same mechanism the flow editor // uses for its Trigger node. Filtered out of rendered edges. - kind: 'lineage-write' | 'lineage-read' | 'trigger-asset' | 'trigger-native' | 'add-anchor' + kind: + | 'lineage-write' + | 'lineage-read' + | 'trigger-asset' + | 'trigger-native' + | 'add-anchor' + | 'data-test' + | 'macro' + | 'test-dependency' unsaved?: boolean + // Muted read edge: a ducklake/s3 input read every run whose (default) + // auto cascade trigger is suppressed by `// mute` / `// mute all`. + // Rendered with a bell-off badge — auto-wiring is the norm, so we mark + // the read that deliberately does NOT cascade, not every derived edge. + muted?: boolean // Edge from a missing-trigger placeholder — styled red dashed to // signal "this script declared `// on kafka` but no trigger row // targets it; create one or remove the annotation". missing?: boolean + // Producer's `// data_test` checks, on the write-edge to the + // materialized asset — rendered as a flask badge on the link. + data_tests?: NonNullable + // Producer's `// column` declared lineage, on the same write-edge — + // rendered as a columns badge on the link. + column_lineage?: NonNullable + // Macro-library edge payload: the macros the consumer calls (or the + // whole lib when pulled in via `// use`) — rendered as a ƒ badge. + macro_names?: string[] + via_use?: boolean } // Graph-id of the script the user just launched (zero-latency hint), @@ -200,10 +262,14 @@ function build(g: AssetGraphResponse) { const nodes: Array<{ id: string - type: 'asset' | 'runnable' | 'trigger' | 'add' + type: 'asset' | 'runnable' | 'trigger' | 'add' | 'data-test' data: any }> = [] const edges: BuiltEdge[] = [] + // Custom (`// data_test `) tests are deployed scripts, so we + // draw each as its own node hanging off the asset it validates (deduped + // by node id across producers). + const addedTestNodes = new Set() const hasAddNode = onAddPipelineScript != null if (hasAddNode) { @@ -259,20 +325,67 @@ downstreamByScript.set(path, set.size) downstreamUnsavedByScript.set(path, [...set].filter((s) => unsavedRunnables.has(s)).length) } + // Read-aware downstream presence for the bounded-run gate. The bounded + // engine treats pure-read `asset → script` edges as downstream, but the + // subscriber-only `downstreamByScript` above does not — so a start whose + // only downstream is a pure reader would otherwise be denied the menu + // item even though its bounded set is non-empty. Mirror of the engine's + // own adjacency (boundedCascade.buildLineageDownstreamMap). + const hasLineageDownstream = new Set(buildLineageDownstreamMap(g).keys()) + + // Producers that declare `// data_test` checks, keyed by runnable id — the + // asset node uses this (plus the producer's run state) to render the + // data-test outcome badge. Tests only assert on ducklake `// materialize` + // targets (v1), so guard status is a ducklake-only concept below. + const producerHasTests = new Set() + // Producer → its declared `// materialize` target, so a multi-output + // producer's guard badge lands only on the table its tests assert on — + // not on its other ducklake outputs. Mirrors the write-edge badge anchor. + const guardMaterializeTarget = new Map< + string, + NonNullable + >() + for (const r of g.runnables) { + if (r.data_tests && r.data_tests.length > 0) producerHasTests.add(`${r.usage_kind}:${r.path}`) + if (r.materialize_target) + guardMaterializeTarget.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } for (const a of g.assets) { const assetId = `asset:${a.kind}:${a.path}` + // Guard/outcome badge inputs: is a producer of this (ducklake) asset + // test-guarded, and did that guarded producer's latest run fail? + let dataTestGuarded = false + let producerFailed = false + if (a.kind === 'ducklake') { + for (const p of producersByAsset.get(`${a.kind}:${a.path}`) ?? []) { + const rid = `${p.kind}:${p.path}` + if (!producerHasTests.has(rid)) continue + // Tests assert on the producer's `// materialize` target; when the + // producer declares one, only that table is guarded (a multi-output + // producer must not badge its other ducklake writes). No declared + // target → single-output producer, so its lone ducklake write is it. + const mt = guardMaterializeTarget.get(rid) + if (mt && !(mt.kind === a.kind && mt.path === a.path)) continue + dataTestGuarded = true + if (runStates?.get(rid)?.status === 'failure') producerFailed = true + } + } nodes.push({ id: assetId, type: 'asset', data: { asset_kind: a.kind, path: a.path, + fork_materialization: a.fork_materialization, + derived_from: a.derived_from, onAddScript: onAddScriptForAsset, pathPrefix, defaultPathSuffix, producers: producersByAsset.get(`${a.kind}:${a.path}`) ?? [], - onRunProducer + onRunProducer, + dataTestGuarded, + producerFailed } }) } @@ -284,6 +397,40 @@ for (const r of g.runnables) { if (r.unsaved) unsavedRunnablePaths.add(r.path) } + // Producer → its `// data_test` checks, keyed by runnable id, so the + // write-edge to the materialized asset can carry the test badge: the + // edge *is* the transformation, and the tests assert on what it produces. + const producerTests = new Map< + string, + NonNullable + >() + for (const r of g.runnables) { + if (r.data_tests && r.data_tests.length > 0) { + producerTests.set(`${r.usage_kind}:${r.path}`, r.data_tests) + } + } + // Producer → its `// column` declared lineage, keyed by runnable id, so + // the write-edge to the materialized asset can carry the columns badge — + // the edge *is* the transformation, and the lineage describes its output. + const producerColumnLineage = new Map< + string, + NonNullable + >() + // The `// materialize` target the lineage describes, so the badge lands + // only on that write-edge (a multi-output script writes several ducklake + // tables) — mirrors the anchor logic in `buildColumnGraph`. + const producerMaterializeTarget = new Map< + string, + NonNullable + >() + for (const r of g.runnables) { + if (r.column_lineage && r.column_lineage.length > 0) { + producerColumnLineage.set(`${r.usage_kind}:${r.path}`, r.column_lineage) + } + if (r.materialize_target) { + producerMaterializeTarget.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } + } for (const r of g.runnables) { const rid = `${r.usage_kind}:${r.path}` // Optimistic badge: the moment a run is launched from this view @@ -305,8 +452,10 @@ in_pipeline: r.in_pipeline ?? false, partition_kind: r.partition_kind, freshness: r.freshness, + last_success_at: r.last_success_at, tag: r.tag, retry: r.retry, + macros: r.macros, unsaved: r.unsaved ?? false, // Same dispatch the asset node uses, only routed when the // runnable is a script (the page handler short-circuits @@ -325,6 +474,16 @@ downstreamCount: downstreamByScript.get(r.path) ?? 0, downstreamUnsavedCount: downstreamUnsavedByScript.get(r.path) ?? 0, runState, + // Bounded-cascade entrypoint: only valid starts (schedule / + // manual roots) with downstream get the "Run downstream up + // to…" menu item. + onStartBoundedRun: + r.usage_kind === 'script' && + onStartBoundedRun && + validStartPaths?.has(r.path) && + hasLineageDownstream.has(r.path) + ? () => onStartBoundedRun(r.path) + : undefined, onRequestRemove: onRunnableMenuRemove ? () => onRunnableMenuRemove({ @@ -337,18 +496,55 @@ }) } + // Read edges of a ducklake/s3 asset with no cascade trigger = muted + // (`// mute` / `// mute all` opted the default auto trigger out). Gated + // on pipeline scripts inside the helper (non-pipeline reads never derive). + const mutedReadKeys = computeMutedReadKeys(g.edges, g.triggers, g.runnables) for (const e of g.edges) { const runnableId = `${e.runnable_kind}:${e.runnable_path}` const assetId = `asset:${e.asset_kind}:${e.asset_path}` const access = e.access_type ?? 'r' if (access === 'w' || access === 'rw') { + // Data tests assert on the `// materialize` target, which is always + // a ducklake asset (v1 enforces this), so only the ducklake + // write-edge carries the badge / custom-test nodes — a producer's + // other (e.g. S3/datatable) outputs must not show them. + const edgeTests = e.asset_kind === 'ducklake' ? producerTests.get(runnableId) : undefined + // Column lineage describes one materialized output, so the badge + // lands only on that asset's write-edge: the declared `// materialize` + // target when known (a multi-output script writes several ducklake + // tables), else the ducklake write-edge as the unambiguous fallback. + const matTarget = producerMaterializeTarget.get(runnableId) + const isOutputEdge = matTarget + ? e.asset_kind === matTarget.kind && e.asset_path === matTarget.path + : e.asset_kind === 'ducklake' + const edgeColumnLineage = isOutputEdge ? producerColumnLineage.get(runnableId) : undefined edges.push({ id: `prod:${runnableId}->${assetId}`, source: runnableId, target: assetId, kind: 'lineage-write', - unsaved: e.unsaved + unsaved: e.unsaved, + data_tests: edgeTests, + column_lineage: edgeColumnLineage }) + // Each custom (`// data_test `) test → its own node + // below the asset it validates, with a dashed "tests" edge. + for (const t of edgeTests ?? []) { + if (t.type !== 'custom') continue + const testNodeId = `datatest:${assetId}:${t.path}` + if (!addedTestNodes.has(testNodeId)) { + addedTestNodes.add(testNodeId) + nodes.push({ id: testNodeId, type: 'data-test', data: { path: t.path } }) + } + edges.push({ + id: `test:${assetId}->${testNodeId}`, + source: assetId, + target: testNodeId, + kind: 'data-test', + unsaved: e.unsaved + }) + } } if (access === 'r' || access === 'rw') { edges.push({ @@ -356,11 +552,53 @@ source: assetId, target: runnableId, kind: 'lineage-read', - unsaved: e.unsaved + unsaved: e.unsaved, + // Only a pure `'r'` read can be muted; `'rw'` is a self-read. + muted: + access === 'r' && + mutedReadKeys.has( + `${e.asset_kind}:${e.asset_path}->${e.runnable_kind}:${e.runnable_path}` + ) }) } } + // Macro-library → consumer edges (runnable→runnable, unlike the + // asset-mediated lineage above). Endpoints must exist as nodes — an + // undeployed `// use` target without a draft has none, so its edge is + // skipped (the annotation still shows in the script body itself). + const runnableNodeIds = new Set(g.runnables.map((r) => `${r.usage_kind}:${r.path}`)) + for (const me of g.macro_edges ?? []) { + const libId = `script:${me.lib_path}` + const consumerId = `script:${me.consumer_path}` + if (!runnableNodeIds.has(libId) || !runnableNodeIds.has(consumerId)) continue + edges.push({ + id: `macro:${libId}->${consumerId}`, + source: libId, + target: consumerId, + kind: 'macro', + unsaved: me.unsaved, + macro_names: me.macro_names, + via_use: me.via_use + }) + } + + // Data-test ordering edges (producer → tested script): the referenced + // asset must be materialized before the test runs, so the cascade orders + // the producer first. Rendered as a dashed "must run after" link, distinct + // from data flow — the tested script doesn't consume the asset's rows. + for (const te of g.test_edges ?? []) { + const producerId = `${te.producer_kind}:${te.producer_path}` + const testedId = `${te.runnable_kind}:${te.runnable_path}` + if (!runnableNodeIds.has(producerId) || !runnableNodeIds.has(testedId)) continue + edges.push({ + id: `testdep:${producerId}->${testedId}`, + source: producerId, + target: testedId, + kind: 'test-dependency' + }) + } + // Non-asset triggers (schedule + native) are rendered as source nodes // above the pipeline script. Real (non-missing) nodes are // deduplicated per (kind, ref) tuple so a single schedule shared @@ -375,7 +613,13 @@ kind: TriggerNodeKind ref: string missing: boolean + // First target script (drives the per-script create/edit flows). runnable_path?: string + // Every target script: a single (kind, ref) — e.g. one schedule — + // can be shared across scripts and dedupes to one node with N + // edges. The bounded-run action must consider all of them, not + // just the first. + runnable_paths: string[] } >() function recordSourceTrigger( @@ -388,9 +632,19 @@ ) { const prev = triggerSourceNodes.get(id) if (!prev) { - triggerSourceNodes.set(id, { allUnsaved: unsaved, kind, ref, missing, runnable_path }) + triggerSourceNodes.set(id, { + allUnsaved: unsaved, + kind, + ref, + missing, + runnable_path, + runnable_paths: runnable_path ? [runnable_path] : [] + }) } else { prev.allUnsaved = prev.allUnsaved && unsaved + if (runnable_path && !prev.runnable_paths.includes(runnable_path)) { + prev.runnable_paths.push(runnable_path) + } } } @@ -449,6 +703,14 @@ }) } for (const [id, info] of triggerSourceNodes) { + // Bounded-run targets among this trigger's scripts: valid starts that + // also have read-aware downstream. A shared schedule may have several + // targets; only offer the action when exactly one qualifies, so the + // run starts from an unambiguous script (rather than the arbitrary + // first-seen one). Multi-eligible nodes suppress it rather than guess. + const eligibleStarts = onStartBoundedRun + ? info.runnable_paths.filter((p) => validStartPaths?.has(p) && hasLineageDownstream.has(p)) + : [] nodes.push({ id, type: 'trigger', @@ -461,11 +723,23 @@ runnable_unsaved: info.runnable_path ? unsavedRunnablePaths.has(info.runnable_path) : false, + // data_upload nodes go green once a file is staged for their + // target script (see readyDataUploadPaths / page dataUploadArgs). + ready: info.runnable_path + ? (readyDataUploadPaths?.has(info.runnable_path) ?? false) + : false, onCreateMissingTrigger, onEditTrigger, onDeleteTrigger, onOpenWebhook, - onOpenDataUpload + onOpenDataUpload, + // View-mode bounded-run entry: offer it on the trigger node + // when exactly one target script is a valid start with + // downstream (see eligibleStarts above). + onStartBoundedRun: + onStartBoundedRun && eligibleStarts.length === 1 + ? () => onStartBoundedRun(eligibleStarts[0]) + : undefined } }) } @@ -518,11 +792,27 @@ // edges + paths → same layout. Renames *do* change the layout for // the renamed entry, since its id moves in the sort, but that's // expected: a rename is a path change, which is part of the input. + // A script with rw access to an asset yields both a write (script → asset) + // and a read/trigger (asset → script) edge — a 2-cycle. The layout resolves + // it producer-above-asset by omitting the backward direction from its + // input; the rendered edges are untouched (both arrows still drawn). + let writeEdgePairs = $derived( + new Set( + model.edges.filter((e) => e.kind === 'lineage-write').map((e) => `${e.source}\n${e.target}`) + ) + ) let layoutInput = $derived({ nodes: model.nodes .map((n) => ({ id: n.id, data: n.data })) .sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)), edges: model.edges + .filter( + (e) => + !( + (e.kind === 'lineage-read' || e.kind === 'trigger-asset') && + writeEdgePairs.has(`${e.target}\n${e.source}`) + ) + ) .map((e) => ({ source: e.source, target: e.target })) .sort((a, b) => a.source === b.source @@ -573,12 +863,22 @@ : assetEmph === 'input' ? 'wm-asset-input' : undefined + // Bounded-run pick overlay takes precedence over the activity rings + // (it's a transient, modal selection). start ring > end mark > in-set + // ring > eligible (clickable, no style) > dimmed (out of reach). + let boundClass: string | undefined + if (boundPick && n.type !== 'add' && n.type !== 'trigger') { + if (n.id === boundPick.start) boundClass = 'wm-bound-start' + else if (boundPick.ends.has(n.id)) boundClass = 'wm-bound-end' + else if (boundPick.bounded.has(n.id)) boundClass = 'wm-bound-in' + else if (!boundPick.eligible.has(n.id)) boundClass = 'wm-bound-dim' + } return { id: n.id, type: n.type, position: { x: p.x + xCenter + xShift, y: p.y + 40 }, data: n.data, - class: runClass ?? assetClass, + class: boundClass ?? runClass ?? assetClass, selected: n.id === selectedId, // All nodes non-draggable: the layout is sugiyama-computed, // dragging would fight the reactive re-layout. Selection is @@ -701,6 +1001,13 @@ style = 'stroke: rgb(156 163 175); stroke-width: 1.25px;' animated = flowAnimated break + case 'data-test': + // Asset → its custom-test script: dashed, muted, no run + // animation (the test isn't a producing step). + style = 'stroke: rgb(156 163 175); stroke-width: 1.25px;' + strokeDasharray = '4 3' + markerColor = 'rgb(156 163 175)' + break case 'trigger-asset': style = 'stroke: rgb(107 114 128); stroke-width: 2px;' animated = flowAnimated @@ -715,6 +1022,26 @@ label = 'triggers' labelStyle = 'fill: rgb(107 114 128); font-size: 10px; font-weight: 600;' break + case 'macro': + // Library → consumer: violet dashed, visually apart from both + // lineage (blue/gray solid) and trigger (gray dashed) families — + // it's a code dependency, not data flow or execution. + style = 'stroke: rgb(139 92 246); stroke-width: 1.25px;' + strokeDasharray = '5 3' + markerColor = 'rgb(139 92 246)' + label = e.via_use ? 'uses lib' : 'macros' + labelStyle = 'fill: rgb(139 92 246); font-size: 10px; font-weight: 600;' + break + case 'test-dependency': + // Producer → tested script: amber dashed ordering link. Not + // data flow (blue/gray) nor execution trigger (gray "triggers") + // — it only says "the test needs this asset to exist first". + style = 'stroke: rgb(217 119 6); stroke-width: 1.25px;' + strokeDasharray = '5 3' + markerColor = 'rgb(217 119 6)' + label = 'test needs' + labelStyle = 'fill: rgb(217 119 6); font-size: 10px; font-weight: 600;' + break default: style = '' } @@ -755,7 +1082,23 @@ source: e.source, target: e.target, type: 'asset', - data: { detourX: detourForEdge(e.source, e.target) }, + data: { + detourX: detourForEdge(e.source, e.target), + // Data-test badge on the producer→asset write-edge. The + // producer's last-run status (the script fails if any test + // fails) tints it green/red; neutral until it has run. + data_tests: e.data_tests, + testsRunStatus: e.data_tests?.length ? runStates?.get(e.source)?.status : undefined, + // Column-lineage badge on the same write-edge (the link is the + // transformation whose output columns the lineage describes). + column_lineage: e.column_lineage, + // Macro-edge badge: which of the library's macros the consumer + // calls (all of them when pulled in via `// use`). + macro_names: e.macro_names, + via_use: e.via_use, + // Muted read edge — bell-off badge on the read link. + muted: e.muted + }, animated, label, labelStyle, @@ -784,7 +1127,8 @@ asset: AssetNode as any, runnable: RunnableNode as any, trigger: TriggerNode as any, - add: AddNode as any + add: AddNode as any, + 'data-test': DataTestNode as any } const edgeTypes = { @@ -792,12 +1136,26 @@ } function handleNodeClick({ node }: { node: Node }) { + // Bounded-run pick mode intercepts clicks: an eligible (downstream) + // node toggles as an end bound; the start, dimmed nodes, and + // triggers/+ are inert. Selection (details pane) is suppressed so the + // modal pick stays focused. + if (boundPick && onPickEnd) { + if (node.type !== 'asset' && node.type !== 'runnable') return + if (node.id === boundPick.start) return + if (!boundPick.eligible.has(node.id)) return + onPickEnd(node.id) + return + } if (!onselect) return const data = node.data as any if (node.type === 'asset') { onselect({ kind: 'asset', asset_kind: data.asset_kind, path: data.path }) } else if (node.type === 'runnable') { onselect({ kind: 'runnable', runnable_kind: data.runnable_kind, path: data.path }) + } else if (node.type === 'data-test') { + // A custom test is a deployed script — open it like any runnable. + onselect({ kind: 'runnable', runnable_kind: 'script', path: data.path }) } // 'schedule' doesn't produce a selection. } @@ -822,21 +1180,37 @@ --background-color={false} >
+ - - n.type === 'asset' - ? 'rgb(96 165 250 / 0.5)' - : n.type === 'trigger' - ? 'rgb(251 191 36 / 0.5)' - : 'rgb(52 211 153 / 0.5)'} - nodeStrokeColor="transparent" - maskColor="rgb(0 0 0 / 0.2)" - /> + {#if showMinimap} + + + n.type === 'asset' + ? 'rgb(59 130 246 / 0.3)' + : n.type === 'trigger' + ? 'rgb(245 158 11 / 0.3)' + : 'rgb(148 163 184 / 0.15)'} + nodeStrokeColor={(n) => + n.type === 'asset' + ? 'rgb(59 130 246 / 0.8)' + : n.type === 'trigger' + ? 'rgb(245 158 11 / 0.8)' + : 'rgb(100 116 139 / 0.7)'} + maskColor="rgb(100 116 139 / 0.12)" + maskStrokeColor="rgb(59 130 246 / 0.5)" + maskStrokeWidth={4} + /> + {/if}
@@ -870,4 +1244,20 @@ :global(.svelte-flow__node.wm-asset-input .drop-shadow-sm) { @apply outline outline-2 outline-gray-400; } + /* Bounded-run pick mode. The start is a solid blue ring; chosen end + bounds get a thicker amber ring; nodes inside the path-between set ring + blue; everything out of reach fades back so the matched subset reads at + a glance. */ + :global(.svelte-flow__node.wm-bound-start .drop-shadow-sm) { + @apply outline outline-2 outline-blue-500; + } + :global(.svelte-flow__node.wm-bound-in .drop-shadow-sm) { + @apply outline outline-2 outline-blue-400/70; + } + :global(.svelte-flow__node.wm-bound-end .drop-shadow-sm) { + @apply outline outline-[3px] outline-amber-500; + } + :global(.svelte-flow__node.wm-bound-dim) { + @apply opacity-30; + } diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte index 39a6a2ae72..425565353a 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte @@ -1,5 +1,6 @@ + {#if data.fork_materialization === 'deferred'} + + + parent + + {:else if data.fork_materialization === 'fork'} + + + fork + + {/if} + + {#if data.derived_from} + + + + {/if}
+ {#if showGuardBadge} + +
+ +
+ {/if} {#if showActions} +

+ {#if running && cancelRequested} + Cancelling — waiting for the current partition to finish; the rest will not run. + {:else if running} + Materializing {slices.filter((s) => s.status === 'success' || s.status === 'failure') + .length}/{slices.length} partitions sequentially — each run gets its partition as an explicit + arg. + {:else} + Backfill finished: {slices.filter((s) => s.status === 'success').length} succeeded, + {slices.filter((s) => s.status === 'failure').length} failed, + {slices.filter((s) => s.status === 'pending').length} not run. + {/if} +

+
+ {#each slices as s (s.partition)} +
+ + {s.status} + + {s.partition} + {#if s.status === 'running'} + + {/if} + {#if s.error} + {s.error} + {/if} +
+ {/each} +
+ {:else} +

+ Re-runs the producing script once per partition in the range, each with an explicit + partition arg. Re-running a partition is idempotent, so this is + safe to repeat. +

+
+
+ From + +
+
+ To + +
+
+ {#if preview.loading} +
+ Computing partitions in range… +
+ {:else if preview.error} +

{errText(preview.error)}

+ {:else if preview.current} +
+

+ {preview.current.partitions.length} + {preview.current.partition_kind} partitions in range — {counts.missing} missing, + {counts.failed} failed, {counts.materialized} materialized. Producer: + {preview.current.producer_path} +

+
+ {#each preview.current.partitions as p (p.partition)} + + {p.partition} + + {/each} +
+ +
+ {/if} + {/if} +
+ + {#snippet actions()} + {#if running} + + {:else if slices?.length} + + + {:else} + + {/if} + {/snippet} + diff --git a/frontend/src/lib/components/assets/AssetGraph/ColumnLineageTrace.svelte b/frontend/src/lib/components/assets/AssetGraph/ColumnLineageTrace.svelte new file mode 100644 index 0000000000..2cf28ba401 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/ColumnLineageTrace.svelte @@ -0,0 +1,173 @@ + + +
+
+ + Column lineage + {#if targetLabel} + + {targetLabel} + {/if} + + {#if selected} + + {:else if component.size > 1} + click a column to trace + {/if} +
+ + {#if component.size === 0} + No column lineage for this asset. + {:else} +
+ + {#each edges as e (`${e.from}->${e.to}`)} + {@const hot = traced !== undefined && traced.has(e.from) && traced.has(e.to)} + {@const cold = traced !== undefined && !hot} + + {/each} + + + {#each [...component] as id (id)} + {@const n = graph.nodes.get(id)} + {@const p = pos.get(id)} + {#if n && p} + {@const isSeed = seedSet.has(id)} + + {/if} + {/each} +
+ {/if} +
diff --git a/frontend/src/lib/components/assets/AssetGraph/DataTestNode.svelte b/frontend/src/lib/components/assets/AssetGraph/DataTestNode.svelte new file mode 100644 index 0000000000..9c46fd1795 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DataTestNode.svelte @@ -0,0 +1,31 @@ + + + +
+ + test + + {data.path} + +
diff --git a/frontend/src/lib/components/assets/AssetGraph/DucklakeAssetPanel.svelte b/frontend/src/lib/components/assets/AssetGraph/DucklakeAssetPanel.svelte new file mode 100644 index 0000000000..5f7ad050e4 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DucklakeAssetPanel.svelte @@ -0,0 +1,123 @@ + + +
+ {#if !qualifiedTable} + + + {:else} +
+ (tab = e.detail)}> + {#snippet children({ item })} + + + + {/snippet} + +
+ +
+ {#if tab === 'partitions'} + + {:else if tab === 'schema'} + + {:else} +
+ + + snapshots.refetch()} + selectedVersion={effectiveVersion} + onSelect={(v) => (selectedVersion = v)} + /> + + +
+ +
+
+
+
+ {/if} +
+ {/if} +
diff --git a/frontend/src/lib/components/assets/AssetGraph/DucklakeResultPreview.svelte b/frontend/src/lib/components/assets/AssetGraph/DucklakeResultPreview.svelte new file mode 100644 index 0000000000..403a100aa2 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DucklakeResultPreview.svelte @@ -0,0 +1,136 @@ + + +
+ {#if partition} +
+ (scope = e.detail)}> + {#snippet children({ item })} + + + {/snippet} + +
+ {/if} + {#if colDefs.loading && !colDefs.current} +
+ +
+ {:else if !tableColDefs} +
+ + Couldn't load a preview of this table. +
+ {:else if dbTableOps} + + {#key [refreshKey, scope]} +
+ +
+ {/key} + {/if} +
diff --git a/frontend/src/lib/components/assets/AssetGraph/DucklakeSnapshotHistory.svelte b/frontend/src/lib/components/assets/AssetGraph/DucklakeSnapshotHistory.svelte new file mode 100644 index 0000000000..e95be118ee --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DucklakeSnapshotHistory.svelte @@ -0,0 +1,79 @@ + + +
+
+ Snapshot history +
+ +
+ {#if loading && !items.length} +
+ Loading snapshots… +
+ {:else if error} +

Failed to load: {error}

+ {:else if !items.length} +

+ No snapshots yet. DuckLake records one on every // materialize write; each becomes a version you can time-travel to. +

+ {:else} +
+ {#each items as s (s.snapshot_id)} + {@const selected = s.snapshot_id === selectedVersion} + + {/each} +
+ {/if} +
+
diff --git a/frontend/src/lib/components/assets/AssetGraph/DucklakeVersionPreview.svelte b/frontend/src/lib/components/assets/AssetGraph/DucklakeVersionPreview.svelte new file mode 100644 index 0000000000..5afb51fe66 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DucklakeVersionPreview.svelte @@ -0,0 +1,133 @@ + + +
+ {#if version == undefined} +
+ Select a snapshot from the list to preview the table as of that version. +
+ {:else} + {#if exampleSql} +
+ + {exampleSql} + +
+ {/if} + {#if ready && dbTableOps} + {#key version} +
+ +
+ {/key} + {:else if columns.error} +
+ + Couldn't load this table at version {version}. +
+ {:else} +
+ +
+ {/if} + {/if} +
diff --git a/frontend/src/lib/components/assets/AssetGraph/InitialFitView.svelte b/frontend/src/lib/components/assets/AssetGraph/InitialFitView.svelte new file mode 100644 index 0000000000..eea1fb306d --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/InitialFitView.svelte @@ -0,0 +1,30 @@ + diff --git a/frontend/src/lib/components/assets/AssetGraph/MacroExplorerDrawer.svelte b/frontend/src/lib/components/assets/AssetGraph/MacroExplorerDrawer.svelte new file mode 100644 index 0000000000..32ac65bb2c --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/MacroExplorerDrawer.svelte @@ -0,0 +1,159 @@ + + + (open = false)}> + (open = false)}> + {#snippet actions()} +
+ {#each macros as m (m.name)} +
+
+ + {macroSignature(m)} + + + {m.is_table ? 'table' : 'scalar'} + +
+
{m.body}
+
+ {/each} +
+ {/each} +
+ {/if} +
+ + diff --git a/frontend/src/lib/components/assets/AssetGraph/PartitionArgControl.svelte b/frontend/src/lib/components/assets/AssetGraph/PartitionArgControl.svelte new file mode 100644 index 0000000000..dac1ecdb02 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PartitionArgControl.svelte @@ -0,0 +1,250 @@ + + +
+ + + Partition + + {spec.kind} + + + + {#if calendarPicker} + onNativeInput(e.currentTarget.value)} + /> + {#if metadataError} + + + + The // partitioned header has {metadataError} — fix it in the + script; no default is filled in. + + + {:else if beforeStart} + + Partitioning starts {spec.start} — defaulted to the first partition. + + {/if} + {:else} + value ?? '', (v) => (value = v)} + size="sm" + inputProps={{ + placeholder: spec.kind === 'dynamic' ? 'Partition key value' : 'Partition bucket' + }} + /> + + {#if spec.kind === 'dynamic'} + Dynamic partition — leave blank to let the run resolve it from the payload. + {:else} + Custom partition format — enter the bucket exactly as the producer renders it. + {/if} + + {/if} + + + {#if calendarPicker && materializeTarget?.kind === 'ducklake' && value} + {#if selectedMaterialized} + + + {value} is already materialized — running replaces it. + + {:else} + + + {value} not materialized yet — running creates it. + + {/if} + {/if} + + + {#if upstreamMissing} + + + + No upstream data for {value} yet — this run may materialize an + empty partition. + + + {/if} + + + {#if recentlyMissing.length > 0} +
+ + {recentlyMissing.length} of the last {recentWindow(spec.kind)} + {spec.kind} partitions are not materialized: + +
+ {#each recentlyMissing.slice(0, MAX_MISSING_CHIPS) as b (b)} + + {/each} + {#if recentlyMissing.length > MAX_MISSING_CHIPS} + + +{recentlyMissing.length - MAX_MISSING_CHIPS} more + + {/if} +
+
+ {/if} +
diff --git a/frontend/src/lib/components/assets/AssetGraph/PartitionStatusGrid.svelte b/frontend/src/lib/components/assets/AssetGraph/PartitionStatusGrid.svelte new file mode 100644 index 0000000000..4be0bb8b6c --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PartitionStatusGrid.svelte @@ -0,0 +1,209 @@ + + +
+
+ Materialized partitions +
+ {#if backfillRunning} + + {:else if backfillSlices?.length} + + {/if} +
+ +
+
+
+ +
+ {#if partitions.loading} +
+ Loading partitions… +
+ {:else if partitions.error} +

Failed to load: {partitions.error.message}

+ {:else if !partitions.current?.length} +

+ No partitions materialized yet. They appear here after a // materialize run. +

+ {:else} +
+ + + + + + + + + + + {#each partitions.current as p (p.partition)} + + + + + + + + {#if p.error} + + {/if} + {/each} + +
PartitionStatusSnapshotRowsMaterialized
{p.partition || '(whole table)'} + + {p.status} + + {p.snapshot_id ?? '—'}{p.row_count ?? '—'}{new Date(p.materialized_at).toLocaleString()}
{p.error}
+ {/if} + + + + (backfillSlices = undefined)} +/> diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineActivityPanel.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineActivityPanel.svelte index 90e531e858..7e6c34adc2 100644 --- a/frontend/src/lib/components/assets/AssetGraph/PipelineActivityPanel.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineActivityPanel.svelte @@ -36,8 +36,12 @@ // History preload hit its page cap before the days cutoff. truncated?: boolean error?: string | undefined - days: number - onDaysChange: (days: number) => void + days?: number + onDaysChange?: (days: number) => void + // Live-only surfaces (local-dev preview) have no historical fetch: hide + // the day-range Select + histogram (both are history-window concepts) and + // show only the live run stream. + liveOnly?: boolean // Hover a run row → emphasize its node(s) on the canvas; a group header // passes the whole cascade's paths. `undefined` clears. onHoverRun?: (paths: string[] | undefined) => void @@ -51,8 +55,9 @@ loading = false, truncated = false, error, - days, + days = 30, onDaysChange, + liveOnly = false, onHoverRun, onSelectRun }: Props = $props() @@ -99,7 +104,7 @@ // Quick reset: drop the brush and return to the default 30-day window. function resetWindow() { selectedRange = undefined - if (days !== 30) onDaysChange(30) + if (days !== 30) onDaysChange?.(30) } function fmtRange(r: { from: number; to: number }): string { const opt: Intl.DateTimeFormatOptions = { @@ -236,7 +241,9 @@ { label: 'Last 30 days', value: 30 }, { label: 'Last 90 days', value: 90 } ] - let windowLabel = $derived(DAY_OPTIONS.find((o) => o.value === days)?.label ?? `Last ${days} days`) + let windowLabel = $derived( + DAY_OPTIONS.find((o) => o.value === days)?.label ?? `Last ${days} days` + ) // Excludes future-scheduled queued jobs (a schedule's next planned run // is not activity) — see isActiveEvent. @@ -341,17 +348,19 @@ {/if} -
- days, (v) => onDaysChange?.(v ?? 30)} + /> +
+ {/if} - {#if events.length > 0} + {#if events.length > 0 && !liveOnly}
{:else}
- No runs in this window ({windowLabel.toLowerCase()}) — executions of this pipeline will - appear here live. + {#if liveOnly} + No runs yet — executions of this pipeline will appear here live. + {:else} + No runs in this window ({windowLabel.toLowerCase()}) — executions of this pipeline will + appear here live. + {/if}
{/if} {:else} @@ -555,7 +568,8 @@ title={`Join fed by ${g.extraTriggers + 1} triggers`}>+{g.extraTriggers} {/if} - {g.members.length} runs + {g.members.length} runs {ago(g.latestAt)} diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineDevView.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineDevView.svelte new file mode 100644 index 0000000000..8b10e54ce3 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineDevView.svelte @@ -0,0 +1,411 @@ + + +
+
+ + f/{folder} + · local pipeline dev + + + {wsState === 'open' + ? 'watching' + : wsState === 'connecting' + ? 'connecting…' + : 'disconnected'} + +
+
+ {#if !bundle} +
+ + Connecting to wmill pipeline dev +
+ {:else if displayGraph.runnables.length === 0} +
+ No // pipeline scripts found in f/{folder}. Mark a script with a + bare + // pipeline comment. +
+ {:else} + (panelHidden = !panelHidden)} + onRunProducer={runProducer} + onRunByPath={(path, args) => runNode(path, args)} + onRunCascadeByPath={(path, args) => runCascadeFrom(path, args)} + downstreamSubscribers={selectionDownstreamCount} + {resolveLocalScript} + localScriptsVersion={bundle} + {selectionProducers} + canRunByPath + onRunCompleted={() => { + activeRunnable = undefined + activeRunnableJobId = undefined + }} + onSelect={handleCanvasSelect} + onClose={() => (pe.selection = undefined)} + > + {#snippet idlePane()} + (activityHoverPaths = p ?? [])} + onSelectRun={(p) => (activitySelectPaths = p ?? [])} + /> + {/snippet} + + {/if} +
+
diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineFolderList.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineFolderList.svelte new file mode 100644 index 0000000000..d35b4f21d6 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineFolderList.svelte @@ -0,0 +1,111 @@ + + +
+
+

+ Existing pipelines +

+ {#if pipelines.loading && !pipelines.current} +
+ + Loading… +
+ {:else if pipelines.error} +
Failed: {pipelines.error.message}
+ {:else if visiblePipelines.length === 0} +
+ {currentFolder + ? 'No other pipelines in this workspace.' + : 'No pipelines yet. A pipeline is any folder whose scripts carry pipeline annotations.'} +
+ {:else} +
+ {#each visiblePipelines as p (p.folder)} + + {/each} +
+ {/if} +
+ + {#if !$userStore?.operator} + +
+

+ Pick or create a folder +

+
+
+ +
+ +
+
+ {/if} +
diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte new file mode 100644 index 0000000000..7db35c2353 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte @@ -0,0 +1,555 @@ + + +
+ + +
+ + {#if boundBar}{@render boundBar()}{/if} + {#if mode === 'edit'} + + {/if} + {#if prefetchingAssets} +
+ + Parsing assets… +
+ {/if} + {#if onTogglePanelHidden && (mode !== 'edit' || editor.selection != undefined || editor.activeDraftPath != undefined)} +
+
+ {/if} +
+ {#if detailsPaneOpen && workspace} + + {#if idleView && idlePane} + {@render idlePane()} + {:else} + onStartBoundedRunForOpen?.(editor.openScriptPath!) + : undefined} + {onRunCompleted} + {onTestStateChange} + {requestRemoveSignal} + {requestRunSignal} + {requestRunCascadeSignal} + {focusUploadSignal} + draftScript={activeDraft?.script} + draftOutputAssets={activeDraft?.outputAssets} + draftInputAssets={activeDraft?.inputAssets} + {pathPrefix} + {onDraftPathChange} + {workspace} + onAnnotationsChange={editor.handleAnnotationsChange} + onAssetsChange={editor.handleAssetsChange} + onContentChange={editor.handleContentChange} + onDraftPersist={editor.handleDraftPersist} + onclose={onClose} + onHide={onTogglePanelHidden} + {onDiscard} + {onDraftSaved} + {onPersistedSaved} + {onScriptRenamed} + {onScriptRemoved} + /> + {/if} + + {/if} +
+
diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineInsertMenu.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineInsertMenu.svelte index 847f682049..e3c6c04378 100644 --- a/frontend/src/lib/components/assets/AssetGraph/PipelineInsertMenu.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineInsertMenu.svelte @@ -231,7 +231,7 @@
pathEl, { timeout: 50 }) })} @@ -240,7 +240,7 @@ {#each visibleOutputKinds.length ? visibleOutputKinds : PIPELINE_OUTPUT_KINDS as k} {@const isSelected = selected.outputId === k.id} - {/each} -
- {/if} - - {/if} - - {#if !$userStore?.operator} - -
-

- Pick or create a folder -

-
-
- -
- -
-
- {/if} -
+ (open = false)} /> diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineRunForm.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineRunForm.svelte new file mode 100644 index 0000000000..4384f125ba --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineRunForm.svelte @@ -0,0 +1,73 @@ + + +
+ {#if partitionSpec} + args.partition, (v) => (args.partition = v)} + {workspace} + {materializeTarget} + {upstreamAssets} + /> + {/if} + +
diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte index bb0fbd4d27..79a1cf6d47 100644 --- a/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte @@ -1,12 +1,13 @@ + +{#if show} +
+
+ +
+

Finish setting up pipelines

+

+ Pipelines materialize data into DuckLake tables backed by object storage. Configure the + following before your first pipeline can run. +

+
+
+ +
    + {#each steps as step (step.title)} + {@const Icon = step.icon} +
  • + {#if step.done === true} + + {:else if step.done === false} + + {:else} + + {/if} +
    + {step.title} + {step.description} +
    + {#if step.done !== true} + + {step.cta} + + + {:else} + Configured + {/if} +
  • + {/each} +
+
+{/if} diff --git a/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte b/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte index 93eeb04163..4cb65ee8f3 100644 --- a/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte @@ -10,7 +10,9 @@ Loader2, Play, RotateCw, + SquareFunction, Tag, + Target, Timer, Trash2, XCircle, @@ -20,12 +22,13 @@ import { preventDefault, stopPropagation } from 'svelte/legacy' import type { GraphUsageKind } from './types' import type { RunnableRunState } from './activeRunnables.svelte' + import { parseDurationSecs } from './parsePipelineAnnotations' import { NODE } from '$lib/components/graph/util' import DropdownV2 from '$lib/components/DropdownV2.svelte' import Popover from '$lib/components/meltComponents/Popover.svelte' import type { Item } from '$lib/utils' import { workspaceStore } from '$lib/stores' - import { sendUserToast } from '$lib/utils' + import { sendUserToast, msToReadableTimeShort } from '$lib/utils' interface Props { data: { @@ -34,8 +37,14 @@ in_pipeline?: boolean partition_kind?: 'daily' | 'hourly' | 'weekly' | 'monthly' | 'dynamic' freshness?: string + // Completion time (ISO) of the newest successful run visible to + // the caller. With `freshness`, drives the fresh/stale chip state. + last_success_at?: string tag?: string retry?: { count: number; delay?: string } + // Macros this script provides (deployed/drafted `// macros` library). + // Non-empty renders the ƒ chip marking the node as a macro library. + macros?: { name: string; params: string; is_table: boolean }[] // Last-run status + run count observed this session (from the // folder queue poll). Undefined until the first observed run. runState?: RunnableRunState @@ -67,6 +76,10 @@ // "Discard" for drafts, "Delete…" (which the page maps to its // archive/delete confirmation flow) for persisted scripts. onRequestRemove?: () => void + // Wired only for valid bounded-run starts (schedule / manual roots + // with downstream). Enters the page's end-node pick mode for a + // bounded cascade rooted at this script. + onStartBoundedRun?: () => void } // SvelteFlow injects this when the user clicks the node. Combined with // hover state to drive the run-button visibility (same pattern as @@ -113,9 +126,47 @@ } } - // Cascade option is surfaced directly on the Run button (via a caret + - // popover) when `downstreamCount > 0`, so the kebab menu stays focused - // on lifecycle actions only. + // Freshness verdict: newest successful run (server `last_success_at`, + // or a newer one the session poll observed) vs the `// freshness` + // window. No verdict (undefined) for drafts — no run history — and for + // unparseable windows; the chip then stays neutral like the other + // annotation chips. + let freshnessWindowS = $derived(data.freshness ? parseDurationSecs(data.freshness) : undefined) + // Ticks so a node crosses fresh→stale while the canvas stays open (the + // graph payload is static between refetches). Armed only when a verdict + // is rendered. + let nowMs = $state(Date.now()) + $effect(() => { + if (freshnessWindowS === undefined || data.unsaved) return + const id = setInterval(() => (nowMs = Date.now()), 30_000) + return () => clearInterval(id) + }) + let lastSuccessMs = $derived.by(() => { + const server = data.last_success_at ? new Date(data.last_success_at).getTime() : undefined + const polled = data.runState?.lastSuccessAt + ? new Date(data.runState.lastSuccessAt).getTime() + : undefined + if (server === undefined) return polled + return polled === undefined ? server : Math.max(server, polled) + }) + let freshnessState = $derived.by((): 'fresh' | 'stale' | undefined => { + if (freshnessWindowS === undefined || data.unsaved) return undefined + if (lastSuccessMs === undefined) return 'stale' + return nowMs - lastSuccessMs <= freshnessWindowS * 1000 ? 'fresh' : 'stale' + }) + let freshnessTooltip = $derived.by(() => { + const base = `// freshness ${data.freshness}` + if (freshnessState === undefined) return base + if (lastSuccessMs === undefined) return `${base} — stale: no successful run yet` + const ago = msToReadableTimeShort(Math.max(0, nowMs - lastSuccessMs)) + return freshnessState === 'fresh' + ? `${base} — fresh: last successful run ${ago} ago` + : `${base} — stale: last successful run ${ago} ago` + }) + + // Cascade + bounded-run options live on the Run button's caret popover + // (whenever there's a cascade OR a bounded-run start — see `hasCaret` + // below), so the kebab menu stays focused on lifecycle actions only. let menuItems: Item[] = $derived( data.onRequestRemove ? [ @@ -155,8 +206,8 @@ + guidelines). Only the freshness chip (when it has a verdict) + and the run-state chip below use semantic colors. --> {#if data.partition_kind}
{data.partition_kind}
{/if} + {#if data.freshness}
{data.freshness} @@ -194,6 +257,17 @@ ×{r.count}
{/if} + {#if data.macros && data.macros.length > 0} +
1 ? 's' : ''}:\n${data.macros + .map((m) => `• ${m.name}(${m.params})${m.is_table ? ' → table' : ''}`) + .join('\n')}`} + > + + ×{data.macros.length} +
+ {/if} {#if data.runState} {@const rs = data.runState}
{@const hasCascade = (data.downstreamCount ?? 0) > 0} + + {@const hasCaret = hasCascade || !!data.onStartBoundedRun}
- {#if hasCascade} + {#if hasCaret}
- + + Let the asset-trigger cascade fan out to the {data.downstreamCount} + subscribed script{data.downstreamCount === 1 ? '' : 's'} after this run succeeds. + + {#if data.unsaved || (data.downstreamUnsavedCount ?? 0) > 0} + + {#if data.unsaved} + Unsaved chain — runs as previews in dependency order. + {:else} + {data.downstreamUnsavedCount} unsaved — chain runs as previews in dependency + order. + {/if} + Deploy to enable automatic triggering. + + {/if} +
+ + {/if} + {#if data.onStartBoundedRun} + + {/if} {/snippet} diff --git a/frontend/src/lib/components/assets/AssetGraph/SchemaHistoryPanel.svelte b/frontend/src/lib/components/assets/AssetGraph/SchemaHistoryPanel.svelte new file mode 100644 index 0000000000..fc49fd0f33 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/SchemaHistoryPanel.svelte @@ -0,0 +1,160 @@ + + +{#snippet columnsTable(cols: SchemaColumn[])} + + + + + + + + + {#each cols as col (col.name)} + + + + + {/each} + +
ColumnType
{col.name}{col.type}
+{/snippet} + +
+
+ Captured schema +
+ +
+ {#if schemas.loading} +
+ Loading schema… +
+ {:else if schemas.error} +

Failed to load: {schemas.error.message}

+ {:else if !schemas.current?.length} +

+ No schema captured yet. The output schema is recorded automatically after a // materialize run. +

+ {:else if !canEvolve} + +
+
+ + This asset's schema is fixed — an append / merge / partitioned materialize INSERTs into + a fixed-schema table, so the columns can't change run-to-run. +
+ {#if selected} + {@render columnsTable(selected.columns)} + {/if} +
+ {:else} +
+ + +
+ {#each schemas.current as s, i (s.version)} + + {/each} +
+
+ +
+ {#if selected} + {@render columnsTable(selected.columns)} + {/if} +
+
+
+
+ {/if} +
+
diff --git a/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte b/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte index 6d8f1224ae..98307013c4 100644 --- a/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte @@ -60,7 +60,7 @@ import { Handle, Position } from '@xyflow/svelte' import { NODE } from '$lib/components/graph/util' import { twMerge } from 'tailwind-merge' - import { AlertTriangle, EllipsisVertical, Trash2 } from 'lucide-svelte' + import { AlertTriangle, CheckCircle2, EllipsisVertical, Target, Trash2 } from 'lucide-svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { stopPropagation, preventDefault } from 'svelte/legacy' import type { Item } from '$lib/utils' @@ -101,6 +101,9 @@ // auto-generated S3 picker lets the user upload + run) instead of // rendering a "missing" placeholder. onOpenDataUpload?: (scriptPath: string) => void + // True once a file is staged for this data_upload entry — renders the + // node green so the user knows the pipeline can run (see WIN-2129). + ready?: boolean // Page-supplied dispatcher to open the matching native trigger // drawer in edit mode for an attached (non-missing) trigger. // `triggerPath` is the trigger row's path (e.g. the mqtt_trigger @@ -113,6 +116,12 @@ // trigger. Confirmation is the caller's responsibility — the // node just exposes the entry point on the kebab menu. onDeleteTrigger?: (kind: NativeTriggerKind, triggerPath: string) => void + // Wired by the canvas only when this trigger's target script is a + // valid bounded-run start (schedule / manual root) with downstream. + // Entering the page's end-node pick mode rooted at that script — the + // View-mode entry point for bounded runs (the script's own Run-button + // caret is Edit-only). + onStartBoundedRun?: () => void } } let { data }: Props = $props() @@ -143,6 +152,10 @@ // data_upload routes through its own handler — clicking opens the target // script's run form (with the auto-generated S3 picker). let canOpenDataUpload = $derived(isDataUpload && !!data.runnable_path && !!data.onOpenDataUpload) + // A staged upload turns the node green (ready to run); before that it stays + // on the neutral surface with the "upload a file" prompt. + let dataUploadReady = $derived(isDataUpload && data.ready === true) + let DataUploadIcon = $derived(dataUploadReady ? CheckCircle2 : style.icon) // Schedule + the other native kinds all have dedicated editors. Webhook and // data_upload are excluded — they route through their own open handlers. let canCreate = $derived( @@ -173,8 +186,17 @@ !!data.onDeleteTrigger ) - let menuItems: Item[] = $derived( - canDelete + let menuItems: Item[] = $derived([ + ...(data.onStartBoundedRun + ? [ + { + displayName: 'Run + downstream…', + icon: Target, + action: () => data.onStartBoundedRun?.() + } + ] + : []), + ...(canDelete ? [ { displayName: 'Delete…', @@ -182,12 +204,12 @@ type: 'delete' as const, action: () => { if (!data.ref || !data.onDeleteTrigger) return - data.onDeleteTrigger(data.kind as NativeTriggerKind, data.ref) + data.onDeleteTrigger?.(data.kind as NativeTriggerKind, data.ref) } } ] - : [] - ) + : []) + ]) function handleMissingClick() { if (!canCreate || !data.runnable_path || !data.onCreateMissingTrigger) return @@ -291,26 +313,47 @@ {:else if canOpenDataUpload} + S3 picker lets the user upload a file and run the pipeline. Goes + green once a file is staged (ready), so "Run pipeline" can proceed; + until then it stays neutral with an "upload a file" prompt. Never + the red "missing" state. --> {:else} diff --git a/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts b/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts index 917e21766a..f6682804c7 100644 --- a/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts +++ b/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts @@ -1,8 +1,15 @@ import { JobService } from '$lib/gen' export type RunStatus = 'running' | 'success' | 'failure' -/** Per-runnable badge state: latest run status + runs observed this session. */ -export type RunnableRunState = { status: RunStatus; runs: number } +/** + * Per-runnable badge state: latest run status + runs observed this session. + * `lastSuccessAt` is the completion time (start + duration when the listing + * carries it, else start as a conservative lower bound) of the newest + * successful run seen by the poll — lets the freshness chip go green right + * after an in-session run, ahead of the next graph refetch (whose + * `last_success_at` would carry it). + */ +export type RunnableRunState = { status: RunStatus; runs: number; lastSuccessAt?: string } export type EventStatus = 'queued' | 'running' | 'success' | 'failure' /** One folder activity-log row (a job observed by the poll). */ @@ -14,6 +21,12 @@ export type PipelineEvent = { /** What started it, as far as the job listing reveals. */ source: 'schedule' | 'run' at: string + /** + * Completion time (start + duration) for completed rows. The freshness + * chip compares against completion — `at` is the start time and would + * read a long run as older than its output actually is. + */ + completedAt?: string /** * Queued jobs: when the job is due to start. A future value means a * scheduled run waiting for its cron tick, not pipeline activity. @@ -45,7 +58,8 @@ function statesEq(a: Map, b: Map() const countedJobIds = new Set() // Job ids we've observed in-flight at least once. The catch-up pulse is @@ -241,10 +255,25 @@ export function useActiveRunnableIds( const prev = completedHistory.get(id) const status: RunStatus = (j as any).success === true ? 'success' : 'failure' const ts = startedTs ?? new Date(pollStartedMs).toISOString() + // Freshness compares against COMPLETION time (that's + // when the output materialized — the server-side + // last_success_at is completed_at too). The listing + // only carries started_at, so add duration_ms; when + // absent, the start is a conservative lower bound + // (errs stale, never false-fresh). + const durationMs = (j as any).duration_ms + const doneTs = + typeof durationMs === 'number' && startedTs + ? new Date(new Date(startedTs).getTime() + durationMs).toISOString() + : ts completedHistory.set(id, { runs: (prev?.runs ?? 0) + 1, lastStatus: !prev || ts >= prev.lastTs ? status : prev.lastStatus, - lastTs: !prev || ts >= prev.lastTs ? ts : prev.lastTs + lastTs: !prev || ts >= prev.lastTs ? ts : prev.lastTs, + lastSuccessTs: + status === 'success' && (!prev?.lastSuccessTs || doneTs >= prev.lastSuccessTs) + ? doneTs + : prev?.lastSuccessTs }) } } @@ -267,6 +296,10 @@ export function useActiveRunnableIds( : 'failure', source: (j as any).schedule_path ? 'schedule' : 'run', at: startedTs ?? new Date(pollStartedMs).toISOString(), + completedAt: + !isQueued && typeof (j as any).duration_ms === 'number' && startedTs + ? new Date(new Date(startedTs).getTime() + (j as any).duration_ms).toISOString() + : undefined, scheduledFor: isQueued ? ((j as any).scheduled_for as string | undefined) : undefined }) } @@ -289,7 +322,11 @@ export function useActiveRunnableIds( // previous badge state until a worker picks the job up. const snap = new Map() for (const [id, h] of completedHistory) { - snap.set(id, { status: runningThisTick.has(id) ? 'running' : h.lastStatus, runs: h.runs }) + snap.set(id, { + status: runningThisTick.has(id) ? 'running' : h.lastStatus, + runs: h.runs, + lastSuccessAt: h.lastSuccessTs + }) } for (const id of runningThisTick) { if (!snap.has(id)) snap.set(id, { status: 'running', runs: 0 }) diff --git a/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.test.ts b/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.test.ts index a5a92f61c2..24e7d4c2da 100644 --- a/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.test.ts @@ -38,7 +38,14 @@ describe('layoutAssetGraph (tidy-tree with join breaks)', () => { // stays strictly left of every node of the right branch. const pos = layoutAssetGraph({ nodes: [n('root'), n('a'), n('b'), n('a1'), n('a2'), n('a3'), n('b1')], - edges: [e('root', 'a'), e('root', 'b'), e('a', 'a1'), e('a', 'a2'), e('a', 'a3'), e('b', 'b1')] + edges: [ + e('root', 'a'), + e('root', 'b'), + e('a', 'a1'), + e('a', 'a2'), + e('a', 'a3'), + e('b', 'b1') + ] }) const leftMax = Math.max(...['a', 'a1', 'a2', 'a3'].map((id) => pos.get(id)!.x)) const rightMin = Math.min(...['b', 'b1'].map((id) => pos.get(id)!.x)) @@ -83,14 +90,37 @@ describe('layoutAssetGraph (tidy-tree with join breaks)', () => { } }) - it('falls back to a grid on cyclic input', () => { + it('lays a 2-cycle out as a chain (feedback edge dropped, no grid)', () => { const pos = layoutAssetGraph({ nodes: [n('a'), n('b')], edges: [e('a', 'b'), e('b', 'a')] }) - expect(pos.size).toBe(2) - expect(pos.get('a')).toBeDefined() - expect(pos.get('b')).toBeDefined() + // First-in-input wins the top slot; the b→a feedback edge is ignored. + expect(pos.get('a')!.y).toBeLessThan(pos.get('b')!.y) + expect(pos.get('a')!.x).toBe(pos.get('b')!.x) + }) + + it('keeps the acyclic part of a graph layered when one cycle exists', () => { + // root → a ⇄ b → leaf: the a⇄b cycle must not degrade root/leaf layering. + const pos = layoutAssetGraph({ + nodes: [n('root'), n('a'), n('b'), n('leaf')], + edges: [e('root', 'a'), e('a', 'b'), e('b', 'a'), e('b', 'leaf')] + }) + expect(pos.get('root')!.y).toBeLessThan(pos.get('a')!.y) + expect(pos.get('a')!.y).toBeLessThan(pos.get('b')!.y) + expect(pos.get('b')!.y).toBeLessThan(pos.get('leaf')!.y) + // A linear chain stays in one column. + expect(new Set(['root', 'a', 'b', 'leaf'].map((id) => pos.get(id)!.x)).size).toBe(1) + }) + + it('handles a longer cycle without dropping nodes', () => { + const pos = layoutAssetGraph({ + nodes: [n('a'), n('b'), n('c')], + edges: [e('a', 'b'), e('b', 'c'), e('c', 'a')] + }) + expect(pos.size).toBe(3) + const ys = ['a', 'b', 'c'].map((id) => pos.get(id)!.y) + expect(new Set(ys).size).toBe(3) }) it('packs disjoint components side by side without overlap', () => { diff --git a/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.ts b/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.ts index cf55db6e0a..c9d7d60355 100644 --- a/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.ts +++ b/frontend/src/lib/components/assets/AssetGraph/assetGraphLayout.ts @@ -53,8 +53,8 @@ interface Band { // // y comes from longest-path layering (same top-down orientation as before: // producers above, assets in the middle, consumers below). Returns positions -// (band centers) normalized so the component's min x,y = 0. Throws on cyclic -// input (caller falls back to a grid for the whole graph). +// (band centers) normalized so the component's min x,y = 0. Cyclic input is +// handled by dropping feedback edges (see the Kahn step below). function layoutComponent( nodes: GraphInput['nodes'], edges: GraphInput['edges'] @@ -76,21 +76,42 @@ function layoutComponent( if (!parents.get(e.target)!.includes(e.source)) parents.get(e.target)!.push(e.source) } - // Kahn topological order — also the cycle guard. + // Kahn topological order. Cycles don't abort the layout: when the queue + // drains with nodes left, the unplaced node with the fewest outstanding + // parents (first in input order on ties) is forced into the order and its + // not-yet-placed parent edges are dropped as feedback edges — layering and + // tree-building then operate on the resulting DAG while the rendered graph + // keeps every edge. (The caller already resolves write⇄read 2-cycles by + // omitting the read direction; this handles any longer cycle.) const indeg = new Map() for (const n of nodes) indeg.set(n.id, parents.get(n.id)!.length) const queue = nodes.filter((n) => indeg.get(n.id) === 0).map((n) => n.id) + const placed = new Set() const topo: string[] = [] - while (queue.length) { + while (topo.length < nodes.length) { + if (queue.length === 0) { + let pick: string | undefined + for (const n of nodes) { + if (placed.has(n.id)) continue + if (pick === undefined || indeg.get(n.id)! < indeg.get(pick)!) pick = n.id + } + parents.set( + pick!, + parents.get(pick!)!.filter((p) => placed.has(p)) + ) + queue.push(pick!) + } const cur = queue.shift()! + if (placed.has(cur)) continue + placed.add(cur) topo.push(cur) for (const c of children.get(cur)!) { + if (placed.has(c)) continue const d = indeg.get(c)! - 1 indeg.set(c, d) if (d === 0) queue.push(c) } } - if (topo.length !== nodes.length) throw new Error('cyclic asset graph') // Longest-path layering: a node sits one layer below its lowest parent. const layer = new Map() @@ -141,8 +162,7 @@ function layoutComponent( out.set(id, { x: left + w / 2, y: layer.get(id)! * LAYER_H }) const kids = treeChildren.get(id)! if (kids.length === 0) return - const kidsW = - kids.reduce((acc, k) => acc + W.get(k)!, 0) + SIBLING_GAP * (kids.length - 1) + const kidsW = kids.reduce((acc, k) => acc + W.get(k)!, 0) + SIBLING_GAP * (kids.length - 1) let cursor = left + (w - kidsW) / 2 for (const k of kids) { placeTree(k, cursor) @@ -221,7 +241,8 @@ function layoutComponent( // disjoint components, so it's excluded from component detection and instead // re-placed centered one layer above the whole packed graph. // -// Falls back to a stable grid if the component layout throws (cyclic inputs). +// Falls back to a stable grid if the component layout throws (defensive — +// cycles are already absorbed by feedback-edge dropping in layoutComponent). export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map { const byId = new Map() if (graph.nodes.length === 0) return byId diff --git a/frontend/src/lib/components/assets/AssetGraph/backfillRun.test.ts b/frontend/src/lib/components/assets/AssetGraph/backfillRun.test.ts new file mode 100644 index 0000000000..69fc6f9728 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/backfillRun.test.ts @@ -0,0 +1,118 @@ +import { describe, expect, it } from 'vitest' +import { runBackfill, type BackfillSliceState } from './backfillRun' + +// Deterministic fake backend: launch resolves with `job:`, +// waitTerminal resolves per the `results` table (default success), recording +// launch order. +function fakeRunner(results: Record = {}) { + const launched: string[] = [] + return { + launched, + launch: async (partition: string) => { + launched.push(partition) + return `job:${partition}` + }, + waitTerminal: async (jobId: string) => results[jobId.slice(4)] ?? ('success' as const) + } +} + +describe('runBackfill', () => { + it('runs slices sequentially in order and reports ok', async () => { + const r = fakeRunner() + const res = await runBackfill({ + partitions: ['2026-06-26', '2026-06-27', '2026-06-29'], + launch: r.launch, + waitTerminal: r.waitTerminal + }) + expect(r.launched).toEqual(['2026-06-26', '2026-06-27', '2026-06-29']) + expect(res.ok).toBe(true) + expect(res.cancelled).toBe(false) + expect(res.slices.map((s) => s.status)).toEqual(['success', 'success', 'success']) + expect(res.slices.map((s) => s.jobId)).toEqual([ + 'job:2026-06-26', + 'job:2026-06-27', + 'job:2026-06-29' + ]) + }) + + it('continues past a failed slice — each slice is independent', async () => { + const r = fakeRunner({ '2026-06-27': 'failure' }) + const res = await runBackfill({ + partitions: ['2026-06-26', '2026-06-27', '2026-06-29'], + launch: r.launch, + waitTerminal: r.waitTerminal + }) + expect(r.launched).toEqual(['2026-06-26', '2026-06-27', '2026-06-29']) + expect(res.ok).toBe(false) + expect(res.slices.map((s) => s.status)).toEqual(['success', 'failure', 'success']) + }) + + it('records a launch error as slice failure and keeps going', async () => { + const r = fakeRunner() + const res = await runBackfill({ + partitions: ['a', 'b'], + launch: async (p) => { + if (p === 'a') throw new Error('boom') + return r.launch(p) + }, + waitTerminal: r.waitTerminal + }) + expect(res.slices[0]).toMatchObject({ status: 'failure', error: 'boom' }) + expect(res.slices[1].status).toBe('success') + }) + + it('stops before the next launch when cancelled, leaving the rest pending', async () => { + const r = fakeRunner() + let done = 0 + const res = await runBackfill({ + partitions: ['a', 'b', 'c'], + launch: r.launch, + waitTerminal: async (id) => { + done++ + return r.waitTerminal(id) + }, + isCancelled: () => done >= 1 + }) + expect(r.launched).toEqual(['a']) + expect(res.cancelled).toBe(true) + expect(res.ok).toBe(false) + expect(res.slices.map((s) => s.status)).toEqual(['success', 'pending', 'pending']) + }) + + it('cancels a job whose launch raced the cancellation', async () => { + let cancelled = false + const cancelledJobs: string[] = [] + const res = await runBackfill({ + partitions: ['a', 'b'], + launch: async (p) => { + // The user clicks cancel while the launch request is in flight — + // there is no job id to cancel yet. + cancelled = true + return `job:${p}` + }, + waitTerminal: async () => 'failure', + isCancelled: () => cancelled, + cancelJob: async (id) => { + cancelledJobs.push(id) + } + }) + expect(cancelledJobs).toEqual(['job:a']) + expect(res.cancelled).toBe(true) + expect(res.slices.map((s) => s.status)).toEqual(['failure', 'pending']) + }) + + it('emits a snapshot per transition, never mutating earlier snapshots', async () => { + const r = fakeRunner() + const snapshots: BackfillSliceState[][] = [] + await runBackfill({ + partitions: ['a'], + launch: r.launch, + waitTerminal: r.waitTerminal, + onUpdate: (s) => snapshots.push(s) + }) + // initial pending, running, running+jobId, terminal + expect(snapshots.map((s) => s[0].status)).toEqual(['pending', 'running', 'running', 'success']) + expect(snapshots[1][0].jobId).toBeUndefined() + expect(snapshots[2][0].jobId).toBe('job:a') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/backfillRun.ts b/frontend/src/lib/components/assets/AssetGraph/backfillRun.ts new file mode 100644 index 0000000000..51fbc9664b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/backfillRun.ts @@ -0,0 +1,83 @@ +// Client-side orchestration of a partition-range backfill (enterprise): one +// deployed run of the producing script per slice, launched with an explicit +// `partition` arg (the worker only resolves a partition when the arg is +// absent, so the caller-provided value wins). Slices run sequentially — +// concurrent materializations of the same ducklake table would contend on +// the catalog commit — and a failed slice does not stop the rest: each slice +// is independent, and the missing/failed set is simply the next worklist. +// +// Pure module (no Svelte runes) so the loop is unit-testable; reactive +// progress is delivered via `onUpdate` snapshots, mirroring +// `cascadeOrchestrator.ts`. + +export type BackfillSliceStatus = 'pending' | 'running' | 'success' | 'failure' + +export type BackfillSliceState = { + partition: string + status: BackfillSliceStatus + jobId?: string + error?: string +} + +export type BackfillRunOptions = { + /** Partition values to materialize, in run order. */ + partitions: string[] + /** Launch one run of the producer with the given partition arg; returns the job id. */ + launch: (partition: string) => Promise + /** Resolve once the job reaches a terminal state. */ + waitTerminal: (jobId: string) => Promise<'success' | 'failure'> + /** Snapshot of all slice states, emitted on every transition. */ + onUpdate?: (slices: BackfillSliceState[]) => void + /** Checked before each launch; a true stop leaves the remaining slices 'pending'. */ + isCancelled?: () => boolean + /** + * Cancel a job whose launch raced the cancellation — the cancel click had + * no job id to act on yet, so the loop cancels it as soon as the id + * arrives. Must not throw (the job may already be terminal). + */ + cancelJob?: (jobId: string) => Promise +} + +export type BackfillRunResult = { + /** True when every slice ran and succeeded. */ + ok: boolean + /** True when the loop stopped early on `isCancelled`. */ + cancelled: boolean + slices: BackfillSliceState[] +} + +export async function runBackfill(opts: BackfillRunOptions): Promise { + const { partitions, launch, waitTerminal, onUpdate, isCancelled, cancelJob } = opts + const slices: BackfillSliceState[] = partitions.map((partition) => ({ + partition, + status: 'pending' + })) + const emit = () => onUpdate?.(slices.map((s) => ({ ...s }))) + emit() + let cancelled = false + for (const slice of slices) { + if (isCancelled?.()) { + cancelled = true + break + } + slice.status = 'running' + emit() + try { + slice.jobId = await launch(slice.partition) + emit() + if (isCancelled?.() && cancelJob) { + await cancelJob(slice.jobId) + } + slice.status = await waitTerminal(slice.jobId) + } catch (e) { + slice.status = 'failure' + slice.error = e instanceof Error ? e.message : String(e) + } + emit() + } + return { + ok: !cancelled && slices.every((s) => s.status === 'success'), + cancelled, + slices + } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts new file mode 100644 index 0000000000..e4692de6b8 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts @@ -0,0 +1,523 @@ +import { describe, expect, it } from 'vitest' +import type { AssetGraphResponse, AssetGraphTrigger, NativeTriggerKind } from './types' +import { + ancestors, + assetUriToNodeId, + boundedSet, + buildLineageDag, + buildLineageDownstreamMap, + descendants, + nonAutorunTriggerScripts, + reachableCutting, + scriptNodeId, + scriptsOf, + validStarts, + validFromStarts +} from './boundedCascade' +import { computeInducedSchedule } from './graphTraversal' + +type W = [script: string, asset: string] // producer write edge (datatable) +type R = [script: string, asset: string] // pure-read edge (datatable) +type S = [script: string, asset: string] // `// on ` subscription +type T = [producer: string, tested: string, asset: string] // `// data_test` ordering edge + +function graph(opts: { + scripts?: string[] + writes?: W[] + reads?: R[] + subs?: S[] + tests?: T[] + native?: Array<[kind: NativeTriggerKind, script: string]> +}): AssetGraphResponse { + const { scripts = [], writes = [], reads = [], subs = [], tests = [], native = [] } = opts + const triggers: AssetGraphTrigger[] = [ + ...subs.map( + ([s, a]) => + ({ + trigger_kind: 'asset', + asset_kind: 'datatable', + asset_path: a, + runnable_kind: 'script', + runnable_path: s + }) as const + ), + ...native.map( + ([kind, s]) => + ({ trigger_kind: kind, runnable_kind: 'script', runnable_path: s }) as AssetGraphTrigger + ) + ] + return { + assets: [], + runnables: scripts.map((p) => ({ path: p, usage_kind: 'script' as const })), + edges: [ + ...writes.map(([s, a]) => ({ + runnable_path: s, + runnable_kind: 'script' as const, + asset_kind: 'datatable' as const, + asset_path: a, + access_type: 'w' as const + })), + ...reads.map(([s, a]) => ({ + runnable_path: s, + runnable_kind: 'script' as const, + asset_kind: 'datatable' as const, + asset_path: a, + access_type: 'r' as const + })) + ], + triggers, + test_edges: tests.map(([producer, tested, a]) => ({ + producer_kind: 'script' as const, + producer_path: producer, + runnable_kind: 'script' as const, + runnable_path: tested, + asset_kind: 'datatable' as const, + asset_path: a + })) + } +} + +const sn = scriptNodeId +const asset = (p: string) => `datatable:${p}` + +describe('buildLineageDag', () => { + it('links producer → asset → subscriber and asset → reader', () => { + // a writes x; b subscribes to x; c reads x. + const g = graph({ writes: [['a', 'x']], subs: [['b', 'x']], reads: [['c', 'x']] }) + const dag = buildLineageDag(g) + expect([...(dag.down.get(sn('a')) ?? [])]).toEqual([asset('x')]) + expect([...(dag.down.get(asset('x')) ?? [])].sort()).toEqual([sn('b'), sn('c')]) + }) + + it('treats rw as production only (no self-cycle through the asset)', () => { + const g: AssetGraphResponse = { + assets: [], + runnables: [{ path: 'u', usage_kind: 'script' }], + edges: [ + { + runnable_path: 'u', + runnable_kind: 'script', + asset_kind: 'datatable', + asset_path: 'x', + access_type: 'rw' + } + ], + triggers: [] + } + const dag = buildLineageDag(g) + expect([...(dag.down.get(sn('u')) ?? [])]).toEqual([asset('x')]) + expect(dag.up.get(sn('u'))).toBeUndefined() // asset is not upstream of its own writer + }) + + it('routes a data_test ordering edge through the referenced asset', () => { + // prod writes x; tested has a `// data_test` against x (prod → tested edge). + // The DAG must place x (and thus prod) upstream of tested so a cascade + // materializes x first. + const g = graph({ + scripts: ['prod', 'tested'], + writes: [['prod', 'x']], + tests: [['prod', 'tested', 'x']] + }) + const dag = buildLineageDag(g) + // asset x → tested (routed through the asset, not a direct prod → tested hop) + expect([...(dag.down.get(asset('x')) ?? [])]).toEqual([sn('tested')]) + // prod → x → tested makes prod an ancestor of tested. + expect(ancestors(dag, sn('tested'))).toEqual(new Set([asset('x'), sn('prod')])) + }) +}) + +describe('ancestors / descendants', () => { + it('walks transitively over scripts and assets', () => { + // a → x → b → y → c + const g = graph({ + writes: [ + ['a', 'x'], + ['b', 'y'] + ], + subs: [ + ['b', 'x'], + ['c', 'y'] + ] + }) + const dag = buildLineageDag(g) + expect(descendants(dag, sn('a'))).toEqual(new Set([asset('x'), sn('b'), asset('y'), sn('c')])) + expect(ancestors(dag, sn('c'))).toEqual(new Set([asset('y'), sn('b'), asset('x'), sn('a')])) + }) + + it('excludes the start node even on a cycle back to it', () => { + // a → x → b → y → a (cycle): descendants(a) must not contain a. + const g = graph({ + writes: [ + ['a', 'x'], + ['b', 'y'] + ], + subs: [ + ['b', 'x'], + ['a', 'y'] + ] + }) + const dag = buildLineageDag(g) + expect(descendants(dag, sn('a')).has(sn('a'))).toBe(false) + expect(ancestors(dag, sn('a')).has(sn('a'))).toBe(false) + }) +}) + +describe('boundedSet', () => { + // a → x → b → y → c → z → d (linear chain through assets) + const chain = () => + graph({ + writes: [ + ['a', 'x'], + ['b', 'y'], + ['c', 'z'] + ], + subs: [ + ['b', 'x'], + ['c', 'y'], + ['d', 'z'] + ] + }) + + it('stops at a single end node (script)', () => { + const dag = buildLineageDag(chain()) + const res = boundedSet(dag, sn('a'), [sn('c')]) + // path a..c includes a,x,b,y,c — not z or d. + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b', 'c']) + expect(res.nodes.has(asset('z'))).toBe(false) + expect(res.droppedEnds).toEqual([]) + }) + + it('supports an asset as the end bound', () => { + const dag = buildLineageDag(chain()) + const res = boundedSet(dag, sn('a'), [asset('y')]) + // up to datatable://y → a, b produced it. + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b']) + }) + + it('unions multiple ends', () => { + // diamond: a → b and a → c, both → d. Bound to {b, c} excludes d. + const g = graph({ + writes: [ + ['a', 'xa'], + ['b', 'xb'], + ['c', 'xc'] + ], + subs: [ + ['b', 'xa'], + ['c', 'xa'], + ['d', 'xb'], + ['d', 'xc'] + ] + }) + const dag = buildLineageDag(g) + const res = boundedSet(dag, sn('a'), [sn('b'), sn('c')]) + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b', 'c']) + }) + + it('drops ends not downstream of start', () => { + const dag = buildLineageDag(chain()) + const res = boundedSet(dag, sn('c'), [sn('a')]) + expect(res.droppedEnds).toEqual([sn('a')]) + expect(res.reachableEnds).toEqual([]) + expect([...res.nodes]).toEqual([sn('c')]) + }) + + it('is cycle-safe', () => { + // b ↔ c cycle downstream of a; bounding to c terminates. + const g = graph({ + writes: [ + ['a', 'x'], + ['b', 'y'], + ['c', 'z'] + ], + subs: [ + ['b', 'x'], + ['c', 'y'], + ['b', 'z'] + ] + }) + const dag = buildLineageDag(g) + const res = boundedSet(dag, sn('a'), [sn('c')]) + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b', 'c']) + }) +}) + +describe('validStarts', () => { + it('includes schedule-rooted scripts', () => { + const g = graph({ scripts: ['s'], native: [['schedule', 's']] }) + expect(validStarts(g)).toEqual(new Set([sn('s')])) + }) + + it('includes manual roots (no trigger, not a subscriber)', () => { + const g = graph({ scripts: ['m'] }) + expect(validStarts(g)).toEqual(new Set([sn('m')])) + }) + + it('excludes event-only roots', () => { + const g = graph({ scripts: ['k'], native: [['kafka', 'k']] }) + expect(validStarts(g).has(sn('k'))).toBe(false) + }) + + it('excludes pure asset subscribers but keeps a schedule subscriber', () => { + // sub is `// on x`; sched is both a subscriber and schedule-triggered. + const g = graph({ + scripts: ['a', 'sub', 'sched'], + writes: [['a', 'x']], + subs: [ + ['sub', 'x'], + ['sched', 'x'] + ], + native: [['schedule', 'sched']] + }) + const starts = validStarts(g) + expect(starts.has(sn('a'))).toBe(true) // manual root + expect(starts.has(sn('sub'))).toBe(false) // event-less but a subscriber + expect(starts.has(sn('sched'))).toBe(true) // schedule overrides subscriber + }) +}) + +describe('validFromStarts (mid-DAG selective execution)', () => { + // a → x → sub (subscriber) → y → reader (pure read). `k` is event-triggered. + const g = () => + graph({ + scripts: ['a', 'sub', 'reader', 'k'], + writes: [ + ['a', 'x'], + ['sub', 'y'] + ], + reads: [['reader', 'y']], + subs: [['sub', 'x']], + native: [['kafka', 'k']] + }) + + it('includes mid-DAG subscribers and pure readers, not just roots', () => { + const from = validFromStarts(g()) + expect(from.has(sn('a'))).toBe(true) // root + expect(from.has(sn('sub'))).toBe(true) // mid-DAG subscriber — NOT a validStart + expect(from.has(sn('reader'))).toBe(true) // pure reader + // The old root-only gate would have rejected the mid-DAG nodes. + expect(validStarts(g()).has(sn('sub'))).toBe(false) + }) + + it('excludes event-triggered scripts (no run-now gesture)', () => { + expect(validFromStarts(g()).has(sn('k'))).toBe(false) + }) + + it('excludes webhook/data_upload mid-DAG subscribers (need caller input)', () => { + // a → x → upload_mid (subscribes x AND `// on data_upload`) → y → consumer. + // upload_mid must NOT be an eligible start (empty-arg run has no S3Object), + // and must be a barrier so `consumer` isn't run with a skipped producer. + const g2 = graph({ + scripts: ['a', 'upload_mid', 'hook_mid', 'consumer'], + writes: [ + ['a', 'x'], + ['upload_mid', 'y'] + ], + subs: [ + ['upload_mid', 'x'], + ['hook_mid', 'x'], + ['consumer', 'y'] + ], + native: [ + ['data_upload', 'upload_mid'], + ['webhook', 'hook_mid'] + ] + }) + const from = validFromStarts(g2) + expect(from.has(sn('upload_mid'))).toBe(false) + expect(from.has(sn('hook_mid'))).toBe(false) + expect(from.has(sn('a'))).toBe(true) // the plain root is still eligible + // and they're barriers, so running downstream from `a` cuts them + consumer. + const bars = nonAutorunTriggerScripts(g2) + expect(bars.has(sn('upload_mid'))).toBe(true) + expect(bars.has(sn('hook_mid'))).toBe(true) + expect(scriptsOf(reachableCutting(buildLineageDag(g2), [sn('a')], bars)).sort()).toEqual(['a']) + }) + + it('keeps a scheduled root that also carries an event trigger', () => { + // schedule wins over the secondary kafka trigger in validStarts, so the + // scheduled root stays --from-eligible (regression guard). + const g2 = graph({ + scripts: ['sched_evt', 'consumer'], + writes: [['sched_evt', 'x']], + subs: [['consumer', 'x']], + native: [ + ['schedule', 'sched_evt'], + ['kafka', 'sched_evt'] + ] + }) + expect(validStarts(g2).has(sn('sched_evt'))).toBe(true) + expect(nonAutorunTriggerScripts(g2).has(sn('sched_evt'))).toBe(true) + expect(validFromStarts(g2).has(sn('sched_evt'))).toBe(true) // union with roots + }) + + it('runs a mid-DAG start plus its downstream WITHOUT re-running upstream', () => { + // Starting at `sub`, the unbounded downstream is {sub, y, reader} — `a`/`x` + // upstream are never pulled in (dbt `--select sub+`). + const dag = buildLineageDag(g()) + const downstream = new Set([sn('sub'), ...descendants(dag, sn('sub'))]) + expect(scriptsOf(downstream).sort()).toEqual(['reader', 'sub']) + expect(downstream.has(sn('a'))).toBe(false) + expect(downstream.has(asset('x'))).toBe(false) + }) +}) + +describe('reachableCutting (barrier cut for "Run + downstream")', () => { + // a → x → k(kafka, also reads x) → z → consumer. Starting at `a` and running + // downstream must cut the event handler `k` AND `consumer` (only reachable + // through it) — else they'd launch with empty args. Mirrors the CLI cut. + const g = () => + graph({ + scripts: ['a', 'k', 'consumer'], + writes: [ + ['a', 'x'], + ['k', 'z'] + ], + reads: [['k', 'x']], + subs: [['consumer', 'z']], + native: [['kafka', 'k']] + }) + + it('detects event-triggered scripts as barriers', () => { + expect(nonAutorunTriggerScripts(g()).has(sn('k'))).toBe(true) + expect(nonAutorunTriggerScripts(g()).has(sn('a'))).toBe(false) + }) + + it('cuts an event descendant and its event-only downstream from a run set', () => { + const dag = buildLineageDag(g()) + const barriers = nonAutorunTriggerScripts(g()) + const runNodes = reachableCutting(dag, [sn('a')], barriers) + expect(scriptsOf(runNodes).sort()).toEqual(['a']) // k + consumer cut + expect(runNodes.has(sn('k'))).toBe(false) + expect(runNodes.has(sn('consumer'))).toBe(false) + }) + + it('protects an explicit start even if it carries an event trigger', () => { + const dag = buildLineageDag(g()) + // start = k itself (user named it); it runs, and so does its downstream. + const barriers = new Set([...nonAutorunTriggerScripts(g())].filter((id) => id !== sn('k'))) + const runNodes = reachableCutting(dag, [sn('k')], barriers) + expect(scriptsOf(runNodes).sort()).toEqual(['consumer', 'k']) + }) + + it('keeps a scheduled event root (and its downstream) reachable from an upstream start', () => { + // a → x → sched_evt(schedule + kafka) → y → consumer. Running downstream + // from `a`, sched_evt is a scheduled root so it must NOT be a barrier even + // though it carries an event trigger — matching the CLI, which excludes all + // valid roots (`starts`), not just the picked start. Mirror of the page's + // barrier construction: nonAutorunTriggerScripts minus validStarts minus start. + const g2 = graph({ + scripts: ['a', 'sched_evt', 'consumer'], + writes: [ + ['a', 'x'], + ['sched_evt', 'y'] + ], + subs: [ + ['sched_evt', 'x'], + ['consumer', 'y'] + ], + native: [ + ['schedule', 'sched_evt'], + ['kafka', 'sched_evt'] + ] + }) + const dag = buildLineageDag(g2) + const roots = validStarts(g2) + const barriers = new Set( + [...nonAutorunTriggerScripts(g2)].filter((id) => !roots.has(id) && id !== sn('a')) + ) + const runNodes = reachableCutting(dag, [sn('a')], barriers) + expect(scriptsOf(runNodes).sort()).toEqual(['a', 'consumer', 'sched_evt']) + // Without the validStarts exclusion, sched_evt (a kafka handler) would be a + // barrier and `consumer` would be dropped — the bug this guards. + const naiveBarriers = new Set([...nonAutorunTriggerScripts(g2)].filter((id) => id !== sn('a'))) + expect(scriptsOf(reachableCutting(dag, [sn('a')], naiveBarriers)).sort()).toEqual(['a']) + }) +}) + +describe('buildLineageDownstreamMap (read-aware scheduling)', () => { + // a writes x; c only *reads* x (no `// on x`). c must still run after a. + const g = graph({ + scripts: ['a', 'c'], + writes: [['a', 'x']], + reads: [['c', 'x']] + }) + + it('links a producer to a pure reader of its asset', () => { + const map = buildLineageDownstreamMap(g) + expect([...(map.get('a') ?? [])]).toEqual(['c']) + }) + + it('orders the reader after the producer through computeInducedSchedule', () => { + const selected = new Set(['a', 'c']) + // Subscriber-only map (default) misses the read dep → c is a stray root. + const subscriberOnly = computeInducedSchedule(g, selected) + expect(subscriberOnly.roots.sort()).toEqual(['a', 'c']) + // Read-aware map orders c strictly after a. + const readAware = computeInducedSchedule(g, selected, buildLineageDownstreamMap(g)) + expect(readAware.roots).toEqual(['a']) + expect(readAware.indegree.get('c')).toBe(1) + expect(readAware.nodes).toEqual(['a', 'c']) + }) + + it('orders a disjoint-root producer before a data_test that references it', () => { + // Two disjoint roots (the HD-1 repro): `dim` produces the dimension + // `dimc`; `fct` produces `fcto` and has a `// data_test relationships` + // against `dimc`. Without the test edge they are unordered and a cold + // cascade can run `fct` first ("table dimc does not exist"). The test + // edge must place `dim` strictly before `fct`. + const g = graph({ + scripts: ['dim', 'fct'], + writes: [ + ['dim', 'dimc'], + ['fct', 'fcto'] + ], + tests: [['dim', 'fct', 'dimc']] + }) + const selected = new Set(['dim', 'fct']) + const map = buildLineageDownstreamMap(g) + expect([...(map.get('dim') ?? [])]).toEqual(['fct']) + const schedule = computeInducedSchedule(g, selected, map) + expect(schedule.roots).toEqual(['dim']) + expect(schedule.indegree.get('fct')).toBe(1) + expect(schedule.nodes).toEqual(['dim', 'fct']) + expect(schedule.cyclic).toEqual([]) + }) + + it('adds no ordering when the referenced asset has no in-pipeline producer', () => { + // `fct` tests against an external `ext` table nothing produces — the + // backend emits no test edge, so the frontend sees none and `fct` stays + // an independent root (the runtime error stands, as designed). + const g = graph({ + scripts: ['dim', 'fct'], + writes: [['fct', 'fcto']], + tests: [] // no producer for `ext` ⇒ backend omitted the edge + }) + const schedule = computeInducedSchedule( + g, + new Set(['dim', 'fct']), + buildLineageDownstreamMap(g) + ) + expect(schedule.roots.sort()).toEqual(['dim', 'fct']) + }) +}) + +describe('assetUriToNodeId', () => { + it('maps s3 prefix to the s3object kind, others verbatim', () => { + expect(assetUriToNodeId('s3://bucket/key')).toBe('s3object:bucket/key') + expect(assetUriToNodeId('datatable://prod/users')).toBe('datatable:prod/users') + expect(assetUriToNodeId('ducklake://lake/t')).toBe('ducklake:lake/t') + expect(assetUriToNodeId('not-a-uri')).toBeUndefined() + }) + it('strips leading slashes from S3 keys so s3:/// and s3:// share a node', () => { + // Mirror of Rust `parse_asset_syntax`: `--to s3:///exports/x` must resolve + // to the same canonical node as the graph's `s3object:exports/x`. + expect(assetUriToNodeId('s3:///exports/x')).toBe('s3object:exports/x') + expect(assetUriToNodeId('s3:///exports/x')).toBe(assetUriToNodeId('s3://exports/x')) + // All leading slashes are stripped so a canonical key never starts with + // `/` (the quad-slash `S3Object(s3="/x")` form collapses to `x`). + expect(assetUriToNodeId('s3:////x')).toBe('s3object:x') + // Hive-partition keys and non-S3 kinds are untouched. + expect(assetUriToNodeId('s3:///t/y=2024/f.parquet')).toBe('s3object:t/y=2024/f.parquet') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts new file mode 100644 index 0000000000..6fcabbe71f --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts @@ -0,0 +1,346 @@ +import type { AssetGraphResponse, NativeTriggerKind } from './types' +import { assetKey, isWriteEdge } from './lib' + +// Bounded-cascade selective execution: instead of a dbt-style `--select` +// grammar (a compile-time file selector dbt needs because it has no runtime +// DAG), Windmill already runs a live cascade. So the primitive here is to +// *bound* that cascade — start at a pipeline entrypoint, fan downstream, but +// stop at one or more chosen end node(s). The matched set is the "path +// between" start and the ends: +// +// descendants(start) ∩ (ancestors(ends) ∪ ends) ∪ {start} +// +// Pure module (no Svelte runes) so the graph algebra is unit-testable and can +// be ported verbatim to the CLI. See `boundedCascade.test.ts`, and the mirror +// at `cli/src/commands/pipeline/boundedCascade.ts` (keep them in sync). + +// Unified lineage-DAG node ids. Assets use `${asset_kind}:${asset_path}` +// (identical to `assetKey`); runnables are prefixed so a script path can never +// collide with an asset key. Flows are excluded — they aren't cascade members +// (mirrors graphTraversal/asset_dispatch). +export const SCRIPT_PREFIX = 'script:' + +export function scriptNodeId(path: string): string { + return `${SCRIPT_PREFIX}${path}` +} +export function isScriptNode(id: string): boolean { + return id.startsWith(SCRIPT_PREFIX) +} +export function scriptPathOf(id: string): string { + return id.slice(SCRIPT_PREFIX.length) +} + +/** Resolve an asset URI (`datatable://x`, `s3://b/k`, …) to its node id. */ +export function assetUriToNodeId(uri: string): string | undefined { + const m = uri.match(/^([a-z0-9_]+):\/\/(.+)$/i) + if (!m) return undefined + const prefix = m[1].toLowerCase() + // `s3` is the URI prefix for the `s3object` asset kind (mirrors the CLI + // `assetUri` and the canvas). All other kinds use their name verbatim. + const kind = prefix === 's3' ? 's3object' : prefix + // Mirror Rust `parse_asset_syntax`: strip all leading slashes from S3 keys so + // `s3:///key` (default storage) and `s3://key` resolve to the same node id + // and a canonical key never starts with `/`. + const path = kind === 's3object' ? m[2].replace(/^\/+/, '') : m[2] + return `${kind}:${path}` +} + +// Native trigger kinds that fan out *per event*: a single event always flows +// through the whole reactive downstream, so "run up to X now" is not a +// meaningful gesture and these are never offered as bounded-run starts. +const EVENT_TRIGGER_KINDS: ReadonlySet = new Set([ + 'kafka', + 'mqtt', + 'nats', + 'postgres', + 'sqs', + 'gcp', + 'email' +]) + +// Trigger kinds a cascade must NOT auto-run: the per-event kinds above PLUS the +// input-only entrypoints `webhook`/`data_upload`, which need caller-supplied +// input (a request body / an uploaded S3Object) and would run the wrong thing +// with empty args. Mirror of the CLI `NON_AUTORUN_TRIGGER_KINDS`. When the +// deployed `/assets/graph` omits a `webhook`/`data_upload` row (it has none for +// them), such a script simply reads as a manual root here — but whenever the +// marker IS visible (editor overlay / draft), it's excluded from bounded-run +// starts and cut as a barrier, exactly as the CLI does. +const NON_AUTORUN_TRIGGER_KINDS: ReadonlySet = new Set([ + ...EVENT_TRIGGER_KINDS, + 'webhook', + 'data_upload' +]) + +export type LineageDag = { + /** upstream node id → set of direct downstream node ids. */ + down: Map> + /** downstream node id → set of direct upstream node ids. */ + up: Map> + /** Every node id (scripts + assets), including isolated ones. */ + nodes: Set +} + +/** + * Build the directed upstream→downstream lineage DAG over scripts ∪ assets: + * - producer script → asset (write / rw edges) + * - asset → reader script (pure-read edges — a data dependency) + * - asset → subscriber script (`// on ` triggers) + * - asset → testing script (`// data_test` ordering edges) + * + * An `rw` edge is treated as production only (script → asset); emitting the + * reverse asset → script too would make every upsert a 2-cycle through its own + * asset. + * + * `test_edges` are modeled through the referenced asset (asset → testing + * script), NOT as a direct producer → testing-script hop: the producer already + * has a write edge to that asset, so this yields producer → asset → testing + * script and keeps the two-hop (script → asset → script) invariant that + * `buildLineageDownstreamMap` relies on. + */ +export function buildLineageDag(g: AssetGraphResponse): LineageDag { + const down = new Map>() + const up = new Map>() + const nodes = new Set() + + const addEdge = (a: string, b: string) => { + if (a === b) return + nodes.add(a) + nodes.add(b) + ;(down.get(a) ?? down.set(a, new Set()).get(a)!).add(b) + ;(up.get(b) ?? up.set(b, new Set()).get(b)!).add(a) + } + + // Register every node up front so isolated scripts/assets still appear. + for (const r of g.runnables ?? []) { + if (r.usage_kind === 'script') nodes.add(scriptNodeId(r.path)) + } + for (const a of g.assets ?? []) nodes.add(`${a.kind}:${a.path}`) + + for (const e of g.edges ?? []) { + if (e.runnable_kind !== 'script') continue + const aid = assetKey(e) + if (isWriteEdge(e)) { + addEdge(scriptNodeId(e.runnable_path), aid) + } else if ((e.access_type ?? 'r') === 'r') { + addEdge(aid, scriptNodeId(e.runnable_path)) + } + } + for (const t of g.triggers ?? []) { + if (t.trigger_kind !== 'asset' || t.runnable_kind !== 'script') continue + addEdge(assetKey(t), scriptNodeId(t.runnable_path)) + } + // Data-test ordering edges: the referenced asset must exist before the + // tested script runs. Routed through the asset node so the existing + // producer → asset write edge extends into producer → asset → testing script. + for (const t of g.test_edges ?? []) { + if (t.runnable_kind !== 'script') continue + addEdge(assetKey(t), scriptNodeId(t.runnable_path)) + } + + return { down, up, nodes } +} + +/** BFS closure over an adjacency map, excluding `start`. */ +function closure(adj: Map>, start: string): Set { + const seen = new Set() + const queue = [start] + while (queue.length > 0) { + const cur = queue.shift()! + for (const n of adj.get(cur) ?? []) { + if (seen.has(n)) continue + seen.add(n) + queue.push(n) + } + } + // A cycle back to `start` would have re-added it; the contract is to + // exclude the node itself. + seen.delete(start) + return seen +} + +/** Transitive downstream of `n` (excludes `n`). Cycle-safe. */ +export function descendants(dag: LineageDag, n: string): Set { + return closure(dag.down, n) +} +/** Transitive upstream of `n` (excludes `n`). Cycle-safe. */ +export function ancestors(dag: LineageDag, n: string): Set { + return closure(dag.up, n) +} + +/** + * Nodes reachable from `starts` over the lineage DAG, treating `barriers` as cut + * points: a barrier node is neither included NOR traversed through, so a node + * reachable ONLY via a barrier is excluded while one also reachable via another + * path stays. Mirror of the CLI `reachableCutting`. Used to keep event handlers + * (and their event-only downstream) out of a cascade run — running such a + * consumer whose producer was skipped would feed it missing/stale inputs. + */ +export function reachableCutting( + dag: LineageDag, + starts: Iterable, + barriers: Set +): Set { + const seen = new Set() + const queue: string[] = [] + for (const s of starts) { + if (barriers.has(s) || seen.has(s)) continue + seen.add(s) + queue.push(s) + } + while (queue.length > 0) { + const n = queue.shift()! + for (const next of dag.down.get(n) ?? []) { + if (barriers.has(next) || seen.has(next)) continue + seen.add(next) + queue.push(next) + } + } + return seen +} + +export type BoundedResult = { + /** Path-between node set (scripts + assets), always including `start`. */ + nodes: Set + /** Ends that are actually reachable downstream of `start`. */ + reachableEnds: string[] + /** Ends passed in that are not downstream of `start` (ignored, surfaced). */ + droppedEnds: string[] +} + +/** + * Nodes on any path from `start` to any of `ends` (inclusive of both). Ends + * not reachable from `start` are dropped and reported. With no reachable end + * the result is just `{start}` — the caller decides whether to warn or fall + * back to the full downstream. + */ +export function boundedSet(dag: LineageDag, start: string, ends: string[]): BoundedResult { + const desc = descendants(dag, start) + const downSet = new Set(desc) + downSet.add(start) + + const reachableEnds = ends.filter((e) => downSet.has(e)) + const droppedEnds = ends.filter((e) => !downSet.has(e)) + if (reachableEnds.length === 0) { + return { nodes: new Set([start]), reachableEnds, droppedEnds } + } + + const upClosure = new Set() + for (const e of reachableEnds) { + upClosure.add(e) + for (const a of ancestors(dag, e)) upClosure.add(a) + } + const nodes = new Set() + for (const n of downSet) if (upClosure.has(n)) nodes.add(n) + nodes.add(start) + return { nodes, reachableEnds, droppedEnds } +} + +/** + * Script node ids eligible to *start* a bounded run: schedule-triggered + * scripts, or manual roots (not an asset subscriber and not event-triggered). + * Event-driven entrypoints are excluded — they have no "run up to X" gesture. + */ +export function validStarts(g: AssetGraphResponse): Set { + const subscribers = new Set() + const scheduleScripts = new Set() + const eventScripts = new Set() + for (const t of g.triggers ?? []) { + if (t.runnable_kind !== 'script') continue + if (t.trigger_kind === 'asset') subscribers.add(t.runnable_path) + else if (t.trigger_kind === 'schedule') scheduleScripts.add(t.runnable_path) + else if (EVENT_TRIGGER_KINDS.has(t.trigger_kind)) eventScripts.add(t.runnable_path) + } + + const out = new Set() + for (const r of g.runnables ?? []) { + if (r.usage_kind !== 'script') continue + const p = r.path + if (scheduleScripts.has(p)) out.add(scriptNodeId(p)) + else if (!subscribers.has(p) && !eventScripts.has(p)) out.add(scriptNodeId(p)) + } + return out +} + +/** + * Script node ids eligible as an EXPLICIT bounded-run start from *anywhere* in + * the DAG (dbt's `--select model+`): every script that can run with empty args — + * i.e. all scripts except event-triggered ones (kafka/mqtt/nats/postgres/sqs/ + * gcp/email fan out per event and have no "run now" gesture). Unlike + * `validStarts` (schedule/manual roots only), this INCLUDES mid-DAG asset + * subscribers and pure readers, so "Run + downstream" can begin at any model — + * that node plus its transitive downstream runs, upstream is never re-run. + * (`webhook`/`data_upload` have no trigger row here — same as `validStarts` they + * read as manual roots and are already included.) + */ +export function validFromStarts(g: AssetGraphResponse): Set { + const nonAutorunScripts = new Set() + for (const t of g.triggers ?? []) { + if (t.runnable_kind !== 'script') continue + if (NON_AUTORUN_TRIGGER_KINDS.has(t.trigger_kind)) nonAutorunScripts.add(t.runnable_path) + } + // Seed with schedule/manual roots: `validStarts` lets a schedule identity win + // over a secondary non-autorun trigger, so a scheduled root that also carries + // e.g. a `// on kafka` stays `--from`-eligible. The mid-DAG loop then adds only + // scripts that can run with empty args — a `webhook`/`data_upload` subscriber + // is NOT added (it needs caller-supplied input). + const out = new Set(validStarts(g)) + for (const r of g.runnables ?? []) { + if (r.usage_kind !== 'script') continue + if (!nonAutorunScripts.has(r.path)) out.add(scriptNodeId(r.path)) + } + return out +} + +/** + * Script node ids carrying a non-autorun trigger (event kinds kafka/mqtt/…/email + * PLUS input-only webhook/data_upload) — they fan out per event or need + * caller-supplied input, so a cascade must cut them (as `barriers` for + * `reachableCutting`) even when they're a lineage descendant of the start. + * Mirror of the CLI `nonAutorunTriggerScripts`. Only detects what the graph + * surfaces: the deployed `/assets/graph` omits `webhook`/`data_upload` rows, so + * such a handler is only cut when its marker is visible (editor overlay / draft) + * — the same limitation as `validStarts`; the CLI closes it via graph enrichment. + */ +export function nonAutorunTriggerScripts(g: AssetGraphResponse): Set { + const out = new Set() + for (const t of g.triggers ?? []) { + if (t.runnable_kind === 'script' && NON_AUTORUN_TRIGGER_KINDS.has(t.trigger_kind)) { + out.add(scriptNodeId(t.runnable_path)) + } + } + return out +} + +/** Project a node-id set to the script paths it contains (run targets). */ +export function scriptsOf(nodes: Iterable): string[] { + const out: string[] = [] + for (const id of nodes) if (isScriptNode(id)) out.push(scriptPathOf(id)) + return out +} + +/** + * Producer→consumer adjacency (script path → downstream script paths) over the + * lineage DAG: one hop through a single asset, following BOTH `// on` + * subscribers *and* pure-read data dependencies. Unlike + * `graphTraversal.buildDownstreamMap` (subscriber-only, which models the + * production dispatch), this orders a script that merely reads an upstream + * asset after its producer — so a bounded selection containing such a reader + * schedules correctly. Mirror of the CLI `topoOrder` adjacency. + */ +export function buildLineageDownstreamMap(g: AssetGraphResponse): Map> { + const dag = buildLineageDag(g) + const map = new Map>() + for (const id of dag.nodes) { + if (!isScriptNode(id)) continue + const s = scriptPathOf(id) + const subs = new Set() + for (const asset of dag.down.get(id) ?? []) { + for (const sub of dag.down.get(asset) ?? []) { + if (isScriptNode(sub) && scriptPathOf(sub) !== s) subs.add(scriptPathOf(sub)) + } + } + if (subs.size > 0) map.set(s, subs) + } + return map +} diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts index 5509839dab..3a0fd4fbd3 100644 --- a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest' -import { runCascade } from './cascadeOrchestrator' -import type { DownstreamClosure } from './graphTraversal' +import { runCascade, runSelection } from './cascadeOrchestrator' +import type { DownstreamClosure, InducedSchedule } from './graphTraversal' function closure(edges: Array<[from: string, to: string]>, nodes: string[]): DownstreamClosure { const e = new Map>() @@ -161,3 +161,101 @@ describe('runCascade', () => { expect(res.statuses.get('c')?.status).toBe('skipped') }) }) + +function schedule( + edges: Array<[from: string, to: string]>, + nodes: string[], + roots: string[] +): InducedSchedule { + const e = new Map>() + const indegree = new Map() + for (const n of nodes) indegree.set(n, 0) + for (const [from, to] of edges) { + const set = e.get(from) ?? new Set() + set.add(to) + e.set(from, set) + indegree.set(to, (indegree.get(to) ?? 0) + 1) + } + return { nodes, edges: e, indegree, roots, cyclic: [] } +} + +describe('runSelection', () => { + it('runs every selected node, seeding all roots', async () => { + // two independent roots a, b each → c. + const sched = schedule( + [ + ['a', 'c'], + ['b', 'c'] + ], + ['a', 'b', 'c'], + ['a', 'b'] + ) + const r = fakeRunner() + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(true) + // Sort a copy for the membership check — mutating `r.launched` here + // would invalidate the launch-order assertions below. + expect([...r.launched].sort()).toEqual(['a', 'b', 'c']) + // c only after both upstreams. + expect(r.launched.indexOf('c')).toBeGreaterThan(r.launched.indexOf('a')) + expect(r.launched.indexOf('c')).toBeGreaterThan(r.launched.indexOf('b')) + }) + + it('skips a failed node’s descendants but keeps independent branches running', async () => { + // Two independent chains: a → b and c → d. `a` fails; `b` (its descendant) + // must be skipped, but `d` depends only on the successful `c`, so it must + // still run — a failure must not stall unrelated branches. + const sched = schedule( + [ + ['a', 'b'], + ['c', 'd'] + ], + ['a', 'b', 'c', 'd'], + ['a', 'c'] + ) + const r = fakeRunner({ a: 'failure' }) + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(false) + expect(res.statuses.get('a')?.status).toBe('failure') + expect(res.statuses.get('b')?.status).toBe('skipped') + expect(res.statuses.get('c')?.status).toBe('success') + expect(res.statuses.get('d')?.status).toBe('success') + expect(r.launched).toContain('d') + expect(r.launched).not.toContain('b') + }) + + it('skips a join node when any one of its upstreams fails', async () => { + // {a, b} → c. `a` fails; `c` needs both, so it must be skipped even though + // `b` succeeds — a poisoned lineage isn’t rescued by a sibling success. + const sched = schedule( + [ + ['a', 'c'], + ['b', 'c'] + ], + ['a', 'b', 'c'], + ['a', 'b'] + ) + const r = fakeRunner({ a: 'failure' }) + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(false) + expect(res.statuses.get('c')?.status).toBe('skipped') + expect(r.launched).not.toContain('c') + }) + + it('stops scheduling a failed node’s chain', async () => { + const sched = schedule([['a', 'b']], ['a', 'b'], ['a']) + const r = fakeRunner({ a: 'failure' }) + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(false) + expect(r.launched).toEqual(['a']) + expect(res.statuses.get('b')?.status).toBe('skipped') + }) + + it('a single-node selection runs that node', async () => { + const sched = schedule([], ['a'], ['a']) + const r = fakeRunner() + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(true) + expect(r.launched).toEqual(['a']) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts index 7e6a6029ed..424f1581ec 100644 --- a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts @@ -1,4 +1,4 @@ -import type { DownstreamClosure } from './graphTraversal' +import type { DownstreamClosure, InducedSchedule } from './graphTraversal' // Client-side orchestration of a pipeline chain dev-run ("Run + downstream" // over a graph that contains drafts). The backend asset-trigger dispatcher @@ -112,3 +112,104 @@ export async function runCascade(opts: CascadeRunOptions): Promise s.status === 'success') return { ok, statuses } } + +export type SelectionRunOptions = { + /** Multi-root induced schedule of the selected scripts (computeInducedSchedule). */ + schedule: InducedSchedule + /** Launch one script (preview for drafts, by-path for deployed); returns the job id. */ + launch: (path: string) => Promise + /** Resolve once the job reaches a terminal state. */ + waitTerminal: (jobId: string) => Promise<'success' | 'failure'> + /** Snapshot of all node states, emitted on every transition. */ + onUpdate?: (statuses: Map) => void +} + +/** + * Execute an arbitrary selected set of scripts (e.g. a bounded-cascade + * selection) in topological order. Unlike `runCascade` there is no single + * privileged root — every `schedule.roots` entry is seeded at once and a node + * runs as soon as its in-set upstreams all succeed. A failure abandons only + * that node's lineage (its transitive descendants end 'skipped'); INDEPENDENT + * branches keep running. In-flight jobs always finish. + */ +export async function runSelection(opts: SelectionRunOptions): Promise { + const { schedule, launch, waitTerminal, onUpdate } = opts + const statuses = new Map() + for (const n of schedule.nodes) statuses.set(n, { status: 'pending' }) + const remaining = new Map(schedule.indegree) + let failed = false + // Nodes a failed prerequisite has made unrunnable — the transitive descendants + // of every failure. Gating scheduling on this (rather than a single global + // fail-fast flag) is what lets independent branches finish: only the lineage + // below a failure is skipped, not every node that happens to become ready + // afterwards. Poisoned nodes are never scheduled, so they end 'skipped' below. + const poisoned = new Set() + const inFlight = new Set>() + + const emit = () => onUpdate?.(new Map(statuses)) + + function poison(path: string) { + const stack = [path] + while (stack.length > 0) { + const n = stack.pop()! + for (const s of schedule.edges.get(n) ?? []) { + if (!poisoned.has(s)) { + poisoned.add(s) + stack.push(s) + } + } + } + } + + function schedule_(path: string) { + const p = runNode(path).finally(() => inFlight.delete(p)) + inFlight.add(p) + } + + async function runNode(path: string): Promise { + statuses.set(path, { status: 'running' }) + emit() + let jobId: string | undefined + try { + jobId = await launch(path) + statuses.set(path, { status: 'running', jobId }) + emit() + const term = await waitTerminal(jobId) + statuses.set(path, { status: term, jobId }) + emit() + if (term === 'failure') { + failed = true + poison(path) + return + } + } catch (e) { + statuses.set(path, { + status: 'failure', + jobId, + error: e instanceof Error ? e.message : String(e) + }) + emit() + failed = true + poison(path) + return + } + for (const s of schedule.edges.get(path) ?? []) { + const d = (remaining.get(s) ?? 0) - 1 + remaining.set(s, d) + if (d === 0 && !poisoned.has(s)) schedule_(s) + } + } + + for (const r of schedule.roots) schedule_(r) + while (inFlight.size > 0) { + await Promise.race(inFlight) + } + + for (const [n, st] of statuses) { + if (st.status === 'pending') statuses.set(n, { status: 'skipped' }) + } + emit() + + const ok = !failed && [...statuses.values()].every((s) => s.status === 'success') + return { ok, statuses } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeRun.test.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeRun.test.ts new file mode 100644 index 0000000000..b180a8cd9f --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeRun.test.ts @@ -0,0 +1,35 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { JobService } from '$lib/gen' +import { makeLaunch } from './cascadeRun' + +afterEach(() => vi.restoreAllMocks()) + +// The client orchestrates the cascade closure, so every launch must carry +// `_wmill_skip_asset_dispatch: true`. A caller arg (e.g. the run form for the +// root node) must NOT be able to override that guard back to false and let the +// backend also dispatch deployed subscribers. +describe('makeLaunch dispatch guard', () => { + it('local preview: a caller `_wmill_skip_asset_dispatch: false` cannot re-enable dispatch', async () => { + const spy = vi.spyOn(JobService, 'runScriptPreview').mockResolvedValue('job-1' as any) + const launch = makeLaunch({ + workspace: 'w', + resolveLocal: () => ({ content: 'x', language: 'bun' as any }), + argsFor: () => ({ _wmill_skip_asset_dispatch: false, foo: 1 }) + }) + await launch('f/x/root') + const body = (spy.mock.calls[0][0] as any).requestBody + expect(body.args._wmill_skip_asset_dispatch).toBe(true) // guard wins + expect(body.args.foo).toBe(1) // other caller args preserved + }) + + it('deployed by-path: a caller `_wmill_skip_asset_dispatch: false` cannot re-enable dispatch', async () => { + const spy = vi.spyOn(JobService, 'runScriptByPath').mockResolvedValue('job-2' as any) + const launch = makeLaunch({ + workspace: 'w', + argsFor: () => ({ _wmill_skip_asset_dispatch: false }) + }) + await launch('f/x/deployed') + const body = (spy.mock.calls[0][0] as any).requestBody + expect(body._wmill_skip_asset_dispatch).toBe(true) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeRun.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeRun.ts new file mode 100644 index 0000000000..ede66ebe4b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeRun.ts @@ -0,0 +1,149 @@ +// Reusable cascade-execution primitives shared by the pipeline route page and +// the local-dev preview (`PipelineDevView`). The backend asset-trigger +// dispatcher only resolves DEPLOYED rows, so whenever a run involves local / +// draft content the client must orchestrate the closure itself: topological +// order over the graph the user is looking at, each node launched with +// `_wmill_skip_asset_dispatch` so the backend never double-fires the deployed +// part of a mixed chain. + +import { JobService, type Preview } from '$lib/gen' +import { + runCascade, + runSelection, + type CascadeRunResult, + type CascadeNodeState +} from './cascadeOrchestrator' +import { computeDownstreamClosure, computeInducedSchedule } from './graphTraversal' +import { buildLineageDownstreamMap } from './boundedCascade' +import type { AssetGraphResponse } from './types' + +export const CASCADE_POLL_INTERVAL_MS = 1000 +export const CASCADE_JOB_TIMEOUT_MS = 30 * 60 * 1000 + +export type LocalScriptContent = { + content: string + language: Preview['language'] + // `// tag ` — routes the preview to that worker (deployed parity). + tag?: string +} + +// Poll a launched cascade job to a terminal state. Capped so a never-terminating +// job can't pin a run guard forever; on timeout it throws, surfaced as a chain +// failure by the orchestrator. +export function makeWaitJobTerminal( + workspace: string +): (jobId: string) => Promise<'success' | 'failure'> { + return async function waitJobTerminal(jobId: string): Promise<'success' | 'failure'> { + const deadline = Date.now() + CASCADE_JOB_TIMEOUT_MS + while (Date.now() < deadline) { + try { + const r = await JobService.getCompletedJobResultMaybe({ + workspace, + id: jobId, + getStarted: false + }) + if (r.completed) return r.success ? 'success' : 'failure' + } catch { + // transient — retry on the next tick + } + await new Promise((res) => setTimeout(res, CASCADE_POLL_INTERVAL_MS)) + } + throw new Error( + `Timed out after ${Math.round(CASCADE_JOB_TIMEOUT_MS / 60000)}min waiting for job ${jobId} to finish` + ) + } +} + +// Build a per-script launch function. When `resolveLocal(path)` yields content, +// the script runs as a preview of that local content (no deploy); otherwise it +// runs the deployed version by path. Always passes `_wmill_skip_asset_dispatch`. +export function makeLaunch(opts: { + workspace: string + resolveLocal?: (path: string) => LocalScriptContent | undefined + tempScriptRefs?: Record + // Extra run args for a specific node (e.g. the uploaded S3Object bound to a + // `data_upload` cascade root). Merged over `_wmill_skip_asset_dispatch`; all + // other nodes run with empty inputs as before. + argsFor?: (path: string) => Record | undefined + onLaunched?: (path: string, jobId: string) => void +}): (path: string) => Promise { + return async function launch(path: string): Promise { + const local = opts.resolveLocal?.(path) + // Caller args (e.g. the run form for the cascade root) must NOT be able to + // re-enable backend asset dispatch while the client orchestrates the closure + // — that would double-run downstream / run deployed subscribers. Drop any + // `_wmill_skip_asset_dispatch` a caller supplied, and always spread it LAST. + const { _wmill_skip_asset_dispatch: _reserved, ...extra } = opts.argsFor?.(path) ?? {} + void _reserved + let jobId: string + if (local) { + if (!local.content || !local.language) { + throw new Error(`local script ${path} has no content/language`) + } + jobId = await JobService.runScriptPreview({ + workspace: opts.workspace, + requestBody: { + content: local.content, + language: local.language, + path, + args: { ...extra, _wmill_skip_asset_dispatch: true }, + ...(local.tag ? { tag: local.tag } : {}), + ...(opts.tempScriptRefs ? { temp_script_refs: opts.tempScriptRefs } : {}) + } + }) + } else { + jobId = await JobService.runScriptByPath({ + workspace: opts.workspace, + path, + requestBody: { ...extra, _wmill_skip_asset_dispatch: true } + }) + } + opts.onLaunched?.(path, jobId) + return jobId + } +} + +// Run `root` plus its full downstream closure over the given graph. +export async function runDownstreamCascade(opts: { + graph: AssetGraphResponse + root: string + launch: (path: string) => Promise + waitTerminal: (jobId: string) => Promise<'success' | 'failure'> + onUpdate?: (statuses: Map) => void +}): Promise { + const closure = computeDownstreamClosure(opts.graph, opts.root) + const res = await runCascade({ + closure, + root: opts.root, + launch: opts.launch, + waitTerminal: opts.waitTerminal, + onUpdate: opts.onUpdate + }) + return { ...res, cyclic: closure.cyclic } +} + +// Run a bounded selection of scripts (the induced schedule over the lineage DAG). +export async function runBoundedCascade(opts: { + graph: AssetGraphResponse + scripts: Set + launch: (path: string) => Promise + waitTerminal: (jobId: string) => Promise<'success' | 'failure'> + onUpdate?: (statuses: Map) => void +}): Promise { + // Read-aware adjacency (NOT the default write-edge map) so a pure-reader + // member runs after its producer — parity with the route page's bounded run + // and the CLI `topoOrder`. `cyclic` is surfaced so callers can warn instead of + // silently dropping scripts stuck on a dependency cycle. + const schedule = computeInducedSchedule( + opts.graph, + opts.scripts, + buildLineageDownstreamMap(opts.graph) + ) + const res = await runSelection({ + schedule, + launch: opts.launch, + waitTerminal: opts.waitTerminal, + onUpdate: opts.onUpdate + }) + return { ...res, cyclic: schedule.cyclic } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts new file mode 100644 index 0000000000..6842dcfd86 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts @@ -0,0 +1,162 @@ +import { describe, expect, it } from 'vitest' +import type { AssetGraphResponse } from './types' +import { + buildColumnGraph, + colNodeId, + traceColumn, + connectedComponent, + assetColumnNodes, + computeDepths +} from './columnLineageGraph' + +// Two scripts chained through an intermediate ducklake table: +// s1: orders.amount -> staging.amt +// s2: staging.amt -> daily.total +// s2: customers.name -> daily.cust (a second source into the sink) +function chainGraph(): AssetGraphResponse { + return { + assets: [], + triggers: [], + runnables: [ + { + path: 's1', + usage_kind: 'script', + column_lineage: [ + { + column: 'amt', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/orders', from_column: 'amount' }] + } + ] + }, + { + path: 's2', + usage_kind: 'script', + column_lineage: [ + { + column: 'total', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/staging', from_column: 'amt' }] + }, + { + column: 'cust', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/customers', from_column: 'name' }] + } + ] + } + ], + edges: [ + { + runnable_path: 's1', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/staging', + access_type: 'w' + }, + { + runnable_path: 's2', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/daily', + access_type: 'w' + } + ] + } +} + +const ORDERS_AMOUNT = colNodeId('ducklake', 'wh/orders', 'amount') +const STAGING_AMT = colNodeId('ducklake', 'wh/staging', 'amt') +const DAILY_TOTAL = colNodeId('ducklake', 'wh/daily', 'total') +const DAILY_CUST = colNodeId('ducklake', 'wh/daily', 'cust') +const CUSTOMERS_NAME = colNodeId('ducklake', 'wh/customers', 'name') + +describe('buildColumnGraph', () => { + it('stitches per-script lineage into a transitive graph via shared columns', () => { + const g = buildColumnGraph(chainGraph()) + // orders.amount feeds staging.amt feeds daily.total + expect(g.up.get(STAGING_AMT)).toEqual(new Set([ORDERS_AMOUNT])) + expect(g.up.get(DAILY_TOTAL)).toEqual(new Set([STAGING_AMT])) + expect(g.down.get(ORDERS_AMOUNT)).toEqual(new Set([STAGING_AMT])) + expect(g.down.get(STAGING_AMT)).toEqual(new Set([DAILY_TOTAL])) + }) + + it('anchors to the // materialize target, not a guessed write-edge', () => { + const graph = chainGraph() + // s1 declares its materialize target and also has unordered extra + // ducklake writes; the lineage must anchor to the declared target. + const s1 = graph.runnables.find((r) => r.path === 's1')! + s1.materialize_target = { kind: 'ducklake', path: 'wh/staging' } + graph.edges.unshift({ + runnable_path: 's1', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/other', + access_type: 'w' + }) + const g = buildColumnGraph(graph) + expect(g.nodes.has(STAGING_AMT)).toBe(true) // anchored to the declared target + expect(g.nodes.has(colNodeId('ducklake', 'wh/other', 'amt'))).toBe(false) + }) + + it('falls back to a ducklake write-edge when there is no materialize target', () => { + // chainGraph's runnables carry no materialize_target, so s1's lineage is + // anchored via its (single) ducklake write-edge. + const g = buildColumnGraph(chainGraph()) + expect(g.nodes.has(STAGING_AMT)).toBe(true) + }) + + it('node ids are collision-proof across `#` / `:` in paths and columns', () => { + // A delimiter-concatenated id would merge these; the JSON-encoded id must not. + expect(colNodeId('ducklake', 'a#b', 'c')).not.toBe(colNodeId('ducklake', 'a', 'b#c')) + expect(colNodeId('ducklake', 'a:b', 'c')).not.toBe(colNodeId('ducklake', 'a', 'b:c')) + }) + + it('skips producers with no ducklake output asset (columns unanchorable)', () => { + const graph = chainGraph() + graph.edges = graph.edges.filter((e) => e.runnable_path !== 's1') // s1 loses its output edge + const g = buildColumnGraph(graph) + // staging.amt is no longer produced as a node by s1... + expect(g.up.has(STAGING_AMT)).toBe(false) + // ...but s2 still anchors daily.total ← staging.amt (staging.amt as a source). + expect(g.up.get(DAILY_TOTAL)).toEqual(new Set([STAGING_AMT])) + }) +}) + +describe('traceColumn', () => { + it('returns the full upstream + downstream impact set of a source column', () => { + const g = buildColumnGraph(chainGraph()) + // from the root source, the whole chain downstream is impacted + expect(traceColumn(ORDERS_AMOUNT, g)).toEqual( + new Set([ORDERS_AMOUNT, STAGING_AMT, DAILY_TOTAL]) + ) + }) + + it('traces backward from a sink to every contributing source', () => { + const g = buildColumnGraph(chainGraph()) + expect(traceColumn(DAILY_TOTAL, g)).toEqual(new Set([DAILY_TOTAL, STAGING_AMT, ORDERS_AMOUNT])) + // the sibling output `cust` and its source are NOT in total's trace + expect(traceColumn(DAILY_TOTAL, g).has(DAILY_CUST)).toBe(false) + expect(traceColumn(DAILY_TOTAL, g).has(CUSTOMERS_NAME)).toBe(false) + }) + + it('traces an intermediate column both directions', () => { + const g = buildColumnGraph(chainGraph()) + expect(traceColumn(STAGING_AMT, g)).toEqual(new Set([STAGING_AMT, ORDERS_AMOUNT, DAILY_TOTAL])) + }) +}) + +describe('connectedComponent + depths', () => { + it('collects the neighborhood of an asset and lays it out by hop depth', () => { + const g = buildColumnGraph(chainGraph()) + const seeds = assetColumnNodes(g, 'ducklake', 'wh/daily') // the sink asset + expect(new Set(seeds)).toEqual(new Set([DAILY_TOTAL, DAILY_CUST])) + const comp = connectedComponent(seeds, g) + expect(comp).toEqual( + new Set([DAILY_TOTAL, DAILY_CUST, STAGING_AMT, ORDERS_AMOUNT, CUSTOMERS_NAME]) + ) + const depths = computeDepths(comp, g) + expect(depths.get(ORDERS_AMOUNT)).toBe(0) + expect(depths.get(STAGING_AMT)).toBe(1) + expect(depths.get(DAILY_TOTAL)).toBe(2) + expect(depths.get(CUSTOMERS_NAME)).toBe(0) + expect(depths.get(DAILY_CUST)).toBe(1) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts new file mode 100644 index 0000000000..50fa1c4e7b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts @@ -0,0 +1,169 @@ +import type { AssetKind } from '$lib/gen' +import type { AssetGraphResponse } from './types' + +// A node in the column-level lineage graph: one column of one asset. +export type ColumnNode = { kind: AssetKind; path: string; column: string } +export type ColumnNodeId = string + +// Collision-proof node id — JSON-encoded tuple, so a `#`/`:` inside a path or +// (quoted) column name can't merge two distinct columns into one node. +export function colNodeId(kind: AssetKind, path: string, column: string): ColumnNodeId { + return JSON.stringify([kind, path, column]) +} + +// The pipeline-wide column-lineage graph, stitched across every producer. Each +// producer's `column_lineage` contributes single-hop edges (its output column ← +// its source columns); shared (asset,column) nodes chain those hops into the +// full transitive graph (`orders.amount → staging.amt → daily.total`). +export type ColumnLineageGraph = { + nodes: Map + // outputColumn → the source columns it derives from (walk upstream). + up: Map> + // sourceColumn → the output columns derived from it (walk downstream). + down: Map> +} + +// Build the column graph from a resolved asset graph. A producer's +// `column_lineage` describes the columns of the asset it materializes; that +// output asset is the ducklake target it writes (v1 materialize target), found +// from its write-edge. Producers without a known ducklake output are skipped +// (their columns can't be anchored to an asset node). +export function buildColumnGraph(graph: AssetGraphResponse): ColumnLineageGraph { + const nodes = new Map() + const up = new Map>() + const down = new Map>() + + const addNode = (n: ColumnNode): ColumnNodeId => { + const id = colNodeId(n.kind, n.path, n.column) + if (!nodes.has(id)) nodes.set(id, n) + return id + } + const addEdge = (src: ColumnNodeId, out: ColumnNodeId) => { + if (src === out) return + ;(up.get(out) ?? up.set(out, new Set()).get(out)!).add(src) + ;(down.get(src) ?? down.set(src, new Set()).get(src)!).add(out) + } + + // The output asset a runnable's `column_lineage` describes. The declared + // `// materialize` target is authoritative (a multi-output script writes + // several ducklake tables, and the deployed write-edges are unordered, so + // picking "a" write-edge can anchor to the wrong asset). Fall back to a + // ducklake write-edge only for producers with no materialize annotation + // (e.g. a literal single-output CTAS). + const outputAsset = new Map() + for (const r of graph.runnables ?? []) { + if (r.materialize_target) { + outputAsset.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } + } + for (const e of graph.edges ?? []) { + const access = e.access_type ?? 'r' + const key = `${e.runnable_kind}:${e.runnable_path}` + if ( + (access === 'w' || access === 'rw') && + e.asset_kind === 'ducklake' && + !outputAsset.has(key) + ) { + outputAsset.set(key, { kind: e.asset_kind, path: e.asset_path }) + } + } + + for (const r of graph.runnables ?? []) { + const lineage = r.column_lineage + if (!lineage || lineage.length === 0) continue + const out = outputAsset.get(`${r.usage_kind}:${r.path}`) + if (!out) continue + for (const cl of lineage) { + const outId = addNode({ kind: out.kind, path: out.path, column: cl.column }) + for (const inp of cl.inputs) { + const srcId = addNode({ + kind: inp.from_kind, + path: inp.from_path, + column: inp.from_column + }) + addEdge(srcId, outId) + } + } + } + + return { nodes, up, down } +} + +// Every node reachable from `start` by following `adj` (transitive closure, +// excluding `start` itself). Iterative to avoid deep-recursion limits. +function reach(start: ColumnNodeId, adj: Map>): Set { + const seen = new Set() + const stack = [start] + while (stack.length) { + const n = stack.pop()! + for (const m of adj.get(n) ?? []) { + if (!seen.has(m)) { + seen.add(m) + stack.push(m) + } + } + } + return seen +} + +// The full transitive trace of a column: itself + all upstream ancestors + all +// downstream descendants. This is the impact set — "everything that feeds, or +// is fed by, this column". +export function traceColumn(id: ColumnNodeId, g: ColumnLineageGraph): Set { + const out = new Set([id]) + for (const a of reach(id, g.up)) out.add(a) + for (const d of reach(id, g.down)) out.add(d) + return out +} + +// The connected neighborhood of a set of seed columns (an asset's columns): +// the seeds plus everything upstream and downstream of any of them. This is the +// subgraph the trace view renders around a selected asset. +export function connectedComponent( + seeds: ColumnNodeId[], + g: ColumnLineageGraph +): Set { + const out = new Set() + for (const s of seeds) { + if (!g.nodes.has(s)) continue + out.add(s) + for (const a of reach(s, g.up)) out.add(a) + for (const d of reach(s, g.down)) out.add(d) + } + return out +} + +// All column-node ids belonging to one asset (its seed set for a trace). +export function assetColumnNodes( + g: ColumnLineageGraph, + kind: AssetKind, + path: string +): ColumnNodeId[] { + const ids: ColumnNodeId[] = [] + for (const [id, n] of g.nodes) if (n.kind === kind && n.path === path) ids.push(id) + return ids +} + +// Longest-path depth of each node within `ids`, sources at depth 0 and depth +// increasing downstream — so a left→right layout reads upstream→downstream. +// Cycle-guarded (lineage is a DAG, but be defensive). +export function computeDepths( + ids: Set, + g: ColumnLineageGraph +): Map { + const depth = new Map() + const visiting = new Set() + const d = (id: ColumnNodeId): number => { + const memo = depth.get(id) + if (memo !== undefined) return memo + if (visiting.has(id)) return 0 + visiting.add(id) + let m = 0 + for (const u of g.up.get(id) ?? []) if (ids.has(u)) m = Math.max(m, d(u) + 1) + visiting.delete(id) + depth.set(id, m) + return m + } + for (const id of ids) d(id) + return depth +} diff --git a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts index 08da865d61..197d7eca6c 100644 --- a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts @@ -1,6 +1,10 @@ import { describe, expect, it } from 'vitest' import type { AssetGraphResponse } from './types' -import { buildDownstreamMap, computeDownstreamClosure } from './graphTraversal' +import { + buildDownstreamMap, + computeDownstreamClosure, + computeInducedSchedule +} from './graphTraversal' /** Direct (one-hop) subscriber script paths of `scriptPath`. */ function downstreamSubscribers(g: AssetGraphResponse, scriptPath: string): string[] { @@ -170,3 +174,54 @@ describe('computeDownstreamClosure', () => { expect(cl.cyclic).toEqual([]) }) }) + +describe('computeInducedSchedule', () => { + // a → b → c → d linear chain through assets. + const chain = () => + graph( + [ + ['a', 'xa'], + ['b', 'xb'], + ['c', 'xc'] + ], + [ + ['b', 'xa'], + ['c', 'xb'], + ['d', 'xc'] + ] + ) + + it('orders a subset topologically and keeps only in-set edges', () => { + const sched = computeInducedSchedule(chain(), new Set(['a', 'b', 'c'])) + expect(sched.nodes).toEqual(['a', 'b', 'c']) + expect(sched.roots).toEqual(['a']) + expect(sched.indegree.get('b')).toBe(1) + expect([...(sched.edges.get('c') ?? [])]).toEqual([]) // d is out of set + expect(sched.cyclic).toEqual([]) + }) + + it('a gap in the selected set produces two independent roots', () => { + // pick a and c (skip b): with b excluded there is no in-set edge, so both + // are roots. + const sched = computeInducedSchedule(chain(), new Set(['a', 'c'])) + expect(sched.roots.sort()).toEqual(['a', 'c']) + expect(sched.indegree.get('c')).toBe(0) + }) + + it('reports cyclic members instead of hanging', () => { + // b ↔ c cycle. + const g = graph( + [ + ['b', 'y'], + ['c', 'z'] + ], + [ + ['c', 'y'], + ['b', 'z'] + ] + ) + const sched = computeInducedSchedule(g, new Set(['b', 'c'])) + expect(sched.nodes).toEqual([]) + expect(sched.cyclic.sort()).toEqual(['b', 'c']) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts index 6b53020ddb..cc59e8f3a8 100644 --- a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts +++ b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts @@ -1,6 +1,31 @@ -import type { AssetGraphResponse } from './types' +import type { AssetGraphResponse, AssetGraphSelection } from './types' import { assetKey, buildAssetSubscribers, isWriteEdge } from './lib' +// Scripts that WRITE the selected asset (its producers), from the graph's +// `w`/`rw` lineage edges. `[]` for a non-asset selection. Shared by the route +// page and the dev preview so "who writes this asset" is defined once and can't +// drift between the two surfaces. +export function assetProducers( + graph: AssetGraphResponse, + selection: AssetGraphSelection | undefined +): Array<{ kind: 'script' | 'flow'; path: string; unsaved?: boolean }> { + if (!selection || selection.kind !== 'asset') return [] + return graph.edges + .filter((e) => { + const access = e.access_type ?? 'r' + return ( + (access === 'w' || access === 'rw') && + e.asset_kind === selection.asset_kind && + e.asset_path === selection.path + ) + }) + .map((e) => ({ + kind: e.runnable_kind as 'script' | 'flow', + path: e.runnable_path, + unsaved: e.unsaved + })) +} + // Execution-DAG traversal over the resolved asset graph (drafts included). // // The execution edges are producer→subscriber: a script S1 *produces* an @@ -117,3 +142,87 @@ export function computeDownstreamClosure(g: AssetGraphResponse, root: string): D } return { nodes: ordered, edges: cleanEdges, indegree: cleanIndegree, cyclic } } + +export type InducedSchedule = { + /** Selected scripts that are schedulable, in a topological order. */ + nodes: string[] + /** In-set dependency edges: `edges.get(p)` = in-set subscribers of `p`. */ + edges: Map> + /** In-set upstream count per node. */ + indegree: Map + /** Schedulable scripts with no in-set upstream — the schedule's seeds. */ + roots: string[] + /** + * Selected scripts on (or fed only through) a cycle. Excluded from + * `nodes`/`edges`/`indegree`/`roots`; surfaced so callers can warn. + */ + cyclic: string[] +} + +/** + * Topological schedule of an arbitrary *set* of scripts (e.g. a bounded-cascade + * selection), respecting only the dependency edges that fall *inside* the set. + * Multi-root: every selected script with no in-set upstream is a seed. + * Cycle-safe in the same way as `computeDownstreamClosure`. + * + * `oneHop` is the producer→consumer adjacency (script path → downstream script + * paths). It defaults to the `// on` subscriber map — but a bounded run passes + * a *read-aware* map (boundedCascade.buildLineageDownstreamMap) so a selected + * script that merely *reads* an upstream asset still runs after its producer, + * matching the CLI's `topoOrder`. Keep the default subscriber-only: it mirrors + * the production asset-trigger dispatch the unbounded cascade simulates. + */ +export function computeInducedSchedule( + g: AssetGraphResponse, + selected: Set, + oneHop: Map> = buildDownstreamMap(g) +): InducedSchedule { + // In-set edges + indegrees. + const edges = new Map>() + const indegree = new Map() + for (const n of selected) indegree.set(n, 0) + for (const p of selected) { + const subs = new Set() + for (const s of oneHop.get(p) ?? []) { + if (!selected.has(s)) continue + subs.add(s) + indegree.set(s, (indegree.get(s) ?? 0) + 1) + } + if (subs.size > 0) edges.set(p, subs) + } + + // Kahn from every indegree-0 node. + const seeds = [...selected].filter((n) => (indegree.get(n) ?? 0) === 0) + const remaining = new Map(indegree) + const ready = [...seeds] + const ordered: string[] = [] + while (ready.length > 0) { + const n = ready.shift()! + ordered.push(n) + for (const s of edges.get(n) ?? []) { + const d = (remaining.get(s) ?? 0) - 1 + remaining.set(s, d) + if (d === 0) ready.push(s) + } + } + + const orderedSet = new Set(ordered) + const cyclic = [...selected].filter((n) => !orderedSet.has(n)) + if (cyclic.length === 0) { + return { nodes: ordered, edges, indegree, roots: seeds, cyclic } + } + + // Strip cyclic nodes from the schedulable structures. + const cyclicSet = new Set(cyclic) + const cleanEdges = new Map>() + const cleanIndegree = new Map() + for (const n of ordered) cleanIndegree.set(n, 0) + for (const [p, subs] of edges) { + if (cyclicSet.has(p)) continue + const kept = new Set([...subs].filter((s) => !cyclicSet.has(s))) + if (kept.size > 0) cleanEdges.set(p, kept) + for (const s of kept) cleanIndegree.set(s, (cleanIndegree.get(s) ?? 0) + 1) + } + const roots = ordered.filter((n) => (cleanIndegree.get(n) ?? 0) === 0) + return { nodes: ordered, edges: cleanEdges, indegree: cleanIndegree, roots, cyclic } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts index dfb5385253..82047e9528 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts @@ -17,7 +17,14 @@ const ASSERTED_TS_FIELDS: Record = { partition: true, freshness: true, tag: true, - retry: true + retry: true, + materialize: true, + dataTests: true, + columnLineage: true, + macros: true, + useLibs: true, + muteAssets: true, + muteAll: true } // Parser-parity guard: this TS parser (drives the live graph preview) and @@ -58,6 +65,34 @@ type Fixture = { freshness: string | null tag: string | null retry: { count: number; delay: string | null } | null + materialize?: { + target_kind: string + target_path: string + manual?: boolean + append?: boolean + unique_key?: string | null + scd2?: boolean + track?: string[] + close_deleted?: boolean + // "warn" | "ignore"; absent === "warn" (the default) + on_schema_change?: string + } | null + // Snake_case form matching the Rust `DataTest` serde output, so the one + // corpus drives both sides. The TS parser emits this shape verbatim + // (snake_case fields), so the comparison is 1:1. Absent === []. + data_tests?: Array> + // Snake_case `ColumnLineage` serde shape — TS parser emits it verbatim, + // so the comparison is 1:1. Absent === []. + column_lineage?: Array> + // `// macros` marker. Absent === false. + macros?: boolean + // `// use ` accumulation, declaration order, deduped. Absent === []. + use_libs?: string[] + // `// mute ` accumulation as `kind:path`, declaration order, deduped. + // Absent === []. + mute?: string[] + // `// mute all` marker. Absent === false. + mute_all?: boolean } } @@ -126,6 +161,53 @@ describe('parsePipelineAnnotations matches the shared Rust fixture corpus', () = expect(got.retry?.count, 'retry count').toBe(f.expected.retry.count) expect(got.retry?.delay, 'retry delay').toEqual(f.expected.retry.delay ?? undefined) } + + if (f.expected.materialize == null) { + expect(got.materialize, 'materialize').toBeUndefined() + } else { + expect(got.materialize?.targetKind, 'materialize target kind').toBe( + f.expected.materialize.target_kind + ) + expect(got.materialize?.targetPath, 'materialize target path').toBe( + f.expected.materialize.target_path + ) + expect(got.materialize?.manual ?? false, 'materialize manual').toBe( + f.expected.materialize.manual ?? false + ) + expect(got.materialize?.append ?? false, 'materialize append').toBe( + f.expected.materialize.append ?? false + ) + expect(got.materialize?.uniqueKey, 'materialize key').toEqual( + f.expected.materialize.unique_key ?? undefined + ) + expect(got.materialize?.scd2 ?? false, 'materialize scd2').toBe( + f.expected.materialize.scd2 ?? false + ) + expect(got.materialize?.track ?? [], 'materialize track').toEqual( + f.expected.materialize.track ?? [] + ) + expect(got.materialize?.closeDeleted ?? false, 'materialize close_deleted').toBe( + f.expected.materialize.close_deleted ?? false + ) + expect(got.materialize?.onSchemaChange ?? 'warn', 'materialize on_schema_change').toBe( + f.expected.materialize.on_schema_change ?? 'warn' + ) + } + + expect(got.dataTests, 'data tests').toEqual(f.expected.data_tests ?? []) + + expect(got.columnLineage, 'column lineage').toEqual(f.expected.column_lineage ?? []) + + expect(got.macros, 'macros').toBe(f.expected.macros ?? false) + + expect(got.useLibs, 'use_libs').toEqual(f.expected.use_libs ?? []) + + expect( + got.muteAssets.map((a) => `${a.kind}:${a.path}`), + 'mute' + ).toEqual(f.expected.mute ?? []) + + expect(got.muteAll, 'mute_all').toBe(f.expected.mute_all ?? false) }) } }) diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts index f73819b959..51aaf97f52 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts @@ -1,5 +1,10 @@ import { describe, expect, it } from 'vitest' -import { parsePipelineAnnotations } from './parsePipelineAnnotations' +import { + mergeColumnLineage, + parseDurationSecs, + parsePipelineAnnotations, + type ColumnLineage +} from './parsePipelineAnnotations' // Unit-test the TS mirror of the backend `parse_pipeline_annotations`. The // two implementations MUST stay behaviorally identical — these tests are @@ -25,6 +30,45 @@ describe('parsePipelineAnnotations: tag', () => { const out = parsePipelineAnnotations('// tagged heavy') expect(out.tag).toBeUndefined() }) + + it('skips a tag value containing whitespace (regular comment false-positive)', () => { + const out = parsePipelineAnnotations('# tag this function so we remember to refactor it later') + expect(out.tag).toBeUndefined() + }) + + it('skips a tag value longer than 50 chars', () => { + const out = parsePipelineAnnotations('// tag ' + 'x'.repeat(51)) + expect(out.tag).toBeUndefined() + }) +}) + +describe('parsePipelineAnnotations: header scan', () => { + it('ignores annotations in the body once code has started', () => { + const code = [ + 'import pandas as pd', + '', + 'def main():', + ' # tag each row with its source so downstream steps can filter', + ' # on s3://should/not/parse', + ' return pd.DataFrame()' + ].join('\n') + const out = parsePipelineAnnotations(code) + expect(out.tag).toBeUndefined() + expect(out.triggerAssets).toHaveLength(0) + }) + + it('tolerates blank lines before code but stops at the first code line', () => { + const code = ['#!/usr/bin/env python', '', '# tag heavy', 'import os', '# tag light'].join('\n') + const out = parsePipelineAnnotations(code) + expect(out.tag).toBe('heavy') + }) + + it('strips trailing key=value opts from a // on asset ref', () => { + // The path must be the bare asset, not `main.orders debounce=60s`, so it + // dedups against body inference and matches the deploy path. + const out = parsePipelineAnnotations('// on ducklake://main.orders debounce=60s') + expect(out.triggerAssets).toEqual([{ kind: 'ducklake', path: 'main.orders' }]) + }) }) describe('parsePipelineAnnotations: retry', () => { @@ -59,6 +103,33 @@ describe('parsePipelineAnnotations: retry', () => { }) }) +describe('parsePipelineAnnotations: macros + use', () => { + it('parses the bare macros marker', () => { + const out = parsePipelineAnnotations('// macros\nCREATE MACRO m(a) AS a;') + expect(out.macros).toBe(true) + }) + + it('macros marker is strict — trailing prose and variants rejected', () => { + expect(parsePipelineAnnotations('// macros are defined below\n').macros).toBe(false) + expect(parsePipelineAnnotations('// macros_v2\n').macros).toBe(false) + expect(parsePipelineAnnotations('-- macros \nSELECT 1;').macros).toBe(true) + }) + + it('use accumulates in order and dedups', () => { + const out = parsePipelineAnnotations( + '// use f/lib/stats\n// use f/lib/dates\n// use f/lib/stats\n' + ) + expect(out.useLibs).toEqual(['f/lib/stats', 'f/lib/dates']) + }) + + it('use rejects prose, slashless and multi-token values', () => { + const out = parsePipelineAnnotations( + '// use this script to compute\n// use standalone\n// use f/lib/ok extra\n' + ) + expect(out.useLibs).toEqual([]) + }) +}) + describe('parsePipelineAnnotations: combined', () => { it('parses all keywords together', () => { const code = [ @@ -99,3 +170,66 @@ describe('parsePipelineAnnotations: data_upload', () => { expect(out.nativeTriggers).toEqual([]) }) }) + +describe('mergeColumnLineage', () => { + const ref = (path: string, col: string): ColumnLineage['inputs'][number] => ({ + from_kind: 'ducklake', + from_path: path, + from_column: col + }) + + it('annotation wins per output column; inferred fills the rest (mirrors Rust)', () => { + const inferred: ColumnLineage[] = [ + { column: 'total', inputs: [ref('w/o', 'amount')] }, + { column: 'qty', inputs: [ref('w/o', 'qty')] } + ] + const annotated: ColumnLineage[] = [{ column: 'total', inputs: [ref('w/manual', 'grand')] }] + const merged = mergeColumnLineage(inferred, annotated) + expect(merged).toEqual([ + { column: 'total', inputs: [ref('w/manual', 'grand')] }, // annotation, first + authoritative + { column: 'qty', inputs: [ref('w/o', 'qty')] } // inferred, not overridden + ]) + }) + + it('returns annotations unchanged when there is no inferred lineage', () => { + const annotated: ColumnLineage[] = [{ column: 'a', inputs: [ref('w/o', 'a')] }] + expect(mergeColumnLineage([], annotated)).toEqual(annotated) + }) +}) + +// Mirror of the Rust `parse_duration_secs` tests (windmill-common assets.rs) +// — the freshness chip's staleness verdict depends on identical parsing. +describe('parseDurationSecs', () => { + it('parses suffixed durations', () => { + expect(parseDurationSecs('30s')).toBe(30) + expect(parseDurationSecs('5m')).toBe(300) + expect(parseDurationSecs('2h')).toBe(7200) + expect(parseDurationSecs('1d')).toBe(86400) + }) + + it('bare integer means seconds', () => { + expect(parseDurationSecs('45')).toBe(45) + }) + + it('tolerates surrounding whitespace', () => { + expect(parseDurationSecs(' 5 m ')).toBe(300) + }) + + it('accepts an explicit plus sign (Rust i64 parsing does)', () => { + expect(parseDurationSecs('+5m')).toBe(300) + expect(parseDurationSecs('+45')).toBe(45) + }) + + it('rejects malformed / non-positive input', () => { + expect(parseDurationSecs('')).toBeUndefined() + expect(parseDurationSecs('h')).toBeUndefined() + expect(parseDurationSecs('1.5h')).toBeUndefined() + expect(parseDurationSecs('-5m')).toBeUndefined() + expect(parseDurationSecs('0')).toBeUndefined() + expect(parseDurationSecs('fast')).toBeUndefined() + }) + + it('rejects values beyond i32 seconds (mirrors backend cap)', () => { + expect(parseDurationSecs('999999999d')).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts index 88e30def4e..f9e53494ff 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts @@ -67,6 +67,28 @@ export type FreshnessSpec = { duration: string } +// Mirrors backend `parse_duration_secs` (windmill-common assets.rs): a bare +// integer means seconds, otherwise `` with an `s`/`m`/`h`/`d` suffix +// (e.g. `30s`, `5m`, `2h`, `1d`). Returns undefined for malformed or +// non-positive input so a typo'd `// freshness` window fails safe (the chip +// stays neutral instead of guessing a staleness verdict). +export function parseDurationSecs(s: string): number | undefined { + const t = s.trim() + if (!t) return undefined + const last = t[t.length - 1] + const mult = + last === 's' ? 1 : last === 'm' ? 60 : last === 'h' ? 3600 : last === 'd' ? 86400 : undefined + const num = (mult !== undefined ? t.slice(0, -1) : t).trim() + // `+?`: Rust's i64 parsing accepts an explicit plus sign (`+5m`), so the + // mirror must too — divergence here would leave the chip neutral for a + // window the deploy path and watchdog honor. + if (mult === undefined && !/^\+?\d+$/.test(t)) return undefined + if (!/^\+?\d+$/.test(num)) return undefined + const secs = Number(num) * (mult ?? 1) + if (!Number.isSafeInteger(secs) || secs <= 0 || secs > 2147483647) return undefined + return secs +} + // `// retry []` — see backend RetrySpec. Delay is kept as the // raw duration string and resolved to seconds at deploy. export type RetrySpec = { @@ -74,6 +96,105 @@ export type RetrySpec = { delay?: string } +// `// materialize [manual] [append] [key=]` — see backend +// MaterializeSpec. Managed by default (the runtime generates the write DDL +// around a single SELECT); `manual` opts out (the script writes its own DDL, +// track-only). `append` / `key` / `history` / `track` are managed-mode strategy +// options; `key= history` (or the `scd2` alias) selects SCD type-2 history, +// and `deletes=close` opts scd2 into hard-delete-close. +export type MaterializeSpec = { + targetKind: AssetKind + targetPath: string + // track-only escape hatch; absent === false (managed) + manual?: boolean + // INSERT-only strategy; absent === false + append?: boolean + // merge key; absent === replace (or append). Also the SCD2 natural key. + uniqueKey?: string + // SCD2 managed history mode (valid_from/valid_to/is_current); absent === false + scd2?: boolean + // SCD2 tracked columns (change ⇒ new version); empty ⇒ all non-key columns + track?: string[] + // SCD2 hard-delete-close (`deletes=close`): close absent keys; absent === false + closeDeleted?: boolean + // `on_schema_change=ignore` opts the produced asset out of downstream + // schema-contract warnings (save-time metadata only). Default `warn`; + // `fail` is deliberately unrecognized in v1 (saves never hard-block). + onSchemaChange?: 'warn' | 'ignore' +} + +// The `_current` SCD2 companion view this managed materialize also +// produces, or `undefined` when it isn't a managed scd2 target. Mirrors Rust +// `MaterializeSpec::scd2_current_target`: managed scd2 creates the base table +// *and* the `_current` view each run; `manual` mode owns its own DDL and creates +// no companion. The graph surfaces register it as a second write of the producer +// so a read of the view links back instead of orphaning. +export function scd2CurrentTargetPath(m: MaterializeSpec): string | undefined { + return m.scd2 && !m.manual ? `${m.targetPath}_current` : undefined +} + +// `// data_test …` — a data-quality assertion run against the +// freshly-materialized asset, failing the run on violation. See backend +// `DataTest`. The first extensible annotation family: a keyword head selects +// the variant, the rest is parsed per-variant; a sibling family (e.g. +// column-lineage) follows the same shape. Built-ins mirror dbt's generic data +// tests; `custom` is dbt's singular-test escape hatch (a DuckDB script path). +// Keyword is `data_test`, NOT `test`, to stay clear of the unrelated `// test:` +// CI-test annotation. +// Field names are snake_case to match the Rust `DataTest` serde output verbatim +// — the same type is populated both by this parser (live drafts) and by the +// backend graph endpoint (deployed nodes), so the two must be wire-identical. +export type DataTest = + | { type: 'unique'; column: string } + | { type: 'not_null'; column: string } + | { type: 'accepted_values'; column: string; values: string[] } + | { + type: 'relationships' + column: string + to_kind: AssetKind + to_path: string + to_column: string + } + | { type: 'custom'; path: string } + +// `// column <- .[, …]` — declared column-level +// lineage: one output column and the upstream source columns it derives from. +// See backend `ColumnLineage`. A sibling of `DataTest` in the extensible +// annotation family — same parse shape, accumulating (one line per output +// column) — but pure metadata: drives the column-lineage graph view, runs no +// probe. Field names are snake_case to match the Rust `ColumnLineage` serde +// output verbatim, so the live-draft parse and the backend graph endpoint +// (deployed nodes) produce wire-identical shapes. +export type ColumnRef = { + from_kind: AssetKind + from_path: string + from_column: string +} +export type ColumnLineage = { + column: string + inputs: ColumnRef[] +} + +// Combine body-inferred column lineage with `// column` annotations, the +// annotation winning per output column. Mirrors the Rust `merge_column_lineage` +// (`asset_parser.rs`) so the live-draft preview matches what deploys: the +// backend already merges inferred + annotated server-side, and the live graph +// must apply the same precedence to the WASM-inferred lineage. +export function mergeColumnLineage( + inferred: ColumnLineage[], + annotated: ColumnLineage[] +): ColumnLineage[] { + const seen = new Set(annotated.map((c) => c.column)) + const out = [...annotated] + for (const c of inferred) { + if (!seen.has(c.column)) { + seen.add(c.column) + out.push(c) + } + } + return out +} + export type PipelineAnnotations = { inPipeline: boolean triggerAssets: PipelineTriggerAsset[] @@ -82,6 +203,26 @@ export type PipelineAnnotations = { freshness?: FreshnessSpec tag?: string retry?: RetrySpec + // `// materialize [manual] [append] [key=]` — target + strategy. + materialize?: MaterializeSpec + // `// data_test …` — accumulating data-quality checks (multiple lines). + dataTests: DataTest[] + // `// column <- .[, …]` — accumulating column lineage. + columnLineage: ColumnLineage[] + // Bare `// macros` (alone on the line, like `// pipeline`) — marks this + // DuckDB script as a workspace macro library. + macros: boolean + // `// use ` — force-inject the named macro library into + // this script's jobs. Accumulating, declaration order, deduped. + useLibs: string[] + // `// mute ` — suppress the auto-derived cascade edge for a read + // that would otherwise trigger this script (a lookup / SCD input). Only + // asset refs; native trigger kinds are never auto-derived. Accumulating, + // deduped. + muteAssets: PipelineTriggerAsset[] + // `// mute all` — opt out of auto-derivation entirely (explicit `// on` + // edges are unaffected). + muteAll: boolean } // Tokenize a `key=value [key="quoted value"] ...` option string. Bare @@ -121,15 +262,224 @@ function parseKvOpts(s: string): Map { return out } +// Drop a `// on` right-hand side's trailing `key=value` opts (e.g. +// `debounce=60s`), returning just the trigger ref. The opts start at the first +// whitespace-delimited token shaped like `=…`; everything before is the +// ref. Mirrors Rust `split_trailing_kv_opts` — asset refs never contain a space +// followed by an `ident=` token, so without this `// on ducklake://t debounce=1s` +// would keep the path as `t debounce=1s` and desync the live canvas from deploy. +function stripTrailingKvOpts(s: string): string { + const m = s.match(/\s([A-Za-z_][A-Za-z0-9_]*)=/) + return (m?.index !== undefined ? s.slice(0, m.index) : s).trimEnd() +} + function parseAssetSyntax(s: string): PipelineTriggerAsset | undefined { for (const [prefix, kind] of ASSET_PREFIXES) { if (s.startsWith(prefix)) { - return { kind, path: s.slice(prefix.length) } + let path = s.slice(prefix.length) + // Mirror the Rust `parse_asset_syntax` S3 canonicalization: strip all + // leading slashes so the SDK object form (`s3:///key`, default + // storage) and DuckDB / `// on s3://key` share one asset path, and a + // canonical key never starts with `/` (so ref reconstruction + // round-trips). Without this the live graph preview would show + // disconnected `/key` and `key` nodes. S3-only; leading slashes only, + // so Hive-partition keys are untouched. + if (kind === 's3object') { + path = path.replace(/^\/+/, '') + } + return { kind, path } } } return undefined } +// Mirror of Rust `parse_asset_syntax(s, enable_default_syntax=true)`: the bare +// words `ducklake` / `datatable` are shorthand for their `…://main` form. Used +// by `// materialize` (but NOT by `// on`, which is default-syntax off, so the +// trigger parser keeps using `parseAssetSyntax`). +function parseAssetSyntaxDefault(s: string): PipelineTriggerAsset | undefined { + if (s === 'datatable') return { kind: 'datatable', path: 'main' } + if (s === 'ducklake') return { kind: 'ducklake', path: 'main' } + return parseAssetSyntax(s) +} + +// Parse a `// materialize [manual] [append] [key=] [history] +// [track=]` right-hand side. Optional leading `manual` word opts out of +// managed mode; a leading `scd2` word is an alias for the `history` flag; the +// next token is the target asset URI (default-syntax shorthands enabled); the +// remainder are strategy options (`append` flag, `key=`, `history` flag, +// `track=`, `deletes=close`, `on_schema_change=ignore`). Missing/empty +// target → undefined (dropped). +function parseMaterializeSpec(s: string): MaterializeSpec | undefined { + // One optional leading mode keyword: `manual` (track-only) or `scd2` (an alias + // for the `history` flag below). + let manual = false + let scd2Kw = false + let rest = s + const afterManual = consumeKeyword(s, 'manual') + const afterScd2 = afterManual === undefined ? consumeKeyword(s, 'scd2') : undefined + if (afterManual !== undefined) { + manual = true + rest = afterManual.trimStart() + } else if (afterScd2 !== undefined) { + scd2Kw = true + rest = afterScd2.trimStart() + } + rest = rest.trim() + const m = rest.match(/^(\S+)(?:\s+(.*))?$/) + if (!m) return undefined + const asset = parseAssetSyntaxDefault(m[1]) + if (!asset || asset.path === '') return undefined + const optsStr = m[2] ?? '' + const optTokens = optsStr.split(/\s+/) + const append = optTokens.some((t) => t === 'append') + // SCD type-2 history: primary spelling is the bare `history` flag on a keyed + // merge; the leading `scd2` keyword is a recognized alias. + const scd2 = scd2Kw || optTokens.some((t) => t === 'history') + const opts = parseKvOpts(optsStr) + const key = opts.get('key') + const uniqueKey = key && key !== '' ? key : undefined + // `track=` (scd2): comma-separated tracked columns; empty ⇒ all. + // The value is whitespace-terminated (like every `=`-option), so it must have + // no spaces (`track=a,b`, not `track=a, b` — the rest is dropped). + const track = (opts.get('track') ?? '') + .split(',') + .map((c) => c.trim()) + .filter((c) => c !== '') + // `deletes=close` (scd2 only) opts into hard-delete-close; any other value + // (or absence) keeps the soft-delete default. + const closeDeleted = opts.get('deletes') === 'close' + // `on_schema_change=ignore` suppresses downstream contract warnings; any + // other value (or absence) keeps the `warn` default, fail-safe like `deletes=`. + const onSchemaChange = opts.get('on_schema_change') === 'ignore' ? 'ignore' : 'warn' + return { + targetKind: asset.kind, + targetPath: asset.path, + manual, + append, + uniqueKey, + scd2, + track, + closeDeleted, + onSchemaChange + } +} + +// A single bare identifier token (column name). Rejects empty / multi-token +// input. Mirrors Rust `single_ident`. +function singleIdent(s: string): string | undefined { + const t = s.trim() + if (t === '' || t.split(/\s+/).length !== 1) return undefined + return t +} + +// Strip one layer of matching surrounding single or double quotes. +function unquote(s: string): string { + if (s.length >= 2 && (s[0] === '"' || s[0] === "'") && s[s.length - 1] === s[0]) { + return s.slice(1, -1) + } + return s +} + +// ` = a,b,c`. Mirrors Rust `parse_accepted_values`. +function parseAcceptedValues(s: string): DataTest | undefined { + const eq = s.indexOf('=') + if (eq < 0) return undefined + const column = singleIdent(s.slice(0, eq)) + if (!column) return undefined + const values = s + .slice(eq + 1) + .split(',') + .map((v) => unquote(v.trim())) + .filter((v) => v !== '') + if (values.length === 0) return undefined + return { type: 'accepted_values', column, values } +} + +// ` -> .`. Mirrors Rust `parse_relationships`. +function parseRelationships(s: string): DataTest | undefined { + const arrow = s.indexOf('->') + if (arrow < 0) return undefined + const column = singleIdent(s.slice(0, arrow)) + if (!column) return undefined + const target = s.slice(arrow + 2).trim() + const dot = target.lastIndexOf('.') + if (dot < 0) return undefined + const toColumn = singleIdent(target.slice(dot + 1)) + if (!toColumn) return undefined + const asset = parseAssetSyntaxDefault(target.slice(0, dot).trim()) + if (!asset || asset.path === '') return undefined + return { + type: 'relationships', + column, + to_kind: asset.kind, + to_path: asset.path, + to_column: toColumn + } +} + +// `.` upstream reference. The column is the segment after the +// final `.`; the rest is the asset URI (default-syntax shorthands enabled). +// Mirrors Rust `parse_column_ref`. +function parseColumnRef(s: string): ColumnRef | undefined { + const dot = s.lastIndexOf('.') + if (dot < 0) return undefined + const fromColumn = singleIdent(s.slice(dot + 1)) + if (!fromColumn) return undefined + const asset = parseAssetSyntaxDefault(s.slice(0, dot).trim()) + if (!asset || asset.path === '') return undefined + return { from_kind: asset.kind, from_path: asset.path, from_column: fromColumn } +} + +// ` <- [, …]`. Individually malformed refs are dropped; the +// line is kept iff ≥1 ref parses (mirrors `parseAcceptedValues`). A missing +// `<-`, a non-ident output column, or zero valid refs drops the line. +// Mirrors Rust `parse_column_lineage_spec`. +function parseColumnLineageSpec(s: string): ColumnLineage | undefined { + const arrow = s.indexOf('<-') + if (arrow < 0) return undefined + const column = singleIdent(s.slice(0, arrow)) + if (!column) return undefined + const inputs = s + .slice(arrow + 2) + .split(',') + .map((r) => parseColumnRef(r.trim())) + .filter((r): r is ColumnRef => r !== undefined) + if (inputs.length === 0) return undefined + return { column, inputs } +} + +// Parse a `// data_test …` right-hand side into one `DataTest`. The +// leading token selects the variant; anything not a built-in keyword is the +// `custom` escape hatch (a single script-path token). Returns `undefined` for +// malformed input so a typo fails safe. Mirrors Rust `parse_data_test_spec`. +function parseDataTestSpec(s: string): DataTest | undefined { + const trimmed = s.trim() + if (trimmed === '') return undefined + const m = trimmed.match(/^(\S+)(?:\s+([\s\S]*))?$/) + if (!m) return undefined + const head = m[1] + const rest = (m[2] ?? '').trim() + switch (head) { + case 'unique': { + const column = singleIdent(rest) + return column ? { type: 'unique', column } : undefined + } + case 'not_null': { + const column = singleIdent(rest) + return column ? { type: 'not_null', column } : undefined + } + case 'accepted_values': + return parseAcceptedValues(rest) + case 'relationships': + return parseRelationships(rest) + default: + // Custom escape hatch: the whole right-hand side must be one path + // token. Trailing content after a non-built-in head is rejected. + return rest === '' ? { type: 'custom', path: head } : undefined + } +} + type ParsedTriggerSpec = | { kind: 'asset'; value: PipelineTriggerAsset } | { kind: 'native'; value: PipelineNativeTrigger } @@ -235,12 +585,23 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { const out: PipelineAnnotations = { inPipeline: false, triggerAssets: [], - nativeTriggers: [] + nativeTriggers: [], + dataTests: [], + columnLineage: [], + macros: false, + useLibs: [], + muteAssets: [], + muteAll: false } for (const rawLine of code.split('\n')) { + // Annotations live in the leading comment header: skip blank lines but + // stop at the first line of actual code, so comments inside the body + // (e.g. a regular `# tag ...` prose comment) can't false-positive. + // Mirrors the Rust parse_pipeline_annotations header scan. + if (rawLine.trim() === '') continue const rest = stripCommentPrefix(rawLine) - if (rest === undefined) continue + if (rest === undefined) break const inner = rest.trimStart() const afterPipeline = consumeKeyword(inner, 'pipeline') @@ -250,6 +611,26 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { continue } + const afterMacros = consumeKeyword(inner, 'macros') + if (afterMacros !== undefined) { + // Strict like `pipeline`: keyword alone on the line, so prose such + // as `// macros are defined below` never false-positives. + if (afterMacros.trim() === '') out.macros = true + continue + } + + // `// use ` — accumulating. The argument must be a + // single whitespace-free token containing `/` (all script paths do), + // so prose like `// use this script to …` is dropped fail-safe. + const afterUse = consumeKeyword(inner, 'use') + if (afterUse !== undefined) { + const path = afterUse.trim() + if (path && !/\s/.test(path) && path.includes('/') && !out.useLibs.includes(path)) { + out.useLibs.push(path) + } + continue + } + const afterPart = consumeKeyword(inner, 'partitioned') if (afterPart !== undefined) { if (!out.partition) { @@ -271,7 +652,10 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { const afterTag = consumeKeyword(inner, 'tag') if (afterTag !== undefined) { const name = afterTag.trim() - if (name && !out.tag) { + // Worker tags are single-word identifiers; a value with whitespace + // or beyond the script.tag column width is almost certainly a + // regular comment starting with "# tag ...". + if (name && !out.tag && !/\s/.test(name) && name.length <= 50) { out.tag = name } continue @@ -286,9 +670,60 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { continue } + const afterMaterialize = consumeKeyword(inner, 'materialize') + if (afterMaterialize !== undefined) { + if (!out.materialize) { + const spec = parseMaterializeSpec(afterMaterialize.trim()) + if (spec) out.materialize = spec + } + continue + } + + // `// mute all` opts out of auto-derived cascade edges entirely; + // `// mute ` suppresses the one edge. Only asset refs are + // muteable — native trigger kinds are never auto-derived. A complete + // word, so prose like `// muted for now` never matches. + const afterMute = consumeKeyword(inner, 'mute') + if (afterMute !== undefined) { + const arg = afterMute.trim() + if (arg === 'all') { + out.muteAll = true + } else { + const spec = parseTriggerSpec(arg) + if ( + spec?.kind === 'asset' && + !out.muteAssets.some((a) => a.kind === spec.value.kind && a.path === spec.value.path) + ) { + out.muteAssets.push(spec.value) + } + } + continue + } + + // `data_test` is a complete word (so it never collides with the `// test:` + // CI annotation, which has no whitespace after `test`). Accumulates. + const afterDataTest = consumeKeyword(inner, 'data_test') + if (afterDataTest !== undefined) { + const spec = parseDataTestSpec(afterDataTest.trim()) + if (spec) out.dataTests.push(spec) + continue + } + + // `column` is a complete word; a body comment that merely starts with + // `column` has no `<-` and is dropped fail-safe. Accumulates. + const afterColumn = consumeKeyword(inner, 'column') + if (afterColumn !== undefined) { + const spec = parseColumnLineageSpec(afterColumn.trim()) + if (spec) out.columnLineage.push(spec) + continue + } + const afterOn = consumeKeyword(inner, 'on') if (afterOn !== undefined) { - const specText = afterOn.trim() + // Strip trailing `key=value` opts (e.g. `debounce=60s`) so the ref + // matches body inference and the deploy path — otherwise the opts + // leak into the asset path and the explicit edge dedups wrong. + const specText = stripTrailingKvOpts(afterOn.trim()) if (!specText) continue const spec = parseTriggerSpec(specText) if (!spec) continue diff --git a/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.test.ts b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.test.ts new file mode 100644 index 0000000000..a8a47af259 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.test.ts @@ -0,0 +1,161 @@ +import { describe, expect, it } from 'vitest' +import { + bucketFor, + bucketFromInputValue, + defaultBucket, + inputValueFromBucket, + isBeforeStart, + isValidStart, + isValidTimeZone, + partitionInputType, + partitionMetadataError, + recentBuckets, + startBucketOf, + usesCalendarPicker +} from './partitionBuckets' +import type { PartitionSpec } from './parsePipelineAnnotations' + +const spec = (kind: PartitionSpec['kind'], extra: Partial = {}): PartitionSpec => + ({ kind, ...extra }) as PartitionSpec + +// A fixed UTC instant: 2026-07-05 14:37Z (a Sunday — ISO week 27 of 2026). +// Explicit UTC so the assertions are independent of the test runner's TZ, and +// they exercise the same UTC default the backend uses when `spec.tz` is absent. +const at = new Date(Date.UTC(2026, 6, 5, 14, 37, 0)) + +describe('bucketFor mirrors backend default_format (UTC)', () => { + it('daily -> %Y-%m-%d', () => { + expect(bucketFor(spec('daily'), at)).toBe('2026-07-05') + }) + it('hourly -> %Y-%m-%dT%H', () => { + expect(bucketFor(spec('hourly'), at)).toBe('2026-07-05T14') + }) + it('monthly -> %Y-%m', () => { + expect(bucketFor(spec('monthly'), at)).toBe('2026-07') + }) + it('weekly -> ISO %G-W%V', () => { + expect(bucketFor(spec('weekly'), at)).toBe('2026-W27') + }) +}) + +describe('bucketFor honours spec.tz', () => { + // 2026-07-05 02:30Z is still 2026-07-04 in America/New_York (UTC-4 in July). + const nearMidnight = new Date(Date.UTC(2026, 6, 5, 2, 30, 0)) + it('shifts the day boundary by the producer tz', () => { + expect(bucketFor(spec('daily'), nearMidnight)).toBe('2026-07-05') // UTC default + expect(bucketFor(spec('daily', { tz: 'America/New_York' }), nearMidnight)).toBe('2026-07-04') + }) + it('shifts the hour bucket by the producer tz', () => { + // 02:30Z -> 22 (previous day) in New York. + expect(bucketFor(spec('hourly', { tz: 'America/New_York' }), nearMidnight)).toBe( + '2026-07-04T22' + ) + }) +}) + +describe('defaultBucket honours the start= anchor', () => { + // `at` is 2026-07-05. + it('seeds the current bucket when at or after start', () => { + expect(defaultBucket(spec('daily'), at)).toBe('2026-07-05') + expect(defaultBucket(spec('daily', { start: '2026-01-01' }), at)).toBe('2026-07-05') + expect(defaultBucket(spec('daily', { start: '2026-07-05' }), at)).toBe('2026-07-05') // == start, not before + }) + it('seeds the start bucket (never a pre-start one) when before start', () => { + expect(defaultBucket(spec('daily', { start: '2026-08-01' }), at)).toBe('2026-08-01') + expect(defaultBucket(spec('monthly', { start: '2026-08-01' }), at)).toBe('2026-08') + expect(defaultBucket(spec('hourly', { start: '2026-08-01' }), at)).toBe('2026-08-01T00') + }) + it('isBeforeStart mirrors the backend date comparison', () => { + expect(isBeforeStart(spec('daily', { start: '2026-08-01' }), at)).toBe(true) + expect(isBeforeStart(spec('daily', { start: '2026-07-05' }), at)).toBe(false) + expect(isBeforeStart(spec('daily'), at)).toBe(false) + }) + it('startBucketOf renders the anchor in the cadence, undefined when unset', () => { + expect(startBucketOf(spec('daily', { start: '2026-08-01' }))).toBe('2026-08-01') + expect(startBucketOf(spec('monthly', { start: '2026-08-15' }))).toBe('2026-08') + expect(startBucketOf(spec('hourly', { start: '2026-08-01' }))).toBe('2026-08-01T00') + expect(startBucketOf(spec('daily'))).toBeUndefined() + }) +}) + +describe('malformed metadata fails safe (parity with backend validation)', () => { + it('isValidTimeZone rejects garbage, accepts real zones and absence', () => { + expect(isValidTimeZone(undefined)).toBe(true) + expect(isValidTimeZone('UTC')).toBe(true) + expect(isValidTimeZone('America/New_York')).toBe(true) + expect(isValidTimeZone('Not/AZone')).toBe(false) + expect(isValidTimeZone('garbage')).toBe(false) + }) + it('isValidStart rejects malformed and JS-normalized dates', () => { + expect(isValidStart(undefined)).toBe(true) + expect(isValidStart('2026-08-01')).toBe(true) + expect(isValidStart('2026-02-31')).toBe(false) // JS would roll to Mar 3 + expect(isValidStart('2026-13-01')).toBe(false) + expect(isValidStart('08/01/2026')).toBe(false) + }) + it('partitionMetadataError reports the first problem, else undefined', () => { + expect(partitionMetadataError(spec('daily'))).toBeUndefined() + expect( + partitionMetadataError(spec('daily', { tz: 'America/New_York', start: '2026-08-01' })) + ).toBeUndefined() + expect(partitionMetadataError(spec('daily', { tz: 'Not/AZone' }))).toContain('timezone') + expect(partitionMetadataError(spec('daily', { start: '2026-02-31' }))).toContain('start date') + }) + it('bucketFor never throws on an invalid tz (falls back to UTC)', () => { + expect(bucketFor(spec('daily', { tz: 'Not/AZone' }), at)).toBe('2026-07-05') + }) + it('an invalid start is treated as no anchor (isBeforeStart/startBucketOf)', () => { + expect(isBeforeStart(spec('daily', { start: '2026-02-31' }), at)).toBe(false) + expect(startBucketOf(spec('daily', { start: '2026-02-31' }))).toBeUndefined() + }) +}) + +describe('partitionInputType', () => { + it('maps each calendar kind to its native input', () => { + expect(partitionInputType(spec('daily'))).toBe('date') + expect(partitionInputType(spec('hourly'))).toBe('datetime-local') + expect(partitionInputType(spec('weekly'))).toBe('week') + expect(partitionInputType(spec('monthly'))).toBe('month') + }) + it('falls back to text for dynamic and custom-format specs', () => { + expect(partitionInputType(spec('dynamic', { key: '$.tenant' }))).toBe('text') + expect(partitionInputType(spec('daily', { format: '%Y/%m/%d' }))).toBe('text') + expect(usesCalendarPicker(spec('dynamic', { key: '$.tenant' }))).toBe(false) + expect(usesCalendarPicker(spec('daily', { format: '%Y/%m/%d' }))).toBe(false) + }) +}) + +describe('native input <-> bucket round-trip', () => { + it('hourly truncates the datetime-local minutes and restores :00', () => { + expect(bucketFromInputValue(spec('hourly'), '2026-07-05T14:37')).toBe('2026-07-05T14') + expect(inputValueFromBucket(spec('hourly'), '2026-07-05T14')).toBe('2026-07-05T14:00') + }) + it('non-hourly kinds pass through unchanged', () => { + expect(bucketFromInputValue(spec('daily'), '2026-07-05')).toBe('2026-07-05') + expect(inputValueFromBucket(spec('weekly'), '2026-W27')).toBe('2026-W27') + }) + it('empty stays empty', () => { + expect(bucketFromInputValue(spec('hourly'), '')).toBe('') + expect(inputValueFromBucket(spec('daily'), '')).toBe('') + }) +}) + +describe('recentBuckets (UTC)', () => { + it('walks back day by day, most recent first', () => { + expect(recentBuckets(spec('daily'), at, 3)).toEqual(['2026-07-05', '2026-07-04', '2026-07-03']) + }) + it('walks back hour by hour', () => { + expect(recentBuckets(spec('hourly'), at, 3)).toEqual([ + '2026-07-05T14', + '2026-07-05T13', + '2026-07-05T12' + ]) + }) + it('walks back month by month across a year boundary without day roll-over', () => { + const jan31 = new Date(Date.UTC(2026, 0, 31, 0, 0, 0)) + expect(recentBuckets(spec('monthly'), jan31, 3)).toEqual(['2026-01', '2025-12', '2025-11']) + }) + it('walks back week by week', () => { + expect(recentBuckets(spec('weekly'), at, 3)).toEqual(['2026-W27', '2026-W26', '2026-W25']) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.ts b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.ts new file mode 100644 index 0000000000..401ef82cae --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.ts @@ -0,0 +1,261 @@ +// Client-side partition-bucket math for the run form's partition picker. +// Produces the same canonical bucket strings the backend renders in +// `windmill-common/src/partition_ee.rs` (`resolve_time_partition` / +// `default_format`): the instant is localized to the spec's timezone +// (`spec.tz`, defaulting to UTC — NOT the browser's zone) and then formatted. +// Getting the zone right matters — a browser in a non-UTC zone would otherwise +// seed a default bucket, and compare against materialized rows, off by a +// day/hour near every boundary and always for an explicit `tz=` spec. +// +// daily %Y-%m-%d -> 2026-07-05 +// hourly %Y-%m-%dT%H -> 2026-07-05T14 +// weekly %G-W%V (ISO) -> 2026-W27 +// monthly %Y-%m -> 2026-07 +// +// Pure module (no Svelte runes) so the mapping is unit-testable. + +import type { PartitionSpec } from './parsePipelineAnnotations' + +export type PartitionInputType = 'date' | 'month' | 'week' | 'datetime-local' | 'text' + +// A spec carrying a custom strftime `format` can't be reproduced by the native +// date pickers (arbitrary strftime), and `dynamic` partitions are a free-form +// key extracted from the payload — both fall back to a plain text input. +export function usesCalendarPicker(spec: PartitionSpec): boolean { + return spec.kind !== 'dynamic' && !spec.format +} + +export function partitionInputType(spec: PartitionSpec): PartitionInputType { + if (!usesCalendarPicker(spec)) return 'text' + switch (spec.kind) { + case 'monthly': + return 'month' + case 'weekly': + return 'week' + case 'hourly': + return 'datetime-local' + default: + return 'date' + } +} + +function pad(n: number): string { + return String(n).padStart(2, '0') +} + +const ZONED_PARTS_OPTS: Intl.DateTimeFormatOptions = { + year: 'numeric', + month: '2-digit', + day: '2-digit', + hour: '2-digit', + minute: '2-digit', + second: '2-digit', + hourCycle: 'h23' +} + +// Re-express `at` as a Date whose UTC fields equal the wall-clock in `tz`, so +// all downstream field reads / calendar arithmetic can use the UTC getters and +// stay in the producer's zone. `hourCycle: 'h23'` keeps hours 00–23. A +// malformed `tz` (rejected by `Intl`) falls back to UTC rather than throwing — +// the backend validates `tz=` and is the source of truth for the error; the +// picker must never crash the graph view (`partitionMetadataError` gates +// auto-seeding so a bogus bucket isn't silently sent). +function zonedAsUtc(at: Date, tz: string): Date { + let parts: Intl.DateTimeFormatPart[] + try { + parts = new Intl.DateTimeFormat('en-US', { ...ZONED_PARTS_OPTS, timeZone: tz }).formatToParts( + at + ) + } catch { + parts = new Intl.DateTimeFormat('en-US', { + ...ZONED_PARTS_OPTS, + timeZone: 'UTC' + }).formatToParts(at) + } + const g = (t: string) => Number(parts.find((p) => p.type === t)?.value) + return new Date( + Date.UTC(g('year'), g('month') - 1, g('day'), g('hour'), g('minute'), g('second')) + ) +} + +// `tz=` is valid iff `Intl` accepts it (absent === UTC === valid). +export function isValidTimeZone(tz?: string): boolean { + if (!tz) return true + try { + new Intl.DateTimeFormat('en-US', { timeZone: tz }) + return true + } catch { + return false + } +} + +// `start=` is valid iff it's a real `YYYY-MM-DD` calendar date. The round-trip +// check rejects dates JS would silently normalize (e.g. `2026-02-31` → Mar 3), +// which the backend's `NaiveDate::parse_from_str` also rejects. +export function isValidStart(start?: string): boolean { + if (!start) return true + const m = start.match(/^(\d{4})-(\d{2})-(\d{2})$/) + if (!m) return false + const y = Number(m[1]) + const mo = Number(m[2]) + const d = Number(m[3]) + const dt = new Date(Date.UTC(y, mo - 1, d)) + return dt.getUTCFullYear() === y && dt.getUTCMonth() === mo - 1 && dt.getUTCDate() === d +} + +// A human-readable reason the `// partitioned` metadata is unusable, or +// undefined when it's sound. Mirrors the backend's `tz=` / `start=` validation +// so the picker can refuse to auto-seed a bucket the backend would reject. +export function partitionMetadataError(spec: PartitionSpec): string | undefined { + if (!isValidTimeZone(spec.tz)) return `invalid timezone "${spec.tz}"` + if (!isValidStart(spec.start)) return `invalid start date "${spec.start}" (want YYYY-MM-DD)` + return undefined +} + +// ISO 8601 week-numbering year + week (chrono's %G / %V) of a UTC-substituted +// date. Standard "nearest Thursday" algorithm, all in UTC. +function isoWeekOf(d: Date): { isoYear: number; week: number } { + const date = new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate())) + const dayNum = (date.getUTCDay() + 6) % 7 // Mon=0 … Sun=6 + date.setUTCDate(date.getUTCDate() - dayNum + 3) // Thursday of this week + const isoYear = date.getUTCFullYear() + const firstThursday = new Date(Date.UTC(isoYear, 0, 4)) + const fdNum = (firstThursday.getUTCDay() + 6) % 7 + firstThursday.setUTCDate(firstThursday.getUTCDate() - fdNum + 3) + const week = 1 + Math.round((date.getTime() - firstThursday.getTime()) / (7 * 86400000)) + return { isoYear, week } +} + +// Render a UTC-substituted date into its canonical bucket for the cadence. +function fmtBucket(kind: PartitionSpec['kind'], d: Date): string { + const y = d.getUTCFullYear() + const m = d.getUTCMonth() + 1 + const day = d.getUTCDate() + const h = d.getUTCHours() + switch (kind) { + case 'hourly': + return `${y}-${pad(m)}-${pad(day)}T${pad(h)}` + case 'monthly': + return `${y}-${pad(m)}` + case 'weekly': { + const { isoYear, week } = isoWeekOf(d) + return `${isoYear}-W${pad(week)}` + } + default: + return `${y}-${pad(m)}-${pad(day)}` + } +} + +export function bucketFor(spec: PartitionSpec, at: Date): string { + return fmtBucket(spec.kind, zonedAsUtc(at, spec.tz ?? 'UTC')) +} + +// The zoned start date (`spec.start`, `YYYY-MM-DD`) as a UTC-substituted Date at +// 00:00, or undefined if unset/malformed. `start` is a plain date in the +// producer's tz — the backend parses it as a NaiveDate and compares by date. +function startDate(spec: PartitionSpec): Date | undefined { + if (!spec.start || !isValidStart(spec.start)) return undefined + const m = spec.start.match(/^(\d{4})-(\d{2})-(\d{2})$/)! + return new Date(Date.UTC(Number(m[1]), Number(m[2]) - 1, Number(m[3]))) +} + +// True when `at` (localized to the producer tz) falls on a date before the +// `start=` anchor — exactly the backend's `local.date_naive() < start_date` +// check (`resolve_time_partition`), which resolves such an instant to no +// partition. +export function isBeforeStart(spec: PartitionSpec, at: Date): boolean { + const start = startDate(spec) + if (!start) return false + const zoned = zonedAsUtc(at, spec.tz ?? 'UTC') + const zonedDate = Date.UTC(zoned.getUTCFullYear(), zoned.getUTCMonth(), zoned.getUTCDate()) + return zonedDate < start.getTime() +} + +// The canonical bucket of the `start=` anchor (its date at 00:00), or undefined +// if unset. Buckets sort lexicographically within a cadence, so callers can +// compare against it to drop pre-start buckets. +export function startBucketOf(spec: PartitionSpec): string | undefined { + const start = startDate(spec) + return start ? fmtBucket(spec.kind, start) : undefined +} + +// The bucket to pre-fill the picker with. Normally the current bucket (matching +// the backend's "absent partition arg -> current bucket" resolution), but when +// the current instant is before the `start=` anchor the backend would resolve +// to NO partition — so default to the first valid bucket (the start) rather +// than a pre-start one the worker would take verbatim and materialize early. +export function defaultBucket(spec: PartitionSpec, at: Date): string { + if (isBeforeStart(spec, at)) { + const start = startDate(spec) + if (start) return fmtBucket(spec.kind, start) + } + return bucketFor(spec, at) +} + +// Native input value -> canonical bucket. Only hourly differs: datetime-local +// carries a minute component the hourly bucket drops. The picked wall-clock is +// taken verbatim as the bucket (the user picks in the producer's frame), so no +// timezone conversion happens here. +export function bucketFromInputValue(spec: PartitionSpec, inputValue: string): string { + if (!inputValue) return '' + if (spec.kind === 'hourly') { + const m = inputValue.match(/^(\d{4}-\d{2}-\d{2}T\d{2})/) + return m ? m[1] : inputValue + } + return inputValue +} + +// Canonical bucket -> native input value. Only hourly differs: datetime-local +// needs a minute component the bucket omits. +export function inputValueFromBucket(spec: PartitionSpec, bucket: string): string { + if (!bucket) return '' + if (spec.kind === 'hourly') { + return /T\d{2}$/.test(bucket) ? `${bucket}:00` : bucket + } + return bucket +} + +// The last `count` buckets ending at (and including) `now`, most-recent first, +// localized to `spec.tz`. Arithmetic walks calendar units in the zoned frame, +// so it's exact across DST (no ±1 drift). Undefined for non-calendar specs +// (the caller guards on `usesCalendarPicker`). +export function recentBuckets(spec: PartitionSpec, now: Date, count: number): string[] { + const base = zonedAsUtc(now, spec.tz ?? 'UTC') + const out: string[] = [] + for (let i = 0; i < count; i++) { + const d = new Date(base) + switch (spec.kind) { + case 'hourly': + d.setUTCHours(d.getUTCHours() - i) + break + case 'weekly': + d.setUTCDate(d.getUTCDate() - 7 * i) + break + case 'monthly': + // Normalize to the 1st first so subtracting months can't roll over + // a short target month (e.g. Mar 31 − 1mo → Mar 3). + d.setUTCDate(1) + d.setUTCMonth(d.getUTCMonth() - i) + break + default: + d.setUTCDate(d.getUTCDate() - i) + } + out.push(fmtBucket(spec.kind, d)) + } + return out +} + +// How many recent buckets the "missing partitions" hint scans, per kind — a +// window that reads as "recent" for each cadence without flooding the hint. +export function recentWindow(kind: PartitionSpec['kind']): number { + switch (kind) { + case 'hourly': + return 24 + case 'weekly': + return 8 + case 'monthly': + return 6 + default: + return 14 + } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.test.ts new file mode 100644 index 0000000000..a1a82870c8 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.test.ts @@ -0,0 +1,143 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { JobService, ScriptService } from '$lib/gen' +import { createPipelineAiHelpers, type PipelineDraft } from './pipelineAiHelpers' +import type { AssetGraphResponse } from './types' + +// Build a helper handle over an in-memory drafts Map, mirroring how the editor +// wires it. `getFolder` returns the bare folder name (as the route/session do). +function makeHandle( + initial: Array<[string, PipelineDraft]> = [], + runnables: Array<{ path: string }> = [] +) { + let drafts = new Map(initial) + let forgotten: string[] = [] + const handle = createPipelineAiHelpers({ + getFolder: () => 'x', + getWorkspace: () => 'w', + getResolvedGraph: () => + ({ assets: [], runnables, edges: [], triggers: [] }) as unknown as AssetGraphResponse, + getDrafts: () => drafts, + setDrafts: (next) => (drafts = next), + newDraftLocalId: () => 'id', + onForgetPath: (p) => forgotten.push(p) + }) + return { handle, drafts: () => drafts, forgotten: () => forgotten } +} + +afterEach(() => vi.restoreAllMocks()) + +const draft = (over: Partial = {}): PipelineDraft => + ({ localId: 'l', script: { content: '' } as any, ...over }) as PipelineDraft + +describe('pipeline AI direct-draft helpers', () => { + it('removeProposedNode discards the unsaved draft at a path', async () => { + const { handle, drafts, forgotten } = makeHandle([ + ['f/x/a', draft()], + ['f/x/b', draft()] + ]) + await handle.removeProposedNode('f/x/a') + expect(drafts().has('f/x/a')).toBe(false) + expect(drafts().has('f/x/b')).toBe(true) + expect(forgotten()).toContain('f/x/a') + }) + + it('removeProposedNode throws when there is no draft to discard', async () => { + const { handle } = makeHandle() + await expect(handle.removeProposedNode('f/x/missing')).rejects.toThrow() + }) + + it('getPipelineContext does not expose any pending/approval state', () => { + const { handle } = makeHandle([['f/x/a', draft()]]) + const ctx = handle.getPipelineContext() + expect(ctx).not.toHaveProperty('pendingProposals') + expect(handle).not.toHaveProperty('acceptAll') + expect(handle).not.toHaveProperty('rejectAll') + }) + + it('testNode on a deployed node never dispatches downstream subscribers', async () => { + // No draft at the path → runs the deployed version, which must carry + // `_wmill_skip_asset_dispatch` so a single-node test can't fire downstream. + const spy = vi.spyOn(JobService, 'runScriptByPath').mockResolvedValue('job-1' as any) + const { handle } = makeHandle() + await handle.testNode('f/x/deployed', { foo: 1 }) + expect(spy).toHaveBeenCalledWith( + expect.objectContaining({ + requestBody: expect.objectContaining({ _wmill_skip_asset_dispatch: true, foo: 1 }) + }) + ) + }) + + it('proposeNode rejects a path outside the open folder', async () => { + const { handle, drafts } = makeHandle() + await expect( + handle.proposeNode({ path: 'f/other/n', language: 'duckdb' as any, content: '' }) + ).rejects.toThrow(/open folder/) + expect(drafts().size).toBe(0) + }) + + it('proposeNode rejects content missing the pipeline annotation', async () => { + const { handle, drafts } = makeHandle() + await expect( + handle.proposeNode({ path: 'f/x/new', language: 'duckdb' as any, content: 'SELECT 1' }) + ).rejects.toThrow(/pipeline annotation/) + expect(drafts().size).toBe(0) + }) + + it('proposeNode rejects a path colliding with an existing draft', async () => { + const { handle } = makeHandle([['f/x/a', draft()]]) + await expect( + handle.proposeNode({ path: 'f/x/a', language: 'duckdb' as any, content: '-- pipeline' }) + ).rejects.toThrow(/already exists/) + }) + + it('proposeNode rejects a path colliding with an existing deployed node', async () => { + const { handle, drafts } = makeHandle([], [{ path: 'f/x/dep' }]) + await expect( + handle.proposeNode({ path: 'f/x/dep', language: 'duckdb' as any, content: '-- pipeline' }) + ).rejects.toThrow(/already exists/) + expect(drafts().size).toBe(0) + }) + + it('proposeNode rejects a path that is an already-deployed script when the graph has not hydrated', async () => { + // Empty graph (session preview can race open_preview), but a deployed script + // exists at the path — the backend probe must still catch it. + const spy = vi.spyOn(ScriptService, 'getScriptByPath').mockResolvedValue({} as any) + const { handle, drafts } = makeHandle() + await expect( + handle.proposeNode({ + path: 'f/x/deployed', + language: 'duckdb' as any, + content: '-- pipeline' + }) + ).rejects.toThrow(/already exists/) + expect(spy).toHaveBeenCalled() + expect(drafts().size).toBe(0) + }) + + it('editNode rejects a path outside the open folder', async () => { + const { handle, drafts } = makeHandle() + await expect(handle.editNode('f/other/foo', '-- pipeline')).rejects.toThrow(/open folder/) + expect(drafts().size).toBe(0) + }) + + it('editNode preserves the deployed script hash/metadata and replaces only content', async () => { + const deployed = { + hash: 'abc123', + path: 'f/x/node', + summary: 'My node', + description: 'desc', + tag: 'custom', + language: 'duckdb', + content: '-- pipeline\nSELECT 1' + } + vi.spyOn(ScriptService, 'getScriptByPath').mockResolvedValue(deployed as any) + const { handle, drafts } = makeHandle() + await handle.editNode('f/x/node', '-- pipeline\nSELECT 2') + const d = drafts().get('f/x/node') + expect(d?.script.hash).toBe('abc123') + expect(d?.script.summary).toBe('My node') + expect(d?.script.description).toBe('desc') + expect(d?.script.tag).toBe('custom') + expect(d?.script.content).toBe('-- pipeline\nSELECT 2') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.ts new file mode 100644 index 0000000000..3658bc5a32 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.ts @@ -0,0 +1,334 @@ +import { JobService, ScriptService, type AssetKind, type Script, type ScriptLang } from '$lib/gen' +import { emptySchema, sendUserToast } from '$lib/utils' +import { inferAssets } from '$lib/infer' +import { + extractReads, + extractWrites, + type AssetWithAltAccessType +} from '$lib/components/assets/lib' +import { assetUri, autoOutputAsset, type PipelineOutputKind } from './pipelineTemplates' +import { parsePipelineAnnotations } from './parsePipelineAnnotations' +import type { AssetGraphResponse } from './types' +import type { + PipelineAIChatHelpers, + PipelineContext, + PipelineNodeSummary +} from '$lib/components/copilot/chat/pipeline/core' + +// ============================================================================ +// Shared data-pipeline AI helper layer. +// +// Both the full-page editor (/pipeline/[folder]) and the in-session preview +// (PipelineEditorView) drive the AI chat's pipeline tools through this factory, +// so the build/edit logic lives in exactly one place. Each caller injects +// accessors for its own draft Map and graph; this module owns the AI behaviour +// (build/edit/discard/test). AI edits apply directly as unsaved drafts — there +// is no separate approve/reject step. +// ============================================================================ + +/** + * An unsaved pipeline node draft. `localId` is a stable per-draft id preserved + * across renames (the page uses it to dedupe concurrent deploys). AI-built nodes + * and manually-created drafts are the same thing — an unsaved node on the canvas. + */ +export type PipelineDraft = { + localId: string + script: Script + outputAssets?: Array<{ kind: AssetKind; path: string }> + /** Body-inferred reads captured with the draft, so an inactive draft keeps + * its input lineage on the canvas. `undefined` = not captured yet (legacy + * bundle / just-seeded draft) — consumers fall back to the session cache; + * an empty array is an authoritative "reads nothing". */ + inputAssets?: Array<{ kind: AssetKind; path: string }> +} + +export type PipelineAiHelperDeps = { + getFolder: () => string + getWorkspace: () => string | undefined + /** The draft-overlaid graph (resolveGraph output) the context summary reads. */ + getResolvedGraph: () => AssetGraphResponse + getDrafts: () => Map + setDrafts: (next: Map) => void + /** Stable id for a freshly-created draft (route page tracks deploys by it). */ + newDraftLocalId: () => string + /** Focus/select the node after it is staged (pan + open in the pane). */ + onProposeNode?: (path: string) => void + /** Throw (or switch to edit mode) when the surface can't accept AI edits. */ + ensureEditable?: () => void + /** Surface the draft overlay if it is hidden (the page's "show drafts" view). */ + onShowDrafts?: () => void + /** Forget per-path state when a draft is discarded. */ + onForgetPath?: (path: string) => void + /** Notify the caller a test run started so it can light up its run UI. */ + onRunStarted?: (jobId: string, path: string) => void +} + +export function makePipelineScript( + language: ScriptLang, + scriptPath: string, + content: string, + createdAt: string +): Script { + // Cast through unknown: a local draft only needs path/language/content/schema; + // the many readonly deployment fields on Script don't matter until createScript. + return { + hash: '', + path: scriptPath, + summary: '', + description: '', + content, + schema: emptySchema(), + is_template: false, + extra_perms: {}, + language, + kind: 'script', + created_by: '', + created_at: createdAt, + archived: false, + deleted: false, + starred: false + } as unknown as Script +} + +async function inferDraftAssets( + language: ScriptLang, + content: string +): Promise<{ + writes: Array<{ kind: AssetKind; path: string }> + reads: Array<{ kind: AssetKind; path: string }> +}> { + try { + const inferred = await inferAssets(language, content) + if (inferred?.status === 'error') return { writes: [], reads: [] } + const assets = (inferred?.assets ?? []) as AssetWithAltAccessType[] + return { writes: extractWrites(assets), reads: extractReads(assets) } + } catch { + return { writes: [], reads: [] } + } +} + +export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIChatHelpers { + // A staged draft is always persisted into the OPEN folder's data_pipeline + // bundle, so a path outside the folder would silently land an unrelated script + // there. Both build and edit must stay scoped to the folder. + function assertInFolder(path: string) { + const folder = deps.getFolder() + if (folder && !path.startsWith(`f/${folder}/`)) { + throw new Error( + `Pipeline nodes must be in the open folder — use a path under 'f/${folder}/' (got '${path}').` + ) + } + } + + // A pipeline node IS its `// pipeline` annotation (it's what makes the deployed + // script a pipeline member). Reject content that lacks it so a staged draft + // isn't a non-member script the model can't see is broken until deploy. + function assertPipelineAnnotation(content: string) { + if (!parsePipelineAnnotations(content).inPipeline) { + throw new Error( + `Pipeline node content must declare the pipeline annotation on its own comment line ` + + `(\`// pipeline\`, or \`-- pipeline\` for SQL / \`# pipeline\` for Python).` + ) + } + } + + function buildContext(): PipelineContext { + const graph = deps.getResolvedGraph() + const drafts = deps.getDrafts() + const nodes: PipelineNodeSummary[] = graph.runnables + .filter((r) => r.usage_kind === 'script') + .map((r) => { + const draft = drafts.get(r.path) + const writes = graph.edges + .filter( + (e) => + e.runnable_kind === 'script' && + e.runnable_path === r.path && + (e.access_type === 'w' || e.access_type === 'rw') + ) + .map((e) => assetUri({ kind: e.asset_kind, path: e.asset_path })) + const reads = graph.edges + .filter( + (e) => + e.runnable_kind === 'script' && + e.runnable_path === r.path && + (e.access_type === 'r' || e.access_type === 'rw') + ) + .map((e) => assetUri({ kind: e.asset_kind, path: e.asset_path })) + const triggers = graph.triggers + .filter((t) => t.runnable_kind === 'script' && t.runnable_path === r.path) + .map((t) => + t.trigger_kind === 'asset' + ? assetUri({ kind: t.asset_kind, path: t.asset_path }) + : t.trigger_kind + ) + return { + path: r.path, + language: draft?.script.language, + unsaved: r.unsaved ?? false, + summary: draft?.script.summary || undefined, + writes: [...new Set(writes)], + reads: [...new Set(reads)], + triggers: [...new Set(triggers)] + } + }) + return { + folder: deps.getFolder(), + mode: 'edit', + nodes, + assets: graph.assets.map((a) => assetUri({ kind: a.kind, path: a.path })) + } + } + + const helpers: PipelineAIChatHelpers = { + getPipelineContext: buildContext, + getNodeBody: async (path) => { + const draft = deps.getDrafts().get(path) + if (draft) return { language: draft.script.language, content: draft.script.content } + const workspace = deps.getWorkspace() + if (!workspace) return undefined + try { + const deployed = await ScriptService.getScriptByPath({ workspace, path }) + return { language: deployed.language, content: deployed.content } + } catch { + return undefined + } + }, + proposeNode: async ({ path, language, content, outputKind }) => { + deps.ensureEditable?.() + // build_pipeline_node creates a NEW node in the OPEN folder. Reject a path + // outside the folder (it would silently stage into this folder's bundle) + // and a path that collides with an existing node (the model should use + // edit_pipeline_node instead of shadowing a deployed node as a draft). + assertInFolder(path) + assertPipelineAnnotation(content) + const drafts = deps.getDrafts() + if (drafts.has(path)) { + throw new Error( + `A draft already exists at '${path}'. Use edit_pipeline_node to change it instead.` + ) + } + if (deps.getResolvedGraph().runnables.some((r) => r.path === path)) { + throw new Error( + `A pipeline node already exists at '${path}'. Use edit_pipeline_node to change it instead.` + ) + } + // Authoritative new-node check: the resolved graph may not have hydrated yet + // (the session preview can race open_preview), and it only lists pipeline + // runnables — so probe the backend. ANY deployed script at this path means + // "build new" would shadow it on deploy; the model should edit instead. + const workspace = deps.getWorkspace() + if (workspace) { + let deployedExists = false + try { + await ScriptService.getScriptByPath({ workspace, path }) + deployedExists = true + } catch { + // 404 → no deployed script at this path, safe to create a new node. + } + if (deployedExists) { + throw new Error( + `A script already exists at '${path}'. Use edit_pipeline_node to change it instead.` + ) + } + } + const inferred = await inferDraftAssets(language, content) + // Fall back to a seeded output (from the declared output_kind) when the + // body doesn't yet write anything inferable. + const seeded = + inferred.writes[0] ?? + (outputKind + ? autoOutputAsset(outputKind as PipelineOutputKind, deps.getFolder(), language) + : undefined) + const next = new Map(drafts) + next.set(path, { + localId: deps.newDraftLocalId(), + script: makePipelineScript(language, path, content, new Date().toISOString()), + outputAssets: inferred.writes.length > 0 ? inferred.writes : seeded ? [seeded] : undefined, + inputAssets: inferred.reads + }) + deps.setDrafts(next) + deps.onShowDrafts?.() + deps.onProposeNode?.(path) + return { path } + }, + editNode: async (path, content) => { + deps.ensureEditable?.() + assertInFolder(path) + assertPipelineAnnotation(content) + const drafts = deps.getDrafts() + const existing = drafts.get(path) + // Base the edit on the existing draft's / deployed script object and replace + // ONLY the content — preserving hash, summary, description, tag, schema, and + // settings. Rebuilding a fresh script would wipe that metadata: deploying + // from the pane (auto_parent) would update the script while clearing it, and + // the route "Save all" path (no parent_hash) could hit the path-conflict + // branch on the occupied path. + let baseScript: Script + if (existing) { + baseScript = existing.script + } else { + const workspace = deps.getWorkspace() + if (!workspace) throw new Error('No workspace is selected.') + baseScript = await ScriptService.getScriptByPath({ workspace, path }) + } + const inferred = await inferDraftAssets(baseScript.language, content) + const next = new Map(drafts) + next.set(path, { + localId: existing?.localId ?? deps.newDraftLocalId(), + script: { ...baseScript, content }, + outputAssets: inferred.writes.length > 0 ? inferred.writes : existing?.outputAssets, + inputAssets: inferred.reads + }) + deps.setDrafts(next) + deps.onShowDrafts?.() + deps.onProposeNode?.(path) + }, + removeProposedNode: async (path) => { + if (!deps.getDrafts().has(path)) { + throw new Error(`No unsaved draft at '${path}' to discard.`) + } + const next = new Map(deps.getDrafts()) + next.delete(path) + deps.setDrafts(next) + deps.onForgetPath?.(path) + }, + testNode: async (path, args) => { + const workspace = deps.getWorkspace() + if (!workspace) return undefined + const draft = deps.getDrafts().get(path) + try { + let jobId: string + if (draft) { + // Un-deployed/edited body: preview-run the draft content so it can be + // tested before deploying. + jobId = await JobService.runScriptPreview({ + workspace, + requestBody: { + path, + content: draft.script.content, + language: draft.script.language, + args: args ?? {} + } + }) + } else { + // test_pipeline_node previews ONE node — never fan out to downstream + // deployed subscribers via the backend asset dispatcher (which would + // run side-effecting deployed scripts the user didn't ask for). + jobId = await JobService.runScriptByPath({ + workspace, + path, + requestBody: { ...(args ?? {}), _wmill_skip_asset_dispatch: true } + }) + } + deps.onRunStarted?.(jobId, path) + return jobId + } catch (e: any) { + sendUserToast(`Run failed: ${e?.body ?? e?.message ?? e}`, true) + return undefined + } + } + } + + return helpers +} diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.svelte.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.svelte.ts new file mode 100644 index 0000000000..086abc1bf8 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.svelte.ts @@ -0,0 +1,201 @@ +import type { AssetKind, Script } from '$lib/gen' +import type { AssetWithAltAccessType } from '$lib/components/assets/lib' +import type { AssetGraphSelection } from './types' +import { + parsePipelineAnnotations, + type ColumnLineage, + type PipelineAnnotations +} from './parsePipelineAnnotations' +import type { PipelineDraft } from './pipelineAiHelpers' + +// ============================================================================ +// Externalized pipeline-editor state — the data-pipeline analogue of the flow +// editor's `flowStore` / `flowStateStore`. It owns the in-flight draft Map, the +// live editor overlays, and the current selection: the substrate the route page +// editor and the in-session preview both render through (via the shared +// ). Persistence, graph resolution, run dispatch, and deploy +// stay with the consumer; this is a plain reactive bag so a consumer can +// read/mutate it without prop plumbing. +// ============================================================================ + +const EMPTY_ANNOTATIONS: PipelineAnnotations = parsePipelineAnnotations('') + +type LiveAnnotations = { scriptPath: string | undefined; annotations: PipelineAnnotations } +type LiveBodyAssets = { + scriptPath: string | undefined + assets: AssetWithAltAccessType[] + columnLineage?: ColumnLineage[] +} +type LiveContent = { scriptPath: string | undefined; content: string } + +export class PipelineEditorState { + /** In-flight drafts keyed by script path (manual + AI-staged). */ + drafts = $state>(new Map()) + /** Draft open in the details pane (mutually exclusive with `selection`). */ + activeDraftPath = $state(undefined) + /** The persisted node/asset selected on the canvas. */ + selection = $state(undefined) + + /** Live-parsed annotations of the open script (refreshed per keystroke). */ + liveAnnotations = $state({ + scriptPath: undefined, + annotations: EMPTY_ANNOTATIONS + }) + /** Live-inferred body read/write assets of the open script. */ + liveBodyAssets = $state({ scriptPath: undefined, assets: [] }) + /** The open draft's live editor buffer. */ + liveContent = $state({ scriptPath: undefined, content: '' }) + + /** Set true once a draft bundle was restored from the DB on load — drives the + * route toolbar's one-shot "Loaded from draft" hint. Written by the editor's + * autosave hydrate when persistence is enabled. */ + loadedFromDbDraft = $state(false) + + /** Folder this state is scoped to. Used by the in-session preview (where one + * instance is reused across editor hide/show) to detect a retarget to a + * different folder and reset, so stale drafts don't bleed across folders. */ + folder = $state(undefined) + + /** True once the DB draft bundle for the current folder has been hydrated + * into this instance. Gated per-instance (not per component mount) so the + * in-session preview hydrates ONCE when its runtime is fresh and then keeps + * the in-memory drafts across editor hide/show — re-reading the DB on every + * remount would race a not-yet-flushed autosave and drop a just-staged draft. + * Reset to false on a folder retarget so the new folder re-hydrates. */ + hydratedFromDb = $state(false) + + /** Clear all in-flight state. Used when the session preview retargets a + * different pipeline folder (a same-folder remount keeps the drafts). */ + reset = () => { + this.drafts = new Map() + this.activeDraftPath = undefined + this.selection = undefined + this.clearLiveOverlays() + this.loadedFromDbDraft = false + // Force a re-hydrate from the DB draft of the newly-targeted folder. + this.hydratedFromDb = false + } + + #nextDraftLocalId = 0 + // Arrow fields so `pe.method` can be passed straight as a callback (the + // details pane takes onDraftPersist / onAnnotationsChange / … by reference). + newDraftLocalId = (): string => { + this.#nextDraftLocalId += 1 + return `pe-${this.#nextDraftLocalId}` + } + + handleAnnotationsChange = (scriptPath: string | undefined, annotations: PipelineAnnotations) => { + this.liveAnnotations = { scriptPath, annotations } + } + handleAssetsChange = ( + scriptPath: string | undefined, + assets: AssetWithAltAccessType[], + columnLineage?: ColumnLineage[] + ) => { + this.liveBodyAssets = { scriptPath, assets, columnLineage } + } + handleContentChange = (scriptPath: string | undefined, content: string) => { + this.liveContent = { scriptPath, content } + } + + clearLiveOverlays = () => { + this.liveAnnotations = { scriptPath: undefined, annotations: EMPTY_ANNOTATIONS } + this.liveBodyAssets = { scriptPath: undefined, assets: [] } + this.liveContent = { scriptPath: undefined, content: '' } + } + + /** Drop per-path editor state when a path goes away. Does NOT touch + * consumer-owned per-path state (e.g. the route page's save errors — the + * route layers that on in its own wrapper). */ + forgetPath = (path: string) => { + if (this.activeDraftPath === path) this.activeDraftPath = undefined + if (this.selection?.kind === 'runnable' && this.selection.path === path) + this.selection = undefined + if (this.liveAnnotations.scriptPath === path) + this.liveAnnotations = { scriptPath: undefined, annotations: EMPTY_ANNOTATIONS } + if (this.liveBodyAssets.scriptPath === path) + this.liveBodyAssets = { scriptPath: undefined, assets: [] } + if (this.liveContent.scriptPath === path) + this.liveContent = { scriptPath: undefined, content: '' } + } + + discardDraft = (path: string) => { + if (!this.drafts.has(path)) return + const next = new Map(this.drafts) + next.delete(path) + this.drafts = next + this.forgetPath(path) + } + + /** Commit body edits + inferred outputs back into the drafts Map on pane + * teardown (deferred a microtask so a same-batch discard doesn't resurrect the + * entry). Verbatim port of the route page's `handleDraftPersist`. */ + handleDraftPersist = ( + p: string, + snapshot: { + content: string + writes: { kind: AssetKind; path: string }[] + // Optional: undefined = reads not captured by this caller — keep + // whatever the draft already carries. + reads?: { kind: AssetKind; path: string }[] + script?: Script + } + ) => { + queueMicrotask(() => { + const d = this.drafts.get(p) + if (!d) { + if (!snapshot.script) return + const next = new Map(this.drafts) + next.set(p, { + localId: this.newDraftLocalId(), + script: snapshot.script, + outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined, + inputAssets: snapshot.reads + }) + this.drafts = next + return + } + // `?? 0` is load-bearing: an undefined `outputAssets` (a no-output draft) + // vs an empty inferred `writes` both mean "no writes". Without the + // coalesce, `undefined === 0` is false, so this never short-circuits — + // every persist re-writes the drafts Map with an equivalent object, + // re-triggering the pane's emit → graph re-derive → persist, an infinite + // microtask loop (hangs the tab without an effect-depth throw). + const refsEqual = ( + a: Array<{ kind: AssetKind; path: string }>, + b: Array<{ kind: AssetKind; path: string }> + ) => + a.length === b.length && a.every((x, i) => x.kind === b[i]?.kind && x.path === b[i]?.path) + const writesEqual = refsEqual(d.outputAssets ?? [], snapshot.writes) + // An uncaptured entry (`inputAssets` undefined) is never "equal" to an + // incoming capture — even `reads: []` must be recorded, or the entry + // stays on the legacy fallback (session cache) and can keep stale read + // edges after the pane closes. + const readsEqual = + snapshot.reads == undefined || + (d.inputAssets != undefined && refsEqual(d.inputAssets, snapshot.reads)) + if (d.script.content === snapshot.content && writesEqual && readsEqual) return + const next = new Map(this.drafts) + next.set(p, { + ...d, + script: { ...d.script, content: snapshot.content }, + outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined, + inputAssets: snapshot.reads ?? d.inputAssets + }) + this.drafts = next + }) + } + + /** The draft open in the pane, if any. */ + get activeDraft(): PipelineDraft | undefined { + return this.activeDraftPath ? this.drafts.get(this.activeDraftPath) : undefined + } + + /** Whichever script is open — the active draft, or a selected persisted script. */ + get openScriptPath(): string | undefined { + if (this.activeDraftPath) return this.activeDraftPath + if (this.selection?.kind === 'runnable' && this.selection.runnable_kind === 'script') + return this.selection.path + return undefined + } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.test.ts new file mode 100644 index 0000000000..2ec06bc8cc --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.test.ts @@ -0,0 +1,106 @@ +import { describe, it, expect } from 'vitest' +import { PipelineEditorState } from './pipelineEditorState.svelte' +import type { PipelineDraft } from './pipelineAiHelpers' +import type { AssetKind } from '$lib/gen' + +// handleDraftPersist defers its commit a microtask (so a same-batch discard can +// win); flush that microtask before asserting. +const flushMicrotasks = () => new Promise((resolve) => queueMicrotask(() => resolve())) + +function draft(content: string, outputAssets?: { kind: AssetKind; path: string }[]): PipelineDraft { + return { + localId: 'pe-1', + script: { path: 'f/x/n', language: 'duckdb', content } as PipelineDraft['script'], + outputAssets + } +} + +describe('PipelineEditorState.handleDraftPersist', () => { + // Regression: a no-output draft has `outputAssets: undefined`; the details pane + // infers an empty `writes: []`. Both mean "no writes", so persisting unchanged + // content+writes must be a no-op. The earlier `undefined === 0` length check made + // it false, so every persist re-wrote the drafts Map with an equivalent object — + // re-triggering the pane's emit → graph re-derive → persist, an infinite + // microtask loop that froze the tab. The Map reference must stay identical. + it('is idempotent for a no-output draft (undefined outputAssets vs empty inferred writes)', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { content: 'SELECT 1', writes: [] }) + await flushMicrotasks() + expect(pe.drafts).toBe(before) + }) + + it('re-writes the drafts Map when the content actually changes', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { content: 'SELECT 2', writes: [] }) + await flushMicrotasks() + expect(pe.drafts).not.toBe(before) + expect(pe.drafts.get('f/x/n')?.script.content).toBe('SELECT 2') + }) + + it('re-writes the drafts Map when the inferred writes actually change', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { + content: 'SELECT 1', + writes: [{ kind: 'resource' as AssetKind, path: 'f/x/out' }] + }) + await flushMicrotasks() + expect(pe.drafts).not.toBe(before) + expect(pe.drafts.get('f/x/n')?.outputAssets).toEqual([{ kind: 'resource', path: 'f/x/out' }]) + }) + + it('stays idempotent when outputAssets and inferred writes match (non-empty)', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([ + ['f/x/n', draft('SELECT 1', [{ kind: 'resource' as AssetKind, path: 'f/x/out' }])] + ]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { + content: 'SELECT 1', + writes: [{ kind: 'resource' as AssetKind, path: 'f/x/out' }] + }) + await flushMicrotasks() + expect(pe.drafts).toBe(before) + }) +}) + +describe('PipelineEditorState.handleDraftPersist — read capture', () => { + it('records an authoritative empty capture on an uncaptured entry (undefined → [])', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { content: 'SELECT 1', writes: [], reads: [] }) + await flushMicrotasks() + // Must rewrite: undefined means "fall back to the session cache", [] means + // "reads nothing" — staying undefined would keep stale cached read edges + // alive after the pane closes. + expect(pe.drafts).not.toBe(before) + expect(pe.drafts.get('f/x/n')?.inputAssets).toEqual([]) + }) + + it('is idempotent once the capture matches ([] vs reads: [])', async () => { + const pe = new PipelineEditorState() + const d = draft('SELECT 1', undefined) + d.inputAssets = [] + pe.drafts = new Map([['f/x/n', d]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { content: 'SELECT 1', writes: [], reads: [] }) + await flushMicrotasks() + expect(pe.drafts).toBe(before) + }) + + it('is idempotent when a legacy caller omits reads', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { content: 'SELECT 1', writes: [] }) + await flushMicrotasks() + expect(pe.drafts).toBe(before) + expect(pe.drafts.get('f/x/n')?.inputAssets).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts index 50ad3e1d6e..d654085179 100644 --- a/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts @@ -80,7 +80,15 @@ export function usePipelineHistory( kind: j.job_kind.startsWith('flow') ? 'flow' : 'script', status: j.success ? 'success' : 'failure', source: j.schedule_path ? 'schedule' : 'run', - at: j.started_at ?? j.created_at + at: j.started_at ?? j.created_at, + // Same completion-time derivation as the live poll — + // the freshness chip compares against completion, and + // `at` (start) would read a long run as older than its + // output actually is. + completedAt: + j.started_at != undefined + ? new Date(new Date(j.started_at).getTime() + j.duration_ms).toISOString() + : undefined }) } sawFullPage = rows.length === PER_PAGE diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.dataTest.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.dataTest.test.ts new file mode 100644 index 0000000000..2acdf2abfb --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.dataTest.test.ts @@ -0,0 +1,56 @@ +import { describe, expect, it } from 'vitest' +import { + autoOutputAsset, + compatibleOutputKinds, + generatePipelineDraft, + PIPELINE_OUTPUT_KINDS +} from './pipelineTemplates' + +// The `data_test` output kind scaffolds a *custom* (singular) data test: a +// standalone DuckDB script referenced from a materialize script's +// `-- data_test ` line. It must be a single SELECT that reads the +// freshly-materialized target through the internal `_wm_target` schema — the +// two rules the backend's self-teaching errors enforce. +describe('data_test scaffold', () => { + it('is a DuckDB-only output kind exposed in the picker', () => { + expect(compatibleOutputKinds('duckdb')).toContain('data_test') + expect(compatibleOutputKinds('python3')).not.toContain('data_test') + expect(PIPELINE_OUTPUT_KINDS.map((k) => k.id)).toContain('data_test') + }) + + it('produces no output asset (it asserts against an existing target)', () => { + expect(autoOutputAsset('data_test', 'folder', 'duckdb')).toBeUndefined() + }) + + it('scaffolds a single SELECT against `_wm_target.`', () => { + const src = generatePipelineDraft({ + language: 'duckdb', + outputKind: 'data_test', + triggers: [] + }) + // starter body is a single SELECT against the internal target alias. + expect(src).toContain('SELECT * FROM _wm_target.your_table WHERE your_condition;') + // exactly one SQL statement (single SELECT) — count statement lines, not + // the word "SELECT" that also appears in the guidance comment. + const stmtLines = src.split('\n').filter((l) => /^\s*SELECT\b/i.test(l)) + expect(stmtLines).toHaveLength(1) + // no `-- materialize ` output annotation — a data test declares no + // asset (the word still appears in the guidance comment, which is fine). + expect(src).not.toMatch(/^--\s*materialize\s/m) + // teaches how to wire it up + the offending-rows convention. + expect(src).toContain('-- data_test ') + expect(src).toContain('offending rows') + }) + + it('seeds the table name from an upstream ducklake asset when present', () => { + const src = generatePipelineDraft({ + language: 'duckdb', + outputKind: 'data_test', + input: { kind: 'ducklake', path: 'analytics/orders' }, + triggers: [] + }) + expect(src).toContain('SELECT * FROM _wm_target.orders WHERE your_condition;') + // no ATTACH of the input — the runtime attaches the target as `_wm_target`. + expect(src).not.toContain('ATTACH') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.test.ts new file mode 100644 index 0000000000..00fdd085ed --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.test.ts @@ -0,0 +1,158 @@ +import { describe, expect, it } from 'vitest' +import type { ScriptLang } from '$lib/gen' +import { + autoOutputAsset, + generatePipelineDraft, + type PipelineOutputKind +} from './pipelineTemplates' + +// The seeded draft asset (`autoOutputAsset`, stored as `outputAssets` and used +// by resolveGraph for inactive-draft node identity) must match the asset +// identity the deploy-time / wasm parser infers from the generated body. The +// parser canonicalizes any S3 URI by stripping the `s3://` prefix and all +// leading slashes (see backend `parse_asset_syntax`); if the seed carried a +// leading slash while the body wrote `s3:///key`, the preview would render a +// duplicate `/key` node and a phantom post-deploy drift. This pins the two in +// lockstep so that class of drift can't regress. + +// Mirror of the parser's S3 canonicalization for a raw `s3://…` URI. +function canonicalS3Key(uri: string): string { + const rest = uri.replace(/^s3:\/\//, '') + return rest.replace(/^\/+/, '') +} + +const S3_KINDS: PipelineOutputKind[] = ['s3_parquet', 's3_object'] +const LANGS: ScriptLang[] = ['bun', 'python3', 'duckdb'] + +describe('pipelineTemplates S3 seed/body parity', () => { + for (const language of LANGS) { + for (const outputKind of S3_KINDS) { + it(`${language} ${outputKind}: seeded asset path matches the body's S3 write URI`, () => { + const output = autoOutputAsset(outputKind, 'demo', language) + expect(output).toBeDefined() + const asset = output! + + // The seed must be a canonical slashless key so it matches the + // identity the parser infers from the generated body. + expect(asset.kind).toBe('s3object') + expect(asset.path.startsWith('/')).toBe(false) + + const body = generatePipelineDraft({ + language, + outputKind, + output: asset, + triggers: [] + }) + + // Every S3 URI the generated body emits must canonicalize back to + // the seeded asset path — the write target especially. + const uris = body.match(/s3:\/\/[^'"`)\s]+/g) ?? [] + expect(uris.length).toBeGreaterThan(0) + for (const uri of uris) { + // Runtime form must stay triple-slash (default storage); a bare + // `s3://key` would target a named storage `key` at run time. + expect(uri.startsWith('s3:///')).toBe(true) + expect(canonicalS3Key(uri)).toBe(asset.path) + } + }) + } + } +}) + +// `{partition}` substitutes to the partition IDENTITY string (e.g. `2026-07-05T23`, +// `2026-W27`, `2026-07`), which is NOT a valid DuckDB TIMESTAMP literal for any +// sub-day / non-daily grain — so a naive `WHERE ts = TIMESTAMP {partition}` +// raises a Conversion Error for hourly/weekly/monthly. The runtime injects a +// `wm_partition(ts)` macro (format from the same source that stamps the +// identity), so the scaffold teaches the one grain-agnostic filter line. +describe('pipelineTemplates materialize partition filter', () => { + const materializeBody = () => + generatePipelineDraft({ + language: 'duckdb', + outputKind: 'materialize', + output: autoOutputAsset('materialize', 'demo', 'duckdb'), + input: { kind: 'ducklake', path: 'main/orders' }, + triggers: [] + }) + + it('teaches the grain-agnostic wm_partition filter', () => { + const body = materializeBody() + expect(body).toContain(`WHERE wm_partition() = {partition}`) + }) + + it('never scaffolds the naive `TIMESTAMP {partition}` cast, nor a raw strftime format', () => { + const body = materializeBody() + // The footgun cast must never appear (comment or SQL) now that the macro + // hides the format entirely. + expect(body).not.toMatch(/TIMESTAMP\s*\{partition\}/) + // No hand-written strftime format to drift from the resolver. + expect(body).not.toContain('strftime') + }) +}) + +describe('pipelineTemplates: auto-derived reads drop the redundant // on', () => { + const draft = ( + input: { kind: 'ducklake' | 's3object' | 'datatable'; path: string } | undefined, + triggers: Parameters[0]['triggers'], + language: ScriptLang = 'duckdb', + outputKind: PipelineOutputKind = 'materialize' + ) => + generatePipelineDraft({ + language, + outputKind, + output: autoOutputAsset(outputKind, 'demo', language), + input, + triggers + }) + + it('omits // on for a ducklake input the body reads (auto-derived)', () => { + const body = draft({ kind: 'ducklake', path: 'main/orders' }, [ + { kind: 'asset', ref: 'ducklake://main/orders' } + ]) + expect(body).not.toContain('on ducklake://main/orders') + // The body still attaches and reads the table (`ducklake://main` → + // `lake.orders`), which is what wires the cascade now that the explicit + // `// on` is gone. + expect(body).toContain(`ATTACH 'ducklake://main'`) + expect(body).toContain('lake.orders') + }) + + it('omits // on for an s3 input the body reads', () => { + const body = draft({ kind: 's3object', path: 'raw/events.parquet' }, [ + { kind: 'asset', ref: 's3://raw/events.parquet' } + ]) + expect(body).not.toContain('on s3://raw/events.parquet') + }) + + it('keeps // on for a datatable input (not auto-derived)', () => { + const body = draft({ kind: 'datatable', path: 'main/dt' }, [ + { kind: 'asset', ref: 'datatable://main/dt' } + ]) + expect(body).toContain('on datatable://main/dt') + }) + + it('keeps // on for a native trigger', () => { + const body = draft(undefined, [{ kind: 'schedule', path: undefined }]) + expect(body).toContain('on schedule') + }) + + it('keeps // on when the body does not read the input (postgres, bash)', () => { + // postgres/bash bodies ignore `input`, so dropping `// on` would leave the + // script with neither an explicit trigger nor an inferred read → no cascade. + const pg = draft( + { kind: 'ducklake', path: 'main/orders' }, + [{ kind: 'asset', ref: 'ducklake://main/orders' }], + 'postgresql', + 'datatable' + ) + expect(pg).toContain('on ducklake://main/orders') + + const bash = draft( + { kind: 's3object', path: 'raw/events.parquet' }, + [{ kind: 'asset', ref: 's3://raw/events.parquet' }], + 'bash', + 's3_object' + ) + expect(bash).toContain('on s3://raw/events.parquet') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts index 44e501ac3e..02607d5f35 100644 --- a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts @@ -3,14 +3,24 @@ import { random_adj } from '$lib/components/random_positive_adjetive' import { parseDbInputFromAssetSyntax } from '$lib/utils' // What kind of asset the new script will produce. Drives the random output -// path scheme and the body skeleton. The output asset is NOT declared in a -// comment annotation — it's reconstructed from the body's SDK calls / SQL by -// the asset parser, same as production scripts, so it can't go stale. +// path scheme and the body skeleton. For most kinds the output asset is NOT +// declared in a comment annotation — it's reconstructed from the body's SDK +// calls / SQL by the asset parser, same as production scripts, so it can't go +// stale. The exception is `materialize`, which declares its target explicitly +// via the `// materialize` annotation (the runtime generates the write). // // `none` is the conservative default — body just has a "fill in" comment. The // other kinds inject their respective wmill SDK calls / SQL setup so the // script is runnable (modulo schema definition) the moment it's created. -export type PipelineOutputKind = 'none' | 'datatable' | 'ducklake' | 's3_parquet' | 's3_object' +export type PipelineOutputKind = + | 'none' + | 'datatable' + | 'ducklake' + | 'materialize' + | 'data_test' + | 's3_parquet' + | 's3_object' + | 'macros' export type PipelineOutputKindMeta = { id: PipelineOutputKind @@ -23,6 +33,16 @@ export type PipelineOutputKindMeta = { // object are the escape hatches for arbitrary blobs; none is last because // picking it disables the whole "auto-generated output" feature. export const PIPELINE_OUTPUT_KINDS: PipelineOutputKindMeta[] = [ + { + id: 'materialize', + label: 'Materialized table', + description: 'Managed DuckLake table — idempotent, versioned, tracked' + }, + { + id: 'data_test', + label: 'Data test', + description: 'Custom assertion — a single SELECT returning offending rows (empty = pass)' + }, { id: 'datatable', label: 'Data table', @@ -31,7 +51,7 @@ export const PIPELINE_OUTPUT_KINDS: PipelineOutputKindMeta[] = [ { id: 'ducklake', label: 'Ducklake', - description: 'DuckDB lakehouse table' + description: 'DuckDB lakehouse table (raw write)' }, { id: 's3_parquet', @@ -43,6 +63,11 @@ export const PIPELINE_OUTPUT_KINDS: PipelineOutputKindMeta[] = [ label: 'S3 Object', description: 'Generic file (JSON/CSV/binary)' }, + { + id: 'macros', + label: 'Macro library', + description: 'Reusable DuckDB macros, callable from every script in the workspace' + }, { id: 'none', label: 'No output', @@ -60,7 +85,20 @@ const LANG_COMPATIBILITY: Record = { bun: ['datatable', 'ducklake', 's3_parquet', 's3_object', 'none'], deno: ['datatable', 'ducklake', 's3_parquet', 's3_object', 'none'], python3: ['datatable', 'ducklake', 's3_parquet', 's3_object', 'none'], - duckdb: ['datatable', 'ducklake', 's3_parquet', 's3_object', 'none'], + // `materialize` is DuckDB-only: it generates the managed write around a + // single SELECT. The Python/TS `wmll.ducklake` helper currently takes a SQL + // SELECT (not in-memory rows), so a polyglot managed materialize is a + // separate follow-up — those langs keep the `ducklake` raw-write kind. + duckdb: [ + 'materialize', + 'data_test', + 'datatable', + 'ducklake', + 's3_parquet', + 's3_object', + 'macros', + 'none' + ], postgresql: ['datatable', 'none'], mysql: ['none'], mssql: ['none'], @@ -135,17 +173,18 @@ export function autoOutputAsset( case 'datatable': return { kind: 'datatable', path: `main/${adj}_${pick(TABLE_NOUNS)}_${slug}` } case 'ducklake': + case 'materialize': return { kind: 'ducklake', path: `main/${adj}_${pick(TABLE_NOUNS)}_${slug}` } - // s3 paths carry the canonical leading slash of a default-storage - // object (`s3:///` parses to path `/`). The deploy-time - // parser stores writes in that form — a slashless seeded path would - // never match it, and the post-deploy drift check would report the - // output as lost (it isn't; the key differs by one '/'). Bodies that - // take a bare key (`{ s3: ... }`) strip the slash via `s3Key`. + // s3 outputs use the canonical slashless key. `parse_asset_syntax` + // normalizes `s3:///` (default storage) and `s3://` to the + // bare ``, so the seeded draft asset must be slashless to match the + // deploy-time inferred identity — otherwise the post-deploy drift check + // would flag the output as a phantom `/`-prefixed node. The generated + // bodies still emit the `s3:///` default-storage URI for runtime I/O. case 's3_parquet': return { kind: 's3object', - path: `/pipelines/${folder}/${adj}_${pick(DATASET_NOUNS)}_${slug}.parquet` + path: `pipelines/${folder}/${adj}_${pick(DATASET_NOUNS)}_${slug}.parquet` } case 's3_object': { // duckdb's natural output for a generic blob is CSV (one COPY TO @@ -155,9 +194,14 @@ export function autoOutputAsset( const ext = language === 'duckdb' ? 'csv' : 'json' return { kind: 's3object', - path: `/pipelines/${folder}/${adj}_${pick(FILE_NOUNS)}_${slug}.${ext}` + path: `pipelines/${folder}/${adj}_${pick(FILE_NOUNS)}_${slug}.${ext}` } } + // A macro library produces no asset — its "output" is the registry + // entries the deploy records. A custom data test produces no asset + // either — it asserts against an existing materialized target. + case 'macros': + case 'data_test': case 'none': return undefined } @@ -178,10 +222,11 @@ export function assetUri(asset: { kind: AssetKind; path: string }): string { return `${ASSET_URI_PREFIX[asset.kind]}${asset.path}` } -// Bare object key for the SDK's `{ s3: }` forms: the canonical asset -// path of a default-storage object has a leading slash, the key must not. +// Bare object key for the SDK's `{ s3: }` / `s3:///` forms. Asset +// paths are already canonical slashless keys; strip stray leading slashes +// defensively so the emitted key never starts with '/'. function s3Key(path: string): string { - return path.replace(/^\//, '') + return path.replace(/^\/+/, '') } // Splits a datatable asset path (`/
` or `/.
`) @@ -286,26 +331,73 @@ export type TemplateContext = { // Header: `// pipeline` + every trigger source as its own annotation line. // Output asset is NOT declared here — it's reconstructed from the body's // SDK calls / SQL by the asset parser, same as production scripts. -function header(language: ScriptLang, triggers: DraftTriggerSource[]): string { +function header(ctx: TemplateContext): string { + const { language, triggers, output, outputKind, input } = ctx const p = commentPrefix(language) - const lines = triggers.map((t) => { + // A custom data test is a standalone script, not a graph node that produces + // an asset — so it gets no `// pipeline` / output annotation. Instead, tell + // the author how to wire it up (the `data_test` reference) and the two rules + // that aren't obvious: single SELECT, returning the offending rows. + if (outputKind === 'data_test') { + return [ + `${p} Custom data test — reference it from a materialize script with \`${p} data_test \`.`, + `${p} It must be a single SELECT returning the offending rows; the run fails if any row comes back.`, + '' + ].join('\n') + } + // A ducklake/s3 read the body performs auto-wires its cascade edge from the + // FROM clause, so an explicit `// on` for that same asset would be redundant + // now that auto-derivation is the default. Only drop it when the generated + // body actually READS the input: bun/deno/python/duckdb emit an input load, + // but postgres/bash/generic bodies (and `data_upload`, which reads the picker + // `file` instead) do not — dropping there would leave the script with no + // cascade at all. Also keep `// on` for kinds inference can't derive + // (datatable/resource/…) and native triggers. + const bodyReadsInput = READS_INPUT_LANGS.has(language) && !isDataUpload(triggers) + const inputRef = input ? assetUri(input) : undefined + const inputAutoDerives = input?.kind === 'ducklake' || input?.kind === 's3object' + const lines = triggers.flatMap((t) => { switch (t.kind) { case 'asset': - return `${p} on ${t.ref}` + if (bodyReadsInput && inputAutoDerives && t.ref === inputRef) return [] + return [`${p} on ${t.ref}`] default: // Native triggers (incl. schedule): marker-only — the // binding lives on the trigger row's own `script_path`. - return `${p} on ${t.kind}` + return [`${p} on ${t.kind}`] } }) - // Discoverability hint — the three annotations users most often miss - // when authoring their first pipeline script. Single line, real - // example values (not placeholders) so users see the syntax. Docs - // link is the canonical reference once they want the details. A blank - // line separates it from the parsed annotations above (`// pipeline`, - // `// on …`) so the editor reads as "real annotations, then a hint". - const more = `${p} More: partitioned daily, freshness 1h, retry 3, tag heavy — https://www.windmill.dev/docs/pipelines/annotations` - return [`${p} pipeline`, ...lines, '', more, ''].join('\n') + // Managed materialization is the one kind that declares its output + // explicitly — the runtime generates the write around the body's SELECT, so + // the target can't be inferred from the body. Emit `// materialize ` + // plus a hint about the strategy options that go on the same line. The hint + // must NOT start with a parser keyword (`materialize`, `on`, …) or it would + // be read as an annotation — `Strategy:` is safe. + const matLine = + outputKind === 'materialize' && output + ? [ + `${p} materialize ${assetUri(output)}`, + `${p} Strategy: add key=to merge (upsert), or append for insert-only; default replaces the partition` + ] + : [] + // Macro library: the `// macros` marker registers every CREATE MACRO below + // into the workspace registry at deploy. The hint must not start with a + // parser keyword — `Consumers` is safe. + const macrosLine = + outputKind === 'macros' + ? [ + `${p} macros`, + `${p} Consumers just call these by name; add \`${p} use \` in a consumer to force-inject the whole library` + ] + : [] + // Discoverability hint — a single pointer to the docs rather than a dump of + // every optional annotation. New users found the old feature list + // (mute/partitioned/freshness/retry/tag on one line) overwhelming; the docs + // are the canonical reference once they want any of it. A blank line + // separates it from the parsed annotations above (`// pipeline`, `// on …`) + // so the editor reads as "real annotations, then a hint". + const more = `${p} Optional: partitioning, freshness, retries & cascade control — https://www.windmill.dev/docs/core_concepts/pipelines` + return [`${p} pipeline`, ...lines, ...matLine, ...macrosLine, '', more, ''].join('\n') } // Bun / Deno bodies. These share the wmill SDK surface, so we treat them @@ -322,6 +414,11 @@ function isDataUpload(triggers: DraftTriggerSource[]): boolean { return triggers.some((t) => t.kind === 'data_upload') } +// Languages whose generated body reads the `input` asset (an SDK load / SQL +// FROM), so a ducklake/s3 input auto-derives its cascade and the explicit +// `// on` can be dropped. postgres/bash/generic bodies ignore `input`. +const READS_INPUT_LANGS: ReadonlySet = new Set(['bun', 'deno', 'python3', 'duckdb']) + function bodyTs(ctx: TemplateContext): string { const { input, output, outputKind } = ctx const dataUpload = isDataUpload(ctx.triggers) @@ -340,28 +437,28 @@ function bodyTs(ctx: TemplateContext): string { if (!input) return '' switch (input.kind) { case 's3object': + // `s3:///` URI — one spelling shared with the `// on + // s3:///…` annotation form (the object literal `{ s3: }` + // is equivalent). return [ - ` // Upstream: ${assetUri(input)}`, - ` const buf = await wmill.loadS3File({ s3: ${JSON.stringify(s3Key(input.path))} })`, + ` const buf = await wmill.loadS3File(${JSON.stringify(`s3:///${s3Key(input.path)}`)})`, ` const rows = JSON.parse(new TextDecoder().decode(buf))`, `` ].join('\n') case 'datatable': return [ - ` // Upstream: ${assetUri(input)}`, ` const src = wmill.datatable(${JSON.stringify(input.path.split('/')[0] ?? 'main')})`, ` const rows = await src\`SELECT * FROM ${input.path.split('/').slice(1).join('_') || 'table_name'}\`.fetch()`, `` ].join('\n') case 'ducklake': return [ - ` // Upstream: ${assetUri(input)}`, ` const lake = wmill.ducklake(${JSON.stringify(input.path.split('/')[0] ?? 'main')})`, ` const rows = await lake\`SELECT * FROM ${input.path.split('/').slice(1).join('_') || 'table_name'}\`.fetch()`, `` ].join('\n') default: - return ` // Upstream: ${assetUri(input)}\n` + return '' } })() @@ -370,9 +467,10 @@ function bodyTs(ctx: TemplateContext): string { switch (outputKind) { case 's3_parquet': case 's3_object': + // `s3:///` URI — see the loadS3File note above. return [ ` const payload = new TextEncoder().encode(JSON.stringify(rows))`, - ` await wmill.writeS3File({ s3: ${JSON.stringify(s3Key(output.path))} }, payload)` + ` await wmill.writeS3File(${JSON.stringify(`s3:///${s3Key(output.path)}`)}, payload)` ].join('\n') case 'datatable': { const dbName = output.path.split('/')[0] ?? 'main' @@ -426,25 +524,25 @@ function bodyPython(ctx: TemplateContext): string { if (!input) return '' switch (input.kind) { case 's3object': + // `s3:///` URI — SDK string params must be s3:// URIs + // (bare keys are rejected), and this form matches the + // `# on s3:///…` annotation spelling. return [ - ` # Upstream: ${assetUri(input)}`, - ` buf = wmill.load_s3_file(${JSON.stringify(s3Key(input.path))})`, + ` buf = wmill.load_s3_file(${JSON.stringify(`s3:///${s3Key(input.path)}`)})`, ` import json; rows = json.loads(buf.decode("utf-8"))` ].join('\n') case 'datatable': return [ - ` # Upstream: ${assetUri(input)}`, ` src = wmill.datatable(${JSON.stringify(input.path.split('/')[0] ?? 'main')})`, ` rows = src.query("SELECT * FROM ${input.path.split('/').slice(1).join('_') || 'table_name'}").fetch()` ].join('\n') case 'ducklake': return [ - ` # Upstream: ${assetUri(input)}`, ` lake = wmill.ducklake(${JSON.stringify(input.path.split('/')[0] ?? 'main')})`, ` rows = lake.query("SELECT * FROM ${input.path.split('/').slice(1).join('_') || 'table_name'}").fetch()` ].join('\n') default: - return ` # Upstream: ${assetUri(input)}` + return '' } })() @@ -453,9 +551,10 @@ function bodyPython(ctx: TemplateContext): string { switch (outputKind) { case 's3_parquet': case 's3_object': + // `s3:///` URI — see the load_s3_file note above. return [ ` import json`, - ` wmill.write_s3_file(${JSON.stringify(s3Key(output.path))}, json.dumps(rows).encode("utf-8"))` + ` wmill.write_s3_file(${JSON.stringify(`s3:///${s3Key(output.path)}`)}, json.dumps(rows).encode("utf-8"))` ].join('\n') case 'datatable': { const dbName = output.path.split('/')[0] ?? 'main' @@ -491,6 +590,22 @@ function bodyPython(ctx: TemplateContext): string { function bodyDuckdb(ctx: TemplateContext): string { const { input, output, outputKind } = ctx const dataUpload = isDataUpload(ctx.triggers) + if (outputKind === 'data_test') { + // Standalone custom data test: it reads ONLY the freshly-materialized + // target, which the runtime attaches under the internal `_wm_target` + // schema — so it emits no ATTACH / input load of its own. When the test + // was created off a ducklake asset, seed its table name; otherwise a + // clear placeholder. This exact shape (single SELECT vs `_wm_target`) is + // what the backend's self-teaching errors ask for. + const testTable = input?.kind === 'ducklake' ? catalogTableRef(input.path) : 'your_table' + return [ + '', + `-- Return the rows that VIOLATE your assertion; an empty result means the test passes.`, + `-- \`_wm_target\` is the freshly-materialized target, attached by the runtime.`, + `SELECT * FROM _wm_target.${testTable} WHERE your_condition;`, + '' + ].join('\n') + } const lines: string[] = [] if (dataUpload) { // `(s3object)` param declaration → the run form renders the S3 picker @@ -498,7 +613,6 @@ function bodyDuckdb(ctx: TemplateContext): string { lines.push(`-- $file (s3object)`) lines.push(`-- \`file\` is uploaded via the S3 picker on the run form.`) } - if (input) lines.push(`-- Upstream: ${assetUri(input)}`) lines.push('') // Resolve the catalog db name to ATTACH. Output's db wins when both sides @@ -532,13 +646,13 @@ function bodyDuckdb(ctx: TemplateContext): string { if (!input) return null switch (input.kind) { case 's3object': - return `read_parquet('s3://${input.path}')` + return `read_parquet('s3:///${input.path}')` case 'datatable': // `pg` is the attached Postgres catalog (see ATTACH above). // Use a 2-part `pg.
` ref so the asset parser maps it - // back to `datatable:///
` — matching the - // `// Upstream` annotation. Schema is only emitted if the - // asset path explicitly includes one (`main/myschema.mytable`). + // back to `datatable:///
` — matching the input asset. + // Schema is only emitted if the asset path explicitly includes + // one (`main/myschema.mytable`). return `pg.${catalogTableRef(input.path)}` case 'ducklake': return `lake.${catalogTableRef(input.path)}` @@ -555,7 +669,7 @@ function bodyDuckdb(ctx: TemplateContext): string { `COPY (`, ` SELECT *`, ` FROM ${fromExpr}`, - `) TO 's3://${output.path}' (FORMAT 'parquet');` + `) TO 's3:///${output.path}' (FORMAT 'parquet');` ) } break @@ -566,7 +680,7 @@ function bodyDuckdb(ctx: TemplateContext): string { `COPY (`, ` SELECT *`, ` FROM ${fromExpr}`, - `) TO 's3://${output.path}' (FORMAT 'csv', HEADER);` + `) TO 's3:///${output.path}' (FORMAT 'csv', HEADER);` ) } break @@ -579,6 +693,25 @@ function bodyDuckdb(ctx: TemplateContext): string { ) } break + case 'materialize': + // Managed materialization: the body is just the SELECT that produces + // the slice — the runtime wraps it into the idempotent write + + // snapshot (see the `// materialize` annotation in the header). No + // CREATE TABLE / INSERT, and the target is NOT attached here (the + // runtime attaches it). + // + // Partitioned? `{partition}` is the slice's identity string. For a + // time grain the runtime injects a `wm_partition(ts)` macro that + // buckets a timestamp with that same identity format, so one + // grain-agnostic filter line targets the active slice — no + // hand-written strftime format, no `= TIMESTAMP {partition}` cast + // (which errors for hourly/weekly/monthly). One commented line keeps + // the scaffold light; docs cover dynamic-grain keys. + lines.push( + `-- Partitioned? Filter the source to the active slice: WHERE wm_partition() = {partition}`, + `SELECT * FROM ${inSql ?? '(SELECT 1 AS placeholder)'};` + ) + break case 'datatable': if (output) { // 2-part `pg.
` so the asset parser resolves the @@ -594,6 +727,18 @@ function bodyDuckdb(ctx: TemplateContext): string { ) } break + case 'macros': + // Library body: only CREATE [OR REPLACE] MACRO statements (plus plain + // setup). One scalar + one table example; bodies may only call macros + // defined EARLIER in the file (DuckDB bind-checks at creation). + lines.push( + `CREATE OR REPLACE MACRO safe_div(a, b, fallback := 0) AS`, + ` CASE WHEN b = 0 THEN fallback ELSE a / b END;`, + ``, + `CREATE OR REPLACE MACRO sample_rows(src, n) AS TABLE`, + ` SELECT * FROM query_table(src) LIMIT n;` + ) + break case 'none': default: // With an uploaded file but no output asset, at least surface its @@ -605,9 +750,8 @@ function bodyDuckdb(ctx: TemplateContext): string { } function bodyPostgres(ctx: TemplateContext): string { - const { input, output, outputKind } = ctx + const { output, outputKind } = ctx const lines: string[] = [] - if (input) lines.push(`-- Upstream: ${assetUri(input)}`) lines.push('') if (outputKind === 'datatable' && output) { @@ -636,9 +780,8 @@ function bodyPostgres(ctx: TemplateContext): string { } function bodyBash(ctx: TemplateContext): string { - const { input, output, outputKind } = ctx + const { output, outputKind } = ctx const lines: string[] = [] - if (input) lines.push(`# Upstream: ${assetUri(input)}`) if (outputKind === 's3_object' && output) { lines.push( ``, @@ -655,7 +798,6 @@ function bodyBash(ctx: TemplateContext): string { function genericBody(ctx: TemplateContext): string { const p = commentPrefix(ctx.language) const lines: string[] = [] - if (ctx.input) lines.push(`${p} Upstream: ${assetUri(ctx.input)}`) lines.push(`${p} Fill in pipeline logic.`) return lines.join('\n') + '\n' } @@ -664,7 +806,7 @@ function genericBody(ctx: TemplateContext): string { // The returned content is ready to drop into a Script as `content` — no // further mutation needed, including for the trigger annotations. export function generatePipelineDraft(ctx: TemplateContext): string { - const head = header(ctx.language, ctx.triggers) + const head = header(ctx) const body = (() => { switch (ctx.language) { case 'bun': diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts index 78199a3871..e35054beab 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest' -import { resolveGraph, type ResolveGraphInput } from './resolveGraph' +import { resolveGraph, computeMutedReadKeys, type ResolveGraphInput } from './resolveGraph' import type { PipelineAnnotations } from './parsePipelineAnnotations' import type { AssetGraphResponse } from './types' import type { AssetWithAltAccessType } from '$lib/components/assets/lib' @@ -8,6 +8,12 @@ const ann = (over: Partial = {}): PipelineAnnotations => ({ inPipeline: false, triggerAssets: [], nativeTriggers: [], + dataTests: [], + columnLineage: [], + macros: false, + useLibs: [], + muteAssets: [], + muteAll: false, ...over }) @@ -33,6 +39,15 @@ const input = (over: Partial = {}): ResolveGraphInput => ({ const s3 = (path: string, access_type: 'r' | 'w' | 'rw'): AssetWithAltAccessType => ({ kind: 's3object', path, access_type }) as AssetWithAltAccessType +const duck = (path: string, access_type: 'r' | 'w' | 'rw'): AssetWithAltAccessType => + ({ kind: 'ducklake', path, access_type }) as AssetWithAltAccessType + +/** kind:path of the unsaved asset-trigger overlays for a runnable. */ +const assetTrigKeys = (r: AssetGraphResponse, path: string): string[] => + r.triggers + .filter((t) => t.trigger_kind === 'asset' && t.runnable_path === path && (t as any).unsaved) + .map((t) => `${(t as any).asset_kind}:${(t as any).asset_path}`) + describe('resolveGraph', () => { it('passes an empty graph through unchanged', () => { const r = resolveGraph(input()) @@ -65,11 +80,14 @@ describe('resolveGraph', () => { expect(resolveGraph(input({ base }))).toEqual(base) }) - it('draft: adds an unsaved runnable + write edge from the static outputAsset', () => { + it('draft: adds an unsaved runnable + write edge from outputAssets', () => { const drafts = new Map([ [ 'f/x/d', - { script: { content: '' }, outputAsset: { kind: 's3object' as const, path: '/out.json' } } + { + script: { content: '' }, + outputAssets: [{ kind: 's3object' as const, path: '/out.json' }] + } ] ]) const r = resolveGraph(input({ drafts })) @@ -92,20 +110,48 @@ describe('resolveGraph', () => { }) }) - it('draft: outputAssets snapshot wins over the static outputAsset', () => { + it('scd2 materialize draft: writes both the base dim and its _current companion view', () => { const drafts = new Map([ [ - 'f/x/d', + 'f/x/dim', { - script: { content: '' }, - outputAsset: { kind: 's3object' as const, path: '/old.json' }, - outputAssets: [{ kind: 's3object' as const, path: '/new.json' }] + script: { + content: '-- materialize ducklake://main/dim_customers key=id history\nselect 1' + } } ] ]) const r = resolveGraph(input({ drafts })) - expect(r.edges.map((e) => e.asset_path)).toContain('/new.json') - expect(r.edges.map((e) => e.asset_path)).not.toContain('/old.json') + // Base dimension: a plain write output node. + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/dim_customers' }) + // Companion `_current` view: same producer, marked derived from the base. + expect(r.assets).toContainEqual({ + kind: 'ducklake', + path: 'main/dim_customers_current', + derived_from: 'main/dim_customers' + }) + for (const path of ['main/dim_customers', 'main/dim_customers_current']) { + expect(r.edges).toContainEqual({ + runnable_path: 'f/x/dim', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: path, + access_type: 'w', + unsaved: true + }) + } + }) + + it('non-scd2 materialize draft: writes only the base dim, no _current companion', () => { + const drafts = new Map([ + [ + 'f/x/dim', + { script: { content: '-- materialize ducklake://main/dim_customers key=id\nselect 1' } } + ] + ]) + const r = resolveGraph(input({ drafts })) + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/dim_customers' }) + expect(r.assets.some((a) => a.path === 'main/dim_customers_current')).toBe(false) }) it('active draft: live body writes are authoritative over the snapshot', () => { @@ -189,6 +235,85 @@ describe('resolveGraph', () => { expect(r.edges.find((e) => e.asset_path === '/should-not.json')).toBeUndefined() }) + it('open buffer: a ducklake/s3 read auto-derives an unsaved cascade trigger', () => { + const liveBodyAssets = { + scriptPath: 'f/x/open', + assets: [duck('main.orders', 'r'), s3('raw/events', 'r'), duck('main.out', 'w')] + } + const liveAnnotations = { scriptPath: 'f/x/open', annotations: ann({ inPipeline: true }) } + const r = resolveGraph(input({ liveBodyAssets, liveAnnotations })) + // The two reads derive edges; the write (main.out) does not (self-edge). + expect(assetTrigKeys(r, 'f/x/open').sort()).toEqual([ + 'ducklake:main.orders', + 's3object:raw/events' + ]) + }) + + it('open buffer: a materialize producer reading its own target does not self-cascade', () => { + // The body `SELECT`s from the table it materializes (incremental model). + // The write is annotation-declared, not in the body, so without excluding + // the materialize target the read would auto-derive a self-cascade edge. + const liveBodyAssets = { scriptPath: 'f/x/p', assets: [duck('main.orders', 'r')] } + const liveAnnotations = { + scriptPath: 'f/x/p', + annotations: ann({ + inPipeline: true, + materialize: { targetKind: 'ducklake' as const, targetPath: 'main.orders' } + }) + } + const r = resolveGraph(input({ liveBodyAssets, liveAnnotations })) + expect(assetTrigKeys(r, 'f/x/p')).toEqual([]) + }) + + it('open buffer: a read that is also written (rw) does not self-trigger', () => { + const liveBodyAssets = { scriptPath: 'f/x/open', assets: [duck('main.self', 'rw')] } + const liveAnnotations = { scriptPath: 'f/x/open', annotations: ann({ inPipeline: true }) } + const r = resolveGraph(input({ liveBodyAssets, liveAnnotations })) + expect(assetTrigKeys(r, 'f/x/open')).toEqual([]) + }) + + it('open buffer: // mute suppresses one derived edge, // mute all suppresses all', () => { + const liveBodyAssets = { + scriptPath: 'f/x/open', + assets: [duck('main.a', 'r'), duck('main.b', 'r')] + } + const muted = resolveGraph( + input({ + liveBodyAssets, + liveAnnotations: { + scriptPath: 'f/x/open', + annotations: ann({ inPipeline: true, muteAssets: [{ kind: 'ducklake', path: 'main.a' }] }) + } + }) + ) + expect(assetTrigKeys(muted, 'f/x/open')).toEqual(['ducklake:main.b']) + + const all = resolveGraph( + input({ + liveBodyAssets, + liveAnnotations: { + scriptPath: 'f/x/open', + annotations: ann({ inPipeline: true, muteAll: true }) + } + }) + ) + expect(assetTrigKeys(all, 'f/x/open')).toEqual([]) + }) + + it('open buffer: an explicit // on is not double-emitted as a derived edge', () => { + const liveBodyAssets = { scriptPath: 'f/x/open', assets: [duck('main.orders', 'r')] } + const liveAnnotations = { + scriptPath: 'f/x/open', + annotations: ann({ + inPipeline: true, + triggerAssets: [{ kind: 'ducklake', path: 'main.orders' }] + }) + } + const r = resolveGraph(input({ liveBodyAssets, liveAnnotations })) + // Exactly one overlay for the table, not one explicit + one derived. + expect(assetTrigKeys(r, 'f/x/open')).toEqual(['ducklake:main.orders']) + }) + it('open-script live annotations add unsaved triggers, deduped vs persisted', () => { const base = baseGraph({ triggers: [ @@ -350,6 +475,55 @@ describe('resolveGraph', () => { expect(r.edges.some((e) => e.asset_path === 'main/out' && e.access_type === 'w')).toBe(true) }) + it('editing a saved scd2 producer keeps both the base and _current persisted write edges', () => { + // Deploy persists a write to both `main/dim` and `main/dim_current`. + // Opening the producer for editing must not judge the companion `_current` + // write stale — otherwise a consumer of only the view orphans mid-edit. + const base = baseGraph({ + runnables: [{ path: 'f/x/dim', usage_kind: 'script' }], + edges: [ + { + runnable_path: 'f/x/dim', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/dim_customers', + access_type: 'w' + }, + { + runnable_path: 'f/x/dim', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/dim_customers_current', + access_type: 'w' + } + ] + }) + const r = resolveGraph( + input({ + base, + liveAnnotations: { + scriptPath: 'f/x/dim', + annotations: ann({ + materialize: { + targetKind: 'ducklake', + targetPath: 'main/dim_customers', + uniqueKey: 'id', + scd2: true + } + }) + }, + liveBodyAssets: { scriptPath: 'f/x/dim', assets: [] } + }) + ) + for (const path of ['main/dim_customers', 'main/dim_customers_current']) { + expect( + r.edges.some( + (e) => e.runnable_path === 'f/x/dim' && e.asset_path === path && e.access_type === 'w' + ) + ).toBe(true) + } + }) + it('selecting a saved script unchanged drops nothing (no stale removal)', () => { const base = baseGraph({ runnables: [{ path: 'f/x/prod', usage_kind: 'script' }], @@ -464,4 +638,389 @@ describe('resolveGraph', () => { missing: true }) }) + + it('macro edges: base passes through; live `// use` adds an unsaved via_use edge', () => { + const base = baseGraph({ + runnables: [ + { + path: 'f/lib/stats', + usage_kind: 'script', + macros: [{ name: 'safe_div', params: 'a, b', is_table: false }] + }, + { path: 'f/x/cons', usage_kind: 'script' } + ], + macro_edges: [ + { + lib_path: 'f/lib/stats', + consumer_path: 'f/x/cons', + macro_names: ['safe_div'], + via_use: false + } + ] + }) + const r = resolveGraph( + input({ + base, + liveAnnotations: { + scriptPath: 'f/x/other', + annotations: ann({ useLibs: ['f/lib/stats'] }) + } + }) + ) + // Detection edge preserved untouched. + expect(r.macro_edges).toContainEqual({ + lib_path: 'f/lib/stats', + consumer_path: 'f/x/cons', + macro_names: ['safe_div'], + via_use: false + }) + // Live `// use` synthesizes an unsaved whole-lib edge with the lib's names. + expect(r.macro_edges).toContainEqual({ + lib_path: 'f/lib/stats', + consumer_path: 'f/x/other', + macro_names: ['safe_div'], + via_use: true, + unsaved: true + }) + }) + + it('macro edges: removing the `// use` line of an overlaid consumer retires its via_use edge', () => { + const base = baseGraph({ + macro_edges: [ + { + lib_path: 'f/lib/stats', + consumer_path: 'f/x/cons', + macro_names: ['safe_div'], + via_use: true + } + ] + }) + const r = resolveGraph( + input({ + base, + liveAnnotations: { scriptPath: 'f/x/cons', annotations: ann() } + }) + ) + expect(r.macro_edges).toEqual([]) + }) + + it('macro edges: draft `// macros` library gets the ƒ badge data from its body', () => { + const drafts = new Map([ + [ + 'f/lib/new', + { + script: { + content: '// macros\nCREATE OR REPLACE MACRO dbl(a) AS a * 2;' + } + } + ] + ]) + const r = resolveGraph(input({ drafts })) + const lib = r.runnables.find((x) => x.path === 'f/lib/new') + expect(lib?.macros).toEqual([{ name: 'dbl', params: 'a', is_table: false }]) + }) +}) + +describe('computeMutedReadKeys', () => { + const readEdge = (asset_kind: any, asset_path: string, access: any = 'r') => ({ + runnable_path: 'f/x/c', + runnable_kind: 'script' as const, + asset_kind, + asset_path, + access_type: access + }) + const assetTrigger = (asset_kind: any, asset_path: string) => ({ + trigger_kind: 'asset' as const, + asset_kind, + asset_path, + runnable_kind: 'script' as const, + runnable_path: 'f/x/c' + }) + // `f/x/c` (the read consumer) is an in-pipeline script. + const pipelineRunnable = [{ path: 'f/x/c', usage_kind: 'script' as const, in_pipeline: true }] + + it('flags a ducklake/s3 read with no cascade trigger as muted', () => { + const muted = computeMutedReadKeys( + [readEdge('ducklake', 'main.orders'), readEdge('s3object', 'raw/events')], + [], + pipelineRunnable + ) + expect([...muted].sort()).toEqual([ + 'ducklake:main.orders->script:f/x/c', + 's3object:raw/events->script:f/x/c' + ]) + }) + + it('does not flag a read that has a cascade trigger', () => { + const muted = computeMutedReadKeys( + [readEdge('ducklake', 'main.orders')], + [assetTrigger('ducklake', 'main.orders')], + pipelineRunnable + ) + expect(muted.size).toBe(0) + }) + + it('ignores rw self-reads and unsupported kinds', () => { + const muted = computeMutedReadKeys( + [ + readEdge('ducklake', 'main.self', 'rw'), // self-read, not muted + readEdge('resource', 'f/db'), // out of scope + readEdge('datatable', 'main.dt') // out of scope + ], + [], + pipelineRunnable + ) + expect(muted.size).toBe(0) + }) + + it('does not flag a read whose script also writes the asset', () => { + // Live-overlay shape of a `// materialize` producer reading its own target: + // a separate `'r'` read edge and `'w'` write edge for the same asset. It's + // the script's own output, not a suppressed input, so it is not muted. + const muted = computeMutedReadKeys( + [readEdge('ducklake', 'main.orders', 'r'), readEdge('ducklake', 'main.orders', 'w')], + [], + pipelineRunnable + ) + expect(muted.size).toBe(0) + }) + + it('does not flag a read by a non-pipeline script (auto-derivation never applied)', () => { + // Same read, but the consumer is a plain script (or a flow), not a + // `// pipeline` member — no auto trigger was ever derived to suppress. + const plainScript = [{ path: 'f/x/c', usage_kind: 'script' as const, in_pipeline: false }] + expect(computeMutedReadKeys([readEdge('ducklake', 'main.orders')], [], plainScript).size).toBe( + 0 + ) + const flow = [{ path: 'f/x/c', usage_kind: 'flow' as const, in_pipeline: true }] + expect(computeMutedReadKeys([readEdge('ducklake', 'main.orders')], [], flow).size).toBe(0) + }) +}) + +describe('live buffer overlays (open script)', () => { + // A deployed producer: `f/x/prod` materializes ducklake main/orders. + const deployedProducer = () => + baseGraph({ + assets: [{ kind: 'ducklake', path: 'main/orders' }], + runnables: [{ path: 'f/x/prod', usage_kind: 'script', in_pipeline: true }], + edges: [ + { + runnable_path: 'f/x/prod', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/orders', + access_type: 'w' + } + ] + }) + + it('open draft: live annotations win over the stale draft snapshot for the materialize target', () => { + const drafts = new Map([ + ['f/x/d', { script: { content: '-- materialize ducklake://main/old_target\nselect 1' } }] + ]) + const r = resolveGraph( + input({ + drafts, + liveBodyAssets: { scriptPath: 'f/x/d', assets: [] }, + liveAnnotations: { + scriptPath: 'f/x/d', + annotations: ann({ + inPipeline: true, + materialize: { targetKind: 'ducklake', targetPath: 'main/new_target' } + }) + } + }) + ) + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/new_target' }) + expect(r.assets).not.toContainEqual({ kind: 'ducklake', path: 'main/old_target' }) + expect(r.edges).toContainEqual({ + runnable_path: 'f/x/d', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/new_target', + access_type: 'w', + unsaved: true + }) + }) + + it('open saved script: retargeting `// materialize` swaps the write edge live', () => { + const r = resolveGraph( + input({ + base: deployedProducer(), + liveBodyAssets: { scriptPath: 'f/x/prod', assets: [] }, + liveAnnotations: { + scriptPath: 'f/x/prod', + annotations: ann({ + inPipeline: true, + materialize: { targetKind: 'ducklake', targetPath: 'main/orders_gold' } + }) + } + }) + ) + // New target surfaces as an unsaved write edge… + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/orders_gold' }) + expect(r.edges).toContainEqual({ + runnable_path: 'f/x/prod', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/orders_gold', + access_type: 'w', + unsaved: true + }) + // …the stale write edge is dropped, but the deployed dataset node stays. + expect( + r.edges.filter((e) => e.asset_path === 'main/orders' && e.runnable_path === 'f/x/prod') + ).toEqual([]) + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/orders' }) + }) + + it('open saved script: unchanged materialize target dedups against the persisted write edge', () => { + const r = resolveGraph( + input({ + base: deployedProducer(), + liveBodyAssets: { scriptPath: 'f/x/prod', assets: [] }, + liveAnnotations: { + scriptPath: 'f/x/prod', + annotations: ann({ + inPipeline: true, + materialize: { targetKind: 'ducklake', targetPath: 'main/orders' } + }) + } + }) + ) + expect( + r.edges.filter( + (e) => + e.runnable_path === 'f/x/prod' && + e.asset_path === 'main/orders' && + (e.access_type === 'w' || e.access_type === 'rw') + ) + ).toHaveLength(1) + }) + + it('open saved script: live body reads/writes overlay as unsaved lineage', () => { + const base = baseGraph({ + assets: [{ kind: 'ducklake', path: 'main/orders' }], + runnables: [{ path: 'f/x/cons', usage_kind: 'script', in_pipeline: true }], + edges: [ + { + runnable_path: 'f/x/cons', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/orders', + access_type: 'r' + } + ] + }) + const r = resolveGraph( + input({ + base, + liveBodyAssets: { + scriptPath: 'f/x/cons', + assets: [duck('main/orders_eu', 'r'), s3('/report.parquet', 'w')] + }, + liveAnnotations: { + scriptPath: 'f/x/cons', + annotations: ann({ inPipeline: true }) + } + }) + ) + // New read + write from the buffer… + expect(r.edges).toContainEqual({ + runnable_path: 'f/x/cons', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/orders_eu', + access_type: 'r', + unsaved: true + }) + expect(r.edges).toContainEqual({ + runnable_path: 'f/x/cons', + runnable_kind: 'script', + asset_kind: 's3object', + asset_path: '/report.parquet', + access_type: 'w', + unsaved: true + }) + // …the no-longer-read persisted edge is dropped. + expect( + r.edges.filter((e) => e.asset_path === 'main/orders' && e.runnable_path === 'f/x/cons') + ).toEqual([]) + }) +}) + +describe('inactive draft input lineage', () => { + it('keeps read edges + derived cascade for a deselected draft via inputAssets', () => { + const drafts = new Map([ + [ + 'f/x/cons', + { + script: { content: '-- pipeline\n-- materialize ducklake://main/agg\nselect 1' }, + outputAssets: [{ kind: 'ducklake' as const, path: 'main/agg' }], + inputAssets: [{ kind: 'ducklake' as const, path: 'main/src' }] + } + ] + ]) + // No live overlay: the draft is NOT the open script. + const r = resolveGraph(input({ drafts })) + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/src' }) + expect(r.edges).toContainEqual({ + runnable_path: 'f/x/cons', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/src', + access_type: 'r', + unsaved: true + }) + // Auto-derived cascade trigger from the captured read. + expect(assetTrigKeys(r, 'f/x/cons')).toContain('ducklake:main/src') + }) + + it('an empty inputAssets array is authoritative — no session-cache fallback', () => { + const drafts = new Map([ + [ + 'f/x/cons', + { + script: { content: '-- pipeline\nselect 1' }, + inputAssets: [] as Array<{ kind: 'ducklake'; path: string }> + } + ] + ]) + const inferredReadsByPath = new Map([ + ['f/x/cons', [{ kind: 'ducklake' as const, path: 'main/stale' }]] + ]) + const r = resolveGraph(input({ drafts, inferredReadsByPath })) + expect(r.assets).not.toContainEqual({ kind: 'ducklake', path: 'main/stale' }) + expect(r.edges.filter((e) => e.asset_path === 'main/stale')).toEqual([]) + }) +}) + +describe('open saved script: live overlay vs inferred-lineage maps', () => { + it('does not duplicate edges when the same live ref reaches both paths', () => { + // The route page mirrors the open pane's liveBodyAssets into + // inferredReads/WritesByPath, so the same refs arrive twice. + const base = baseGraph({ + runnables: [{ path: 'f/x/cons', usage_kind: 'script', in_pipeline: true }] + }) + const live = [duck('main/src', 'r'), s3('/out.parquet', 'w')] + const r = resolveGraph( + input({ + base, + liveBodyAssets: { scriptPath: 'f/x/cons', assets: live }, + liveAnnotations: { scriptPath: 'f/x/cons', annotations: ann({ inPipeline: true }) }, + inferredReadsByPath: new Map([ + ['f/x/cons', [{ kind: 'ducklake' as const, path: 'main/src' }]] + ]), + inferredWritesByPath: new Map([ + ['f/x/cons', [{ kind: 's3object' as const, path: '/out.parquet' }]] + ]) + }) + ) + expect( + r.edges.filter((e) => e.asset_path === 'main/src' && e.runnable_path === 'f/x/cons') + ).toHaveLength(1) + expect( + r.edges.filter((e) => e.asset_path === '/out.parquet' && e.runnable_path === 'f/x/cons') + ).toHaveLength(1) + }) }) diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts index 6e4f512000..8ee0ca53d7 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts @@ -1,5 +1,11 @@ -import type { AssetGraphResponse, NativeTriggerKind } from './types' -import { parsePipelineAnnotations, type PipelineAnnotations } from './parsePipelineAnnotations' +import type { AssetGraphMacroEdge, AssetGraphResponse, NativeTriggerKind } from './types' +import { + mergeColumnLineage, + parsePipelineAnnotations, + scd2CurrentTargetPath, + type ColumnLineage, + type PipelineAnnotations +} from './parsePipelineAnnotations' import { extractWrites, extractReads, @@ -10,8 +16,10 @@ import { /** Minimal structural shape of a pipeline draft `resolveGraph` needs. */ export type GraphDraft = { script: { content: string } - outputAsset?: { kind: AssetKind; path: string } outputAssets?: Array<{ kind: AssetKind; path: string }> + /** Reads captured on pane teardown. `undefined` = not captured (legacy + * bundle) → fall back to the session cache; `[]` = authoritative none. */ + inputAssets?: Array<{ kind: AssetKind; path: string }> } export type ResolveGraphInput = { @@ -20,7 +28,12 @@ export type ResolveGraphInput = { /** In-flight drafts keyed by script path. */ drafts: Map /** Body assets inferred for the currently-open script (live keystrokes). */ - liveBodyAssets: { scriptPath: string | undefined; assets: AssetWithAltAccessType[] } + liveBodyAssets: { + scriptPath: string | undefined + assets: AssetWithAltAccessType[] + /** Body-inferred column lineage (DuckDB SQL AST) for the open script. */ + columnLineage?: ColumnLineage[] + } /** Pipeline annotations parsed from the currently-open buffer. */ liveAnnotations: { scriptPath: string | undefined; annotations: PipelineAnnotations } /** Sticky session caches of inferred body writes/reads per script path. */ @@ -60,6 +73,108 @@ function persistedNativeKinds(base: AssetGraphResponse, path: string): Set = new Set(['ducklake', 's3object']) + +/** `kind:path` refs of a script's `// materialize` write target(s) (base + + * the scd2 `_current` companion), which the body `SELECT` doesn't express. */ +function materializeWriteRefs(parsed: PipelineAnnotations): string[] { + const m = parsed.materialize + if (!m) return [] + const refs = [`${m.targetKind}:${m.targetPath}`] + const current = scd2CurrentTargetPath(m) + if (current) refs.push(`${m.targetKind}:${current}`) + return refs +} + +/** + * Backend-mirror of `derive_pipeline_asset_trigger_refs` (windmill-common + * assets.rs): a pipeline script's ducklake/s3 read auto-wires a cascade + * trigger edge from the FROM clause, so `// on ` is only needed for + * edges inference can't see. Excluded: assets the script also writes (`writes` + * covers `w`/`rw`, so an `rw` self-read can't loop-trigger), muted assets, + * `// mute all`, and any explicit `// on` (which already emits its own edge). + * Returns the `{kind, path}` refs to overlay as unsaved asset triggers, deduped. + */ +function deriveAutoAssetTriggers( + reads: Array<{ kind: AssetKind; path: string }>, + writes: Array<{ kind: AssetKind; path: string }>, + parsed: PipelineAnnotations +): Array<{ kind: AssetKind; path: string }> { + // Auto-derivation is scoped to `// pipeline` scripts (backend gates on + // `in_pipeline`); `// mute all` opts a pipeline script back out. + if (!parsed.inPipeline || parsed.muteAll) return [] + const skip = new Set([ + ...writes.map((w) => `${w.kind}:${w.path}`), + // The `// materialize` target(s) are this script's own output — the body + // `SELECT` doesn't express the write, so an incremental model that reads + // its own target would otherwise auto-derive a self-cascade edge (backend + // parity: the deploy path upgrades the same read to `rw`). + ...materializeWriteRefs(parsed), + ...parsed.muteAssets.map((a) => `${a.kind}:${a.path}`), + ...parsed.triggerAssets.map((a) => `${a.kind}:${a.path}`) + ]) + const out: Array<{ kind: AssetKind; path: string }> = [] + for (const r of reads) { + const key = `${r.kind}:${r.path}` + if (!AUTO_TRIGGER_KINDS.has(r.kind) || skip.has(key)) continue + skip.add(key) // dedup within reads too + out.push({ kind: r.kind, path: r.path }) + } + return out +} + +/** Edge key matching a lineage/trigger edge's `(asset, runnable)` pair. */ +function edgeKey(a: { + asset_kind: string + asset_path: string + runnable_kind: string + runnable_path: string +}): string { + return `${a.asset_kind}:${a.asset_path}->${a.runnable_kind}:${a.runnable_path}` +} + +/** + * Keys of "muted read" edges: a ducklake/s3 asset a script reads read-*only* + * (`access_type === 'r'`) yet has NO cascade trigger for. Auto-derivation is the + * default, so a supported read with no trigger means the author opted it out + * with `// mute ` / `// mute all` (or it's an explicit non-triggering + * read). The canvas badges these — the exception — instead of every derived + * edge. Self-reads are excluded: a script that also writes the asset carries + * `'rw'` on the deployed graph (one edge) or a separate `'w'` edge in the live + * overlay (a `// materialize` producer reading its own target), and neither + * should badge as muted — it's the script's own output, not a suppressed input. + * + * Only `// pipeline` scripts are considered: auto-derivation never applies to a + * plain script or a flow, so a non-pipeline runnable reading a ducklake/s3 + * asset has no auto trigger to suppress and must render as ordinary lineage. + */ +export function computeMutedReadKeys( + edges: AssetGraphResponse['edges'], + triggers: AssetGraphResponse['triggers'], + runnables: AssetGraphResponse['runnables'] +): Set { + const pipelineScripts = new Set( + runnables.filter((r) => r.usage_kind === 'script' && r.in_pipeline).map((r) => r.path) + ) + const cascaded = new Set( + triggers.filter((t) => t.trigger_kind === 'asset').map((t) => edgeKey(t)) + ) + const written = new Set( + edges.filter((e) => e.access_type === 'w' || e.access_type === 'rw').map((e) => edgeKey(e)) + ) + const muted = new Set() + for (const e of edges) { + if (e.access_type !== 'r' || !AUTO_TRIGGER_KINDS.has(e.asset_kind)) continue + if (e.runnable_kind !== 'script' || !pipelineScripts.has(e.runnable_path)) continue + const key = edgeKey(e) + if (!cascaded.has(key) && !written.has(key)) muted.add(key) + } + return muted +} + /** `kind:path` keys of persisted asset (`// on `) triggers for `path`. */ function persistedAssetKeys(base: AssetGraphResponse, path: string): Set { return new Set( @@ -137,15 +252,94 @@ export function resolveGraph(input: ResolveGraphInput): AssetGraphResponse { return false return true }) + // Mirror the backend's skip-if-empty: no `macro_edges` key at all when + // there is nothing to show (also keeps the no-macros response shape + // byte-identical to before the feature). + const macroEdges = resolveMacroEdges(input) return { ...base, assets: acc.assets, runnables: acc.runnables, edges: acc.edges, - triggers: [...baseTriggers, ...acc.extraTriggers] + triggers: [...baseTriggers, ...acc.extraTriggers], + ...(macroEdges.length > 0 || base.macro_edges ? { macro_edges: macroEdges } : {}) } } +/** + * Macro-library → consumer edges: the deployed base edges, with `// use` + * declarations of overlaid scripts (drafts + the open buffer) taking over + * their consumer's `via_use` edges so adding/removing a `// use` line updates + * the canvas live. Detection-based edges (macro calls in the deployed body) + * are backend-owned and only refresh on redeploy. + */ +function resolveMacroEdges(input: ResolveGraphInput): AssetGraphMacroEdge[] { + const { base, drafts, liveAnnotations } = input + const libMacroNames = new Map() + for (const r of base.runnables) { + if (r.usage_kind === 'script' && r.macros?.length) { + libMacroNames.set( + r.path, + r.macros.map((m) => m.name) + ) + } + } + const useByPath = new Map() + for (const [path, d] of drafts) { + useByPath.set(path, parsePipelineAnnotations(d.script.content).useLibs) + } + if (liveAnnotations.scriptPath) { + // `?? []` — callers may hand a minimal annotations object (tests, older + // call sites) that predates the field. + useByPath.set(liveAnnotations.scriptPath, liveAnnotations.annotations.useLibs ?? []) + } + const out: AssetGraphMacroEdge[] = [] + for (const e of base.macro_edges ?? []) { + if (e.via_use && useByPath.has(e.consumer_path)) continue + out.push({ ...e }) + } + for (const [path, libs] of useByPath) { + for (const lib of libs) { + const existing = out.find((e) => e.lib_path === lib && e.consumer_path === path) + if (existing) { + // Upgrade the detection edge in place: `// use` pulls in the whole + // library, so the edge covers every macro the lib defines. + existing.via_use = true + existing.unsaved = true + existing.macro_names = [ + ...new Set([...existing.macro_names, ...(libMacroNames.get(lib) ?? [])]) + ] + } else { + out.push({ + lib_path: lib, + consumer_path: path, + macro_names: libMacroNames.get(lib) ?? [], + via_use: true, + unsaved: true + }) + } + } + } + return out +} + +// Light regex extraction of a draft macro library's definitions for the live +// node badge. The strict grammar lives in the Rust `parse_macro_library` at +// deploy; this only needs name/params/table-ness for display (nested parens +// in a default value may truncate the shown signature, never the deploy). +const MACRO_DEF_RE = + /create\s+(?:or\s+replace\s+)?(?:temp(?:orary)?\s+)?(?:macro|function)\s+([a-zA-Z_][a-zA-Z0-9_]*)\s*\(([^)]*)\)\s*as\s+(table\b)?/gi + +export function extractDraftMacros( + content: string +): { name: string; params: string; is_table: boolean }[] { + const out: { name: string; params: string; is_table: boolean }[] = [] + for (const m of content.matchAll(MACRO_DEF_RE)) { + out.push({ name: m[1].toLowerCase(), params: m[2].trim(), is_table: m[3] !== undefined }) + } + return out +} + type ResolveContext = { draftedPaths: Set isDrafted: (kind: string, p: string) => boolean @@ -173,6 +367,20 @@ function makeContext(input: ResolveGraphInput): ResolveContext { if (liveAnnotations.scriptPath === openPath) { for (const a of liveAnnotations.annotations.triggerAssets) liveRefKeys.add(`${a.kind}:${a.path}`) + // The `// materialize ` target is a declared *output*, but it + // lives in an annotation (not the SQL body), so neither triggerAssets + // (inputs) nor the body-inferred assets cover it. Without this its + // persisted write-edge is judged stale and dropped the moment the + // script is selected/edited — leaving the output asset unlinked. A + // managed scd2 producer persists a second write to the `_current` + // companion view, so keep that too or a consumer of only the view + // orphans while the producer is open for editing. + const m = liveAnnotations.annotations.materialize + if (m) { + liveRefKeys.add(`${m.targetKind}:${m.targetPath}`) + const currentPath = scd2CurrentTargetPath(m) + if (currentPath) liveRefKeys.add(`${m.targetKind}:${currentPath}`) + } } for (const a of liveBodyAssets.assets) liveRefKeys.add(`${a.kind}:${a.path}`) } @@ -203,6 +411,95 @@ function seedAccumulator(input: ResolveGraphInput, ctx: ResolveContext): Accumul } } +/** + * The write output(s) a `// materialize ` annotation declares: the + * target itself, plus — for a managed scd2 materialize — the `_current` + * companion view (mirrors the deploy path), so a consumer of only the view + * links back to this producer instead of orphaning. + */ +function materializeOuts( + parsed: PipelineAnnotations +): Array<{ kind: AssetKind; path: string; derivedFrom?: string }> { + if (!parsed.materialize) return [] + const outs: Array<{ kind: AssetKind; path: string; derivedFrom?: string }> = [ + { kind: parsed.materialize.targetKind, path: parsed.materialize.targetPath } + ] + const currentPath = scd2CurrentTargetPath(parsed.materialize) + if (currentPath) { + outs.push({ + kind: parsed.materialize.targetKind, + path: currentPath, + derivedFrom: parsed.materialize.targetPath + }) + } + return outs +} + +/** Add an output asset node + unsaved write edge for `runnablePath`, deduped + * against the assets/edges already accumulated (base survivors included). */ +function pushWriteOut( + acc: Accumulator, + runnablePath: string, + out: { kind: AssetKind; path: string; derivedFrom?: string } +) { + const existing = acc.assets.find((a) => a.kind === out.kind && a.path === out.path) + if (!existing) { + acc.assets.push({ + kind: out.kind, + path: out.path, + ...(out.derivedFrom ? { derived_from: out.derivedFrom } : {}) + }) + } else if (out.derivedFrom && existing.derived_from == undefined) { + existing.derived_from = out.derivedFrom + } + const hasWriteEdge = acc.edges.some( + (e) => + e.runnable_kind === 'script' && + e.runnable_path === runnablePath && + e.asset_kind === out.kind && + e.asset_path === out.path && + (e.access_type === 'w' || e.access_type === 'rw') + ) + if (hasWriteEdge) return + acc.edges.push({ + runnable_path: runnablePath, + runnable_kind: 'script', + asset_kind: out.kind, + asset_path: out.path, + access_type: 'w', + unsaved: true + }) +} + +/** Add an input asset node + unsaved read edge for `runnablePath`, deduped + * against the assets/edges already accumulated. */ +function pushReadIn( + acc: Accumulator, + runnablePath: string, + inp: { kind: AssetKind; path: string } +) { + if (!acc.assets.some((a) => a.kind === inp.kind && a.path === inp.path)) { + acc.assets.push({ kind: inp.kind, path: inp.path }) + } + const hasReadEdge = acc.edges.some( + (e) => + e.runnable_kind === 'script' && + e.runnable_path === runnablePath && + e.asset_kind === inp.kind && + e.asset_path === inp.path && + (e.access_type === 'r' || e.access_type === 'rw') + ) + if (hasReadEdge) return + acc.edges.push({ + runnable_path: runnablePath, + runnable_kind: 'script', + asset_kind: inp.kind, + asset_path: inp.path, + access_type: 'r', + unsaved: true + }) +} + /** * Every draft contributes: a runnable, output asset(s), a write edge, live * read lineage for the active draft, plus its seeded asset/native triggers @@ -210,11 +507,31 @@ function seedAccumulator(input: ResolveGraphInput, ctx: ResolveContext): Accumul * `drafts` map so multiple concurrent drafts all render at once. */ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { - const { base, drafts, liveBodyAssets } = input - const { runnables, assets, edges, extraTriggers } = acc + const { base, drafts, liveBodyAssets, inferredReadsByPath, inferredWritesByPath } = input + const { runnables, assets, extraTriggers } = acc for (const [path, d] of drafts) { - const parsed = parsePipelineAnnotations(d.script.content) + // The open draft's annotations come from the live buffer, not the draft + // snapshot — the snapshot only syncs on pane teardown, so parsing it here + // would pin annotation-derived outputs (`// materialize` target, badges) + // to their pre-edit values until the user clicks away. + const parsed = + path === input.liveAnnotations.scriptPath + ? input.liveAnnotations.annotations + : parsePipelineAnnotations(d.script.content) + // For the open script, fold in the WASM-inferred column lineage (DuckDB + // SQL AST) under the same annotation-wins precedence the backend applies + // on deploy, so the live preview matches what deploys. Only the open + // script carries live inference (`liveBodyAssets`); other drafts stay + // annotation-only until they deploy (the backend infers then). + const inferredCL = + path === liveBodyAssets.scriptPath ? (liveBodyAssets.columnLineage ?? []) : [] + const mergedCL = mergeColumnLineage(inferredCL, parsed.columnLineage) + // The `// materialize` target this draft's column lineage describes, so + // the column graph anchors to it rather than guessing a write-edge. + const materializeTarget = parsed.materialize + ? { kind: parsed.materialize.targetKind, path: parsed.materialize.targetPath } + : undefined // A draft can coexist with a base entry — during save the refetch // lands before drafts cleanup, and a user re-editing a deployed // script also produces both. In that case we mutate the existing @@ -222,6 +539,9 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { // duplicate (which would crash svelte-flow's keyed each), so the // canvas + trigger-node labels reflect that there's pending body // editing for this path. + // `// macros` library draft: extract the definitions for the live node + // badge (regex-light; the strict parse happens at deploy). + const draftMacros = parsed.macros ? extractDraftMacros(d.script.content) : [] const baseIdx = runnables.findIndex((r) => r.usage_kind === 'script' && r.path === path) if (baseIdx === -1) { runnables.push({ @@ -232,87 +552,58 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { freshness: parsed.freshness?.duration, tag: parsed.tag, retry: parsed.retry, + data_tests: parsed.dataTests.length > 0 ? parsed.dataTests : undefined, + column_lineage: mergedCL.length > 0 ? mergedCL : undefined, + materialize_target: materializeTarget, + macros: draftMacros.length > 0 ? draftMacros : undefined, unsaved: true }) } else { - runnables[baseIdx] = { ...runnables[baseIdx], unsaved: true } + // Refresh annotation-derived badges from the live parse too, so + // adding/removing `// data_test` / `// column` lines on an + // already-deployed script updates the badge immediately (not only + // after redeploy/refetch). + runnables[baseIdx] = { + ...runnables[baseIdx], + data_tests: parsed.dataTests.length > 0 ? parsed.dataTests : undefined, + column_lineage: mergedCL.length > 0 ? mergedCL : undefined, + materialize_target: materializeTarget, + macros: draftMacros.length > 0 ? draftMacros : undefined, + unsaved: true + } } - // Output asset(s): three-tier resolution. + // Output asset(s): two-tier resolution. // 1. Active draft (the body the user is editing right now): // live body inference is authoritative — renaming a // CREATE TABLE target or writeS3File path retires the // old output node and surfaces the new one as the user // types. - // 2. Inactive draft with a captured `outputAssets` snapshot - // (taken on the last pane transition): use those, so a - // draft the user already edited keeps its renamed outputs - // after they've clicked elsewhere. - // 3. Fallback to the static `outputAsset` seeded at draft - // creation — covers fresh drafts and parser misses (e.g. - // WIN-1943: wmill.writeS3File({s3, storage}) object form - // not yet detected by the TS parser). + // 2. Inactive draft: its captured `outputAssets` (inferred at + // creation/last edit, or the seeded output for a fresh draft + // whose body doesn't yet write anything inferable). const liveForThisDraft = liveBodyAssets.scriptPath === path - const writeOuts: Array<{ kind: AssetKind; path: string }> = [] + const writeOuts: Array<{ kind: AssetKind; path: string; derivedFrom?: string }> = [] if (liveForThisDraft) { writeOuts.push(...extractWrites(liveBodyAssets.assets)) } else if (d.outputAssets) { writeOuts.push(...d.outputAssets) } - if (writeOuts.length === 0 && d.outputAsset) { - writeOuts.push(d.outputAsset) - } + // `// materialize ` declares a write output via annotation, not + // the SQL body, so the body-inference tiers above miss it (pushWriteOut + // dedups against existing assets/edges). + writeOuts.push(...materializeOuts(parsed)) for (const out of writeOuts) { - const hasAsset = assets.some((a) => a.kind === out.kind && a.path === out.path) - if (!hasAsset) assets.push({ kind: out.kind, path: out.path }) - // Dedup against edges already in the overlay (this draft's base - // edges were dropped above, so this only guards against duplicate - // writeOuts entries — not against the persisted version). - const hasWriteEdge = edges.some( - (e) => - e.runnable_kind === 'script' && - e.runnable_path === path && - e.asset_kind === out.kind && - e.asset_path === out.path && - (e.access_type === 'w' || e.access_type === 'rw') - ) - if (hasWriteEdge) continue - edges.push({ - runnable_path: path, - runnable_kind: 'script', - asset_kind: out.kind, - asset_path: out.path, - access_type: 'w', - unsaved: true - }) + pushWriteOut(acc, path, out) } - // Live read lineage for the active draft (body reads like loadS3File / - // SELECT). Only the open draft has live-inferred assets; inactive drafts - // fall back to their `// on ` annotations below for inputs. Without - // this, dropping the persisted base edges above would lose the input - // edges of a saved script the moment the user starts editing it. - if (liveForThisDraft) { - for (const inp of extractReads(liveBodyAssets.assets)) { - if (!assets.some((a) => a.kind === inp.kind && a.path === inp.path)) { - assets.push({ kind: inp.kind, path: inp.path }) - } - const hasReadEdge = edges.some( - (e) => - e.runnable_kind === 'script' && - e.runnable_path === path && - e.asset_kind === inp.kind && - e.asset_path === inp.path && - (e.access_type === 'r' || e.access_type === 'rw') - ) - if (hasReadEdge) continue - edges.push({ - runnable_path: path, - runnable_kind: 'script', - asset_kind: inp.kind, - asset_path: inp.path, - access_type: 'r', - unsaved: true - }) - } + // Read lineage: live inference for the open draft, the captured + // `inputAssets` snapshot for inactive ones (with the session cache as a + // legacy fallback). Without the inactive tier, merely selecting another + // node would drop this draft's input edges from the canvas. + const draftReads = liveForThisDraft + ? extractReads(liveBodyAssets.assets) + : (d.inputAssets ?? inferredReadsByPath.get(path) ?? []) + for (const inp of draftReads) { + pushReadIn(acc, path, inp) } // Seed trigger edges from the draft's template so the graph stays // stable when the user clicks off this draft. Live annotations @@ -334,6 +625,28 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { const hasTriggerAsset = assets.some((x) => x.kind === a.kind && x.path === a.path) if (!hasTriggerAsset) assets.push({ kind: a.kind, path: a.path }) } + // Auto-derived cascade edges (backend parity): a ducklake/s3 read wires + // the edge from the body alone. Reads reuse the tier above; writes come + // from the active draft's live inference, else the captured snapshot / + // session cache. The open buffer's derived edges are re-computed + // authoritatively in applyLiveBufferOverlay (which strips this path's + // seeded triggers first), same as the explicit `// on` triggers above. + const draftWrites = liveForThisDraft + ? extractWrites(liveBodyAssets.assets) + : (d.outputAssets ?? inferredWritesByPath.get(path) ?? []) + for (const a of deriveAutoAssetTriggers(draftReads, draftWrites, parsed)) { + extraTriggers.push({ + trigger_kind: 'asset', + asset_kind: a.kind, + asset_path: a.path, + runnable_kind: 'script', + runnable_path: path, + unsaved: true + }) + if (!assets.some((x) => x.kind === a.kind && x.path === a.path)) { + assets.push({ kind: a.kind, path: a.path }) + } + } // Native trigger annotations on a draft are "missing" until a // matching trigger row exists. A brand-new draft never has one (the // script isn't deployed yet), but a draft promoted from unsaved @@ -394,6 +707,49 @@ function applyLiveBufferOverlay(acc: Accumulator, input: ResolveGraphInput, ctx: assets.push({ kind: a.kind, path: a.path }) } } + // Auto-derived cascade edges (backend parity): a ducklake/s3 read wires + // the edge from the FROM clause alone, keystroke-live. The open buffer's + // live body inference is authoritative for its reads/writes. `// mute` / + // `// mute all` and explicit `// on` (above) suppress a derived edge; a + // derived edge already persisted for a non-draft open script is deduped + // via `assetKeys`. + if (input.liveBodyAssets.scriptPath === livePath) { + const reads = extractReads(input.liveBodyAssets.assets) + const writes = extractWrites(input.liveBodyAssets.assets) + for (const a of deriveAutoAssetTriggers(reads, writes, liveAnnotations.annotations)) { + if (assetKeys.has(`${a.kind}:${a.path}`)) continue + extraTriggers.push({ + trigger_kind: 'asset', + asset_kind: a.kind, + asset_path: a.path, + runnable_kind: 'script', + runnable_path: livePath, + unsaved: true + }) + if (!assets.some((x) => x.kind === a.kind && x.path === a.path)) { + assets.push({ kind: a.kind, path: a.path }) + } + } + } + // Lineage overlay for an open SAVED script. seedDraftOverlays owns drafts, + // but a deployed script's unsaved edits are only promoted to a draft on + // pane teardown — until then the buffer's lineage lives here. staleForOpen + // already dropped the base edges the buffer no longer references; this adds + // the ones it now does. Without it, retargeting `// materialize` (or a body + // write/read) shows no output edge until the user clicks away. + if (!ctx.draftedPaths.has(livePath)) { + for (const out of materializeOuts(liveAnnotations.annotations)) { + pushWriteOut(acc, livePath, out) + } + if (input.liveBodyAssets.scriptPath === livePath) { + for (const out of extractWrites(input.liveBodyAssets.assets)) { + pushWriteOut(acc, livePath, out) + } + for (const inp of extractReads(input.liveBodyAssets.assets)) { + pushReadIn(acc, livePath, inp) + } + } + } // Native trigger annotations: kinds for which a matching trigger // row was found in the backend response. If the live buffer // declares `// on kafka` and at least one kafka_trigger row points @@ -442,7 +798,7 @@ function crossCheckSweptScripts(acc: Accumulator, input: ResolveGraphInput) { * changes — not just while selected. */ function overlayInferredLineage(acc: Accumulator, input: ResolveGraphInput) { - const { base, drafts, inferredWritesByPath, inferredReadsByPath } = input + const { drafts, inferredWritesByPath, inferredReadsByPath } = input const { assets, edges } = acc const overlayLineage = ( @@ -451,19 +807,21 @@ function overlayInferredLineage(acc: Accumulator, input: ResolveGraphInput) { ) => { for (const [scriptPath, refs] of byPath) { if (drafts.has(scriptPath)) continue - const persisted = new Set( - base.edges - .filter( - (e) => - e.runnable_path === scriptPath && - e.runnable_kind === 'script' && - (e.access_type === access || e.access_type === 'rw') - ) - .map((e) => `${e.asset_kind}:${e.asset_path}`) - ) for (const a of refs) { - const key = `${a.kind}:${a.path}` - if (persisted.has(key)) continue + // Dedup against the ACCUMULATED edges, not just base: for the open + // script these same refs may already be overlaid by + // applyLiveBufferOverlay (which feeds the maps on the route page), + // and a duplicate would collide on the canvas's endpoint-derived + // edge ids. + const hasEdge = edges.some( + (e) => + e.runnable_path === scriptPath && + e.runnable_kind === 'script' && + e.asset_kind === a.kind && + e.asset_path === a.path && + (e.access_type === access || e.access_type === 'rw') + ) + if (hasEdge) continue if (!assets.some((x) => x.kind === a.kind && x.path === a.path)) { assets.push({ kind: a.kind, path: a.path }) } diff --git a/frontend/src/lib/components/assets/AssetGraph/schemaContracts.test.ts b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.test.ts new file mode 100644 index 0000000000..9247b23d9b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.test.ts @@ -0,0 +1,276 @@ +import { describe, expect, it } from 'vitest' +import { + buildSchemaContractContext, + diffSchemaContracts, + mapWarningsToMarkers, + normalizeAssetPath, + referencedDucklakePaths, + type CapturedSchemaLite +} from './schemaContracts' +import { parsePipelineAnnotations } from './parsePipelineAnnotations' +import type { AssetWithAltAccessType } from '../lib' + +// Mirrors backend/windmill-common/src/schema_contracts.rs unit tests — the two +// diffs must apply the same rules or the editor previews a different verdict +// than the save-time check returns. + +function schema(cols: [string, string][], version = 2): CapturedSchemaLite { + return { + columns: cols.map(([name, type]) => ({ name, type })), + version, + capturedAt: '2026-01-01T00:00:00Z' + } +} + +function readAsset(path: string, cols: string[]): AssetWithAltAccessType { + return { + path, + kind: 'ducklake', + access_type: 'r', + columns: Object.fromEntries(cols.map((c) => [c, 'r' as const])) + } +} + +const NO_ANN = { columnLineage: [], dataTests: [] } + +describe('diffSchemaContracts', () => { + it('warns on a missing read column, matching case-insensitively', () => { + const schemas = new Map([ + [ + 'lake/orders', + schema([ + ['Order_ID', 'BIGINT'], + ['amount_usd', 'DOUBLE'] + ]) + ] + ]) + const w = diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['order_id', 'amount'])], + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(1) + expect(w[0].kind).toBe('missing_column') + expect(w[0].column).toBe('amount') + expect(w[0].schema_version).toBe(2) + }) + + it('skips unknown-column assets, "*" and reserved columns', () => { + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const noColumns: AssetWithAltAccessType = { + path: 'lake/orders', + kind: 'ducklake', + access_type: 'r' + } + expect( + diffSchemaContracts({ ...NO_ANN, assets: [noColumns], schemas, ignored: new Set() }) + ).toEqual([]) + expect( + diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['*', '_wm_partition', 'id'])], + schemas, + ignored: new Set() + }) + ).toEqual([]) + }) + + it('is silent for assets without a captured schema', () => { + expect( + diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/unknown', ['whatever'])], + schemas: new Map(), + ignored: new Set() + }) + ).toEqual([]) + }) + + it('normalizes the {partition} token before lookup', () => { + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const w = diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders/{partition}', ['gone'])], + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(1) + expect(w[0].asset_path).toBe('lake/orders') + }) + + it('warns on broken // column lineage refs', () => { + const ann = parsePipelineAnnotations( + '// column total <- ducklake://lake/orders.amount\nSELECT 1;' + ) + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const w = diffSchemaContracts({ + assets: [], + columnLineage: ann.columnLineage, + dataTests: [], + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(1) + expect(w[0].kind).toBe('missing_lineage_source') + }) + + it('flags missing relationship columns and captured-type differences', () => { + const ann = parsePipelineAnnotations( + '// materialize ducklake://lake/orders\n' + + '// data_test relationships customer_id -> ducklake://lake/customers.id\n' + + '// data_test relationships customer_id -> ducklake://lake/customers.uuid\n' + + 'SELECT 1;' + ) + const schemas = new Map([ + ['lake/customers', schema([['id', 'VARCHAR']])], + ['lake/orders', schema([['customer_id', 'BIGINT']])] + ]) + const w = diffSchemaContracts({ + assets: [], + columnLineage: ann.columnLineage, + dataTests: ann.dataTests, + materialize: ann.materialize, + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(2) + expect( + w.some( + (x) => + x.kind === 'relationship_type_mismatch' && + x.expected_type === 'BIGINT' && + x.found_type === 'VARCHAR' + ) + ).toBe(true) + expect(w.some((x) => x.kind === 'missing_relationship_column' && x.column === 'uuid')).toBe( + true + ) + }) + + it('suppresses ignored assets down to one informational note', () => { + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const ignored = new Set(['lake/orders']) + const w = diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['a', 'b'])], + schemas, + ignored + }) + expect(w).toHaveLength(1) + expect(w[0].kind).toBe('suppressed') + expect( + diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['id'])], + schemas, + ignored + }) + ).toEqual([]) + }) +}) + +describe('referencedDucklakePaths', () => { + it('collects paths from reads, lineage, relationships and materialize', () => { + const ann = parsePipelineAnnotations( + '// materialize ducklake://lake/out\n' + + '// column total <- ducklake://lake/a.amount\n' + + '// data_test relationships k -> ducklake://lake/b.id\n' + + 'SELECT 1;' + ) + const refs = referencedDucklakePaths({ + assets: [readAsset('lake/c/{partition}', ['x'])], + columnLineage: ann.columnLineage, + dataTests: ann.dataTests, + materialize: ann.materialize + }) + expect(refs.sort()).toEqual(['lake/a', 'lake/b', 'lake/c', 'lake/out']) + }) +}) + +describe('buildSchemaContractContext', () => { + it('derives ignored assets and scd2 _current bases from graph runnables', () => { + const ctx = buildSchemaContractContext([ + { + materialize_target: { kind: 'ducklake', path: 'lake/dim' }, + materialize_strategy: 'scd2', + materialize_on_schema_change: 'ignore' + }, + { + materialize_target: { kind: 'ducklake', path: 'lake/orders' }, + materialize_strategy: 'replace' + }, + // non-ducklake and absent targets are ignored + { materialize_target: { kind: 's3object', path: 'x/y' }, materialize_strategy: 'scd2' }, + {} + ]) + expect(ctx.ignoredAssets).toEqual(['lake/dim', 'lake/dim_current']) + expect(ctx.scd2CurrentBases).toEqual({ 'lake/dim_current': 'lake/dim' }) + }) + + it('ignores _current only for scd2 producers (backend spec.scd2 gate)', () => { + const ctx = buildSchemaContractContext([ + { + materialize_target: { kind: 'ducklake', path: 'lake/t' }, + materialize_strategy: 'replace', + materialize_on_schema_change: 'ignore' + } + ]) + // a non-scd2 producer's `_current` is an unrelated asset — it must + // keep warning, exactly like the server-side check + expect(ctx.ignoredAssets).toEqual(['lake/t']) + expect(ctx.scd2CurrentBases).toEqual({}) + }) +}) + +describe('mapWarningsToMarkers', () => { + it('anchors annotation warnings to their lines and body reads to the identifier', () => { + const code = + '-- pipeline\n' + + '-- on ducklake://lake/orders\n' + + '-- column total <- ducklake://lake/orders.amount\n' + + '-- data_test relationships k -> ducklake://lake/customers.uuid\n' + + 'SELECT amount FROM dl.orders;' + const markers = mapWarningsToMarkers(code, [ + { + kind: 'missing_lineage_source', + asset_path: 'lake/orders', + column: 'amount', + message: 'm1' + }, + { + kind: 'missing_relationship_column', + asset_path: 'lake/customers', + column: 'uuid', + message: 'm2' + }, + { kind: 'missing_column', asset_path: 'lake/orders', column: 'amount', message: 'm3' }, + { kind: 'suppressed', asset_path: 'lake/orders', message: 'hidden' } + ]) + expect(markers).toHaveLength(3) + expect(markers[0].startLineNumber).toBe(3) + expect(markers[1].startLineNumber).toBe(4) + // body-read warning anchors to the first occurrence of the identifier, + // which is the annotation line mentioning `amount` (line 3) + expect(markers[2].startLineNumber).toBe(3) + // token range is tight around the identifier, not the whole line + const line3 = '-- column total <- ducklake://lake/orders.amount' + expect(markers[2].startColumn).toBe(line3.indexOf('amount') + 1) + }) + + it('falls back to the line mentioning the asset path', () => { + const code = '# pipeline\n# on ducklake://lake/orders\nprint(1)' + const markers = mapWarningsToMarkers(code, [ + { kind: 'missing_column', asset_path: 'lake/orders', column: 'zzz', message: 'm' } + ]) + expect(markers[0].startLineNumber).toBe(2) + }) +}) + +describe('normalizeAssetPath', () => { + it('strips the partition token and trailing slashes', () => { + expect(normalizeAssetPath('lake/orders/{partition}')).toBe('lake/orders') + expect(normalizeAssetPath('lake/orders_{partition}')).toBe('lake/orders_') + expect(normalizeAssetPath('lake/orders/')).toBe('lake/orders') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/schemaContracts.ts b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.ts new file mode 100644 index 0000000000..4a1de04e13 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.ts @@ -0,0 +1,440 @@ +// Client-side mirror of the save-time schema-contract check (pipelines gap +// #2b, backend/windmill-common/src/schema_contracts.rs). The backend endpoint +// (`checkSchemaContracts`) is the authoritative check run on save; this mirror +// drives the *live* editor surface (Monaco warning markers + completions) from +// the WASM parse that already runs on the open buffer, so the two must apply +// the same rules: ducklake-only, case-insensitive column names, `columns` +// absent ⇒ skip, `_wm_partition` whitelisted, `{partition}` token stripped, +// annotation-declared lineage only, asset without captured schema ⇒ silent. + +import { AssetService, ScriptService, type ContractWarning, type ScriptLang } from '$lib/gen' +import { sendUserToast } from '$lib/toast' +import type { AssetWithAltAccessType } from '../lib' +import { + parsePipelineAnnotations, + type ColumnLineage, + type DataTest, + type MaterializeSpec +} from './parsePipelineAnnotations' + +// Columns the materialize engine manages; excluded from the captured schema on +// purpose, so reads of them must not warn. +const RESERVED_COLUMNS = ['_wm_partition'] + +const PARTITION_TOKEN = '{partition}' + +export type CapturedSchemaLite = { + columns: { name: string; type: string }[] + version: number + capturedAt: string +} + +// Strip the `{partition}` token a declared URI may carry so lookups hit the +// captured path (mirrors `normalize_asset_path`). +export function normalizeAssetPath(path: string): string { + return path + .replaceAll('/' + PARTITION_TOKEN, '') + .replaceAll(PARTITION_TOKEN, '') + .replace(/\/+$/, '') +} + +function isReserved(name: string): boolean { + return RESERVED_COLUMNS.some((r) => r.toLowerCase() === name.toLowerCase()) +} + +function findColumn( + schema: CapturedSchemaLite, + name: string +): { name: string; type: string } | undefined { + const lower = name.toLowerCase() + return schema.columns.find((c) => c.name.toLowerCase() === lower) +} + +export type SchemaContractInputs = { + // Per-asset column reads/writes from the WASM parse (entries without a + // `columns` map are skipped — wildcard/unknown access). + assets: AssetWithAltAccessType[] + // Annotation-declared `// column` lineage ONLY (not merged AST-inferred + // lineage — redundant with body reads and alias-attribution can misfire). + columnLineage: ColumnLineage[] + dataTests: DataTest[] + materialize?: MaterializeSpec + // Latest captured schema per normalized ducklake path (after any + // `_current` → base-table fallback the caller resolved). + schemas: Map + // Normalized paths whose producer declares `on_schema_change=ignore`. + ignored: Set +} + +// Mirrors backend `diff_contract` — same warning kinds and suppression +// semantics, minus the human message wording (the editor renders its own). +export function diffSchemaContracts(input: SchemaContractInputs): ContractWarning[] { + const { assets, columnLineage, dataTests, materialize, schemas, ignored } = input + const warnings: ContractWarning[] = [] + + // W1 — body-read/written columns missing from the captured schema. + for (const a of assets) { + if (a.kind !== 'ducklake' || a.columns == undefined) continue + const path = normalizeAssetPath(a.path) + const schema = schemas.get(path) + if (!schema) continue + for (const col of Object.keys(a.columns)) { + if (col === '*' || isReserved(col)) continue + if (!findColumn(schema, col)) { + warnings.push({ + kind: 'missing_column', + asset_path: path, + column: col, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `column \`${col}\` of ducklake://${path} is not in its captured schema (v${schema.version}, columns: ${schema.columns.map((c) => c.name).join(', ')})` + }) + } + } + } + + // W2 — `// column` lineage source refs. + for (const cl of columnLineage) { + for (const input of cl.inputs) { + if (input.from_kind !== 'ducklake' || isReserved(input.from_column)) continue + const path = normalizeAssetPath(input.from_path) + const schema = schemas.get(path) + if (!schema) continue + if (!findColumn(schema, input.from_column)) { + warnings.push({ + kind: 'missing_lineage_source', + asset_path: path, + column: input.from_column, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `\`// column ${cl.column}\` reads \`${input.from_column}\` from ducklake://${path}, which is not in its captured schema (v${schema.version})` + }) + } + } + } + + // W3 — relationships refs: missing column, and captured-type difference + // when the consumer's own materialize target has a capture. Types still + // coerce at run time, so a difference is "differs", never "will fail". + const ownSchema = + materialize?.targetKind === 'ducklake' + ? schemas.get(normalizeAssetPath(materialize.targetPath)) + : undefined + for (const dt of dataTests) { + if (dt.type !== 'relationships' || dt.to_kind !== 'ducklake') continue + const path = normalizeAssetPath(dt.to_path) + const schema = schemas.get(path) + if (!schema) continue + const refCol = findColumn(schema, dt.to_column) + if (!refCol) { + warnings.push({ + kind: 'missing_relationship_column', + asset_path: path, + column: dt.to_column, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `\`// data_test relationships ${dt.column}\` references ducklake://${path}.${dt.to_column}, which is not in its captured schema (v${schema.version})` + }) + } else { + const ownCol = ownSchema && findColumn(ownSchema, dt.column) + if (ownCol && ownCol.type.toLowerCase() !== refCol.type.toLowerCase()) { + warnings.push({ + kind: 'relationship_type_mismatch', + asset_path: path, + column: dt.to_column, + expected_type: ownCol.type, + found_type: refCol.type, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `\`// data_test relationships ${dt.column}\` joins \`${dt.column}\` (${ownCol.type}) to ducklake://${path}.${dt.to_column} (${refCol.type}) — captured types differ` + }) + } + } + } + + // W4 — producer `on_schema_change=ignore`: drop the asset's warnings, + // leaving one informational entry per suppressed asset. + if (ignored.size > 0) { + const suppressed: string[] = [] + const kept = warnings.filter((w) => { + if (ignored.has(w.asset_path)) { + if (!suppressed.includes(w.asset_path)) suppressed.push(w.asset_path) + return false + } + return true + }) + warnings.length = 0 + warnings.push(...kept) + for (const path of suppressed) { + warnings.push({ + kind: 'suppressed', + asset_path: path, + message: `schema mismatches on ducklake://${path} suppressed by its producer's \`on_schema_change=ignore\`` + }) + } + } + + return warnings +} + +// The ducklake paths a buffer references in ways the contract check inspects — +// what the editor needs captured schemas for. +export function referencedDucklakePaths( + input: Pick +): string[] { + const paths = new Set() + for (const a of input.assets) { + if (a.kind === 'ducklake' && a.columns != undefined) paths.add(normalizeAssetPath(a.path)) + } + for (const cl of input.columnLineage) { + for (const i of cl.inputs) { + if (i.from_kind === 'ducklake') paths.add(normalizeAssetPath(i.from_path)) + } + } + for (const dt of input.dataTests) { + if (dt.type === 'relationships' && dt.to_kind === 'ducklake') + paths.add(normalizeAssetPath(dt.to_path)) + } + if (input.materialize?.targetKind === 'ducklake') + paths.add(normalizeAssetPath(input.materialize.targetPath)) + return [...paths] +} + +// --- Captured-schema cache ------------------------------------------------ + +// Short-TTL cache so per-keystroke recomputes and completion requests don't +// re-fetch. Captured schemas only change when a producer materializes, so a +// briefly stale hit is fine — the authoritative save-time check re-reads. +const SCHEMA_TTL_MS = 30_000 +const schemaCache = new Map() + +export async function fetchLatestSchema( + workspace: string, + path: string +): Promise { + const key = `${workspace}:${path}` + const hit = schemaCache.get(key) + if (hit && Date.now() - hit.at < SCHEMA_TTL_MS) return hit.value + let value: CapturedSchemaLite | undefined = undefined + try { + const versions = await AssetService.listAssetSchemas({ workspace, path }) + const latest = versions[0] + if (latest) { + value = { + columns: latest.columns, + version: latest.version, + capturedAt: latest.captured_at + } + } + } catch (e) { + console.error('failed to fetch captured asset schema', path, e) + } + schemaCache.set(key, { at: Date.now(), value }) + return value +} + +// Resolve the schema map for a set of referenced paths, applying the scd2 +// `_current` → base-table fallback when the graph identifies the view's +// producer as a managed scd2 materializer (the view is `SELECT * … WHERE +// is_current`, so columns are identical). +export async function fetchSchemasForPaths( + workspace: string, + paths: string[], + scd2CurrentBase?: (path: string) => string | undefined +): Promise> { + const out = new Map() + await Promise.all( + paths.map(async (p) => { + let schema = await fetchLatestSchema(workspace, p) + if (!schema && p.endsWith('_current')) { + const base = scd2CurrentBase?.(p) + if (base) schema = await fetchLatestSchema(workspace, base) + } + if (schema) out.set(p, schema) + }) + ) + return out +} + +// --- Pipeline-graph context --------------------------------------------------- + +// Producer-side facts the contract mirror needs but cannot derive from the +// open buffer: which assets are muted (`on_schema_change=ignore`) and which +// `_current` views map to an scd2 base table. Built by the pipeline page +// from the resolved graph; absent outside the pipeline editor (standalone +// script editor), where suppression simply doesn't apply client-side — the +// save-time server check remains authoritative either way. +export type SchemaContractGraphContext = { + // Normalized asset paths whose producer declares `on_schema_change=ignore`. + ignoredAssets: string[] + // `_current` → base for managed scd2 producers in the graph. + scd2CurrentBases: Record +} + +export function buildSchemaContractContext( + runnables: Pick< + import('./types').AssetGraphRunnableNode, + 'materialize_target' | 'materialize_strategy' | 'materialize_on_schema_change' + >[] +): SchemaContractGraphContext { + const ignoredAssets: string[] = [] + const scd2CurrentBases: Record = {} + for (const r of runnables) { + const t = r.materialize_target + if (!t || t.kind !== 'ducklake') continue + const base = normalizeAssetPath(t.path) + if (r.materialize_on_schema_change === 'ignore') { + ignoredAssets.push(base) + // The `_current` companion is the producer's own view only for scd2 — + // mirroring the backend's `spec.scd2` gate; for any other strategy a + // `_current` ref is an unrelated asset that must keep warning. + if (r.materialize_strategy === 'scd2') { + ignoredAssets.push(`${base}_current`) + } + } + if (r.materialize_strategy === 'scd2') { + scd2CurrentBases[`${base}_current`] = base + } + } + return { ignoredAssets, scd2CurrentBases } +} + +// --- Save-time surface ------------------------------------------------------ + +// Run the authoritative backend check for just-deployed content and toast the +// result. Never throws — a failed check must not taint a successful deploy. +export async function notifyContractWarnings( + workspace: string, + language: ScriptLang, + content: string +): Promise { + // Every checkable ref carries the `ducklake` token (URIs and the bare + // default-syntax shorthand alike) — skip the round-trip for the vast + // majority of saves that can't produce a warning. + if (!content.includes('ducklake')) return + try { + const { warnings } = await ScriptService.checkSchemaContracts({ + workspace, + requestBody: { language, content } + }) + const real = warnings.filter((w) => w.kind !== 'suppressed') + if (real.length === 0) return + sendUserToast( + `Schema contract: ${real.length} warning${real.length > 1 ? 's' : ''}`, + 'warning', + [], + real.map((w) => `• ${w.message}`).join('\n'), + 10000 + ) + } catch (e) { + console.error('schema-contract check failed', e) + } +} + +// End-to-end live-editor check: parse the buffer's annotations, resolve the +// captured schemas for everything it references, diff, and anchor the result +// to source positions. Cheap per keystroke — the annotation parse is a line +// scan and schema fetches hit the short-TTL cache. +export async function computeContractMarkers( + workspace: string, + code: string, + assets: AssetWithAltAccessType[], + context?: SchemaContractGraphContext +): Promise { + const ann = parsePipelineAnnotations(code) + const inputs = { + assets, + // Annotation-declared lineage only — body-inferred lineage is redundant + // with the body-read check and its alias attribution can misfire. + columnLineage: ann.columnLineage, + dataTests: ann.dataTests, + materialize: ann.materialize + } + const refs = referencedDucklakePaths(inputs) + if (refs.length === 0) return [] + const schemas = await fetchSchemasForPaths( + workspace, + refs, + context ? (p) => context.scd2CurrentBases[p] : undefined + ) + if (schemas.size === 0) return [] + const warnings = diffSchemaContracts({ + ...inputs, + schemas, + ignored: new Set(context?.ignoredAssets ?? []) + }) + return mapWarningsToMarkers(code, warnings) +} + +// --- Editor marker mapping --------------------------------------------------- + +export type ContractMarker = { + message: string + startLineNumber: number + startColumn: number + endLineNumber: number + endColumn: number +} + +// Best-effort source anchoring: annotation-family warnings anchor to their +// annotation line; body-read warnings anchor to the first occurrence of the +// column identifier; fallback is the `// on`/first line mentioning the asset. +export function mapWarningsToMarkers(code: string, warnings: ContractWarning[]): ContractMarker[] { + const lines = code.split('\n') + + function lineMatching(pred: (line: string) => boolean): number | undefined { + const idx = lines.findIndex(pred) + return idx >= 0 ? idx + 1 : undefined + } + + function tokenRange( + lineNumber: number, + token: string + ): { startColumn: number; endColumn: number } { + const line = lines[lineNumber - 1] ?? '' + const idx = line.toLowerCase().indexOf(token.toLowerCase()) + if (idx < 0) return { startColumn: 1, endColumn: line.length + 1 } + return { startColumn: idx + 1, endColumn: idx + 1 + token.length } + } + + return warnings + .filter((w) => w.kind !== 'suppressed') + .map((w) => { + let lineNumber: number | undefined + let token: string | undefined = w.column ?? undefined + switch (w.kind) { + case 'missing_lineage_source': + lineNumber = lineMatching( + (l) => /^\s*(\/\/|--|#)\s*column\s/.test(l) && !!w.column && l.includes(w.column) + ) + break + case 'missing_relationship_column': + case 'relationship_type_mismatch': + lineNumber = lineMatching( + (l) => + /^\s*(\/\/|--|#)\s*data_test\s+relationships\s/.test(l) && l.includes(w.asset_path) + ) + break + case 'missing_column': { + // first body occurrence of the column identifier + const re = new RegExp(`\\b${w.column?.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\b`, 'i') + lineNumber = w.column ? lineMatching((l) => re.test(l)) : undefined + break + } + } + if (lineNumber == undefined) { + // fallback: the `// on …` (or any) line mentioning the asset path + lineNumber = lineMatching((l) => l.includes(w.asset_path)) ?? 1 + token = w.asset_path + } + const range = token + ? tokenRange(lineNumber, token) + : { startColumn: 1, endColumn: (lines[lineNumber - 1]?.length ?? 0) + 1 } + return { + message: w.message, + startLineNumber: lineNumber, + endLineNumber: lineNumber, + ...range + } + }) +} diff --git a/frontend/src/lib/components/assets/AssetGraph/types.ts b/frontend/src/lib/components/assets/AssetGraph/types.ts index 5f3aa4b52d..0e055abca2 100644 --- a/frontend/src/lib/components/assets/AssetGraph/types.ts +++ b/frontend/src/lib/components/assets/AssetGraph/types.ts @@ -1,10 +1,21 @@ import type { AssetKind } from '$lib/gen' +import type { ColumnLineage, DataTest } from './parsePipelineAnnotations' export type GraphUsageKind = 'script' | 'flow' export interface AssetGraphAssetNode { kind: AssetKind path: string + // Fork workspaces only: 'fork' when this ducklake asset was materialized in + // the fork itself, 'deferred' when reads fall back to the parent workspace's + // current table via a defer view. Absent outside forks / for other kinds / + // when never materialized anywhere. Lockstep with Rust `GraphAssetNode`. + fork_materialization?: 'fork' | 'deferred' + // Base dimension path this node is the SCD2 `_current` companion view of + // (its producer declares `// materialize … history` on ``). Set only on + // the `_current` node; lets the canvas mark it as a derived "current view" + // rather than an unrelated table. Lockstep with Rust `GraphAssetNode`. + derived_from?: string } export interface AssetGraphRunnableNode { @@ -21,6 +32,11 @@ export interface AssetGraphRunnableNode { // Raw `// freshness ` value, e.g. "1h", "30m". Surfaced for // the badge; the runtime parses it as needed. freshness?: string + // Completion time (ISO) of the newest successful run of this pipeline + // member visible to the caller. The freshness chip compares it against + // the `// freshness` window to render fresh/stale. Absent = no + // successful run found (or none visible under job RLS). + last_success_at?: string // `// tag ` worker-tag override. Surfaced for the badge so users // can see which worker pool will pick this script up at a glance. tag?: string @@ -28,9 +44,39 @@ export interface AssetGraphRunnableNode { // duration string (`"5s"`, `"30s"`); absent = back-to-back. Surfaced as // a badge so retry-enabled scripts are visible without opening the pane. retry?: { count: number; delay?: string } + // `// data_test …` data-quality checks run against the materialized + // asset. Surfaced as a count badge (with a per-test breakdown in the title) + // so test coverage is visible on the node without opening the pane. + data_tests?: DataTest[] + // `// column <- .` declared column-level lineage for this + // script's materialized output. Surfaced as a count badge on the write-edge + // and as a column-to-column diagram in the asset details pane. + column_lineage?: ColumnLineage[] + // `// materialize ` target — the asset `column_lineage` describes. + // Lets the column graph anchor lineage to the exact output instead of + // guessing a ducklake write-edge (a multi-output script writes several). + materialize_target?: { kind: AssetKind; path: string } + // Managed `// materialize` write strategy. Absent for non-materializing or + // `manual` scripts. Used (with `partition_kind`) to decide whether a + // produced asset's schema can evolve: only whole-table `replace` can, since + // `append`/`merge`/`scd2`/partitioned writes INSERT into a fixed-schema + // table. `scd2` also identifies the producer of a `_current` companion + // view for the schema-contract `_current` → base-table fallback. + materialize_strategy?: 'replace' | 'append' | 'merge' | 'scd2' + // `on_schema_change=ignore` on the managed materialize — the producer's + // opt-out from downstream schema-contract warnings. Only present when set + // to `ignore` (default `warn` is absent). Threaded into the editor's + // contract mirror so it suppresses the same warnings the server check does. + materialize_on_schema_change?: string + // Macros this script provides to the workspace registry (deployed + // `// macros` library). Non-empty marks the node as a macro library; + // drives the "defines N macros" badge and the details-pane signature + // list. `params` is the verbatim parameter list. + macros?: { name: string; params: string; is_table: boolean }[] // Synthesized by the page from a local draft; the script doesn't exist // in the DB yet. Drives a dashed/lower-opacity rendering to mirror how // unsaved triggers are styled — visually distinct from persisted nodes. + // AI-built nodes are plain drafts too (no separate pending/approval state). unsaved?: boolean } @@ -94,11 +140,41 @@ export type AssetGraphTrigger = missing?: boolean } +// Macro-library → consumer edge: the consumer calls `macro_names` of +// `lib_path`'s macros (deploy-recorded detection), or pulls in the whole +// library via `// use` (`via_use`, macro_names then lists the full library). +// `unsaved: true` marks a draft's `// use` overlay. +export interface AssetGraphMacroEdge { + lib_path: string + consumer_path: string + macro_names: string[] + via_use: boolean + unsaved?: boolean +} + +// Ordering-only "must-run-after" edge: `runnable_path`'s `// data_test` +// (a `relationships` ref, or a custom test reading a pipeline asset) needs +// `asset` materialized before the tested script runs — but the tested script +// doesn't consume the asset's rows, so this is NOT a lineage edge. Resolved +// server-side to the referenced asset's in-pipeline producer; fed into the +// cascade topo-sort (buildLineageDag) so a cold cascade orders the referenced +// dimension first, and rendered dashed on the canvas (like macro edges). +export interface AssetGraphTestEdge { + producer_kind: GraphUsageKind + producer_path: string + runnable_kind: GraphUsageKind + runnable_path: string + asset_kind: AssetKind + asset_path: string +} + export interface AssetGraphResponse { assets: AssetGraphAssetNode[] runnables: AssetGraphRunnableNode[] edges: AssetGraphEdge[] triggers: AssetGraphTrigger[] + macro_edges?: AssetGraphMacroEdge[] + test_edges?: AssetGraphTestEdge[] } export type AssetGraphNodeData = diff --git a/frontend/src/lib/components/assets/workspaceMacros.ts b/frontend/src/lib/components/assets/workspaceMacros.ts new file mode 100644 index 0000000000..d25f9d1bea --- /dev/null +++ b/frontend/src/lib/components/assets/workspaceMacros.ts @@ -0,0 +1,32 @@ +import { AssetService, type ListWorkspaceMacrosResponse } from '$lib/gen' + +export type WorkspaceMacro = ListWorkspaceMacrosResponse[number] + +// Workspace macros are late-bound (the worker reads the registry per job), so +// mild staleness in editor surfaces is harmless — a short TTL keeps repeated +// editor mounts / drawer opens from refetching on every keystroke-driven +// remount while still picking up a lib deploy within seconds. +const TTL_MS = 30_000 +const cache = new Map() + +export async function listWorkspaceMacrosCached(workspace: string): Promise { + const hit = cache.get(workspace) + if (hit && Date.now() - hit.at < TTL_MS) return hit.items + const items = await AssetService.listWorkspaceMacros({ workspace }) + cache.set(workspace, { at: Date.now(), items }) + return items +} + +export function invalidateWorkspaceMacros(workspace: string) { + cache.delete(workspace) +} + +/** `name(params)` display signature, with the table-macro arrow. */ +export function macroSignature(m: WorkspaceMacro): string { + return `${m.name}(${m.params})${m.is_table ? ' → table' : ''}` +} + +/** Full `CREATE` statement for the copy button / documentation preview. */ +export function macroDefinitionSql(m: WorkspaceMacro): string { + return `CREATE OR REPLACE MACRO ${m.name}(${m.params}) AS ${m.is_table ? 'TABLE ' : ''}${m.body};` +} diff --git a/frontend/src/lib/components/common/ScrollableX.svelte b/frontend/src/lib/components/common/ScrollableX.svelte new file mode 100644 index 0000000000..836439bfbb --- /dev/null +++ b/frontend/src/lib/components/common/ScrollableX.svelte @@ -0,0 +1,17 @@ + + +
+ {@render children()} +
diff --git a/frontend/src/lib/components/common/badge/Badge.svelte b/frontend/src/lib/components/common/badge/Badge.svelte index cf7a7cad66..c57d41029c 100644 --- a/frontend/src/lib/components/common/badge/Badge.svelte +++ b/frontend/src/lib/components/common/badge/Badge.svelte @@ -90,7 +90,7 @@ const hovers: Partial> = { gray: 'hover:bg-surface-hover', - blue: 'hover:bg-blue-200 dark:hover:bg-blue-700/40', + blue: 'hover:bg-blue-100 dark:hover:bg-blue-700/60', red: 'hover:bg-red-200 dark:hover:bg-red-500/25', green: 'hover:bg-green-200 dark:hover:bg-green-500/25', yellow: 'hover:bg-yellow-200 dark:hover:bg-yellow-500/25', diff --git a/frontend/src/lib/components/common/button/CopyButton.svelte b/frontend/src/lib/components/common/button/CopyButton.svelte new file mode 100644 index 0000000000..4eaed95b9b --- /dev/null +++ b/frontend/src/lib/components/common/button/CopyButton.svelte @@ -0,0 +1,44 @@ + + + {/if} - {#if !isCloudHosted() && !isRuleActive('DisableWorkspaceForking') && (!showEditButton || !app.canWrite)} + {#if !isCloudHosted() && editInForkAllowed($workspaceStore, $userWorkspaces) && (!showEditButton || !app.canWrite)}
{/if} @@ -235,10 +234,13 @@ hide: $userStore?.operator }, { - displayName: 'Edit in workspace fork', + displayName: editInForkLabel($workspaceStore, $userWorkspaces), icon: GitFork, href: buildForkEditUrl(app.raw_app ? 'raw_app' : 'app', path), - hide: $userStore?.operator || isCloudHosted() || isRuleActive('DisableWorkspaceForking') + hide: + $userStore?.operator || + isCloudHosted() || + !editInForkAllowed($workspaceStore, $userWorkspaces) }, { displayName: 'Move/Rename', diff --git a/frontend/src/lib/components/common/table/FlowRow.svelte b/frontend/src/lib/components/common/table/FlowRow.svelte index 89a08b412b..a19715686f 100644 --- a/frontend/src/lib/components/common/table/FlowRow.svelte +++ b/frontend/src/lib/components/common/table/FlowRow.svelte @@ -8,7 +8,7 @@ import DraftBadge from '$lib/components/DraftBadge.svelte' import type ShareModal from '$lib/components/ShareModal.svelte' import { FlowService, type Flow } from '$lib/gen' - import { userStore, workspaceStore } from '$lib/stores' + import { userStore, userWorkspaces, workspaceStore } from '$lib/stores' import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { createEventDispatcher } from 'svelte' import Badge from '../badge/Badge.svelte' @@ -36,8 +36,7 @@ import FlowHistory from '$lib/components/flows/FlowHistory.svelte' import InheritedLabels from '$lib/components/InheritedLabels.svelte' import { getDeployUiSettings } from '$lib/components/home/deploy_ui' - import { isRuleActive } from '$lib/workspaceProtectionRules.svelte' - import { buildForkEditUrl } from '$lib/utils/editInFork' + import { buildForkEditUrl, editInForkAllowed, editInForkLabel } from '$lib/utils/editInFork' import { isCloudHosted } from '$lib/cloud' interface Props { @@ -191,7 +190,7 @@ {/if} - {#if !isCloudHosted() && !isRuleActive('DisableWorkspaceForking') && (!showEditButton || !flow.canWrite)} + {#if !isCloudHosted() && editInForkAllowed($workspaceStore, $userWorkspaces) && (!showEditButton || !flow.canWrite)}
{/if} @@ -252,10 +251,13 @@ hide: $userStore?.operator }, { - displayName: 'Edit in workspace fork', + displayName: editInForkLabel($workspaceStore, $userWorkspaces), icon: GitFork, href: buildForkEditUrl('flow', path), - hide: $userStore?.operator || isCloudHosted() || isRuleActive('DisableWorkspaceForking') + hide: + $userStore?.operator || + isCloudHosted() || + !editInForkAllowed($workspaceStore, $userWorkspaces) }, { displayName: 'Audit logs', diff --git a/frontend/src/lib/components/common/table/RawAppRow.svelte b/frontend/src/lib/components/common/table/RawAppRow.svelte index e759a1d9fe..3f74356506 100644 --- a/frontend/src/lib/components/common/table/RawAppRow.svelte +++ b/frontend/src/lib/components/common/table/RawAppRow.svelte @@ -33,7 +33,7 @@ {:else if effectiveKind === 'app' || effectiveKind === 'raw_app'} + {:else if effectiveKind === 'raw_app_file'} + {:else if effectiveKind === 'script'} {:else if effectiveKind === 'variable'} @@ -121,6 +129,8 @@ {:else if effectiveKind === 'data_pipeline'} + {:else if effectiveKind === 'datatable_migration'} + {:else}
{/if} diff --git a/frontend/src/lib/components/common/table/ScriptRow.svelte b/frontend/src/lib/components/common/table/ScriptRow.svelte index 1d87d4c8d5..507986294f 100644 --- a/frontend/src/lib/components/common/table/ScriptRow.svelte +++ b/frontend/src/lib/components/common/table/ScriptRow.svelte @@ -9,7 +9,7 @@ import type ShareModal from '$lib/components/ShareModal.svelte' import { ScriptService, type Script } from '$lib/gen' - import { hubBaseUrlStore, userStore, workspaceStore } from '$lib/stores' + import { hubBaseUrlStore, userStore, userWorkspaces, workspaceStore } from '$lib/stores' import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { createEventDispatcher } from 'svelte' @@ -48,8 +48,7 @@ import Tooltip from '$lib/components/Tooltip.svelte' import { getDeployUiSettings } from '$lib/components/home/deploy_ui' import { scriptToHubUrl } from '$lib/hub' - import { isRuleActive } from '$lib/workspaceProtectionRules.svelte' - import { buildForkEditUrl } from '$lib/utils/editInFork' + import { buildForkEditUrl, editInForkAllowed, editInForkLabel } from '$lib/utils/editInFork' import { isCloudHosted } from '$lib/cloud' interface Props { @@ -251,7 +250,7 @@ {/if} {/if} - {#if !isCloudHosted() && !isRuleActive('DisableWorkspaceForking') && (!showEditButton || !script.canWrite)} + {#if !isCloudHosted() && editInForkAllowed($workspaceStore, $userWorkspaces) && (!showEditButton || !script.canWrite)}
{/if} @@ -334,10 +333,13 @@ hide: $userStore?.operator }, { - displayName: 'Edit in workspace fork', + displayName: editInForkLabel($workspaceStore, $userWorkspaces), icon: GitFork, href: buildForkEditUrl('script', script.path), - hide: $userStore?.operator || isCloudHosted() || isRuleActive('DisableWorkspaceForking') + hide: + $userStore?.operator || + isCloudHosted() || + !editInForkAllowed($workspaceStore, $userWorkspaces) }, { displayName: 'Move/Rename', diff --git a/frontend/src/lib/components/common/tabs/DraggableTabs.svelte b/frontend/src/lib/components/common/tabs/DraggableTabs.svelte index 937191d272..905759f391 100644 --- a/frontend/src/lib/components/common/tabs/DraggableTabs.svelte +++ b/frontend/src/lib/components/common/tabs/DraggableTabs.svelte @@ -25,8 +25,8 @@ import { dndzone, type DndEvent } from '@windmill-labs/svelte-dnd-action' import { X } from 'lucide-svelte' import { twMerge } from 'tailwind-merge' - import { createScrollArea, melt } from '@melt-ui/svelte' import { untrack } from 'svelte' + import ScrollableX from '../ScrollableX.svelte' interface Props { tabs: TabItem[] @@ -36,11 +36,28 @@ onReorder?: (newOrder: TabItem[]) => void /** Extra classes for the outer tab strip. */ class?: string - /** Render after the right-pinned tabs (e.g. a "Split with Preview" toggle). */ + /** Render inside the scroll row, right after the last tab (e.g. a "+" new-tab + * button) — scrolls with the tabs, unlike `trailing`. */ + afterTabs?: import('svelte').Snippet + /** Render after the right-pinned tabs, outside the scroll area so it stays + * pinned (e.g. a "Split with Preview" toggle). */ trailing?: import('svelte').Snippet + /** Render inside each tab, after the label and before the close button (e.g. a + * per-tab chevron/breadcrumb picker). Receives the tab and whether it's active. */ + tabAccessory?: import('svelte').Snippet<[TabItem, boolean]> } - let { tabs, activeId, onSelect, onClose, onReorder, class: c = '', trailing }: Props = $props() + let { + tabs, + activeId, + onSelect, + onClose, + onReorder, + class: c = '', + afterTabs, + trailing, + tabAccessory + }: Props = $props() const pinnedLeft = $derived(tabs.filter((t) => t.pinned === 'left')) const middle = $derived(tabs.filter((t) => !t.pinned)) @@ -60,27 +77,6 @@ if (!isDragging) dndMiddle = next }) - // `type: 'hover'` shows the custom bar only while hovering/scrolling the strip. - const { - elements: { root, viewport, content, scrollbarX, thumbX } - } = createScrollArea({ type: 'hover', hideDelay: 600, dir: 'ltr' }) - - // melt only re-measures the thumb when its *content* resizes, not the - // viewport — so a pane resize leaves the thumb stale. Detect width changes - // via `bind:clientWidth` and nudge melt by perturbing the 0×0 sentinel's box. - let viewportWidth = $state(0) - let resizeSentinel: HTMLSpanElement | undefined = $state(undefined) - $effect(() => { - void viewportWidth - const el = untrack(() => resizeSentinel) - if (!el) return - el.style.width = '1px' - const raf = requestAnimationFrame(() => { - el.style.width = '0px' - }) - return () => cancelAnimationFrame(raf) - }) - function handleConsider(e: CustomEvent>) { isDragging = true dndMiddle = e.detail.items @@ -144,6 +140,15 @@ {/if} {tab.label} + {#if tabAccessory} + + + + e.stopPropagation()}> + {@render tabAccessory(tab, isActive)} + + {/if} {#if tab.closable !== false} {/snippet} diff --git a/frontend/src/lib/components/copilot/chat/AIChat.svelte b/frontend/src/lib/components/copilot/chat/AIChat.svelte index add2c7b44f..038751f4b6 100644 --- a/frontend/src/lib/components/copilot/chat/AIChat.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChat.svelte @@ -19,7 +19,9 @@ forceDisabledMessage = '', wideLayout = false, emptyHint, - inputPreface + inputPreface, + initialInstructions = undefined, + onDraftChange = undefined }: { hideHeader?: boolean hideModeSelector?: boolean @@ -35,6 +37,9 @@ wideLayout?: boolean emptyHint?: import('svelte').Snippet inputPreface?: import('svelte').Snippet + // Seed / observe the composer's draft text (forwarded to AIChatDisplay). + initialInstructions?: string + onDraftChange?: (text: string) => void } = $props() const isAdmin = $derived($userStore?.is_admin || $userStore?.is_super_admin) @@ -163,7 +168,8 @@ {headerLeft} hasDiff={aiChatManager.scriptEditorOptions && !!aiChatManager.scriptEditorOptions.lastDeployedCode && - aiChatManager.scriptEditorOptions.lastDeployedCode !== aiChatManager.scriptEditorOptions.code} + aiChatManager.scriptEditorOptions.lastDeployedCode !== + aiChatManager.scriptEditorOptions.getCode()} diffMode={aiChatManager.scriptEditorOptions?.diffMode ?? false} {disabled} {disabledMessage} @@ -173,4 +179,6 @@ {wideLayout} {emptyHint} {inputPreface} + {initialInstructions} + {onDraftChange} > diff --git a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte index e9ac5eed16..3845c20435 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte @@ -10,6 +10,8 @@ ChevronDown, ChevronsRight, CheckIcon, + FileText, + Folder, Hand, HistoryIcon, Hourglass, @@ -26,9 +28,8 @@ import { isActiveUserQuestion, type DisplayMessage } from './shared' import type { ContextElement } from './context' import ChatQuickActions from './ChatQuickActions.svelte' - import ProviderModelSelector from './ProviderModelSelector.svelte' import ContextUsageIndicator from './ContextUsageIndicator.svelte' - import AIChatSettingsMenu from './AIChatSettingsMenu.svelte' + import AIChatModelSettings from './AIChatModelSettings.svelte' import ChatMode from './ChatMode.svelte' import DatatableCreationPolicy from './DatatableCreationPolicy.svelte' import Tooltip from '$lib/components/meltComponents/Tooltip.svelte' @@ -39,8 +40,18 @@ import ChatTypingIndicator from './ChatTypingIndicator.svelte' import AIChatInput from './AIChatInput.svelte' import QueuedMessageChip from './QueuedMessageChip.svelte' + import JobsSegment from './JobsSegment.svelte' import { getModifierKey } from '$lib/utils' import type { SelectedContext } from './app/core' + import AttachedFilesBar from './files/AttachedFilesBar.svelte' + import { type FileToAttach } from './files/attachedFiles.svelte' + import { + hasFileSystemAccess, + pickDirectory, + handlesFromDataTransfer, + readDroppedEntries + } from './files/fsAccess' + import { sendUserToast } from '$lib/toast' const MAX_YOLO_TOOLTIP_TOOLS = 8 const aiChatManager = getAiChatManager() @@ -105,7 +116,9 @@ hideModeSelector = false, wideLayout = false, emptyHint, - inputPreface + inputPreface, + initialInstructions = undefined, + onDraftChange = undefined }: { messages: DisplayMessage[] pastChats: { id: string; title: string }[] @@ -132,6 +145,9 @@ wideLayout?: boolean emptyHint?: Snippet inputPreface?: Snippet + // Seed / observe the main composer's draft text (see AIChatInput). + initialInstructions?: string + onDraftChange?: (text: string) => void } = $props() let aiChatInput: AIChatInput | undefined = $state() @@ -242,16 +258,142 @@ const showTypingIndicator = $derived(aiChatManager.loading) - // `@` context picker is offered in modes that accept workspace/script/flow - // references (SCRIPT, FLOW, GLOBAL → workspace items + code blocks) or in - // APP mode (datatables, frontend files, etc.). Other modes (NAVIGATOR, - // ASK, API) don't accept @-context. + // The manual `@` context-picker button. Shown in SCRIPT/FLOW (workspace items + + // code blocks) and APP (datatables, frontend files). Hidden in GLOBAL — there + // `@`-context is still invoked inline by typing `@` in the input, so the button + // is redundant. NAVIGATOR/ASK/API don't take @-context at all. const showContextPicker = $derived( aiChatManager.mode === AIMode.SCRIPT || aiChatManager.mode === AIMode.FLOW || - aiChatManager.mode === AIMode.GLOBAL || aiChatManager.mode === AIMode.APP ) + + // File attachment is GLOBAL-mode only. + const canAttachFiles = $derived(aiChatManager.mode === AIMode.GLOBAL && !disabled) + // Steers the OS file picker toward text formats (soft hint; content sniff is authoritative). + const TEXT_FILE_ACCEPT = + 'text/*,.txt,.csv,.tsv,.json,.jsonl,.ndjson,.md,.markdown,.log,.yaml,.yml,.toml,.ini,.cfg,.conf,.env,.xml,.html,.htm,.css,.js,.mjs,.cjs,.ts,.tsx,.jsx,.py,.rb,.rs,.go,.java,.kt,.c,.h,.cpp,.cc,.cs,.php,.sh,.bash,.zsh,.sql,.svelte,.vue,.dockerfile' + let fileInputEl = $state(null) + let folderInputEl = $state(null) + let dragDepth = $state(0) + const isDraggingFiles = $derived(dragDepth > 0) + // File System Access API → live re-grantable folder handles (refreshed each turn). + // Otherwise folders are snapshotted into the browser (via webkitdirectory / dropped-entry + // walk), same as files. Either way folders display identically. + const canUseFsAccess = hasFileSystemAccess() + + function reportAddResult(added: string[], rejected: { name: string; reason: string }[]) { + if (rejected.length === 0) return + // Single rejected file (e.g. one dropped image): show the precise reason. + if (added.length === 0 && rejected.length === 1) { + sendUserToast(`Could not attach "${rejected[0].name}": ${rejected[0].reason}`, true) + return + } + // Otherwise (folders / multi-select): summarize to avoid a flood of toasts. The only + // per-file rejection left is non-text content (binary files are skipped). + const lead = added.length + ? `Attached ${added.length}, skipped ${rejected.length}` + : `Skipped ${rejected.length} file${rejected.length === 1 ? '' : 's'}` + sendUserToast(`${lead} (non-text).`, added.length === 0) + } + + async function handleAddFiles(files: FileList | FileToAttach[]) { + const { added, rejected } = await aiChatManager.attachedFiles.addFiles(files) + reportAddResult(added, rejected) + } + + async function addDirHandle(dir: FileSystemDirectoryHandle) { + const { added, rejected } = await aiChatManager.attachedFiles.addFolder(dir) + reportAddResult(added, rejected) + } + + function linkFiles() { + // Files are always snapshotted (every browser), so the universal picker is fine. + fileInputEl?.click() + } + + async function linkFolder() { + if (!canUseFsAccess) { + // No File System Access API → pick a folder via the directory input; its files are + // snapshotted into the browser (no live handle), grouped under the folder name. + folderInputEl?.click() + return + } + let dir: FileSystemDirectoryHandle | undefined + try { + dir = await pickDirectory() + } catch (e) { + // The picker threw instead of opening — surface why (e.g. a browser/enterprise + // policy blocking the File System Access API) rather than appearing to do nothing. + sendUserToast( + `Couldn't open the folder picker: ${e instanceof Error ? e.message : String(e)}`, + true + ) + return + } + if (dir) await addDirHandle(dir) + } + + function dragHasFiles(e: DragEvent): boolean { + return Array.from(e.dataTransfer?.types ?? []).includes('Files') + } + + function onPanelDragEnter(e: DragEvent) { + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + dragDepth++ + } + function onPanelDragOver(e: DragEvent) { + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + if (e.dataTransfer) e.dataTransfer.dropEffect = 'copy' + } + function onPanelDragLeave(_e: DragEvent) { + if (!canAttachFiles) return + dragDepth = Math.max(0, dragDepth - 1) + } + async function onPanelDrop(e: DragEvent) { + dragDepth = 0 + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + const dt = e.dataTransfer + if (!dt) return + if (canUseFsAccess) { + // getAsFileSystemHandle calls are kicked off synchronously inside this call. + const handles = await handlesFromDataTransfer(dt) + for (const h of handles) { + if (h.kind === 'directory') { + // Folders link as a live handle. + await addDirHandle(h as FileSystemDirectoryHandle) + } else { + // Files are always snapshotted (handle discarded). + await handleAddFiles([{ file: await (h as FileSystemFileHandle).getFile() }]) + } + } + } else { + // Fallback (no File System Access API): snapshot dropped files AND folders by walking + // the legacy webkitGetAsEntry tree. readDroppedEntries reads the entries synchronously + // (they're only valid during this event) before its first await; if it yields nothing + // (no entry API), fall back to the flat dt.files. + const entries = await readDroppedEntries(Array.from(dt.items ?? [])) + if (entries.length > 0) await handleAddFiles(entries) + else if (dt.files.length > 0) await handleAddFiles(dt.files) + } + } + + function onFileInputChange(e: Event) { + const input = e.currentTarget as HTMLInputElement + if (input.files && input.files.length > 0) void handleAddFiles(input.files) + input.value = '' // allow re-selecting the same file + } + + function onFolderInputChange(e: Event) { + const input = e.currentTarget as HTMLInputElement + // webkitdirectory files carry webkitRelativePath (`folder/sub/file`); addFiles groups + // them under the folder and skips junk paths. Snapshot, like a dropped folder. + if (input.files && input.files.length > 0) void handleAddFiles(input.files) + input.value = '' + } const availableAutonomyModeOptions = $derived.by(() => autonomyModeOptions.filter((option) => isAutonomyModeAvailable( @@ -339,7 +481,28 @@ -
+
+ {#if isDraggingFiles} +
+
+ + Drop files to attach +
+
+ {/if} {#if !hideHeader}
{#if messages.length > 0}
-
+
{/if}
+ {#if aiChatManager.mode === AIMode.GLOBAL && !aiChatManager.isSessionChat} + +
+ +
+ {/if} + {#if aiChatManager.mode === AIMode.GLOBAL} + + + {/if} {#if inputPreface} {@render inputPreface()} {/if} @@ -544,6 +724,9 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. --> bind:this={aiChatInput} bind:selectedContext {availableContext} + {initialInstructions} + {onDraftChange} + showContext={aiChatManager.mode !== AIMode.GLOBAL} disabled={disabled || hasActiveUserQuestion} isFirstMessage={messages.length === 0} /> @@ -595,11 +778,76 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. --> setShowing={(showing) => { if (!showing) close() }} + onSelectFile={(name) => { + aiChatInput?.insertFileMention(name) + close() + }} /> {/if} {/snippet} {/if} + {#if canAttachFiles} + [ + { displayName: 'Attach file', icon: FileText, action: () => linkFiles() }, + { + // A real (live) link needs the File System Access API; without it the + // folder is only snapshotted, so call it "Add folder", not "Link folder". + displayName: canUseFsAccess ? 'Link folder' : 'Add folder', + icon: Folder, + tooltip: canUseFsAccess + ? 'Linked live — the assistant reads the folder’s current files from disk and refreshes each turn.' + : 'Loaded as a snapshot — the folder’s files are copied into your browser (they won’t auto-update). For a live link that refreshes from disk, use a Chromium-based browser (Chrome, Edge).', + action: () => linkFolder() + } + ]} + placement="bottom-start" + fixedHeight={false} + > + {#snippet buttonReplacement()} + +
{/if}
-
- -
{#if (aiChatManager.mode === AIMode.NAVIGATOR || aiChatManager.mode === AIMode.ASK) && suggestions.length > 0 && messages.filter((m) => m.role === 'user').length === 0 && !disabled}
diff --git a/frontend/src/lib/components/copilot/chat/AIChatInput.svelte b/frontend/src/lib/components/copilot/chat/AIChatInput.svelte index 3d9e5c550f..8f39b78a27 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatInput.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChatInput.svelte @@ -6,6 +6,7 @@ import type { ContextElement } from './context' import { AIMode } from './AIChatManager.svelte' import { CHAT_INPUT_PADDING, getAiChatManager } from './aiChatManagerContext' + import { formatMention } from './mention' import { twMerge } from 'tailwind-merge' import { tick, untrack, type Snippet } from 'svelte' import Portal from '$lib/components/Portal.svelte' @@ -41,6 +42,10 @@ loading?: boolean // Called when the user clicks Stop. Defaults to `aiChatManager.cancel()`. onCancel?: () => void + // Observe the composer draft as it changes (the text is local state — + // `aiChatManager.instructions` only carries programmatic prompts). Used by + // sessions to persist the typed-but-unsent prompt with the session draft. + onDraftChange?: (text: string) => void } let { @@ -60,7 +65,8 @@ bottomRightSnippet, onKeyDown = undefined, loading, - onCancel + onCancel, + onDraftChange = undefined }: Props = $props() // GLOBAL-mode suggestion pool. We pick one at mount-time so each new @@ -117,6 +123,10 @@ let contextTextareaComponent: ContextTextarea | undefined = $state() let instructionsTextareaComponent: HTMLTextAreaElement | undefined = $state() let instructions = $state(untrack(() => initialInstructions)) + $effect(() => { + const text = instructions + untrack(() => onDraftChange?.(text)) + }) // Collapsed big-paste blobs referenced by tokens in `instructions`. let pastes = $state(untrack(() => initialPastes ?? [])) @@ -196,6 +206,13 @@ focusInput() } + /** Insert a plain @filename mention for an attached file (used by the @ menu Files category). */ + export function insertFileMention(name: string) { + const sep = instructions.length === 0 || instructions.endsWith(' ') ? '' : ' ' + instructions = `${instructions}${sep}${formatMention(name)} ` + focusInput() + } + function clickOutside(node: HTMLElement) { function handleClick(event: MouseEvent) { if (node && !node.contains(event.target as Node)) { @@ -222,7 +239,9 @@ // Workspace items are fetched on-demand and not in availableContext, // so skip the availableContext check for them const isWorkspaceItem = - contextElement.type === 'workspace_script' || contextElement.type === 'workspace_flow' + contextElement.type === 'workspace_script' || + contextElement.type === 'workspace_flow' || + contextElement.type === 'workspace_app' if ( !isWorkspaceItem && !availableContext.find( diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts index 0eb9a79707..c58c3311c3 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts @@ -1,5 +1,5 @@ import type { ScriptLang } from '$lib/gen/types.gen' -import { WorkspaceService } from '$lib/gen' +import { WorkspaceService, JobService, type CompletedJob } from '$lib/gen' import type { FlowOptions, ScriptOptions } from './ContextManager.svelte' import { flowTools, @@ -19,7 +19,15 @@ import { type DisplayMessage, type Tool, type ToolCallbacks, - type ToolDisplayMessage + type ToolDisplayMessage, + type UserQuestionDisplay, + type ChatJob, + type ChatJobInit, + type ChatJobStatus, + completedJobToolStatus, + backgroundJobCompletionNote, + deriveChatJobStatus, + trimJob } from './shared' import type { ChatCompletionMessageParam, @@ -44,6 +52,9 @@ import { buildSummaryMessageContent } from './compactionPrompt' import { dfs } from '$lib/components/flows/previousResults' +import { SvelteSet } from 'svelte/reactivity' +import type { UserDraftItemKind } from '$lib/gen' +import { maskKey } from '$lib/components/sessions/modifiedItemsMask' import { getStringError } from './utils' import { type PasteAttachment } from './pasteTokens' import { chatDraft, expanded } from './chatDraft' @@ -55,6 +66,7 @@ import { BROWSER } from 'esm-env' import { workspaceStore, type DBSchemas } from '$lib/stores' import { askTools, prepareAskSystemMessage, prepareAskUserMessage } from './ask/core' import { readDocsPageTool, searchDocsTool } from './docs/core' +import { TypewriterReveal } from './typewriterReveal' import { chatState, DEFAULT_SIZE, triggerablesByAi } from './sharedChatState.svelte' import { createAppBackendRunnableContextElement, @@ -67,24 +79,43 @@ import type { Selection } from 'monaco-editor' import type AIChatInput from './AIChatInput.svelte' import { prepareApiSystemMessage, prepareApiUserMessage } from './api/core' import { runChatLoop, truncateToToolPairedPrefix } from './chatLoop' +import { sanitizeToolCallArguments } from './toolCallArguments' import { normalizeContextUsage } from './tokenUsage' import type { ReviewChangesOpts } from './monaco-adapter' import { getCurrentModel, tryGetCurrentModel, getCombinedCustomPrompt, + getCustomPromptParts, + getUserCustomPrompts, + setUserCustomPrompts, isWebSearchEnabledForProvider } from '$lib/aiStore' import type { WorkspaceMutationTarget } from './workspaceTools' import { globalToolsFor, + loadWorkspaceSkills, prepareGlobalSystemMessage, prepareGlobalUserMessage, + type AiSkillListItem, type GlobalToolHelpers } from './global/core' +import { formatChatJobCompletion } from './datatableTools' import { isGlobalAiEnabled } from './global/gate' +import { + pipelineTools, + getPipelinePromptSection, + type PipelineAIChatHelpers +} from './pipeline/core' import { scopedKey, onUserChange, migrateLegacyLocalStorage } from '$lib/userScopedStorage' import { getLocalSetting, storeLocalSetting } from '$lib/utils' +import { AttachedFilesStore } from './files/attachedFiles.svelte' +import { appendAttachedFilesRoster } from './files/fileTools' + +// SSR and users who prefer reduced motion get no typewriter pacing. +function prefersInstantReveal(): boolean { + return !BROWSER || (window.matchMedia?.('(prefers-reduced-motion: reduce)').matches ?? false) +} // Compaction of the stored history: once the projected request size // (contextTokens — the provider's report when current, a fresh chars/4 @@ -109,6 +140,15 @@ const MAX_CONSECUTIVE_COMPACTION_FAILURES = 3 // (panel teardown, save-and-clear) pass their own reason, so the queued-message // flush can tell "the user wants to move on" from "the turn was torn down". const USER_CANCEL_REASON = 'user_cancelled' +// Built-in `/compact` session command — summarizes the conversation locally +// instead of sending a turn to the model. Matched on the whole input so a +// regular message that merely mentions "/compact" mid-sentence is unaffected. +const COMPACT_COMMAND_NAME = 'compact' +const COMPACT_COMMAND_RE = /^\/compact\s*$/ +// Built-in `/clear` session command — saves the conversation to history and +// resets to a fresh chat (the "New chat" action), instead of sending a turn. +const CLEAR_COMMAND_NAME = 'clear' +const CLEAR_COMMAND_RE = /^\/clear\s*$/ const AI_AUTONOMY_MODE_STORAGE_KEY = 'ai-chat-autonomy-mode' const LEGACY_AUTO_ACCEPT_TOOL_CONFIRMATIONS_STORAGE_KEY = 'ai-chat-yolo-mode' const WEB_SEARCH_ERROR_HINT = @@ -230,12 +270,15 @@ function getSendRequestErrorMessage(err: unknown, webSearchUnavailable: boolean) export class AIChatManager { contextManager = new ContextManager() historyManager = new HistoryManager() + /** Files the user attached to the current GLOBAL-mode conversation. */ + attachedFiles = new AttachedFilesStore() abortController: AbortController | undefined = undefined inlineAbortController: AbortController | undefined = undefined // Flag to skip Responses API if it's not available (e.g., Azure region doesn't support it) skipResponsesApi = false mode = $state(AIMode.NAVIGATOR) + pipelineAiChatHelpers = $state(undefined) readonly isOpen = $derived(chatState.size > 0) savedSize = $state(0) instructions = $state('') @@ -245,10 +288,52 @@ export class AIChatManager { // the turn finishes (clean completion or user cancel). Ephemeral — never // saved to displayMessages or history. queuedMessage = $state('') + // Jobs the chat started that detached into the background (global/sessions + // chat only). Rendered in the jobs tray, persisted with the chat, and advanced + // by a single background poller. See registerJob / #pollBackgroundJobs. + backgroundJobs = $state([]) + // Completion notes for finished background jobs awaiting delivery to the model. + // Drained as a preamble into the next turn — either the user's next message, or, + // when the chat is idle, an auto-resume turn started for them (see + // #maybeAutoResumeFromJobs). Ephemeral like queuedMessage — not persisted. + pendingJobNotes = $state([]) + // Guards #maybeAutoResumeFromJobs against re-entering while its own turn spins up. + #autoResuming = false + #jobPollTimer: ReturnType | undefined = undefined + #jobPollDelay = 2000 + // True while a #pollBackgroundJobs pass is executing. #stopJobPoller only clears + // the scheduled timer, not an in-flight poll, so without this a refreshBackgroundJobs + // mid-poll (cancel / approval close) would start a second concurrent poll chain and + // double the poll rate. The guard makes such a refresh coalesce into the running pass. + #isPolling = false + // Bumped on every conversation switch (clearBackgroundJobs). An in-flight poll + // captures it before its awaits and bails if it changed, so a getJob that + // resolves after the user switched chats can't mutate the newly-loaded one. + #jobPollGeneration = 0 + // Consecutive getJob failures per background job, so a vanished/404 job can be + // drained instead of polled forever. Ephemeral, keyed by jobId. + #jobPollFailures = new Map() + /** Opens a run in the sessions preview pane. Set by the session runtime; + * undefined in the global side-panel chat, where the tray falls back to opening + * the run in a new browser tab. */ + openRunInPreview?: (a: { jobId: string; workspace: string; label: string }) => void loading = $state(false) currentReply = $state('') currentReasoning = $state('') currentReasoningActive = $state(false) + // Smooths the provider's bursty delivery into continuous typing by revealing + // buffered text a slice per frame. The reply and the reasoning/thinking stream + // each get their own reveal (independent buffers, both append to their own + // $state). Reduced-motion (sampled once — the pref never changes mid-session) + // and SSR fall back to instant. + private replyReveal = new TypewriterReveal({ + onReveal: (chunk) => (this.currentReply += chunk), + instant: prefersInstantReveal() + }) + private reasoningReveal = new TypewriterReveal({ + onReveal: (chunk) => (this.currentReasoning += chunk), + instant: prefersInstantReveal() + }) displayMessages = $state([]) messages = $state([]) /** Provider-reported context size of the last committed turn (prompt + @@ -305,7 +390,7 @@ export class AIChatManager { cachedDatatables = $state([]) private confirmationCallbacks = new Map void>() - private userQuestionCallbacks = new Map void>() + private userQuestionCallbacks = new Map void>() private appDatatablesRefreshTimeout: ReturnType | undefined = undefined disabledModes: Partial> = $state({}) @@ -317,6 +402,433 @@ export class AIChatManager { // tool `helpers` in GLOBAL mode so the preview/deploy tools dispatch to THIS // session rather than the UI-active one — keeps backgrounded sessions isolated. sessionId: string | undefined = undefined + // Resolves the workspace this chat operates on. Session chats set it to their + // own (possibly forked) workspace so the chat targets it WITHOUT switching the + // global workspaceStore. Undefined for the global side-panel chat, which + // follows the active workspace. Always read via `operatingWorkspace`. + workspaceResolver: (() => string | undefined) | undefined = undefined + + // The workspace every workspace-scoped chat action targets — skills, tool + // loop, logging, user-message context, and commit. Session-resolved when a + // resolver is set, else the globally-active workspace. + get operatingWorkspace(): string | undefined { + return this.workspaceResolver?.() ?? get(workspaceStore) + } + + // Fired whenever the active chat id changes away from the one the consumer + // knows (a "/clear" rotation or a history switch). Session runtimes wire this + // to keep the session record's chatId aligned — the compare-page handoff + // (`from_session`) reads it, and a stale id would preselect the previous + // chat's items. Set here (not imported) to avoid a copilot→sessions cycle. + onChatRotated: ((chatId: string) => void) | undefined = undefined + + // Workspace items the CURRENT chat modified via AI tool calls, as + // `${UserDraftItemKind}:${storagePath}` keys (see modifiedItemsMask.ts). + // undefined = untracked: only the global side-panel chat (never initialised), + // which falls back to the show-all bar. Session chats are always tracked (a + // SvelteSet, even empty) — see loadPastChat/initRuntime — so their Edits + // surface never claims drafts the session didn't touch. Reactive so the + // session bar updates as tools record mid-turn. + modifiedItems = $state | undefined>(undefined) + + // Start tracking for a brand-new session chat (empty = "tracked, nothing yet"). + initModifiedItemsTracking() { + this.modifiedItems = new SvelteSet() + } + + // Record an item an AI tool call created/edited/deleted. No-op when untracked + // (the global singleton never initialises the set), so it stays unaffected. + recordModifiedItem(itemKind: UserDraftItemKind, storagePath: string) { + this.modifiedItems?.add(maskKey(itemKind, storagePath)) + } + + // Un-record an item whose chat-made change was discarded — without this the + // still-existing deployed item would keep reading as this chat's "Deployed" + // edit. Persisted immediately: unlike recordModifiedItem (whose persistence + // rides on the turn's saveChat), a discard can fire from the review dock + // outside any turn, and waiting would resurrect the entry on reload. + async removeModifiedItem(itemKind: UserDraftItemKind, storagePath: string) { + if (!this.modifiedItems?.delete(maskKey(itemKind, storagePath))) return + await this.#persistModifiedItems() + } + + // Move a mask entry to the path a draft actually deployed to. A draft-only + // flow/app parks at a synthetic `draft_{uuid}` storage path and deploys to + // its chosen path — without the move, the existence check at the synthetic + // path fails after reload and the deployed row vanishes from the dock. + async renameModifiedItem(itemKind: UserDraftItemKind, fromPath: string, toPath: string) { + if (fromPath === toPath) return + if (!this.modifiedItems?.delete(maskKey(itemKind, fromPath))) return + this.modifiedItems.add(maskKey(itemKind, toPath)) + await this.#persistModifiedItems() + } + + // Serialized, snapshot-at-write-time persistence: two rapid dock actions + // would otherwise race their saveChat writes, and the earlier (staler) + // snapshot could land last — dropping the later mutation until the next + // turn-end save. + #maskPersistQueue: Promise = Promise.resolve() + #persistModifiedItems(): Promise { + this.#maskPersistQueue = this.#maskPersistQueue.then(() => + this.historyManager + .saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) + // Swallow (and log) a failed write so it can't wedge the queue as a + // rejected link — the next persist snapshots the full current set, so + // a lost write self-heals on the next mutation or turn-end save. + .catch((e) => console.error('Failed to persist modified-items mask', e)) + ) + return this.#maskPersistQueue + } + + // ===== Background jobs (global/sessions chat only) ===== + // + // A test-run tool that doesn't finish within the inline wait detaches: it + // returns a "still running" handle to the model and registers the job here. + // A single poller advances all detached jobs; on completion it fills the tool + // card and queues a notify-only note for the model's next turn. + + private isJobNonTerminal(status: ChatJobStatus): boolean { + // suspended (awaiting approval) and scheduled are non-terminal — the poller + // MUST keep watching them, else an approval would never clear from the tray. + return ( + status === 'queued' || + status === 'running' || + status === 'suspended' || + status === 'scheduled' + ) + } + + /** Record a job the moment it starts, so the tray shows it while it is still + * inline-waiting. Idempotent on jobId. The init carries the serializable + * `resultFormat` (persisted), so completion formatting survives a reload. */ + registerJob = (init: ChatJobInit) => { + if (this.backgroundJobs.some((j) => j.jobId === init.jobId)) return + this.backgroundJobs = [ + ...this.backgroundJobs, + { ...init, createdAt: Date.now(), status: 'queued', detached: false, reported: false } + ] + } + + /** Merge a partial update into a tracked job by id. */ + updateJob = (jobId: string, update: Partial) => { + const idx = this.backgroundJobs.findIndex((j) => j.jobId === jobId) + if (idx === -1) return + this.backgroundJobs[idx] = { ...this.backgroundJobs[idx], ...update } + this.backgroundJobs = [...this.backgroundJobs] + } + + /** A job left the inline wait — hand it to the background poller. */ + markJobDetached = (jobId: string) => { + this.updateJob(jobId, { detached: true }) + this.#ensureJobPoller() + void this.#persistBackgroundJobs() + } + + /** User-facing cancel from the jobs tray. */ + cancelJob = async (jobId: string) => { + const job = this.backgroundJobs.find((j) => j.jobId === jobId) + if (!job) return + try { + await JobService.cancelQueuedJob({ workspace: job.workspace, id: jobId, requestBody: {} }) + // Don't mark terminal here: a bare `status: 'canceled'` would (a) leave the + // `job` snapshot that drives JobStatusIcon stale (badge stuck on running) + // and (b) make isJobNonTerminal false so the poller stops before it can + // refresh either. Let the poller observe the canceled CompletedJob and set + // status + job together; poke it so the tray converges within a tick. + this.refreshBackgroundJobs() + } catch (e) { + console.error('Failed to cancel job', jobId, e) + sendUserToast('Failed to cancel job', true) + } + } + + /** Remove a finished job from the tray. */ + dismissJob = (jobId: string) => { + this.backgroundJobs = this.backgroundJobs.filter((j) => j.jobId !== jobId) + void this.#persistBackgroundJobs() + } + + /** Force an immediate background-job poll (e.g. right after an approval) instead + * of waiting for the next scheduled tick. */ + refreshBackgroundJobs = () => { + this.#stopJobPoller() + this.#jobPollDelay = 2000 + void this.#pollBackgroundJobs() + } + + #ensureJobPoller() { + if (this.#jobPollTimer !== undefined) return + // A poll pass is running (it cleared #jobPollTimer on entry). It reschedules + // from the current job set when it finishes, so the job that just detached is + // already covered. Scheduling here instead would create a second timer that the + // end-of-pass reschedule overwrites WITHOUT clearing — orphaning it into a + // duplicate, self-perpetuating poll chain. Coalesce into the active pass. + if (this.#isPolling) return + if (!this.backgroundJobs.some((j) => j.detached && this.isJobNonTerminal(j.status))) return + this.#jobPollDelay = 2000 + this.#scheduleJobPoll() + } + + #scheduleJobPoll() { + this.#jobPollTimer = setTimeout(() => void this.#pollBackgroundJobs(), this.#jobPollDelay) + } + + #stopJobPoller() { + if (this.#jobPollTimer !== undefined) { + clearTimeout(this.#jobPollTimer) + this.#jobPollTimer = undefined + } + } + + // Guarded entry point for every poll trigger (scheduled tick, #ensureJobPoller, + // and refreshBackgroundJobs): if a pass is already running, coalesce into it + // instead of starting a second concurrent chain that would double the poll rate. + async #pollBackgroundJobs() { + if (this.#isPolling) return + this.#isPolling = true + try { + await this.#runBackgroundJobsPoll() + } finally { + this.#isPolling = false + } + } + + async #runBackgroundJobsPoll() { + this.#jobPollTimer = undefined + const gen = this.#jobPollGeneration + const pending = this.backgroundJobs.filter((j) => j.detached && this.isJobNonTerminal(j.status)) + if (pending.length === 0) return + + let anyTerminal = false + for (const job of pending) { + try { + const fetched = await JobService.getJob({ + workspace: job.workspace, + id: job.jobId, + noLogs: false, + noCode: true + }) + // The user switched conversations while this getJob was in flight; its + // result belongs to a chat that's gone. Drop it rather than mutate the + // newly-loaded one (which re-armed its own poller on load). + if (gen !== this.#jobPollGeneration) return + this.#jobPollFailures.delete(job.jobId) + if (fetched.type === 'CompletedJob') { + anyTerminal = true + this.#onBackgroundJobComplete(job, fetched as CompletedJob) + } else { + // Store the derived status and the trimmed Job together so the tray + // badge (JobStatusIcon) and the scalar status can never drift. + this.updateJob(job.jobId, { + status: deriveChatJobStatus(fetched), + job: trimJob(fetched) + }) + } + } catch (e) { + // Same generation guard as the success path — a switch during the failing + // getJob means this result is for a conversation that's gone. + if (gen !== this.#jobPollGeneration) return + // A vanished job (404) or repeated failures must not keep the poller + // alive forever — now that suspended/scheduled are polled too, drain it + // as failed so isJobNonTerminal lets the poller stop. + const httpStatus = (e as { status?: number })?.status + const failures = (this.#jobPollFailures.get(job.jobId) ?? 0) + 1 + this.#jobPollFailures.set(job.jobId, failures) + if (httpStatus === 404 || failures >= 5) { + this.#jobPollFailures.delete(job.jobId) + // Vanished (404) or unreachable after repeated polls. Mark it failed WITH + // a snapshot + tool-card patch (mirroring #onBackgroundJobComplete) so + // neither the tray badge nor the launching tool card stays frozen on + // "running" — a bare `status: 'failure'` with no `job` would render the + // orange queued badge (JobsSegment's `!job.job` fallback). The synthetic + // failed CompletedJob keeps the `success`-key discriminant so JobStatusIcon + // and deriveChatJobStatus agree. No model note/auto-resume: a vanished job + // isn't a meaningful completion to react to (usually transient infra). + const gone = { + type: 'CompletedJob', + id: job.jobId, + success: false, + canceled: false + } as unknown as CompletedJob + this.updateJob(job.jobId, { status: 'failure', reported: true, job: trimJob(gone) }) + this.applyToolStatus(job.toolCallId, { + content: 'Background job could not be retrieved (it may have been removed)', + error: `Job ${job.jobId} was unreachable` + }) + anyTerminal = true + } else { + console.error('Failed to poll background job', job.jobId, e) + } + } + } + + if (anyTerminal) { + void this.#persistBackgroundJobs() + } + + // Reschedule while anything is still in flight, backing off up to 5s. + if (this.backgroundJobs.some((j) => j.detached && this.isJobNonTerminal(j.status))) { + this.#jobPollDelay = Math.min(this.#jobPollDelay + 1000, 5000) + this.#scheduleJobPoll() + } + + // Something finished this cycle — if the chat is idle, react to it now + // instead of waiting for the user's next message. Fire-and-forget so the + // poller loop above isn't blocked by the turn. + if (anyTerminal) void this.#maybeAutoResumeFromJobs() + } + + #onBackgroundJobComplete(job: ChatJob, completed: CompletedJob) { + const status = deriveChatJobStatus(completed) + this.updateJob(job.jobId, { + status, + durationMs: completed.duration_ms, + reported: true, + job: trimJob(completed) + }) + // If the launching tool stamped a resultFormat, reconstruct its shaped card + + // model text so the detached path reports the same contract the inline path + // would (row-capped rows, friendly datatable errors) — even after a reload, + // since resultFormat is persisted on the job. A canceled job skips formatting: + // its card is the neutral "canceled" state, not a result. + const formatted = + status === 'canceled' || !job.resultFormat + ? undefined + : formatChatJobCompletion(completed, job.resultFormat) + // Fill the tool card that launched it (we run outside a turn here). + this.applyToolStatus(job.toolCallId, formatted?.card ?? completedJobToolStatus(completed)) + // A user-canceled job needs no model note or auto-resume: the user stopped it + // deliberately, so announcing it (as "FAILED", since a canceled job isn't a + // success) or burning a turn on it would be noise. + if (status === 'canceled') return + // Queue a completion note for the model. Delivered on the next turn — + // either the user's next message or an idle auto-resume (fired by the poller). + this.pendingJobNotes = [ + ...this.pendingJobNotes, + backgroundJobCompletionNote(job.jobId, job.label, completed, formatted?.llmText) + ] + } + + /** + * Stage 2 wake: when a background job finishes and the chat is otherwise idle, + * start a turn on the user's behalf so the model reacts to the result (reports + * it, continues the plan) instead of waiting for the next manual message. The + * rich completion note reaches the model via the pendingJobNotes preamble in + * sendRequest; the visible bubble is just a short, clearly-automated line. + * + * Bounded so it can't run away: fires only when idle (no in-flight turn) and + * only when notes exist — and sendRequest drains the notes, so a turn that + * doesn't spawn a new job leaves nothing to re-trigger on. A turn that DOES + * spawn another job resumes again when that one finishes, which is the point. + */ + async #maybeAutoResumeFromJobs() { + if (this.#autoResuming) return + // Global/sessions chat only (the only mode with a jobs tray + preamble). + if (this.mode !== AIMode.GLOBAL) return + // Mid-turn: the notes will ride that turn's preamble, so don't start another. + if (this.loading) return + if (this.pendingJobNotes.length === 0) return + // Nothing to continue (empty chat), or the user is mid-compose — don't + // clobber their draft or auto-send it. Their eventual send carries the notes. + if (this.messages.length === 0 || this.instructions.trim()) return + this.#autoResuming = true + try { + const count = this.pendingJobNotes.length + this.instructions = + count === 1 ? 'A background job just finished.' : `${count} background jobs just finished.` + await this.sendRequest() + } catch (e) { + console.error('Auto-resume after background job failed', e) + } finally { + this.#autoResuming = false + } + } + + // Serialized snapshot-at-write persistence, mirroring #persistModifiedItems. + // Omits the modified-items mask so a concurrent mask write isn't clobbered + // (saveChat keeps the prior mask when it is undefined). + #jobPersistQueue: Promise = Promise.resolve() + #persistBackgroundJobs(): Promise { + this.#jobPersistQueue = this.#jobPersistQueue.then(() => + this.historyManager + .saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + undefined, + $state.snapshot(this.backgroundJobs) + ) + .catch((e) => console.error('Failed to persist background jobs', e)) + ) + return this.#jobPersistQueue + } + + /** Reset background-job state on conversation switch. */ + private clearBackgroundJobs() { + this.#stopJobPoller() + // Invalidate any in-flight poll so its post-await continuation can't write + // into the conversation we're switching to. + this.#jobPollGeneration++ + this.backgroundJobs = [] + this.pendingJobNotes = [] + } + + /** Merge a status patch into the tool card identified by tool_call_id, or + * create it. Shared by the per-turn setToolStatus callback and the background + * job poller (which runs outside a turn). */ + applyToolStatus = (id: string, metadata?: Partial) => { + const existingIdx = this.displayMessages.findIndex( + (m) => m.role === 'tool' && m.tool_call_id === id + ) + if (existingIdx !== -1) { + const existing = this.displayMessages[existingIdx] as ToolDisplayMessage + if (existing.content.length === 0 && metadata?.error) { + this.displayMessages[existingIdx].content = metadata.error + } + this.displayMessages[existingIdx] = { + ...existing, + ...(metadata || {}) + } as ToolDisplayMessage + } else { + const newMessage: ToolDisplayMessage = { + role: 'tool', + tool_call_id: id, + content: metadata?.content ?? metadata?.error ?? '', + ...(metadata || {}) + } + this.displayMessages.push(newMessage) + } + } + + // Workspace AI skills (name + description) advertised in the GLOBAL system + // prompt and surfaced as slash commands in session chat. Loaded + // asynchronously when entering GLOBAL mode; the system message is rebuilt + // once they resolve. + globalSkills = $state([]) + private globalSkillsRefreshId = 0 + + // Built-in session-chat slash commands, listed in the command picker + // alongside workspace skills. Unlike a skill, these run locally and never + // reach the model; the submit path intercepts them first, so they shadow any + // workspace skill of the same name. + readonly sessionBuiltinCommands: AiSkillListItem[] = [ + { name: COMPACT_COMMAND_NAME, description: 'Summarize the conversation to free up context' }, + { name: CLEAR_COMMAND_NAME, description: 'Clear the conversation and start a new chat' } + ] + + // Built-ins followed by workspace skills, with any skill whose name collides + // with a built-in dropped: the picker keys leaves by name, so a duplicate + // would break its keyed list and ambiguous-resolve nav. Built-ins win — they + // already shadow same-named skills at execution (the submit interception). + sessionCommands: AiSkillListItem[] = $derived([ + ...this.sessionBuiltinCommands, + ...this.globalSkills.filter((s) => !this.sessionBuiltinCommands.some((b) => b.name === s.name)) + ]) allowedModes: Record = $derived({ script: @@ -419,6 +931,67 @@ export class AIChatManager { return freed } + /** + * Core summarize + rewrite, shared by automatic and manual compaction. Sends + * the prefix to the summarizer, then replaces the summarized prefix with a + * single summary message in `messages` (as a user message) and + * `displayMessages` (as a `summary` boundary). Surviving tail user messages + * have their restart `index` re-based onto the new history: the summary + * occupies slot 0, so a tail user message that was at `keepFrom` lands at slot + * 1. `displayKeepFrom` is where the kept tail begins in `displayMessages`. + * + * Owns only the `compacting` flag and the history rewrite; callers own trigger + * policy (circuit breaker, gates) and persistence. Returns the outcome — + * 'aborted' is a user Stop (history left untouched), distinct from 'error'. + */ + private runSummarization = async ( + prefix: ChatCompletionMessageParam[], + tail: ChatCompletionMessageParam[], + keepFrom: number, + displayKeepFrom: number, + abortController: AbortController + ): Promise<'ok' | 'empty' | 'aborted' | 'error'> => { + this.compacting = true + try { + const raw = await getNonStreamingCompletion( + [ + ...sanitizeToolCallArguments(prefix), + { role: 'user', content: getCompactionSummaryPrompt() } + ], + abortController + ) + const formatted = formatCompactSummary(raw ?? '') + if (!formatted) { + return 'empty' + } + + this.messages = [{ role: 'user', content: buildSummaryMessageContent(formatted) }, ...tail] + + // Replace the summarized display prefix with the boundary marker and + // re-base the surviving tail's restart indices (the summary occupies + // slot 0, so the tail now starts at slot 1). + this.displayMessages = [ + { role: 'summary', content: formatted }, + ...this.displayMessages + .slice(displayKeepFrom) + .map((m) => (m.role === 'user' ? { ...m, index: m.index - keepFrom + 1 } : m)) + ] + + // The provider report described the pre-compaction history; the new + // history is much smaller, so clear it and let readers re-estimate. + this.contextUsage = undefined + return 'ok' + } catch (err) { + if (abortController.signal.aborted) { + return 'aborted' + } + console.error('Conversation summarization failed', err) + return 'error' + } finally { + this.compacting = false + } + } + /** * Summary-based partial compaction. Summarizes the older PREFIX of the stored * history into a single user message and keeps the recent tail verbatim, @@ -493,45 +1066,92 @@ export class AIChatManager { return false } - this.compacting = true - try { - const raw = await getNonStreamingCompletion( - [...prefix, { role: 'user', content: getCompactionSummaryPrompt() }], - abortController - ) - const formatted = formatCompactSummary(raw ?? '') - if (!formatted) { - this.consecutiveCompactionFailures++ - return false - } - - this.messages = [{ role: 'user', content: buildSummaryMessageContent(formatted) }, ...tail] - - // Replace the summarized display prefix with the boundary marker and - // re-base the surviving tail's restart indices (the summary occupies - // slot 0, so the tail now starts at slot 1). - this.displayMessages = [ - { role: 'summary', content: formatted }, - ...this.displayMessages - .slice(displayKeepFrom) - .map((m) => (m.role === 'user' ? { ...m, index: m.index - keepFrom + 1 } : m)) - ] - - // The provider report described the pre-compaction history; the new - // history is much smaller, so clear it and let readers re-estimate. - this.contextUsage = undefined + const result = await this.runSummarization( + prefix, + tail, + keepFrom, + displayKeepFrom, + abortController + ) + if (result === 'ok') { this.consecutiveCompactionFailures = 0 return true - } catch (err) { - // A user Stop aborts the in-flight summary — that's a turn cancel, not a - // compaction failure, so it doesn't count toward the circuit breaker. - if (!abortController.signal.aborted) { - console.error('Conversation summarization failed', err) - this.consecutiveCompactionFailures++ + } + // 'aborted' is a user Stop during the in-flight summary — a turn cancel, not + // a compaction failure, so it doesn't count toward the circuit breaker. + if (result === 'empty' || result === 'error') { + this.consecutiveCompactionFailures++ + } + return false + } + + /** + * Manual compaction (the `/compact` session command): summarize the ENTIRE + * stored history into a single summary message and keep nothing verbatim, so + * the next message continues from the summary alone. Unlike the automatic + * trigger it ignores the context-window budget, the circuit breaker, and the + * prefix-size gate — the user asked for it explicitly — and runs on its own + * abort controller so the Stop button (`cancel`) can interrupt the in-flight + * summary, leaving history untouched. + */ + compactManually = async (): Promise => { + if (this.loading) { + return + } + // A summary round-trip only pays off once there's a prior exchange to fold + // in; a single message (or none) has nothing to compact. + if (this.messages.length < 2) { + sendUserToast('Nothing to compact yet.') + return + } + + const abortController = new AbortController() + this.abortController = abortController + this.loading = true + let result: 'ok' | 'empty' | 'aborted' | 'error' = 'error' + try { + // Everything is the prefix, nothing is kept verbatim: keepFrom and + // displayKeepFrom point past the end so the kept tail is empty. + result = await this.runSummarization( + [...this.messages], + [], + this.messages.length, + this.displayMessages.length, + abortController + ) + switch (result) { + case 'ok': + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) + sendUserToast('Conversation compacted.') + break + case 'empty': + sendUserToast('Compaction produced an empty summary — conversation left unchanged.', true) + break + case 'error': + sendUserToast('Failed to compact the conversation.', true) + break + // 'aborted' (user Stop): history untouched, no toast. } - return false } finally { - this.compacting = false + this.loading = false + } + + // Flush a message typed while compaction ran. Mirrors the send-turn + // epilogue (loading gated its capture): auto-send after a successful + // compaction or a deliberate user cancel — the user is ready to move on — + // while a failed/empty compaction or a programmatic cancel leaves it queued. + if ((result === 'ok' || this.wasCancelledByUser()) && this.queuedMessage) { + const next = this.queuedMessage + this.queuedMessage = '' + const accepted = await this.sendRequest({ instructions: next }) + if (accepted === false) { + this.queuedMessage = next + } } } @@ -621,35 +1241,40 @@ export class AIChatManager { requestUserQuestion = ( toolId: string, - _question: { question: string; choices: string[] } - ): Promise => { + _question: UserQuestionDisplay + ): Promise => { return new Promise((resolve) => { this.userQuestionCallbacks.set(toolId, resolve) }) } - handleUserQuestionAnswer = (toolId: string, choice: string) => { + handleUserQuestionAnswer = (toolId: string, choices: string[]) => { const callback = this.userQuestionCallbacks.get(toolId) if (!callback) { return } + // Display-only readback for the collapsed tool-header: a compact comma list. + // The model-facing return (bare string / newline-bulleted) is built by the + // tool fn from the resolved choices below. + const answerSummary = choices.join(', ') + this.displayMessages = this.displayMessages.map((message) => { if (message.role === 'tool' && message.tool_call_id === toolId && message.userQuestion) { return { ...message, - content: `User answered question: ${choice}`, + content: `User answered question: ${answerSummary}`, isLoading: false, userQuestion: { ...message.userQuestion, - selectedChoice: choice + selectedChoices: choices } } } return message }) - callback(choice) + callback(choices) this.userQuestionCallbacks.delete(toolId) } @@ -760,7 +1385,7 @@ export class AIChatManager { this.helpers = { getScriptOptions: () => { return { - code: this.scriptEditorOptions?.code ?? '', + code: this.scriptEditorOptions?.getCode() ?? '', lang: lang, path: this.scriptEditorOptions?.path ?? '', args: this.scriptEditorOptions?.args ?? {} @@ -808,15 +1433,8 @@ export class AIChatManager { this.tools = [searchDocsTool, readDocsPageTool, ...this.apiTools] this.helpers = {} } else if (mode === AIMode.GLOBAL) { - const customPrompt = getCombinedCustomPrompt(mode) - this.systemMessage = prepareGlobalSystemMessage(customPrompt, { - previewTools: this.isSessionChat - }) - this.tools = globalToolsFor({ sessionPreview: this.isSessionChat }) - this.helpers = { - ...(this.isSessionChat ? { sessionId: this.sessionId } : {}), - testActiveFlow: async (args?: Record) => this.flowAiChatHelpers?.testFlow(args) - } satisfies GlobalToolHelpers + this.configureGlobalMode() + void this.refreshGlobalSkills() } else if (mode === AIMode.APP) { const customPrompt = getCombinedCustomPrompt(mode) this.systemMessage = prepareAppSystemMessage(customPrompt) @@ -825,6 +1443,100 @@ export class AIChatManager { } } + // Fetch the workspace's AI skills and, if GLOBAL mode is still active, rebuild + // the system message so the next chat-loop iteration advertises them. Ignore + // stale resolves so workspace changes cannot overwrite newer skills. + // Build the global-mode system message, tools, and helpers, layering on the + // pipeline surface when a /pipeline editor has registered helpers. Centralized + // so changeMode, refreshGlobalSkills, and setPipelineHelpers stay consistent — + // each rebuild would otherwise drop the pipeline augmentation the others added. + private configureGlobalMode = () => { + const systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { + previewTools: this.isSessionChat, + skills: this.globalSkills + }) + const baseHelpers: GlobalToolHelpers = { + // A session targets its own fixed (possibly forked) workspace, so capture it for + // permission gating. The global side-panel chat follows the live navigation + // workspace instead, so leave it unset there — allowedOpenPages reads the store. + ...(this.isSessionChat + ? { sessionId: this.sessionId, operatingWorkspace: this.operatingWorkspace } + : {}), + testActiveFlow: async (args?: Record) => this.flowAiChatHelpers?.testFlow(args), + attachedFiles: this.attachedFiles, + getUserInstructions: () => getUserCustomPrompts()[AIMode.GLOBAL] ?? '', + setUserInstructions: (instructions: string) => { + const prompts = getUserCustomPrompts() + if (instructions.trim()) { + prompts[AIMode.GLOBAL] = instructions + } else { + delete prompts[AIMode.GLOBAL] + } + setUserCustomPrompts(prompts) + this.rebuildGlobalSystemMessage() + } + } + const pipeline = this.pipelineAiChatHelpers + if (pipeline) { + systemMessage.content += getPipelinePromptSection(pipeline.getPipelineContext()) + this.tools = [...globalToolsFor({ sessionPreview: this.isSessionChat }), ...pipelineTools] + this.helpers = { ...baseHelpers, pipeline } + } else { + this.tools = globalToolsFor({ sessionPreview: this.isSessionChat }) + this.helpers = baseHelpers + } + this.systemMessage = systemMessage + } + + refreshGlobalSkills = async (workspace = this.operatingWorkspace ?? '') => { + const refreshId = ++this.globalSkillsRefreshId + const skills = await loadWorkspaceSkills(workspace) + if (refreshId !== this.globalSkillsRefreshId) { + return + } + this.globalSkills = skills + if (this.mode === AIMode.GLOBAL) { + this.configureGlobalMode() + } + } + + // Rebuild the GLOBAL system message in place so an updated user instruction (persisted by + // the update_user_instructions tool) is picked up on the next chat-loop iteration, which + // re-reads this.systemMessage via a getter. + rebuildGlobalSystemMessage = () => { + if (this.mode !== AIMode.GLOBAL) { + return + } + const systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { + previewTools: this.isSessionChat, + skills: this.globalSkills + }) + // Preserve the active pipeline-editor augmentation that configureGlobalMode + // adds — otherwise update_user_instructions (which calls this) would drop the + // /pipeline/ context + direct-draft/materialize guidance mid-session. + const pipeline = this.pipelineAiChatHelpers + if (pipeline) { + systemMessage.content += getPipelinePromptSection(pipeline.getPipelineContext()) + } + this.systemMessage = systemMessage + } + + private expandGlobalSkillCommand = (instructions: string): string => { + if (!this.isSessionChat || this.mode !== AIMode.GLOBAL || !instructions.startsWith('/')) { + return instructions + } + const match = /^\/([a-z0-9-]+)(?:\s+([\s\S]*))?$/.exec(instructions) + if (!match) { + return instructions + } + const skill = this.globalSkills.find((s) => s.name === match[1]) + if (!skill) { + return instructions + } + const rest = match[2]?.trim() + return rest ? `Use the "${skill.name}" skill. ${rest}` : `Use the "${skill.name}" skill.` + } + canApplyCode = $derived(this.allowedModes.script && this.mode === AIMode.SCRIPT) private changeModeTool = { @@ -1015,7 +1727,13 @@ export class AIChatManager { messages, addedMessages, get systemMessage() { - return systemMessageOverride ?? self.systemMessage + const base = systemMessageOverride ?? self.systemMessage + // Inject the attached-files roster at request time (re-read each iteration) + // so it always reflects the live file list without reactive bookkeeping. + if (self.mode === AIMode.GLOBAL && self.attachedFiles.count > 0) { + return appendAttachedFilesRoster(base, self.attachedFiles) + } + return base }, get tools() { return self.tools @@ -1031,11 +1749,19 @@ export class AIChatManager { get webSearch() { return isWebSearchEnabledForProvider(getCurrentModel().provider) }, - clients: { - openai: workspaceAIClients.getOpenaiClient(), - anthropic: workspaceAIClients.getAnthropicClient() + // Build the proxy clients against the operating workspace, not the global + // singleton: a session deliberately leaves workspaceStore untouched, so the + // singleton (init'd only on global workspace changes) would route the LLM + // request through the navigation workspace's /ai/proxy instead of the + // session's — sending it to the wrong workspace's AI credentials. + get clients() { + const ws = self.operatingWorkspace ?? '' + return { + openai: workspaceAIClients.createOpenaiClient(ws), + anthropic: workspaceAIClients.createAnthropicClient(ws) + } }, - workspace: get(workspaceStore) ?? '', + workspace: this.operatingWorkspace ?? '', skipResponsesApi: this.skipResponsesApi, onSkipResponsesApi: () => { this.skipResponsesApi = true @@ -1060,7 +1786,7 @@ export class AIChatManager { return prepareGlobalUserMessage( pendingPrompt, this.contextManager.getSelectedContext(), - { workspace: get(workspaceStore) } + { workspace: this.operatingWorkspace } ) } return undefined @@ -1189,9 +1915,11 @@ export class AIChatManager { isPreprocessor?: boolean } = {} ) => { - // Returns whether the message was actually turned into a chat turn — - // the queue flush uses this to restore messages dropped by an early - // return instead of silently losing them. + // Returns whether the input was consumed: true when it was sent as a chat + // turn OR handled as a local built-in command, false when it was dropped + // without being acted on (mode hidden, empty, beforeSend failed). The + // queue flush restores the queued message only on false, so a consumed + // command isn't re-queued and re-fired into the next conversation. const requestedMode = options.mode ?? this.mode if (!isAIModeVisible(requestedMode)) { return false @@ -1206,6 +1934,40 @@ export class AIChatManager { if (!this.instructions.trim()) { return false } + // Built-in session commands run locally instead of becoming a chat turn. + // Intercepted here — before the beforeSend workspace commit, file regrants, + // and skill expansion. Scoped to session chat GLOBAL mode, where the + // slash-command UI lives. Return true (consumed, not dropped) so that a + // command flushed from the queue isn't restored and re-fired into the next + // conversation. + if (this.isSessionChat && this.mode === AIMode.GLOBAL) { + const trimmed = this.instructions.trim() + // `/compact`: summarize the conversation locally to free up context. + if (COMPACT_COMMAND_RE.test(trimmed)) { + this.instructions = '' + await this.compactManually() + return true + } + // `/clear`: save the conversation to history and start a fresh chat. + if (CLEAR_COMMAND_RE.test(trimmed)) { + this.instructions = '' + await this.saveAndClear() + return true + } + } + // Re-grant any locked File System Access handles within this send gesture, so the + // file tools can read the live files. requestPermission() needs a user gesture, and + // this runs before the first await/network call while the Send click is still active. + // Attachment upkeep must never block the send — affected files just stay locked/stale + // and the tools report their status to the model. + try { + await this.attachedFiles.regrantLocked() + // Re-enumerate linked folders so on-disk changes (renamed/added/removed/edited + // files) are reflected in the roster + indexes before this turn runs. + await this.attachedFiles.refreshFolders() + } catch (e) { + console.error('Attached-files upkeep failed before send', e) + } if (this.beforeSend) { try { await this.beforeSend() @@ -1224,6 +1986,11 @@ export class AIChatManager { return false } } + // Session chats commit their workspace in beforeSend; skills must match the + // committed workspace before the system prompt is sent. + if (this.mode === AIMode.GLOBAL) { + await this.refreshGlobalSkills(this.operatingWorkspace ?? '') + } const isFirstUserTurn = !this.displayMessages.some((message) => message.role === 'user') // Declared outside `try` so the catch can recover what the loop produced // before a failure: the structured messages and the latest streamed text @@ -1250,7 +2017,7 @@ export class AIChatManager { const model = tryGetCurrentModel() if (model) { WorkspaceService.logAiChat({ - workspace: get(workspaceStore) ?? '', + workspace: this.operatingWorkspace ?? '', requestBody: { session_id: this.historyManager.getCurrentChatId(), provider: model.provider, @@ -1297,6 +2064,19 @@ export class AIChatManager { // The LLM gets the full pasted content; the display message above keeps // the compact tokens + registry so the bubble can render/expand chips. const oldInstructions = expanded(chatDraft(this.instructions, pastes)) + // Deliver background-job completions to the model as a preamble on this + // turn (notify-only wake). Folded into the model-facing text only — the + // display bubble keeps this.instructions, and no extra message is added, so + // the display↔messages index pairing above stays intact. Ephemeral. + const jobNotesPreamble = + this.mode === AIMode.GLOBAL && this.pendingJobNotes.length > 0 + ? this.pendingJobNotes.join('\n\n') + '\n\n' + : '' + if (jobNotesPreamble) this.pendingJobNotes = [] + const modelInstructions = + this.mode === AIMode.GLOBAL + ? jobNotesPreamble + this.expandGlobalSkillCommand(oldInstructions) + : oldInstructions this.instructions = '' if (this.mode === AIMode.SCRIPT && !this.scriptEditorOptions && !options.lang) { @@ -1329,8 +2109,8 @@ export class AIChatManager { userMessage = prepareApiUserMessage(oldInstructions) break case AIMode.GLOBAL: - userMessage = prepareGlobalUserMessage(oldInstructions, oldSelectedContext, { - workspace: get(workspaceStore) + userMessage = prepareGlobalUserMessage(modelInstructions, oldSelectedContext, { + workspace: this.operatingWorkspace }) break case AIMode.APP: @@ -1350,8 +2130,15 @@ export class AIChatManager { const projectedContextTokens = this.contextTokens + this.estimateMessagesTokens([userMessage]) this.messages.push(userMessage) - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) + this.replyReveal.reset() + this.reasoningReveal.reset() this.currentReply = '' this.currentReasoning = '' this.currentReasoningActive = false @@ -1386,7 +2173,12 @@ export class AIChatManager { this.contextUsage = Math.max(0, this.contextUsage - freed) } } - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) } } // Rollback anchors for restoreUnsentTurn: captured after compaction so @@ -1407,10 +2199,16 @@ export class AIChatManager { messages: [...this.messages], abortController: this.abortController, callbacks: { - onNewToken: (token) => (this.currentReply += token), - onReasoningDelta: (token) => (this.currentReasoning += token), + onNewToken: (token) => this.replyReveal.push(token), + onReasoningDelta: (token) => this.reasoningReveal.push(token), onReasoningStart: () => (this.currentReasoningActive = true), onMessageEnd: () => { + // Drain any un-revealed backlog into currentReply first, so the reads + // below see the full text. This funnel covers clean completion, tool + // boundaries, and abort/error — flush-before-read is the invariant that + // keeps text from being lost or duplicated on any exit path. + this.replyReveal.flush() + this.reasoningReveal.flush() // Keep the streamed text for the abort/error paths. Non-empty only: // parsers flush (and reset) when a tool call starts after text, and // the catch's later empty call would wipe it — stale keeps are @@ -1436,31 +2234,17 @@ export class AIChatManager { this.currentReasoning = '' this.currentReasoningActive = false }, - setToolStatus: (id, metadata) => { - const existingIdx = this.displayMessages.findIndex( - (m) => m.role === 'tool' && m.tool_call_id === id - ) - if (existingIdx !== -1) { - // Update existing tool message with metadata - const existing = this.displayMessages[existingIdx] as ToolDisplayMessage - if (existing.content.length === 0 && metadata?.error) { - this.displayMessages[existingIdx].content = metadata.error + setToolStatus: this.applyToolStatus, + // Job-tracking hooks enable detach-into-background; wire them only in + // GLOBAL mode (global chat + sessions). In-editor script/flow/pipeline + // chats leave these undefined, so their test runs keep blocking. + ...(this.mode === AIMode.GLOBAL + ? { + onJobStarted: (job) => this.registerJob(job), + onJobStatus: (jobId, update) => this.updateJob(jobId, update), + onJobDetached: (jobId) => this.markJobDetached(jobId) } - this.displayMessages[existingIdx] = { - ...existing, - ...(metadata || {}) - } as ToolDisplayMessage - } else { - // Create new tool message with metadata - const newMessage: ToolDisplayMessage = { - role: 'tool', - tool_call_id: id, - content: metadata?.content ?? metadata?.error ?? '', - ...(metadata || {}) - } - this.displayMessages.push(newMessage) - } - }, + : {}), removeToolStatus: (id) => { const existingIdx = this.displayMessages.findIndex( (m) => m.role === 'tool' && m.tool_call_id === id @@ -1472,7 +2256,10 @@ export class AIChatManager { }, requestConfirmation: this.requestConfirmation, shouldAutoAcceptToolConfirmations: () => this.autoAcceptToolConfirmationsActive, - requestUserQuestion: this.requestUserQuestion + requestUserQuestion: this.requestUserQuestion, + onItemModified: (kind, path) => this.recordModifiedItem(kind, path), + onItemDeployed: (kind, from, to) => void this.renameModifiedItem(kind, from, to), + onItemDiscarded: (kind, path) => void this.removeModifiedItem(kind, path) } } @@ -1508,7 +2295,12 @@ export class AIChatManager { this.contextUsage = result?.lastIterationUsage ? result.lastIterationUsage.prompt + result.lastIterationUsage.completion : undefined - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) // Still counts as the saved first turn — skipping the hook here would // permanently miss it (the next turn isn't "first" anymore). if (isFirstUserTurn && this.afterFirstTurnSaved) { @@ -1538,7 +2330,12 @@ export class AIChatManager { // user message on reload. Remove it instead. this.historyManager.deletePastChat(this.historyManager.getCurrentChatId()) } else { - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) } if (!wasAborted) { sendUserToast('The model returned no response — your message was restored to the input.') @@ -1557,7 +2354,12 @@ export class AIChatManager { if (this.autoAcceptEditsActive) { this.acceptPendingFlowEdits() } - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) // Only this branch is a clean send: the queued-message flush below // auto-sends the next message after it (set after saveChat so a // persistence failure falls through to the restore path instead). @@ -1581,7 +2383,12 @@ export class AIChatManager { // compaction on the next send instead of failing the same way again. this.contextUsage = undefined try { - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) } catch (saveErr) { console.error('Failed to persist partial chat after error', saveErr) } @@ -1590,6 +2397,11 @@ export class AIChatManager { sendUserToast(getSendRequestErrorMessage(err, webSearchUnavailable), true) } finally { this.loading = false + // Turn teardown: cancel any in-flight reveal frame and drop leftover + // backlog. onMessageEnd already flushed on every outcome, so this only + // releases the loop; it never discards uncommitted text. + this.replyReveal.reset() + this.reasoningReveal.reset() } // Flush the queued message. Send it after a cleanly committed turn OR a // deliberate user cancel (Esc / Stop) — in both cases the user is ready @@ -1607,6 +2419,11 @@ export class AIChatManager { this.queuedMessage = next } } + // A background job may have finished mid-turn: its note missed this turn's + // preamble (captured at the start) and the poller skipped auto-resume while + // we were loading. Now that we're idle, deliver it via an auto-resume. Skips + // itself if the queued-message flush above already carried the notes. + void this.#maybeAutoResumeFromJobs() return true } @@ -1710,10 +2527,28 @@ export class AIChatManager { // Drop any message queued in this conversation so it can't auto-send into // the fresh chat or linger as a card across the switch. this.queuedMessage = '' - await this.historyManager.save(this.displayMessages, this.messages, this.contextUsage) + // The tray + poller belong to the conversation being left; the just-saved + // chat keeps its persisted jobs (save() omits the arg → fallback preserves). + this.clearBackgroundJobs() + await this.historyManager.save( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) this.displayMessages = [] this.messages = [] this.contextUsage = undefined + // The mask belongs to the conversation just saved — the fresh chat starts + // its own (empty) tracking; carrying entries over would claim the previous + // conversation's edits for the new one. Untracked chats stay untracked. + if (this.modifiedItems) this.modifiedItems = new SvelteSet() + // In an AI session, linked files are session-scoped: they persist across conversations + // (cleared only when the session is deleted). The ephemeral global side-panel chat has no + // session, so "New chat" must clear them — otherwise the next, unrelated conversation + // would still get the previous file roster and could read/search it. + if (!this.isSessionChat) this.attachedFiles.clear() + this.onChatRotated?.(this.historyManager.getCurrentChatId()) } loadPastChat = async (id: string) => { @@ -1722,10 +2557,35 @@ export class AIChatManager { // Drop any message queued in the current conversation so it doesn't // auto-send into the loaded one or linger as a card across the switch. this.queuedMessage = '' + // Stop the poller for the conversation being left before swapping in the + // loaded chat's jobs below. + this.clearBackgroundJobs() + // Same isolation as saveAndClear: the ephemeral global chat's attachments belong to + // the conversation being left, not the one being loaded; sessions keep them. + if (!this.isSessionChat) this.attachedFiles.clear() this.displayMessages = chat.displayMessages this.messages = chat.actualMessages this.contextUsage = normalizeContextUsage(chat.contextUsage) + // Seed the modified-items mask from the stored chat. A session's Edits + // surface is scoped strictly to what this session edited, so it must never + // fall back to showing every draft in the (possibly forked) workspace: a + // legacy chat with no stored mask seeds an empty tracked set, not undefined. + // The global side-panel chat never tracks, so leave it untouched there. + if (this.isSessionChat) { + const stored = this.historyManager.getModifiedItems(id) + this.modifiedItems = new SvelteSet(stored ?? []) + } + // Rebuild the jobs tray from the loaded chat, and re-attach the poller to + // any job that was still in flight when it was last persisted. + const storedJobs = this.historyManager.getBackgroundJobs(id) + this.backgroundJobs = storedJobs ? storedJobs.map((j) => ({ ...j })) : [] + for (const j of this.backgroundJobs) { + if (this.isJobNonTerminal(j.status)) j.detached = true + } + if (this.backgroundJobs.length > 0) this.backgroundJobs = [...this.backgroundJobs] + this.#ensureJobPoller() this.#automaticScroll = true + this.onChatRotated?.(id) } } @@ -1856,14 +2716,13 @@ export class AIChatManager { lastDeployedCode: undefined, lastSavedCode: undefined } - return { args: moduleState?.previewArgs ?? {}, error: moduleState && !moduleState.previewSuccess ? getStringError(moduleState.previewResult) : undefined, - code: module.value.content, + getCode: () => (module.value.type === 'rawscript' ? module.value.content : ''), lang: module.value.language, path: module.id, ...editorRelated @@ -1907,6 +2766,28 @@ export class AIChatManager { } } + // Registered by the /pipeline editor while it is mounted. Rebuilds the global + // tool set so the pipeline tools appear (and disappear on unregister). Pipeline + // AI edits apply directly as drafts, so there is nothing to auto-accept. + // Returns a cleanup that tears the registration back down. + setPipelineHelpers = (pipelineHelpers: PipelineAIChatHelpers) => { + this.pipelineAiChatHelpers = pipelineHelpers + untrack(() => { + if (this.mode === AIMode.GLOBAL) { + this.configureGlobalMode() + } + }) + + return () => { + this.pipelineAiChatHelpers = undefined + untrack(() => { + if (this.mode === AIMode.GLOBAL) { + this.configureGlobalMode() + } + }) + } + } + /** * Refresh cached datatables from the app helpers (async) * Creates one context element per table (not per datatable) diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts index 16a08d7344..94270d0e6a 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts @@ -6,6 +6,18 @@ import type { ChatCompletionMessageParam } from 'openai/resources/chat/completio import { AIChatManager, AIMode, AIAutonomyMode } from './AIChatManager.svelte' import { runChatLoop } from './chatLoop' +// This suite forces esm-env BROWSER=true (below). That makes @sveltejs/kit's +// client runtime (pulled transitively via $lib/navigation) evaluate browser-only +// globals at import time and throw "location is not defined" under the node test +// env. Stub the two $app modules $lib/navigation needs so kit's client runtime is +// never loaded. File-local: no other suite is affected. +vi.mock('$app/navigation', () => ({ + goto: vi.fn(), + afterNavigate: vi.fn(), + beforeNavigate: vi.fn() +})) +vi.mock('$app/paths', () => ({ base: '', assets: '' })) + const mocks = vi.hoisted(() => ({ getCurrentModel: vi.fn(), tryGetCurrentModel: vi.fn(), @@ -15,7 +27,10 @@ const mocks = vi.hoisted(() => ({ getOpenaiClient: vi.fn(), getAnthropicClient: vi.fn(), getNonStreamingCompletion: vi.fn(), - runChatLoop: vi.fn() + runChatLoop: vi.fn(), + listAiSkills: vi.fn(), + getJob: vi.fn(), + workspace: 'test_workspace' as string | undefined })) vi.mock('monaco-editor', () => ({ @@ -24,26 +39,44 @@ vi.mock('monaco-editor', () => ({ vi.mock('$lib/gen', () => ({ WorkspaceService: { - logAiChat: mocks.logAiChat + logAiChat: mocks.logAiChat, + listAiSkills: mocks.listAiSkills }, ScriptService: {}, FlowService: {}, - JobService: {} + JobService: { + getJob: mocks.getJob + } })) // Autonomy mode is now namespaced by the logged-in user's email (see // userScopedStorage); the mock emits one so scopedKey() resolves. const TEST_EMAIL = 'admin@test' -vi.mock('$lib/stores', () => ({ - workspaceStore: { subscribe: () => () => undefined }, - userStore: { - subscribe: (run: (value: { username: string; email: string }) => void) => { - run({ username: 'admin', email: 'admin@test' }) +vi.mock('$lib/stores', () => { + // A minimal readable store: get(store) reads this value synchronously. Defined + // inside the factory since vi.mock is hoisted above module-scope declarations. + const readable = (value: T) => ({ + subscribe: (run: (v: T) => void) => { + run(value) return () => undefined } + }) + return { + workspaceStore: { + subscribe: (run: (value: string | undefined) => void) => { + run(mocks.workspace) + return () => undefined + } + }, + userStore: readable({ username: 'admin', email: 'admin@test', is_admin: true }), + // Read eagerly at module load by the open_page tool's allowedOpenPages / + // allowedTriggerKinds (global/core.ts) as the manager's tools are built. + superadmin: readable(false), + userWorkspaces: readable([] as unknown[]), + enterpriseLicense: readable(undefined) } -})) +}) vi.mock('$lib/toast', () => ({ sendUserToast: mocks.sendUserToast @@ -53,6 +86,9 @@ vi.mock('$lib/aiStore', () => ({ getCurrentModel: mocks.getCurrentModel, tryGetCurrentModel: mocks.tryGetCurrentModel, getCombinedCustomPrompt: () => '', + getCustomPromptParts: () => ({}), + getUserCustomPrompts: () => ({}), + setUserCustomPrompts: () => {}, isWebSearchEnabledForProvider: mocks.isWebSearchEnabledForProvider })) @@ -95,6 +131,8 @@ beforeEach(() => { mocks.logAiChat.mockResolvedValue(undefined) mocks.getOpenaiClient.mockReturnValue({}) mocks.getAnthropicClient.mockReturnValue({}) + mocks.listAiSkills.mockResolvedValue([]) + mocks.workspace = 'test_workspace' mocks.runChatLoop.mockResolvedValue({ addedMessages: [], tokenUsage: { prompt: 0, completion: 0, total: 0 }, @@ -185,6 +223,82 @@ describe('AIChatManager request errors', () => { }) }) +describe('AIChatManager global skills', () => { + const model = { provider: 'openai', model: 'gpt-4o' } + + beforeEach(() => { + localStorage.clear() + mocks.getCurrentModel.mockReturnValue(model) + mocks.tryGetCurrentModel.mockReturnValue(model) + }) + + it('loads skills after beforeSend commits the session workspace', async () => { + let resolveParentSkills: ((skills: { name: string; description: string }[]) => void) | undefined + const parentSkills = new Promise<{ name: string; description: string }[]>((resolve) => { + resolveParentSkills = resolve + }) + mocks.workspace = 'parent' + mocks.listAiSkills.mockImplementation(({ workspace }: { workspace: string }) => { + if (workspace === 'parent') { + return parentSkills + } + return Promise.resolve([{ name: 'child-skill', description: 'child workspace skill' }]) + }) + mocks.runChatLoop.mockImplementation(async (config: any) => { + expect(config.workspace).toBe('child') + expect(config.systemMessage.content).toContain('child-skill') + expect(config.systemMessage.content).not.toContain('parent-skill') + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + + const manager = new AIChatManager() + manager.isSessionChat = true + manager.beforeSend = () => { + mocks.workspace = 'child' + } + + await manager.sendRequest({ instructions: 'first', mode: AIMode.GLOBAL }) + resolveParentSkills?.([{ name: 'parent-skill', description: 'parent workspace skill' }]) + await Promise.resolve() + + expect(mocks.listAiSkills).toHaveBeenCalledWith({ workspace: 'parent' }) + expect(mocks.listAiSkills).toHaveBeenCalledWith({ workspace: 'child' }) + expect(manager.systemMessage.content).toContain('child-skill') + expect(manager.systemMessage.content).not.toContain('parent-skill') + }) + + it('expands a leading slash skill command for the model while preserving the displayed text', async () => { + mocks.listAiSkills.mockResolvedValue([ + { name: 'review-code', description: 'review code for bugs' } + ]) + mocks.runChatLoop.mockImplementation(async (config: any) => { + const userMessage = config.messages[config.messages.length - 1] + expect(userMessage.content).toContain('Use the "review-code" skill. find bugs') + expect(userMessage.content).not.toContain('/review-code find bugs') + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + + const manager = new AIChatManager() + manager.isSessionChat = true + + await manager.sendRequest({ instructions: '/review-code find bugs', mode: AIMode.GLOBAL }) + + expect(manager.displayMessages[0]?.content).toBe('/review-code find bugs') + }) +}) + describe('AIChatManager autonomy mode', () => { beforeEach(() => { localStorage.clear() @@ -514,6 +628,78 @@ describe('AIChatManager queued messages', () => { await manager.loadPastChat('chat-b') expect(manager.queuedMessage).toBe('') }) + + it('clears attachments on New chat / load past chat (non-session), keeps them in a session', async () => { + const txt = (n: string) => new File(['hello\n'], n, { type: 'text/plain' }) + + // Non-session global chat: New chat must clear the previous conversation's attachments. + const manager = createManager(createInputMock()) + await manager.attachedFiles.addFiles([txt('a.txt')]) + expect(manager.attachedFiles.count).toBe(1) + await manager.saveAndClear() + expect(manager.attachedFiles.count).toBe(0) + + // ...and loading a past chat clears them too. + vi.spyOn(manager.historyManager, 'loadPastChat').mockReturnValue({ + id: 'chat-c', + title: 'Chat C', + displayMessages: [], + actualMessages: [], + lastModified: 0 + } as unknown as ReturnType) + await manager.attachedFiles.addFiles([txt('c.txt')]) + expect(manager.attachedFiles.count).toBe(1) + await manager.loadPastChat('chat-c') + expect(manager.attachedFiles.count).toBe(0) + + // Session chat: attachments are session-scoped — they survive New chat. + const session = createManager(createInputMock()) + session.isSessionChat = true + await session.attachedFiles.addFiles([txt('b.txt')]) + await session.saveAndClear() + expect(session.attachedFiles.count).toBe(1) + }) + + it('tracks (empty mask) a session chat loaded with no stored modified-items', async () => { + // A legacy session chat has no persisted mask. It must NOT stay untracked + // (undefined) — that makes the Edits surface fall back to showing every + // draft in the (possibly forked) workspace. Seed an empty tracked set so the + // session only ever surfaces what it actually edited. + const manager = createManager(createInputMock()) + manager.isSessionChat = true + vi.spyOn(manager.historyManager, 'loadPastChat').mockReturnValue({ + id: 'legacy-session-chat', + title: 'Legacy', + displayMessages: [], + actualMessages: [], + lastModified: 0 + } as unknown as ReturnType) + vi.spyOn(manager.historyManager, 'getModifiedItems').mockReturnValue(undefined) + + await manager.loadPastChat('legacy-session-chat') + + expect(manager.modifiedItems).toBeInstanceOf(Set) + expect(manager.modifiedItems?.size).toBe(0) + }) + + it('seeds a session chat mask from its stored modified-items', async () => { + const manager = createManager(createInputMock()) + manager.isSessionChat = true + vi.spyOn(manager.historyManager, 'loadPastChat').mockReturnValue({ + id: 'tracked-session-chat', + title: 'Tracked', + displayMessages: [], + actualMessages: [], + lastModified: 0 + } as unknown as ReturnType) + vi.spyOn(manager.historyManager, 'getModifiedItems').mockReturnValue([ + 'script:u/admin/hello_world' + ]) + + await manager.loadPastChat('tracked-session-chat') + + expect([...(manager.modifiedItems ?? [])]).toEqual(['script:u/admin/hello_world']) + }) }) describe('AIChatManager context compaction', () => { @@ -573,7 +759,9 @@ describe('AIChatManager context compaction', () => { expect(manager.messages[0]).toMatchObject({ role: 'user', content: 'c'.repeat(400) }) // Mid-turn, the report is debited by the freed estimate (visible in the // compaction-time save) so a rolled-back turn keeps a consistent value - expect(saveChat).toHaveBeenCalledWith(expect.anything(), expect.anything(), 650_000) + // 4th arg: the modified-items mask rides on every save (undefined here — + // this bare manager never initialised tracking). + expect(saveChat).toHaveBeenCalledWith(expect.anything(), expect.anything(), 650_000, undefined) // At commit, the no-report turn clears the stored value; the readable // number falls back to estimating the now-tiny compacted history expect(manager.contextUsage).toBeUndefined() @@ -823,7 +1011,7 @@ describe('AIChatManager context compaction', () => { // The request that went out begins with the summary user message, then the // recent tail verbatim, then the new question. - const sent = mocks.runChatLoop.mock.calls[0][0].messages + const sent = mocks.runChatLoop.mock.calls[mocks.runChatLoop.mock.calls.length - 1][0].messages expect(sent).toHaveLength(4) expect(sent[0].role).toBe('user') expect(sent[0].content).toContain('SUMMARY TEXT') @@ -885,12 +1073,10 @@ describe('AIChatManager context compaction', () => { mocks.tryGetCurrentModel.mockReturnValue(gpt4oModel) // The user hits Stop while the summary request is in flight: it aborts the // turn's controller and rejects. - mocks.getNonStreamingCompletion.mockImplementation( - async (_msgs: any, ac: AbortController) => { - ac.abort('user_cancelled') - throw new Error('aborted') - } - ) + mocks.getNonStreamingCompletion.mockImplementation(async (_msgs: any, ac: AbortController) => { + ac.abort('user_cancelled') + throw new Error('aborted') + }) // With the controller already aborted, the real request returns nothing; // mirror that so the turn takes the cancel/rollback path. mocks.runChatLoop.mockImplementation(async () => ({ @@ -914,6 +1100,244 @@ describe('AIChatManager context compaction', () => { }) }) +describe('AIChatManager manual compaction', () => { + const model = { provider: 'openai', model: 'gpt-4o' } + + beforeEach(() => { + localStorage.clear() + vi.clearAllMocks() + mocks.getCurrentModel.mockReturnValue(model) + mocks.tryGetCurrentModel.mockReturnValue(model) + // changeMode(GLOBAL) refreshes workspace skills; keep it a no-op here. + mocks.listAiSkills.mockResolvedValue([]) + }) + + function seedExchange(manager: AIChatManager) { + manager.messages = [ + { role: 'user', content: 'q1' }, + { role: 'assistant', content: 'a1' }, + { role: 'user', content: 'q2' }, + { role: 'assistant', content: 'a2' } + ] + manager.displayMessages = [ + { role: 'user', content: 'q1', index: 0 }, + { role: 'assistant', content: 'a1' }, + { role: 'user', content: 'q2', index: 2 }, + { role: 'assistant', content: 'a2' } + ] + } + + it('folds the whole history into a single summary boundary, keeping nothing verbatim', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('MANUAL SUMMARY') + const manager = new AIChatManager() + seedExchange(manager) + manager.contextUsage = 123 + const saveChat = vi.spyOn(manager.historyManager, 'saveChat') + + await manager.compactManually() + + // The summarizer saw the entire history, then the summary instruction. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + const summaryReq = mocks.getNonStreamingCompletion.mock.calls[0][0] + expect(summaryReq).toHaveLength(5) + expect(summaryReq[0].content).toBe('q1') + expect(summaryReq[3].content).toBe('a2') + expect(summaryReq[4].content).toContain('detailed summary') + + // Nothing kept verbatim: messages collapse to just the summary user message. + expect(manager.messages).toHaveLength(1) + expect(manager.messages[0].role).toBe('user') + expect(manager.messages[0].content).toContain('MANUAL SUMMARY') + expect(manager.messages[0].content).toContain('continued from a previous conversation') + expect(manager.messages[0].content).not.toContain('') + + // The transcript shows one summary boundary in place of the old bubbles. + expect(manager.displayMessages).toHaveLength(1) + expect(manager.displayMessages[0]).toMatchObject({ role: 'summary', content: 'MANUAL SUMMARY' }) + + expect(manager.contextUsage).toBeUndefined() + expect(saveChat).toHaveBeenCalled() + expect(mocks.sendUserToast).toHaveBeenCalledWith('Conversation compacted.') + expect(manager.loading).toBe(false) + expect(manager.compacting).toBe(false) + }) + + it('no-ops with a toast when there is nothing worth compacting', async () => { + const manager = new AIChatManager() + manager.messages = [{ role: 'user', content: 'only one' }] + + await manager.compactManually() + + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + expect(mocks.sendUserToast).toHaveBeenCalledWith('Nothing to compact yet.') + expect(manager.messages).toHaveLength(1) + }) + + it('leaves history untouched when the user stops mid-summary', async () => { + mocks.getNonStreamingCompletion.mockImplementation(async (_msgs: any, ac: AbortController) => { + ac.abort('user_cancelled') + throw new Error('aborted') + }) + const manager = new AIChatManager() + seedExchange(manager) + + await manager.compactManually() + + expect(manager.messages).toHaveLength(4) + expect(manager.displayMessages.some((m) => m.role === 'summary')).toBe(false) + // An abort is a user cancel, not a failure — no toast, no destructive change. + expect(mocks.sendUserToast).not.toHaveBeenCalled() + expect(manager.loading).toBe(false) + }) + + it('routes the /compact session command to manual compaction instead of the model', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('VIA COMMAND') + const manager = new AIChatManager() + manager.isSessionChat = true + seedExchange(manager) + + const sent = await manager.sendRequest({ instructions: '/compact', mode: AIMode.GLOBAL }) + + // Consumed as a local command (true so the queue flush won't re-fire it), + // without ever reaching the model loop... + expect(sent).toBe(true) + expect(mocks.runChatLoop).not.toHaveBeenCalled() + // ...it ran the summarizer and compacted in place, clearing the composer. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + expect(manager.displayMessages[0]).toMatchObject({ role: 'summary', content: 'VIA COMMAND' }) + expect(manager.instructions).toBe('') + }) + + it('auto-sends a message queued while compaction was running', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('S') + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = true + manager.changeMode(AIMode.GLOBAL) + seedExchange(manager) + // A message typed while loading was true gets queued, not sent. + manager.queuedMessage = 'follow-up question' + + await manager.compactManually() + + // Compaction ran once, then the queued message went out as a real turn. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + const sent = mocks.runChatLoop.mock.calls[0][0].messages + expect(sent[sent.length - 1].content).toContain('follow-up question') + expect(manager.queuedMessage).toBe('') + }) + + it('routes the /clear session command to a fresh chat instead of the model', async () => { + const manager = new AIChatManager() + manager.isSessionChat = true + seedExchange(manager) + + const sent = await manager.sendRequest({ instructions: '/clear', mode: AIMode.GLOBAL }) + + // Consumed as a local command (true so the queue flush won't re-fire it), + // without ever reaching the model... + expect(sent).toBe(true) + expect(mocks.runChatLoop).not.toHaveBeenCalled() + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + // ...it reset the conversation and cleared the composer. + expect(manager.displayMessages).toEqual([]) + expect(manager.messages).toEqual([]) + expect(manager.instructions).toBe('') + }) + + it('consumes a /clear flushed from the queue without re-queuing it', async () => { + // A normal turn that commits cleanly, so its epilogue flushes the queue. + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = true + manager.changeMode(AIMode.GLOBAL) + seedExchange(manager) + // `/clear` typed while the turn was streaming gets queued, not sent. + manager.queuedMessage = '/clear' + + await manager.sendRequest({ instructions: 'a normal message', mode: AIMode.GLOBAL }) + + // The committed turn's flush ran `/clear` (resetting the conversation) and, + // because the command reports itself as consumed, did NOT restore it — so a + // stale `/clear` can't re-fire and wipe the next conversation. + expect(manager.queuedMessage).toBe('') + expect(manager.displayMessages).toEqual([]) + expect(manager.messages).toEqual([]) + }) + + it('does not intercept /clear outside session chat', async () => { + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = false + + await manager.sendRequest({ instructions: '/clear', mode: AIMode.GLOBAL }) + + // Without the session-chat command surface, /clear is a normal message. + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + }) + + it('does not intercept /compact outside session chat', async () => { + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = false + + await manager.sendRequest({ instructions: '/compact', mode: AIMode.GLOBAL }) + + // Without the session-chat command surface, /compact is a normal message. + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + }) + + it('shadows a workspace skill that collides with a built-in command', () => { + const manager = new AIChatManager() + manager.globalSkills = [ + { name: 'compact', description: 'a workspace skill that happens to be named compact' }, + { name: 'review-code', description: 'review code for bugs' } + ] + + // Built-ins come first and the colliding skill is dropped, so the picker + // never renders two leaves with the same `skill:compact` key. + const names = manager.sessionCommands.map((c) => c.name) + expect(names).toEqual(['compact', 'clear', 'review-code']) + expect(manager.sessionCommands[0].description).toBe( + 'Summarize the conversation to free up context' + ) + }) +}) + const assistantToolCall = (id: string): ChatCompletionMessageParam => ({ role: 'assistant', content: '', @@ -1202,3 +1626,95 @@ describe('AIChatManager sendRequest lifecycle', () => { expect(manager.loading).toBe(false) }) }) + +describe('AIChatManager background job completion', () => { + const completed = (over: Record = {}) => + ({ + type: 'CompletedJob', + id: 'job-1', + success: true, + canceled: false, + result: [{ n: 1 }], + duration_ms: 1234, + logs: 'ran', + ...over + }) as any + + beforeEach(() => { + localStorage.clear() + vi.clearAllMocks() + mocks.getCurrentModel.mockReturnValue({ provider: 'openai', model: 'gpt-4o' }) + }) + + // Drive a registered+detached job to completion through the public poller entry + // (refreshBackgroundJobs polls immediately) and wait until the poller reports it. + async function completeDetachedJob(manager: AIChatManager) { + manager.markJobDetached('job-1') + manager.refreshBackgroundJobs() + await vi.waitFor(() => expect(manager.backgroundJobs[0]?.reported).toBe(true)) + } + + // A ChatJob carrying only its serializable resultFormat (no in-memory closure) — + // exactly the shape a job has after being rehydrated from IndexedDB on reload. + const datatableJob = { + jobId: 'job-1', + toolCallId: 'tc-1', + kind: 'script' as const, + label: 'SQL · main', + workspace: 'ws', + resultFormat: { kind: 'datatable' as const, datatableName: 'main' } + } + + it('reconstructs the datatable result contract from the persisted resultFormat', async () => { + const manager = new AIChatManager() + manager.registerJob(datatableJob) + const applyToolStatus = vi.spyOn(manager, 'applyToolStatus') + mocks.getJob.mockResolvedValue(completed({ result: [{ n: 1 }, { n: 2 }] })) + + await completeDetachedJob(manager) + + // No live closure is involved: the descriptor alone reshapes both the tool card + // and the model note, so a job that detached and survived a reload still reports + // the SQL contract (row count + shaped rows) rather than generic job output. + expect(applyToolStatus).toHaveBeenCalledWith('tc-1', { + content: 'Query returned 2 row(s)', + result: JSON.stringify([{ n: 1 }, { n: 2 }], null, 2) + }) + expect(manager.pendingJobNotes).toHaveLength(1) + expect(manager.pendingJobNotes[0]).toContain('"rowCount": 2') + }) + + it('skips reconstruction and emits no note for a canceled detached job', async () => { + const manager = new AIChatManager() + manager.registerJob(datatableJob) + const applyToolStatus = vi.spyOn(manager, 'applyToolStatus') + mocks.getJob.mockResolvedValue(completed({ success: false, canceled: true })) + + await completeDetachedJob(manager) + + // A user cancel isn't a result to shape or a completion to announce. + expect(manager.pendingJobNotes).toHaveLength(0) + expect(manager.backgroundJobs[0]?.status).toBe('canceled') + expect(applyToolStatus).toHaveBeenCalledWith('tc-1', { + content: 'Background job canceled', + logs: expect.anything() + }) + }) + + it('falls back to the generic note when the job has no resultFormat', async () => { + const manager = new AIChatManager() + manager.registerJob({ + jobId: 'job-1', + toolCallId: 'tc-1', + kind: 'script', + label: 'run', + workspace: 'ws' + }) + mocks.getJob.mockResolvedValue(completed()) + + await completeDetachedJob(manager) + + expect(manager.pendingJobNotes).toHaveLength(1) + expect(manager.pendingJobNotes[0]).toContain('Background job job-1 for "run" succeeded') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte new file mode 100644 index 0000000000..0a911134b6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte @@ -0,0 +1,456 @@ + + +{#snippet externalLinkIcon()} + +{/snippet} + + + {#snippet buttonReplacement()} +
+ +
+ {/snippet} + {#snippet menu({ item, builders, close })} +
+ + + +
+
Model
+
+ {#each models as m (m.provider + m.model)} + selectModel(m)} + > + {m.model} + {#if m.model === providerModel.model && m.provider === providerModel.provider} + + {/if} + + {/each} +
+ +
+ {#if capability.supported} + + +
+ Thinking + {currentStop} +
+ {#if stops.length > 1} + +
+ selectReasoning(stops[+e.currentTarget.value])} + use:isolatePointer + class="lean-range no-default-style w-full" + aria-label="Reasoning effort" + /> +
+ {/if} +
+ {:else} + +
+
Thinking
+
Not supported by this model
+
+ {/if} +
+ {/snippet} +
+ + + + diff --git a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte b/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte deleted file mode 100644 index 683bf3c586..0000000000 --- a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte +++ /dev/null @@ -1,186 +0,0 @@ - - -{#snippet externalLinkIcon()} - -{/snippet} - - - {#snippet buttonReplacement()} - {/each} @@ -182,12 +299,27 @@ variant="subtle" unifiedSize="sm" iconOnly - title="Send" - startIcon={{ icon: ArrowUp }} + title={multiSelect ? 'Add answer' : 'Send'} + startIcon={{ icon: multiSelect ? Plus : ArrowUp }} disabled={!canSubmitCustomAnswer} onClick={submitCustomAnswer} btnClasses="shrink-0" />
+ + {#if multiSelect} + + {/if}
diff --git a/frontend/src/lib/components/copilot/chat/AssistantMessage.svelte b/frontend/src/lib/components/copilot/chat/AssistantMessage.svelte index dd6e6a73a2..80cd446d13 100644 --- a/frontend/src/lib/components/copilot/chat/AssistantMessage.svelte +++ b/frontend/src/lib/components/copilot/chat/AssistantMessage.svelte @@ -3,6 +3,7 @@ import { gfmPlugin } from 'svelte-exmarkdown/gfm' import { twMerge } from 'tailwind-merge' import { Brain, ChevronDown, ChevronRight, Loader2 } from 'lucide-svelte' + import { slide } from 'svelte/transition' import type { DisplayMessage } from './shared' import CodeDisplay from './script/CodeDisplay.svelte' import LinkRenderer from './LinkRenderer.svelte' @@ -95,6 +96,7 @@ {#if reasoningExpanded}
+ import { Sparkles } from 'lucide-svelte' + import DrillPicker from '$lib/components/DrillPicker.svelte' + import type { DrillLeaf, DrillNode } from '$lib/components/drillPicker' + import type { AiSkillListItem } from './global/core' + + interface Props { + skills: AiSkillListItem[] + onSelect: (skill: AiSkillListItem) => void + setShowing?: (showing: boolean) => void + externalFilter?: string + autoFocus?: boolean + } + + let { skills, onSelect, setShowing, externalFilter, autoFocus = true }: Props = $props() + + type DrillPickerHandle = { + handleKeydown: (e: KeyboardEvent) => void + } + + let inner = $state(undefined) + + const tree = $derived[]>( + skills.map((skill) => ({ + type: 'leaf' as const, + key: `skill:${skill.name}`, + label: `/${skill.name}`, + secondary: skill.description, + searchableText: `${skill.name} ${skill.description}`, + data: skill + })) + ) + + export function handleKeydown(e: KeyboardEvent) { + inner?.handleKeydown(e) + } + + function handlePick(leaf: DrillLeaf) { + onSelect(leaf.data) + } + + function onDocumentKeydown(e: KeyboardEvent) { + if (e.key === 'Escape' && !e.defaultPrevented) { + setShowing?.(false) + } + } + + $effect(() => { + document.addEventListener('keydown', onDocumentKeydown) + return () => document.removeEventListener('keydown', onDocumentKeydown) + }) + + +{#snippet skillIcon(_leaf: DrillLeaf)} + +{/snippet} + +
+ +
diff --git a/frontend/src/lib/components/copilot/chat/ChatContextPicker.svelte b/frontend/src/lib/components/copilot/chat/ChatContextPicker.svelte index c083f9eab8..45a81113a3 100644 --- a/frontend/src/lib/components/copilot/chat/ChatContextPicker.svelte +++ b/frontend/src/lib/components/copilot/chat/ChatContextPicker.svelte @@ -4,21 +4,23 @@ AI chat `@`-mention dropdown. Mounts the generic `DrillPicker` with a unified tree: Diffs / Modules / Databases / Workspace - ├── All / Flows / Scripts + ├── All / Flows / Scripts / Apps │ └── f/scope/sub/leaf … The Diffs / Modules / Databases branches are synthesized from the chat's in-memory `availableContext`. The Workspace branch delegates to `buildWorkspaceTree` so the picker shares the workspace caching machinery -with the standalone picker used by `EditorHeader`. +with the standalone picker used by `EditorHeader`. The Apps branch only +appears in GLOBAL chat and lists raw (code-based) apps; visual apps are +excluded. On a workspace-leaf pick, emits a reference-only `WorkspaceScriptElement` / -`WorkspaceFlowElement` (path + title + summary). Content is materialized -at message-prep time by `AIChatManager` — see PR #9216. +`WorkspaceFlowElement` / `WorkspaceAppElement` (path + title + summary). +Content is materialized at message-prep time by `AIChatManager` — see PR #9216. --> {#if visible} - - Context usage: ~{formatTokenCount(usedTokens)}{contextWindow - ? ` / ${formatTokenCount(contextWindow)}` - : ''} - + + +
+
+
+
+
+ {#snippet text()} +
+

Context usage

+

+ ~{formatTokenCount(usedTokens)}{contextWindow + ? ` / ${formatTokenCount(contextWindow)}` + : ''}{fillPct !== undefined ? ` (${fillPct}%)` : ''} +

+ {#if ratio !== undefined && ratio >= COMPACTION_TRIGGER_RATIO} +

History will be compacted soon to free up space.

+ {/if} + {#if canCompact} +

+ Type /compact to summarize and free up space now. +

+ {/if} +
+ {/snippet} +
{/if} diff --git a/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts b/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts index 9173e4c441..d24246b711 100644 --- a/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts @@ -1,5 +1,5 @@ import { type DBSchema as IDBSchema, type IDBPDatabase } from 'idb' -import type { DisplayMessage } from './shared' +import type { ChatJob, DisplayMessage } from './shared' import { expanded, messageDraft } from './chatDraft' import { createLongHash } from '$lib/editorLangUtils' import { userScopedDb, type UserScopedDbMigrateDeps } from '$lib/userScopedDb' @@ -25,6 +25,16 @@ interface ChatSchema extends IDBSchema { // New writes store the plain reported token count; chats persisted by // earlier versions may still hold the legacy anchor object. contextUsage?: PersistedContextUsage + // Workspace items this chat modified via AI tool calls, as + // `${UserDraftItemKind}:${storagePath}` keys. Persisted out-of-band from + // the message arrays so it survives compaction. Absent (undefined) on + // chats predating this feature → consumers fall back to showing all + // workspace drafts; a defined array (even empty) means "tracked". + modifiedItems?: string[] + // Jobs this chat started that detached into the background, so an + // in-flight job's tray row and completion survive a reload. Absent on + // chats predating this feature. Persisted out-of-band like modifiedItems. + backgroundJobs?: ChatJob[] } } } @@ -80,6 +90,29 @@ export function __resetLegacyChatClaimForTesting(): void { legacyChatClaim = undefined } +// Read a chat's modified-items mask by chatId WITHOUT mounting an AIChatManager, +// for the standalone /forks/compare route. Returns undefined for a legacy chat +// (no field) so the page falls back to selecting all items; a defined array +// (even empty) narrows the preselection. Opens a throwaway user-scoped handle; +// the `get` is O(1) on the `id` keyPath. +export async function readChatModifiedItems(chatId: string): Promise { + const dbh = userScopedDb(DB_NAME, { + version: 1, + upgrade: createChatStore, + migrate: migrateLegacyChatDb + }) + try { + const db = await dbh.whenReady() + const chat = await db?.get('chats', chatId) + return chat?.modifiedItems + } catch (err) { + console.error('Could not read chat modified items', err) + return undefined + } finally { + dbh.close() + } +} + export default class HistoryManager { // Per-instance handle to the shared per-user DB lifecycle. There is one // HistoryManager per AIChatManager (the singleton + one per session runtime), @@ -100,6 +133,8 @@ export default class HistoryManager { lastModified: number sessionId?: string contextUsage?: PersistedContextUsage + modifiedItems?: string[] + backgroundJobs?: ChatJob[] } > = $state({}) @@ -173,10 +208,20 @@ export default class HistoryManager { return Object.values(this.savedChats) } + getModifiedItems(id: string): string[] | undefined { + return this.savedChats[id]?.modifiedItems + } + + getBackgroundJobs(id: string): ChatJob[] | undefined { + return this.savedChats[id]?.backgroundJobs + } + async saveChat( displayMessages: DisplayMessage[], messages: ChatCompletionMessageParam[], - contextUsage?: number + contextUsage?: number, + modifiedItems?: string[], + backgroundJobs?: ChatJob[] ) { if (displayMessages.length > 0) { // Compaction replaces the original first message with a summary boundary. @@ -203,7 +248,30 @@ export default class HistoryManager { id: this.currentChatId, lastModified: Date.now(), ...(this.sessionId ? { sessionId: this.sessionId } : {}), - ...(contextUsage !== undefined ? { contextUsage } : {}) + ...(contextUsage !== undefined ? { contextUsage } : {}), + // Only persist when the caller passes a defined array — an untracked + // chat (the global side-panel chat, mask still undefined) must not be + // stamped with [], which would flip it to the filtered view. Session + // chats are always tracked (see AIChatManager.loadPastChat), so they do + // pass a defined array and persist it. Since `put` replaces the whole + // record, a caller that omits the argument must not ERASE a tracked + // chat's stored mask — fall back to the previously saved field. + // Snapshot the fallback: savedChats is $state, so the stored value is a + // proxy that structuredClone (used by IndexedDB put) cannot clone. + ...(modifiedItems !== undefined + ? { modifiedItems } + : this.savedChats[this.currentChatId]?.modifiedItems !== undefined + ? { modifiedItems: $state.snapshot(this.savedChats[this.currentChatId].modifiedItems) } + : {}), + // Same "don't erase on omit" guard as modifiedItems: a turn-end save + // that doesn't pass backgroundJobs must keep the tray's stored jobs. + ...(backgroundJobs !== undefined + ? { backgroundJobs } + : this.savedChats[this.currentChatId]?.backgroundJobs !== undefined + ? { + backgroundJobs: $state.snapshot(this.savedChats[this.currentChatId].backgroundJobs) + } + : {}) } this.savedChats = { ...this.savedChats, @@ -218,9 +286,11 @@ export default class HistoryManager { async save( displayMessages: DisplayMessage[], messages: ChatCompletionMessageParam[], - contextUsage?: number + contextUsage?: number, + modifiedItems?: string[], + backgroundJobs?: ChatJob[] ) { - await this.saveChat(displayMessages, messages, contextUsage) + await this.saveChat(displayMessages, messages, contextUsage, modifiedItems, backgroundJobs) this.currentChatId = createLongHash() } diff --git a/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts b/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts index 38fb7ecf0d..540dfaf72e 100644 --- a/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts +++ b/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts @@ -141,3 +141,30 @@ describe('HistoryManager title across compaction', () => { expect(hm.getAllSavedChats().find((c) => c.id === id)?.title).toBe('original first question') }) }) + +describe('HistoryManager modified-items mask persistence', () => { + const msgs = [{ role: 'user', content: 'hello', index: 0 }] as DisplayMessage[] + + it('a save without the argument preserves a previously stored mask', async () => { + const hm = new HistoryManager() + await hm.init() + const id = hm.getCurrentChatId() + + await hm.saveChat(msgs, [] as ChatCompletionMessageParam[], undefined, ['script:u/a/x']) + expect(hm.getModifiedItems(id)).toEqual(['script:u/a/x']) + + // e.g. manual compaction re-saving the transcript: the whole record is + // rewritten, but the tracked mask must survive. + await hm.saveChat(msgs, [] as ChatCompletionMessageParam[]) + expect(hm.getModifiedItems(id)).toEqual(['script:u/a/x']) + }) + + it('never retroactively stamps an untracked chat', async () => { + const hm = new HistoryManager() + await hm.init() + const id = hm.getCurrentChatId() + + await hm.saveChat(msgs, [] as ChatCompletionMessageParam[]) + expect(hm.getModifiedItems(id)).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/JobsSegment.svelte b/frontend/src/lib/components/copilot/chat/JobsSegment.svelte new file mode 100644 index 0000000000..0bcb315bda --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/JobsSegment.svelte @@ -0,0 +1,384 @@ + + +{#if jobs.length > 0} + +
{announcement}
+ + {#snippet trigger()} + + Jobs + + + + {#if runningJob}{runningJob.label}{:else}{segment.text}{/if} + + {#if chipElapsed} + + {chipElapsed} + + {/if} + + + {/snippet} + + {#snippet content()} +
+
Jobs this session
+
+ {#each sortedJobs as job (job.jobId)} +
+ {#if job.status === 'queued' || !job.job} + + + {:else} + + {/if} + {job.label} + {elapsedLabel(job)} +
+ {#if job.status === 'suspended'} + + {/if} + {#if !isTerminal(job.status)} + + {/if} +
+
+ {/each} +
+
+ {/snippet} +
+ + + + + {#if approvalJob} + { + // Approving resumes the flow (back to running); optimistically drop the + // suspended status so the segment updates instantly. Closing the modal + // triggers the on-close effect above, which re-polls to reconcile. + if (approved && approvalJob) { + aiChatManager.updateJob(approvalJob.id, { status: 'running' }) + } + approvalOpen = false + }} + /> + {:else} +
Loading approval…
+ {/if} +
+
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/ToolMessageActions.svelte b/frontend/src/lib/components/copilot/chat/ToolMessageActions.svelte index 0a2fd1ab79..afee03bcf2 100644 --- a/frontend/src/lib/components/copilot/chat/ToolMessageActions.svelte +++ b/frontend/src/lib/components/copilot/chat/ToolMessageActions.svelte @@ -1,15 +1,24 @@ + +
(showDelete = true)} + onmouseleave={() => (showDelete = false)} + role="listitem" + title={`${file.name} — ${detail}`} +> + + {file.name} +
diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte new file mode 100644 index 0000000000..50b961c090 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte @@ -0,0 +1,86 @@ + + +{#snippet chip(card: Card)} + {#if card.kind === 'folder'} + removeCard(card)} /> + {:else} + removeCard(card)} /> + {/if} +{/snippet} + +{#if cards.length > 0} +
+ {#each visible as card (card.key)} + {@render chip(card)} + {/each} + + {#if overflow.length > 0} + + {#snippet trigger()} +
+ +{overflow.length} +
+ {/snippet} + {#snippet content()} +
+ {#each overflow as card (card.key)} + {@render chip(card)} + {/each} +
+ {/snippet} +
+ {/if} + + {#if lockedCount > 0} + + {/if} +
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte new file mode 100644 index 0000000000..068447a878 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte @@ -0,0 +1,48 @@ + + +
(showDelete = true)} + onmouseleave={() => (showDelete = false)} + role="listitem" + title={hoverList} +> + + {folder.name} +
diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts new file mode 100644 index 0000000000..124b390ec2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts @@ -0,0 +1,602 @@ +/** + * Session-scoped store of files/folders the user has linked to the GLOBAL AI chat. + * + * Persistence model (survives reload, keyed by session in ./attachedFilesDB): + * - FILES are always stored as a full-byte Blob snapshot — same on every browser, + * no permission re-grant, never "locked". + * - FOLDERS link as a live File System Access directory handle where the API exists + * (one record, re-enumerated live on restore — folder files are read through the + * handle, not copied). Where it doesn't (Firefox/Safari), a dropped/picked folder's + * files are snapshotted individually (each carrying its `folder`/`relPath`) so they + * regroup into the same folder chip on restore. + * + * Storage is bounded by the real browser quota (writes that exceed it are caught and + * the item simply isn't persisted — it stays usable for the session). Persistence is + * gated on the session being persisted (non-transient); links in a transient session + * are buffered and flushed on the first send. + */ +import { createLongHash } from '$lib/editorLangUtils' +import { buildLineIndex, isTextFile, type FileEntry } from './fileEngine' +import { + putItem, + deleteItem, + getItemsForSession, + ensurePersistentStorage, + type PersistedAttachedItem +} from './attachedFilesDB' +import { enumerateDir, isIgnoredPath, queryReadPermission, requestReadPermission } from './fsAccess' + +export type AttachedFileStatus = 'indexing' | 'ready' | 'error' | 'locked' | 'unavailable' + +export interface AttachedFile extends FileEntry { + size: number + status: AttachedFileStatus + error?: string + /** Top-level folder this file came from (first path segment), if part of a folder. */ + folder?: string + /** Persisted source-record id. Folder children share the folder's record id. */ + sourceId: string + /** Parent directory handle (folder children only) — used to re-grant / re-enumerate. */ + handle?: FileSystemDirectoryHandle + /** Relative path within the folder (folder children only) — stable key for refresh diffing. */ + relPath?: string + /** + * Internal: a single placeholder row standing in for a not-yet-expanded folder + * (locked/unavailable). Consumers should read `store.folders` instead of testing this. + */ + isFolderRoot?: boolean +} + +/** A linked folder as a first-class object — consumers read this instead of re-grouping rows. */ +export interface AttachedFolder { + name: string + /** Aggregate status (locked > unavailable > indexing > error > ready). */ + status: AttachedFileStatus + /** Child files; empty while the folder is locked/unavailable after a reload. */ + files: AttachedFile[] +} + +/** Aggregate a folder's rows (children + a possible placeholder) into one status. */ +function folderStatus(rows: AttachedFile[]): AttachedFileStatus { + for (const status of ['locked', 'unavailable', 'indexing', 'error'] as const) { + if (rows.some((f) => f.status === status)) return status + } + return 'ready' +} + +export interface AddFilesResult { + added: string[] + rejected: { name: string; reason: string }[] +} + +/** A file to link: a raw File, or `{ file, path? }` (path = relative display name). */ +export type FileToAttach = File | { file: File; path?: string } + +const EMPTY = new Blob([]) + +export class AttachedFilesStore { + files = $state([]) + + /** Session context, set by the runtime; persistence writes are gated on `#persisted`. */ + sessionId: string | undefined = undefined + #persisted = false + /** Records buffered while the session is transient (flushed on first send). */ + #pending: PersistedAttachedItem[] = [] + + list(): AttachedFile[] { + return this.files + } + get(name: string): AttachedFile | undefined { + // Resolve to a real file — a folder-root placeholder may share the folder's name. + return this.files.find((f) => f.name === name && !f.isFolderRoot) + } + readyFiles(): AttachedFile[] { + // Folder-root placeholders aren't real files — never expose them to the read/search tools. + return this.files.filter((f) => f.status === 'ready' && !f.isFolderRoot) + } + get count(): number { + return this.files.length + } + + /** Linked folders, children grouped and status aggregated (placeholder rows hidden). */ + folders: AttachedFolder[] = $derived.by(() => { + const byName = new Map() + for (const f of this.files) { + if (!f.folder) continue + const rows = byName.get(f.folder) + if (rows) rows.push(f) + else byName.set(f.folder, [f]) + } + return [...byName.entries()].map(([name, rows]) => ({ + name, + status: folderStatus(rows), + files: rows.filter((f) => !f.isFolderRoot) + })) + }) + + /** Files linked on their own (not as part of a folder). */ + standalone: AttachedFile[] = $derived.by(() => this.files.filter((f) => !f.folder)) + + /** Number of locked folders needing a re-grant. */ + get lockedCount(): number { + return this.folders.filter((f) => f.status === 'locked').length + } + + clear(): void { + this.files = [] + this.#pending = [] + } + + removeFile(name: string): void { + // Target the real file only — never a folder-root placeholder that happens to share + // the name (those are managed via removeFolder), else removing a same-named standalone + // file would also drop the folder's placeholder. + const f = this.files.find((x) => x.name === name && !x.isFolderRoot) + if (!f) return + this.files = this.files.filter((x) => !(x.name === name && !x.isFolderRoot)) + void this.#deleteRecord(f.sourceId) + } + + /** Remove every file linked as part of the given folder (and its persisted record). */ + removeFolder(folder: string): void { + const ids = new Set(this.files.filter((f) => f.folder === folder).map((f) => f.sourceId)) + this.files = this.files.filter((f) => f.folder !== folder) + for (const id of ids) void this.#deleteRecord(id) + } + + // ---------------------------------------------------------------- linking + + /** + * Link individual files — always stored as a Blob snapshot. Items carrying a folder + * path (`folder/sub/file`, from a dropped/picked folder) are grouped into a folder and + * have their junk paths (node_modules/.git/dotfiles) skipped; a loose single file is + * kept as-is (so an explicitly attached `.env` isn't filtered out). + */ + async addFiles(input: FileList | FileToAttach[]): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + + for (const item of Array.from(input as ArrayLike)) { + const file = item instanceof File ? item : item.file + const desired = + (item instanceof File ? '' : (item.path ?? '')) || + (file as File & { webkitRelativePath?: string }).webkitRelativePath || + file.name || + 'file' + + const folder = desired.includes('/') ? desired.split('/')[0] : undefined + if (folder && isIgnoredPath(desired)) continue // skip junk inside folders + + if (this.#isDuplicate(desired, file)) continue // silent no-op on re-link + + const reason = await this.#preflight(file) + if (reason) { + result.rejected.push({ name: desired, reason }) + continue + } + + const name = this.#uniqueName(desired) + const relPath = folder ? desired : undefined + const sourceId = createLongHash() + + this.#pushIndexing({ name, file, folder, sourceId, relPath }) + result.added.push(name) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'snapshot', + name, + folder, + relPath, + blob: file, + size: file.size, + lastModified: file.lastModified, + addedAt: Date.now() + }) + } + + return result + } + + /** + * Link a folder via a live directory handle (File System Access path only). + * Enumerates the handle internally (junk-filtered, capped) — the same walk used + * on restore and refresh, so callers never pre-enumerate. + */ + async addFolder(dirHandle: FileSystemDirectoryHandle): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + const folder = dirHandle.name + const existing = this.files.filter((f) => f.folder === folder) + if (existing.length > 0) { + const placeholder = existing.length === 1 ? existing.find((f) => f.isFolderRoot) : undefined + if (placeholder) { + // Re-picking a folder that sits locked/unavailable after a reload is a natural + // recovery gesture — replace the stale link with the freshly-granted handle. + this.files = this.files.filter((f) => f.sourceId !== placeholder.sourceId) + void this.#deleteRecord(placeholder.sourceId) + } else { + // Same basename, possibly a different directory — surface it instead of a silent no-op. + result.rejected.push({ name: folder, reason: 'A folder with this name is already linked' }) + return result + } + } + + const files = await enumerateDir(dirHandle) + const sourceId = createLongHash() + for (const { file, path } of files) { + if (!(await this.#sniffText(file))) { + result.rejected.push({ name: path, reason: 'Not a text file' }) + continue + } + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + result.added.push(name) + } + // Keep the folder represented even when it links empty (or all-binary): a placeholder + // carries the handle so the chip stays and refreshFolders picks up files added later. + // Persist unconditionally so an empty-at-link folder also survives a reload. + this.#ensureFolderRow(sourceId, folder, dirHandle) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'dir-handle', + name: folder, + folder, + handle: dirHandle, + addedAt: Date.now() + }) + return result + } + + // ------------------------------------------------------------- persistence + + /** Set session context and load any persisted items for it (called on activation). */ + async restore(sessionId: string, persisted: boolean): Promise { + this.sessionId = sessionId + this.#persisted = persisted + this.files = [] + this.#pending = [] + + const items = await getItemsForSession(sessionId) + for (const item of items) { + try { + if (item.kind === 'snapshot') { + if (!item.blob) { + this.#pushPlaceholder(item, 'unavailable') + continue + } + this.#pushIndexing({ + name: item.name, + file: item.blob, + folder: item.folder, + relPath: item.relPath, + sourceId: item.id + }) + } else { + // dir-handle (folder) + const handle = item.handle as FileSystemDirectoryHandle + if ((await queryReadPermission(handle)) === 'granted') { + await this.#expandFolder(handle, item.id) + } else { + this.#pushPlaceholder(item, 'locked', true) + } + } + } catch { + this.#pushPlaceholder(item, 'unavailable', item.kind === 'dir-handle') + } + } + } + + /** Re-grant any locked folder handles. MUST be called within a user gesture (e.g. on send). */ + async regrantLocked(): Promise { + const sources = new Map() + for (const f of this.files) { + if (f.status === 'locked' && f.handle) sources.set(f.sourceId, f) + } + if (sources.size === 0) return + + // Kick off all permission requests within the gesture, then process. A rejected + // request (requestReadPermission never rejects, but stay defensive) counts as denied. + const decided = await Promise.all( + [...sources.values()].map((f) => + requestReadPermission(f.handle!).then( + (perm) => ({ f, perm }), + () => ({ f, perm: 'denied' as PermissionState }) + ) + ) + ) + for (const { f, perm } of decided) { + if (perm !== 'granted') continue + try { + await this.#expandFolder(f.handle as FileSystemDirectoryHandle, f.sourceId) + // Children are in — drop the locked placeholder row, then restore a ready + // placeholder if the folder came back empty/all-binary (else dropping the only + // handle-bearing row would unlink the folder and stop it ever refreshing). + this.files = this.files.filter((x) => !(x.sourceId === f.sourceId && x.isFolderRoot)) + this.#ensureFolderRow(f.sourceId, f.folder ?? f.name, f.handle as FileSystemDirectoryHandle) + } catch { + // Enumeration failed (folder moved/deleted on disk): drop any partially-added + // children and keep the placeholder so the chip shows "unavailable". + this.files = this.files.filter((x) => x.sourceId !== f.sourceId || x.isFolderRoot) + this.#patchSource(f.sourceId, { status: 'unavailable' }) + } + } + } + + /** Flush buffered links once the session becomes persistent (first send). */ + async flushPending(): Promise { + this.#persisted = true + if (!this.sessionId) return + const pending = this.#pending + this.#pending = [] + if (pending.length === 0) return + void ensurePersistentStorage() + for (const item of pending) { + try { + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file', e) + } + } + } + + async #persist(item: PersistedAttachedItem): Promise { + if (this.#persisted && this.sessionId) { + void ensurePersistentStorage() + try { + // A QuotaExceededError just means it won't survive a reload — the item + // stays usable for this session. Swallow + log rather than fail the link. + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file (kept for this session)', e) + } + } else { + this.#pending.push(item) + } + } + + async #deleteRecord(sourceId: string): Promise { + this.#pending = this.#pending.filter((p) => p.id !== sourceId) + if (this.#persisted) { + try { + await deleteItem(sourceId) + } catch { + /* ignore */ + } + } + } + + // ------------------------------------------------------------- internals + + /** Identical re-link (same name, or same File identity) → silent no-op. */ + #isDuplicate(desired: string, file: File): boolean { + return this.files.some( + (f) => + // Folder-root placeholders aren't real files — they must not block attaching a + // standalone file that happens to share the folder's name. + !f.isFolderRoot && + (f.name === desired || + // Identical re-drop at the SAME relative path (its row name may have been + // auto-suffixed). Keyed on the path, NOT the basename — otherwise two distinct + // files sharing a basename under different folder subdirs (proj/a/index.ts vs + // proj/b/index.ts) would be wrongly deduped and silently dropped. + ((f.relPath ?? f.name) === desired && + f.size === file.size && + f.file instanceof File && + f.file.lastModified === file.lastModified)) + ) + } + + /** Returns a rejection reason, or undefined if the file may be linked. */ + async #preflight(file: File): Promise { + if (!(await this.#sniffText(file))) return 'Not a text file' + return undefined + } + + async #sniffText(file: Blob): Promise { + try { + return await isTextFile(file) + } catch { + return false + } + } + + #pushIndexing(p: { + name: string + file: File | Blob + folder?: string + sourceId: string + handle?: FileSystemDirectoryHandle + relPath?: string + }): void { + this.files = [ + ...this.files, + { + name: p.name, + file: p.file, + size: p.file.size, + lineIndex: [], + lineCount: 0, + status: 'indexing', + folder: p.folder, + sourceId: p.sourceId, + handle: p.handle, + relPath: p.relPath + } + ] + void this.#indexFile(p.name, p.file) + } + + #pushPlaceholder( + item: PersistedAttachedItem, + status: AttachedFileStatus, + isFolderRoot = false + ): void { + this.files = [ + ...this.files, + { + name: item.name, + file: EMPTY, + size: item.size ?? 0, + lineIndex: [], + lineCount: 0, + status, + folder: item.folder, + sourceId: item.id, + handle: item.handle as FileSystemDirectoryHandle | undefined, + isFolderRoot + } + ] + } + + async #expandFolder(dirHandle: FileSystemDirectoryHandle, sourceId: string): Promise { + const folder = dirHandle.name + const children = await enumerateDir(dirHandle) + for (const { file, path } of children) { + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + } + this.#ensureFolderRow(sourceId, folder, dirHandle) + } + + /** + * Re-enumerate granted folder handles to reflect on-disk changes since they were + * linked/last refreshed: added/removed/renamed files and content edits. Called on + * each send so the AI sees the folder's current state. Unchanged files are left as-is + * (diffed by relative path + lastModified); only changed files are re-indexed. + */ + async refreshFolders(): Promise { + const sources = new Map() + for (const f of this.files) { + // Include folder-root placeholders (an emptied folder keeps only its placeholder), + // else the source is lost and the folder never re-enumerates again. + if (f.folder && f.handle) { + sources.set(f.sourceId, { handle: f.handle, folder: f.folder }) + } + } + for (const [sourceId, { handle, folder }] of sources) { + try { + if ((await queryReadPermission(handle)) !== 'granted') continue + const children = await enumerateDir(handle) + await this.#reconcileFolder(sourceId, folder, handle, children) + } catch { + this.#patchSource(sourceId, { status: 'unavailable' }) + } + } + } + + async #reconcileFolder( + sourceId: string, + folder: string, + handle: FileSystemDirectoryHandle, + children: { file: File; path: string }[] + ): Promise { + const existing = new Map() + for (const f of this.files) if (f.sourceId === sourceId && f.relPath) existing.set(f.relPath, f) + const seen = new Set() + + for (const { file, path } of children) { + seen.add(path) + const cur = existing.get(path) + if (!cur) { + // newly added on disk + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle, relPath: path }) + } else { + const curMod = cur.file instanceof File ? cur.file.lastModified : undefined + if (file.size !== cur.size || file.lastModified !== curMod) { + // content changed → re-read + re-index + this.#patch(cur.name, { file, size: file.size, status: 'indexing' }) + void this.#indexFile(cur.name, file) + } + } + } + // removed/renamed-away on disk → drop from memory + const removed = [...existing.values()].filter((f) => f.relPath && !seen.has(f.relPath)) + if (removed.length > 0) { + const names = new Set(removed.map((f) => f.name)) + this.files = this.files.filter((f) => !names.has(f.name)) + } + this.#ensureFolderRow(sourceId, folder, handle) + } + + /** + * Keep a linked folder represented even with no readable children: leave one + * handle-carrying placeholder row so the chip stays visible AND `refreshFolders` + * keeps the live source (without it, an emptied folder vanishes and never + * re-enumerates). Drop the placeholder as soon as real children exist again. + */ + #ensureFolderRow(sourceId: string, folder: string, handle: FileSystemDirectoryHandle): void { + const hasChild = this.files.some((f) => f.sourceId === sourceId && !f.isFolderRoot) + const hasPlaceholder = this.files.some((f) => f.sourceId === sourceId && f.isFolderRoot) + if (!hasChild && !hasPlaceholder) { + this.files = [ + ...this.files, + { + name: folder, + file: EMPTY, + size: 0, + lineIndex: [], + lineCount: 0, + status: 'ready', + folder, + sourceId, + handle, + isFolderRoot: true + } + ] + } else if (hasChild && hasPlaceholder) { + this.files = this.files.filter((f) => !(f.sourceId === sourceId && f.isFolderRoot)) + } + } + + async #indexFile(name: string, file: File | Blob): Promise { + try { + const { lineIndex, lineCount } = await buildLineIndex(file) + this.#patchFile(name, file, { lineIndex, lineCount, status: 'ready' }) + } catch (e) { + this.#patchFile(name, file, { + status: 'error', + error: e instanceof Error ? e.message : String(e) + }) + } + } + + #patch(name: string, changes: Partial): void { + this.files = this.files.map((f) => (f.name === name ? { ...f, ...changes } : f)) + } + /** + * Patch the row for `name` ONLY while it still holds the exact `file` we indexed. + * `buildLineIndex` is async and unawaited; between its start and finish the row's + * file can be swapped (remove + re-add a same-named file, or a folder refresh + * re-indexing an edited file). Without the identity check a stale completion would + * stamp the wrong lineIndex/lineCount on the new file, and read_file would then slice + * the new Blob with the old offsets. + */ + #patchFile(name: string, file: File | Blob, changes: Partial): void { + this.files = this.files.map((f) => + f.name === name && f.file === file ? { ...f, ...changes } : f + ) + } + #patchSource(sourceId: string, changes: Partial): void { + this.files = this.files.map((f) => (f.sourceId === sourceId ? { ...f, ...changes } : f)) + } + + #uniqueName(original: string): string { + // Uniqueness is only among real files — folder-root placeholders may share a name + // with a standalone file and must not push it to a "(2)" suffix. + const taken = (n: string) => this.files.some((f) => f.name === n && !f.isFolderRoot) + if (!taken(original)) return original + const dot = original.lastIndexOf('.') + const base = dot > 0 ? original.slice(0, dot) : original + const ext = dot > 0 ? original.slice(dot) : '' + let n = 2 + let candidate = `${base} (${n})${ext}` + while (taken(candidate)) { + n++ + candidate = `${base} (${n})${ext}` + } + return candidate + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts new file mode 100644 index 0000000000..c81bc1f9c4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts @@ -0,0 +1,476 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// Mock persistence + File System Access so we exercise the in-memory store logic. +vi.mock('./attachedFilesDB', () => ({ + putItem: vi.fn(async () => {}), + deleteItem: vi.fn(async () => {}), + getItemsForSession: vi.fn(async () => []), + ensurePersistentStorage: vi.fn(async () => {}) +})) + +const enumerateDirMock = vi.fn<(h: unknown) => Promise<{ file: File; path: string }[]>>() +vi.mock('./fsAccess', () => ({ + enumerateDir: (h: unknown) => enumerateDirMock(h), + isIgnoredPath: (p: string) => + p.split('/').some((s) => s.startsWith('.') || ['node_modules', 'dist', '.git'].includes(s)), + queryReadPermission: vi.fn(async () => 'granted'), + requestReadPermission: vi.fn(async () => 'granted') +})) + +// buildLineIndex is real by default; a single test flips to 'manual' to control +// completion ordering and exercise the stale-index race guard. +type BuildResult = { lineIndex: number[]; lineCount: number } +const buildDeferreds: Array<{ file: Blob; resolve: (r: BuildResult) => void }> = [] +let buildMode: 'real' | 'manual' = 'real' +vi.mock('./fileEngine', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + buildLineIndex: (file: Blob) => + buildMode === 'real' + ? actual.buildLineIndex(file) + : new Promise((resolve) => buildDeferreds.push({ file, resolve })) + } +}) + +import { AttachedFilesStore } from './attachedFiles.svelte' + +function file(name: string, content: string, lastModified = 1): File { + return new File([content], name, { type: 'text/plain', lastModified }) +} + +const dir = { kind: 'directory', name: 'proj' } as unknown as FileSystemDirectoryHandle + +async function settle(store: AttachedFilesStore) { + for (let i = 0; i < 100 && store.list().some((f) => f.status === 'indexing'); i++) { + await new Promise((r) => setTimeout(r, 2)) + } +} + +const names = (store: AttachedFilesStore) => + store + .list() + .map((f) => f.name) + .sort() + +describe('AttachedFilesStore', () => { + let store: AttachedFilesStore + beforeEach(async () => { + store = new AttachedFilesStore() + await store.restore('s1', false) + }) + + it('links and indexes individual files as snapshots', async () => { + await store.addFiles([file('a.txt', 'one\ntwo\n')]) + await settle(store) + const f = store.get('a.txt') + expect(f?.status).toBe('ready') + expect(f?.lineCount).toBe(2) + expect(f?.handle).toBeUndefined() // files never carry a handle + }) + + it('removes a file', async () => { + await store.addFiles([file('a.txt', 'x')]) + store.removeFile('a.txt') + expect(store.count).toBe(0) + }) + + it('links a folder via a directory handle (enumerating it internally)', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n'), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + expect(enumerateDirMock).toHaveBeenCalledWith(dir) + expect(names(store)).toEqual(['proj/app.ts', 'proj/old.ts']) + expect(store.get('proj/app.ts')?.folder).toBe('proj') + }) + + it('refreshFolders detects rename, add, edit, and delete', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n', 1), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n', 1), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + + // On disk: app.ts edited (mtime bumped), old.ts renamed → new.ts, readme.md added. + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\nedited\n', 2), path: 'proj/app.ts' }, + { file: file('new.ts', 'y\n', 1), path: 'proj/new.ts' }, + { file: file('readme.md', '# hi\n', 1), path: 'proj/readme.md' } + ]) + await store.refreshFolders() + await settle(store) + + // old.ts dropped (renamed away); new.ts + readme.md added; app.ts kept. + expect(names(store)).toEqual(['proj/app.ts', 'proj/new.ts', 'proj/readme.md']) + // edited file re-indexed to its new content (2 lines). + expect(store.get('proj/app.ts')?.status).toBe('ready') + expect(store.get('proj/app.ts')?.lineCount).toBe(2) + }) + + it('exposes folders and standalone as structured views', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await store.addFolder(dir) + await store.addFiles([file('solo.txt', 'one\n')]) + await settle(store) + + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + expect(store.folders[0].status).toBe('ready') + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/app.ts', + 'proj/sub/b.ts' + ]) + expect(store.standalone.map((f) => f.name)).toEqual(['solo.txt']) + expect(store.lockedCount).toBe(0) + }) + + it('a locked folder surfaces as one folder with no files', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + + expect(s2.folders).toEqual([{ name: 'proj', status: 'locked', files: [] }]) + expect(s2.standalone).toEqual([]) + expect(s2.lockedCount).toBe(1) + }) + + it('re-picking a locked folder relinks it instead of silently no-oping', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.folders[0]?.status).toBe('locked') + + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + const result = await s2.addFolder(dir) + await settle(s2) + expect(result.added).toEqual(['proj/app.ts']) + expect(s2.folders).toHaveLength(1) + expect(s2.folders[0].status).toBe('ready') + }) + + it('rejects linking a second folder with the same name (visible, not silent)', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + const result = await store.addFolder(dir) + expect(result.added).toEqual([]) + expect(result.rejected[0]?.reason).toMatch(/already linked/) + expect(store.folders).toHaveLength(1) + }) + + it('regrant keeps the folder visible as unavailable when enumeration fails', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Permission re-granted, but the directory is gone from disk. + enumerateDirMock.mockRejectedValueOnce(new Error('directory removed')) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'unavailable', files: [] }]) + }) + + it('regrant of an empty folder keeps it linked and refreshing (not unlinked)', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Access re-granted, but the folder is currently empty — it must stay linked (ready + // placeholder), not vanish when the locked placeholder is dropped. + enumerateDirMock.mockResolvedValueOnce([]) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(s2.lockedCount).toBe(0) + + // A file added afterward is picked up — the handle survived. + enumerateDirMock.mockResolvedValueOnce([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s2.refreshFolders() + await settle(s2) + expect(s2.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) + + it('removeFolder drops all of a folder’s files', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/b.ts' } + ]) + await store.addFolder(dir) + store.removeFolder('proj') + expect(store.count).toBe(0) + }) + + it('snapshots a folder via addFiles (paths), grouping it and persisting folder + relPath', async () => { + const { putItem } = await import('./attachedFilesDB') + // A persisted (non-transient) session writes through to IndexedDB immediately. + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + expect(s.folders.map((f) => f.name)).toEqual(['proj']) + expect(s.folders[0].files.map((f) => f.relPath).sort()).toEqual(['proj/a.ts', 'proj/sub/b.ts']) + expect(s.standalone).toEqual([]) + const rec = (putItem as ReturnType).mock.calls + .map((c) => c[0]) + .find((r) => r.name === 'proj/a.ts') + expect(rec).toMatchObject({ kind: 'snapshot', folder: 'proj', relPath: 'proj/a.ts' }) + }) + + it('keeps same-basename files from different folder subdirs (dedup by path, not basename)', async () => { + // Two distinct files with the same basename, size and lastModified, different subdirs. + const res = await store.addFiles([ + { file: file('index.ts', 'a\n', 5), path: 'proj/a/index.ts' }, + { file: file('index.ts', 'a\n', 5), path: 'proj/b/index.ts' } + ]) + await settle(store) + expect(res.added.sort()).toEqual(['proj/a/index.ts', 'proj/b/index.ts']) + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/a/index.ts', + 'proj/b/index.ts' + ]) + }) + + it('skips junk paths (node_modules/.git/dotfiles) inside a snapshotted folder', async () => { + const res = await store.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('dep.js', 'z\n'), path: 'proj/node_modules/dep.js' }, + { file: file('cfg', 'w\n'), path: 'proj/.git/config' } + ]) + await settle(store) + expect(res.added).toEqual(['proj/a.ts']) + expect(store.folders[0].files).toHaveLength(1) + }) + + it('keeps an explicitly attached standalone dotfile (filter is folder-only)', async () => { + const res = await store.addFiles([file('.env', 'SECRET=1\n')]) + await settle(store) + expect(res.added).toEqual(['.env']) + expect(store.standalone.map((f) => f.name)).toEqual(['.env']) + }) + + it('restores a snapshot folder grouped from its persisted folder/relPath', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 's-a', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/a.ts', + folder: 'proj', + relPath: 'proj/a.ts', + blob: file('a.ts', 'x\n'), + addedAt: 0 + }, + { + id: 's-b', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/b.ts', + folder: 'proj', + relPath: 'proj/b.ts', + blob: file('b.ts', 'y\n'), + addedAt: 0 + } + ]) + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + await settle(s2) + expect(s2.folders.map((f) => f.name)).toEqual(['proj']) + expect(s2.folders[0].files).toHaveLength(2) + expect(s2.standalone).toEqual([]) + }) + + it('imposes no file-count cap on a folder', async () => { + enumerateDirMock.mockResolvedValue( + Array.from({ length: 150 }, (_, i) => ({ + file: file(`f${i}.ts`, 'x\n'), + path: `proj/f${i}.ts` + })) + ) + await store.addFolder(dir) + await settle(store) + expect(store.folders[0].files.length).toBe(150) + }) + + it('removeFolder deletes every snapshot record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + const ids = s + .list() + .filter((f) => f.folder === 'proj') + .map((f) => f.sourceId) + expect(ids.length).toBe(2) + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + const deleted = (deleteItem as ReturnType).mock.calls.map((c) => c[0]) + for (const id of ids) expect(deleted).toContain(id) + }) + + it('a stale index completion does not corrupt a re-added same-named file', async () => { + buildMode = 'manual' + try { + const A = file('a.txt', 'AAA\n') + const B = file('a.txt', 'BBB\nBBB\nBBB\n') + await store.addFiles([A]) // row 'a.txt' (file A) → buildLineIndex(A) pending + store.removeFile('a.txt') + await store.addFiles([B]) // new row 'a.txt' (file B) → buildLineIndex(B) pending + + // The old (stale) index for A resolves last — it must NOT touch the row now holding B. + buildDeferreds.find((d) => d.file === A)!.resolve({ lineIndex: [0], lineCount: 99 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('indexing') + expect(store.get('a.txt')?.lineCount).not.toBe(99) + + // B's own index applies normally. + buildDeferreds.find((d) => d.file === B)!.resolve({ lineIndex: [0, 4, 8], lineCount: 3 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('ready') + expect(store.get('a.txt')?.lineCount).toBe(3) + } finally { + buildMode = 'real' + buildDeferreds.length = 0 + } + }) + + it('removeFolder deletes the live folder record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.addFolder(dir) + await settle(s) + const sourceId = s.list().find((f) => f.folder === 'proj')?.sourceId + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + expect((deleteItem as ReturnType).mock.calls.map((c) => c[0])).toContain(sourceId) + }) + + it('an emptied live folder stays visible and refreshes when files return', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Folder emptied on disk → the last child is removed but the folder persists (placeholder). + enumerateDirMock.mockResolvedValue([]) + await store.refreshFolders() + await settle(store) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(store.get('proj/app.ts')).toBeUndefined() + expect(store.readyFiles()).toEqual([]) // placeholder is never a tool target + + // A file added back on disk is picked up — the live source survived the empty state. + enumerateDirMock.mockResolvedValue([{ file: file('new.ts', 'y\n'), path: 'proj/new.ts' }]) + await store.refreshFolders() + await settle(store) + expect(store.folders[0].files.map((f) => f.relPath)).toEqual(['proj/new.ts']) + }) + + it('an empty-folder placeholder does not collide with a same-named standalone file', async () => { + enumerateDirMock.mockResolvedValue([]) // empty folder "proj" → creates a placeholder named "proj" + await store.addFolder(dir) + await settle(store) + + // A standalone file literally named "proj" must NOT be deduped by the placeholder. + const res = await store.addFiles([file('proj', 'hello\n')]) + await settle(store) + expect(res.added).toEqual(['proj']) + expect(store.standalone.map((f) => f.name)).toEqual(['proj']) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Removing that standalone leaves the folder's placeholder intact. + store.removeFile('proj') + expect(store.standalone).toEqual([]) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + }) + + it('links an initially empty live folder (kept visible, persisted, refreshes)', async () => { + const { putItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([]) // folder is empty at link time + const res = await s.addFolder(dir) + await settle(s) + expect(res.added).toEqual([]) + expect(s.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + // persisted as a dir-handle so it survives a reload despite being empty + const persisted = (putItem as ReturnType).mock.calls.map((c) => c[0]) + expect(persisted.some((r) => r.kind === 'dir-handle' && r.folder === 'proj')).toBe(true) + + // a file added later is picked up — the source existed from the start. + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.refreshFolders() + await settle(s) + expect(s.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts new file mode 100644 index 0000000000..9673e29edc --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from 'vitest' +import { + getItemsForSession, + putItem, + deleteItem, + deleteItemsForSession, + ensurePersistentStorage +} from './attachedFilesDB' + +// IndexedDB is unavailable in the node test env. The module must degrade gracefully +// (open fails → reads return [], writes/deletes are no-ops) rather than throwing. +describe('attachedFilesDB without IndexedDB', () => { + it('returns [] for reads', async () => { + expect(await getItemsForSession('s1')).toEqual([]) + }) + + it('does not throw on writes/deletes', async () => { + await expect( + putItem({ id: 'a', sessionId: 's1', kind: 'snapshot', name: 'x.txt', addedAt: 0 }) + ).resolves.toBeUndefined() + await expect(deleteItem('a')).resolves.toBeUndefined() + await expect(deleteItemsForSession('s1')).resolves.toBeUndefined() + }) + + it('does not throw when requesting persistent storage', async () => { + await expect(ensurePersistentStorage()).resolves.toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts new file mode 100644 index 0000000000..36f9ac4fbe --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts @@ -0,0 +1,118 @@ +/** + * IndexedDB persistence for AI-chat linked files, keyed by session id. + * + * Two kinds of records survive a reload (see the persistence plan): + * - handle records ('file-handle' / 'dir-handle'): a re-grantable File System + * Access handle is stored (structured-clone), re-read live on restore. + * - 'snapshot' records: a full-byte Blob copy (fallback when the File System + * Access API is unavailable). + * + * Mirrors the `idb` usage in HistoryManager.svelte.ts. + */ +import { openDB, type DBSchema as IDBSchema, type IDBPDatabase } from 'idb' + +export type AttachedItemKind = 'snapshot' | 'dir-handle' + +export interface PersistedAttachedItem { + /** Stable record id. */ + id: string + sessionId: string + /** 'snapshot' = a file copied into IndexedDB; 'dir-handle' = a live folder handle. */ + kind: AttachedItemKind + /** Display name: relative path for files, folder name for dir-handle records. */ + name: string + /** Top-level folder (for grouping); equals `name` for dir-handle records. */ + folder?: string + /** Folder-relative path (snapshot folder children) — restores the folder grouping/tree. */ + relPath?: string + /** Live directory handle (for 'dir-handle'). */ + handle?: FileSystemDirectoryHandle + /** Full-content copy (for 'snapshot'). */ + blob?: Blob + size?: number + lastModified?: number + addedAt: number +} + +interface AttachedFilesSchema extends IDBSchema { + items: { + key: string + value: PersistedAttachedItem + indexes: { 'by-session': string } + } +} + +let dbPromise: Promise | undefined> | undefined + +function getDB(): Promise | undefined> { + if (!dbPromise) { + try { + dbPromise = openDB('copilot-attached-files', 1, { + upgrade(db) { + if (!db.objectStoreNames.contains('items')) { + const store = db.createObjectStore('items', { keyPath: 'id' }) + store.createIndex('by-session', 'sessionId') + } + } + }).catch((err) => { + console.error('Could not open attached-files database', err) + return undefined + }) + } catch (err) { + // IndexedDB unavailable (e.g. private mode / no DOM) — degrade gracefully. + console.error('Could not open attached-files database', err) + dbPromise = Promise.resolve(undefined) + } + } + return dbPromise +} + +export async function putItem(item: PersistedAttachedItem): Promise { + const db = await getDB() + await db?.put('items', item) +} + +export async function getItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return [] + try { + return await db.getAllFromIndex('items', 'by-session', sessionId) + } catch (err) { + console.error('Could not read attached files', err) + return [] + } +} + +export async function deleteItem(id: string): Promise { + const db = await getDB() + await db?.delete('items', id) +} + +export async function deleteItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return + try { + const tx = db.transaction('items', 'readwrite') + const index = tx.store.index('by-session') + let cursor = await index.openCursor(sessionId) + while (cursor) { + await cursor.delete() + cursor = await cursor.continue() + } + await tx.done + } catch (err) { + console.error('Could not delete attached files for session', err) + } +} + +/** Ask the browser to keep our storage from being evicted (best-effort, once). */ +let persistRequested = false +export async function ensurePersistentStorage(): Promise { + if (persistRequested) return + persistRequested = true + try { + await navigator.storage?.persist?.() + } catch { + // best-effort; ignore + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts new file mode 100644 index 0000000000..8ef6d5b827 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts @@ -0,0 +1,286 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + buildLineIndex, + readFile, + searchFiles, + searchFilesInWorker, + isTextFile, + numberLines, + type FileEntry +} from './fileEngine' + +function makeFile(content: string | Uint8Array, name = 'f.txt'): File { + return new File([content as BlobPart], name) +} + +async function makeEntry(content: string, name = 'f.txt'): Promise { + const file = makeFile(content, name) + const { lineIndex, lineCount } = await buildLineIndex(file) + return { name, file, lineIndex, lineCount } +} + +describe('buildLineIndex', () => { + it('counts lines without a trailing newline', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('does not count a single trailing newline as an extra line', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb\n')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('handles an empty file', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('')) + expect(lineCount).toBe(0) + expect(lineIndex).toEqual([]) + }) + + it('handles CRLF line endings (offsets by byte)', async () => { + // bytes: a=0 \r=1 \n=2 b=3 → line starts at 0 and 3 + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\r\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 3]) + }) + + it('counts lines correctly across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => `line ${i}`) + const { lineCount } = await buildLineIndex(makeFile(lines.join('\n'))) + expect(lineCount).toBe(5000) + }) +}) + +describe('readFile', () => { + it('reads a bounded window and reports pagination', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 2 }) + expect(res.text).toBe('l1\nl2\n') + expect(res.startLine).toBe(1) + expect(res.endLine).toBe(2) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=3') + }) + + it('reads the final line to end of file', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 3 }) + expect(res.text).toBe('l3') + expect(res.truncated).toBe(false) + }) + + it('clamps the window to maxLines', async () => { + const entry = await makeEntry(Array.from({ length: 100 }, (_, i) => `l${i}`).join('\n')) + const res = await readFile(entry, { startLine: 1, maxLines: 10 }) + expect(res.endLine).toBe(10) + expect(res.truncated).toBe(true) + }) + + it('char-caps a degenerate single long line', async () => { + const entry = await makeEntry('x'.repeat(10000)) + const res = await readFile(entry, { maxChars: 8000 }) + expect(res.text.length).toBe(8000) + expect(res.truncated).toBe(true) + expect(res.note).toContain('8000 characters') + }) + + it('bounds the byte decode on a newline-sparse window (never reads past maxChars*4 bytes)', async () => { + // One 200k-char line; with maxChars=1000 only ≤4000 bytes are sliced before decode. + const entry = await makeEntry('y'.repeat(200_000)) + const res = await readFile(entry, { maxChars: 1000 }) + expect(res.text).toBe('y'.repeat(1000)) + expect(res.truncated).toBe(true) + }) + + it('clamps a start line beyond the end', async () => { + const entry = await makeEntry('l1\nl2') + const res = await readFile(entry, { startLine: 99 }) + expect(res.startLine).toBe(2) + expect(res.text).toBe('l2') + }) + + it('returns empty for an empty file', async () => { + const entry = await makeEntry('') + const res = await readFile(entry) + expect(res.text).toBe('') + expect(res.totalLines).toBe(0) + }) + + it('char-truncation inside the first line resumes the note at the next line', async () => { + // Line 1 exceeds maxChars; lines 2-3 follow. The window only returns line 1's prefix, + // so the note must report line 1 and point the model at line 2 (not claim lines 1-3). + const entry = await makeEntry('x'.repeat(20000) + '\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 3, maxChars: 5000 }) + expect(res.endLine).toBe(1) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=2') + }) + + it('char-truncation after some whole lines resumes at the truncated line', async () => { + const entry = await makeEntry('a\nb\n' + 'x'.repeat(20000) + '\nd') + const res = await readFile(entry, { startLine: 1, endLine: 4, maxChars: 5000 }) + expect(res.endLine).toBe(2) // a, b whole; line 3 (xxx) cut + expect(res.note).toContain('start_line=3') + // the partial line 3 must NOT leak into the body — it would contradict the note + expect(res.text).toBe('a\nb\n') + expect(numberLines(res.text, res.startLine)).toBe('1→a\n2→b') + }) +}) + +describe('searchFiles', () => { + it('finds matches with 1-based line numbers', async () => { + const entry = await makeEntry('alpha\nbeta\ngamma beta') + const res = await searchFiles([entry], 'beta') + expect(res.error).toBeUndefined() + expect(res.hits).toEqual([ + { file: 'f.txt', line: 2, text: 'beta' }, + { file: 'f.txt', line: 3, text: 'gamma beta' } + ]) + }) + + it('searches across multiple files', async () => { + const a = await makeEntry('needle here', 'a.txt') + const b = await makeEntry('nope\nneedle', 'b.txt') + const res = await searchFiles([a, b], 'needle') + expect(res.hits.map((h) => `${h.file}:${h.line}`)).toEqual(['a.txt:1', 'b.txt:2']) + }) + + it('restricts to a single file with pathFilter', async () => { + const a = await makeEntry('needle', 'a.txt') + const b = await makeEntry('needle', 'b.txt') + const res = await searchFiles([a, b], 'needle', { pathFilter: 'b.txt' }) + expect(res.hits).toEqual([{ file: 'b.txt', line: 1, text: 'needle' }]) + }) + + it('reports an unknown pathFilter as an error', async () => { + const a = await makeEntry('needle', 'a.txt') + const res = await searchFiles([a], 'needle', { pathFilter: 'missing.txt' }) + expect(res.error).toContain('missing.txt') + }) + + it('truncates at maxHits', async () => { + const entry = await makeEntry(Array.from({ length: 10 }, () => 'match').join('\n')) + const res = await searchFiles([entry], 'match', { maxHits: 3 }) + expect(res.hits.length).toBe(3) + expect(res.truncated).toBe(true) + }) + + it('supports case-insensitive flags', async () => { + const entry = await makeEntry('Hello\nworld') + const res = await searchFiles([entry], 'hello', { flags: 'i' }) + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'Hello' }]) + }) + + it('strips trailing CR from matched CRLF lines', async () => { + const entry = await makeEntry('foo\r\nbar') + const res = await searchFiles([entry], 'foo') + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'foo' }]) + }) + + it('returns a friendly error for an invalid regex', async () => { + const entry = await makeEntry('anything') + const res = await searchFiles([entry], '(') + expect(res.error).toContain('Invalid regex') + expect(res.hits).toEqual([]) + }) + + it('matches across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => (i === 4999 ? 'TARGET' : `line ${i}`)) + const entry = await makeEntry(lines.join('\n')) + const res = await searchFiles([entry], 'TARGET') + expect(res.hits).toEqual([{ file: 'f.txt', line: 5000, text: 'TARGET' }]) + }) + + it('a global flag does not drop matches via a stale lastIndex', async () => { + // `.test()` is stateful under the `g` flag; without resetting lastIndex, lines after + // the first match would be tested from a stale offset and silently miss. + const entry = await makeEntry('match\nmatch\nmatch') + const res = await searchFiles([entry], 'match', { flags: 'g' }) + expect(res.hits.map((h) => h.line)).toEqual([1, 2, 3]) + }) +}) + +describe('searchFilesInWorker', () => { + // A controllable stand-in for the search Worker — `reply`, `error`, or `hang` (never responds). + class MockWorker { + onmessage: ((e: MessageEvent) => void) | null = null + onerror: ((e: unknown) => void) | null = null + static mode: 'reply' | 'error' | 'hang' = 'reply' + static reply: unknown = { hits: [], truncated: false } + static terminated = false + constructor(_url: URL | string, _opts?: unknown) {} + postMessage(): void { + if (MockWorker.mode === 'reply') + queueMicrotask(() => this.onmessage?.({ data: MockWorker.reply } as MessageEvent)) + else if (MockWorker.mode === 'error') queueMicrotask(() => this.onerror?.({})) + // 'hang' → never responds, exercising the timeout path. + } + terminate(): void { + MockWorker.terminated = true + } + } + + beforeEach(() => { + MockWorker.terminated = false + vi.stubGlobal('Worker', MockWorker) + }) + afterEach(() => vi.unstubAllGlobals()) + + const entry: FileEntry = { name: 'a.txt', file: makeFile('x'), lineIndex: [], lineCount: 0 } + + it('resolves with the worker result and terminates the worker', async () => { + MockWorker.mode = 'reply' + MockWorker.reply = { hits: [{ file: 'a.txt', line: 1, text: 'x' }], truncated: false } + const res = await searchFilesInWorker([entry], 'x') + expect(res.hits).toEqual([{ file: 'a.txt', line: 1, text: 'x' }]) + expect(MockWorker.terminated).toBe(true) + }) + + it('times out on a non-responding (pathological) pattern and terminates the worker', async () => { + MockWorker.mode = 'hang' + const res = await searchFilesInWorker([entry], '^(a+)+$', {}, 20) + expect(res.error).toContain('timed out') + expect(MockWorker.terminated).toBe(true) + }) +}) + +describe('isTextFile', () => { + it('accepts UTF-8 text', async () => { + expect(await isTextFile(makeFile('hello © world'))).toBe(true) + }) + + it('accepts an empty file', async () => { + expect(await isTextFile(makeFile(''))).toBe(true) + }) + + it('rejects content with NUL bytes', async () => { + expect(await isTextFile(makeFile(new Uint8Array([104, 0, 105]), 'b.bin'))).toBe(false) + }) +}) + +describe('numberLines', () => { + it('prefixes each line with its absolute 1-based number', () => { + expect(numberLines('l3\nl4\n', 3)).toBe('3→l3\n4→l4') + }) + + it('right-aligns numbers to a common width', () => { + expect(numberLines('a\nb', 9)).toBe(' 9→a\n10→b') + }) + + it('numbers a final line that has no trailing newline', () => { + expect(numberLines('only', 5)).toBe('5→only') + }) + + it('matches a readFile window (numbers the returned lines, no phantom line)', async () => { + const file = makeFile('l1\nl2\nl3') + const { lineIndex, lineCount } = await buildLineIndex(file) + const res = await readFile( + { name: 'f.txt', file, lineIndex, lineCount }, + { startLine: 1, endLine: 2 } + ) + expect(numberLines(res.text, res.startLine)).toBe('1→l1\n2→l2') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts new file mode 100644 index 0000000000..5dbceb77e2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts @@ -0,0 +1,430 @@ +/** + * Storage-agnostic streaming engine for reading and searching attached files. + * + * Files are kept as `File` handles (lazy references to bytes on disk). Nothing is + * decoded into the JS heap wholesale: we stream in chunks, so a large file never + * freezes the tab or blows up memory. The only per-file state held in RAM is a + * line-offset index (a flat array of byte offsets, ~8 bytes per line). + * + * Line semantics match `String.split('\n')` except a single trailing newline does + * NOT add an empty final line (so "a\nb\n" is 2 lines, like `wc -l`). Lines are + * 1-based in the public read/search API. + */ + +/** Minimal shape the engine needs. The attached-files store extends this with reactive status. */ +export interface FileEntry { + name: string + /** A File (live link) or a Blob (restored snapshot) — both stream/slice identically. */ + file: File | Blob + lineIndex: number[] + lineCount: number +} + +const CHUNK_NEWLINE = 0x0a // '\n' — in UTF-8 this byte never appears inside a multibyte sequence + +export const DEFAULT_READ_MAX_LINES = 200 +export const DEFAULT_READ_MAX_CHARS = 8000 +export const DEFAULT_SEARCH_MAX_HITS = 50 +/** + * Per-line cap on how much of a degenerate long line the regex is tested against. + * This bounds work for linear-time patterns; it does NOT prevent catastrophic + * backtracking — a nested-quantifier pattern can still go exponential within the + * capped prefix (search runs on the main thread, so that is a self-DoS of the tab). + */ +export const DEFAULT_SEARCH_LINE_SCAN_CAP = 100_000 +/** How much of a matching line we echo back, to keep search results bounded. */ +export const DEFAULT_SEARCH_LINE_ECHO_CAP = 500 + +/** + * Stream the file once and record the byte offset at which each line starts. + * Scans raw bytes for '\n' (no decode needed — 0x0A is unambiguous in UTF-8). + */ +export async function buildLineIndex( + file: Blob +): Promise<{ lineIndex: number[]; lineCount: number }> { + const fileSize = file.size + if (fileSize === 0) { + return { lineIndex: [], lineCount: 0 } + } + + const lineIndex: number[] = [0] + let offset = 0 + const reader = file.stream().getReader() + try { + while (true) { + const { done, value } = await reader.read() + if (done) break + const chunk = value as Uint8Array + for (let i = 0; i < chunk.length; i++) { + if (chunk[i] === CHUNK_NEWLINE) { + lineIndex.push(offset + i + 1) + } + } + offset += chunk.length + } + } finally { + reader.releaseLock() + } + + // A trailing newline points one past the end (a phantom empty line) — drop it. + if (lineIndex.length > 1 && lineIndex[lineIndex.length - 1] === fileSize) { + lineIndex.pop() + } + + return { lineIndex, lineCount: lineIndex.length } +} + +export interface ReadResult { + text: string + startLine: number + endLine: number + totalLines: number + truncated: boolean + note: string +} + +/** + * Read a bounded window of lines, reading only the relevant byte range from disk. + * Clamps the window to `maxLines` and the returned text to `maxChars` (protects + * against degenerate single-line files). Returns a self-describing pagination note. + */ +export async function readFile( + entry: FileEntry, + opts: { + startLine?: number + endLine?: number + maxLines?: number + maxChars?: number + } = {} +): Promise { + const totalLines = entry.lineCount + const maxLines = opts.maxLines ?? DEFAULT_READ_MAX_LINES + const maxChars = opts.maxChars ?? DEFAULT_READ_MAX_CHARS + + if (totalLines === 0) { + return { + text: '', + startLine: 0, + endLine: 0, + totalLines: 0, + truncated: false, + note: 'File is empty.' + } + } + + let start = opts.startLine ?? 1 + if (start < 1) start = 1 + if (start > totalLines) start = totalLines + + const requestedEnd = opts.endLine ?? start + maxLines - 1 + let end = requestedEnd + if (end < start) end = start + const cappedByLines = end - start + 1 > maxLines + if (cappedByLines) end = start + maxLines - 1 + if (end > totalLines) end = totalLines + + const byteStart = entry.lineIndex[start - 1] + const byteEnd = end < totalLines ? entry.lineIndex[end] : entry.file.size + // Bound the decode for newline-sparse files (minified JS, single-line JSONL): the + // window can span the whole file, but we only ever return maxChars characters, and + // a UTF-8 character is at most 4 bytes — so never materialize more than that. + const byteCap = byteStart + maxChars * 4 + const byteCapped = byteCap < byteEnd + + let text: string + try { + text = await entry.file.slice(byteStart, byteCapped ? byteCap : byteEnd).text() + } catch (e) { + throw new FileReadError(entry.name, e instanceof Error ? e.message : String(e)) + } + + let cappedByChars = byteCapped + if (text.length > maxChars) { + text = text.slice(0, maxChars) + cappedByChars = true + } + + // When the char cap truncates the window short of `end`, the text holds fewer lines + // than requested — so the note must report the last line actually returned and resume + // at the next unread one (otherwise it claims lines it didn't return and skips them). + let lastLine = end + let resumeAt: number | undefined = end < totalLines ? end + 1 : undefined + if (cappedByChars) { + const completeLines = (text.match(/\n/g) || []).length + if (completeLines >= 1) { + // lines start..start+completeLines-1 are whole; the next line was cut mid-content. + // Trim that partial line off the returned text so the body matches the note (and + // the model doesn't see a line the note says it'll get on the next read). + lastLine = start + completeLines - 1 + resumeAt = start + completeLines + text = text.slice(0, text.lastIndexOf('\n') + 1) + } else { + // the cap fell inside line `start` itself — it can't be returned in full, so + // advance past it rather than re-truncating the same line forever. + lastLine = start + resumeAt = start + 1 + } + if (resumeAt > totalLines) resumeAt = undefined + } + + const truncated = cappedByChars || resumeAt !== undefined + + let note = `Showing lines ${start}-${lastLine} of ${totalLines}.` + if (cappedByChars) { + note += ` Output truncated to ${maxChars} characters (line(s) very long).` + } + if (resumeAt !== undefined) { + note += ` Call read_file again with start_line=${resumeAt} for more.` + } + + return { text, startLine: start, endLine: lastLine, totalLines, truncated, note } +} + +/** + * Prefix each line of a read window with its absolute 1-based number (``), + * so the model can quote/reference exact lines. `startLine` is the window's first line. + */ +export function numberLines(text: string, startLine: number): string { + const lines = text.split('\n') + // readFile's window ends with the trailing newline of its last line when more lines + // follow, so split yields a phantom empty element — drop it before numbering. + if (lines.length > 1 && lines[lines.length - 1] === '') lines.pop() + const width = String(startLine + lines.length - 1).length + return lines.map((l, i) => `${String(startLine + i).padStart(width)}→${l}`).join('\n') +} + +export interface SearchHit { + file: string + line: number + text: string +} + +export interface SearchResult { + hits: SearchHit[] + truncated: boolean + error?: string +} + +/** + * Run a regex across one or more files, streaming each (no full-file load), and + * return matching lines with 1-based line numbers. Stops at `maxHits`. + */ +export async function searchFiles( + entries: FileEntry[], + pattern: string, + opts: { + flags?: string + pathFilter?: string + maxHits?: number + lineScanCap?: number + lineEchoCap?: number + } = {} +): Promise { + const maxHits = opts.maxHits ?? DEFAULT_SEARCH_MAX_HITS + const lineScanCap = opts.lineScanCap ?? DEFAULT_SEARCH_LINE_SCAN_CAP + const lineEchoCap = opts.lineEchoCap ?? DEFAULT_SEARCH_LINE_ECHO_CAP + + let regex: RegExp + try { + regex = new RegExp(pattern, opts.flags ?? '') + } catch (e) { + return { + hits: [], + truncated: false, + error: `Invalid regex: ${e instanceof Error ? e.message : String(e)}` + } + } + + const targets = opts.pathFilter ? entries.filter((e) => e.name === opts.pathFilter) : entries + if (opts.pathFilter && targets.length === 0) { + return { hits: [], truncated: false, error: `No attached file named "${opts.pathFilter}".` } + } + + const hits: SearchHit[] = [] + let truncated = false + + for (const entry of targets) { + if (hits.length >= maxHits) { + truncated = true + break + } + try { + await streamLines(entry.file, (line, lineNo) => { + // Bound backtracking on pathological long lines by only testing a prefix. + const scanned = line.length > lineScanCap ? line.slice(0, lineScanCap) : line + // `regex` may carry a caller-supplied `g`/`y` flag, which makes `.test()` + // stateful (it advances `lastIndex`) — reset so each line matches from 0. + regex.lastIndex = 0 + if (regex.test(scanned)) { + hits.push({ + file: entry.name, + line: lineNo, + text: line.length > lineEchoCap ? line.slice(0, lineEchoCap) + '…' : line + }) + } + return hits.length < maxHits // continue? + }) + } catch (e) { + return { + hits, + truncated, + error: `Error reading "${entry.name}": ${e instanceof Error ? e.message : String(e)}` + } + } + if (hits.length >= maxHits) { + truncated = true + break + } + } + + return { hits, truncated } +} + +/** + * Run `searchFiles` off the main thread. A model-supplied regex can backtrack + * catastrophically (e.g. /^(a+)+$/) and `regex.test()` can't be interrupted — so we run + * it in a Worker and `terminate()` it on a timeout, keeping the tab responsive instead of + * frozen. Falls back to a main-thread search where Workers aren't available (best effort). + */ +export function searchFilesInWorker( + entries: FileEntry[], + pattern: string, + opts: { flags?: string; pathFilter?: string; maxHits?: number } = {}, + timeoutMs = 3000 +): Promise { + let worker: Worker + try { + worker = new Worker(new URL('./searchWorker.ts', import.meta.url), { type: 'module' }) + } catch { + return searchFiles(entries, pattern, opts) + } + return new Promise((resolve) => { + const finish = (r: SearchResult) => { + clearTimeout(timer) + worker.terminate() + resolve(r) + } + const timer = setTimeout( + () => + finish({ + hits: [], + truncated: false, + error: 'Search timed out — the pattern is too expensive. Try a simpler regex.' + }), + timeoutMs + ) + worker.onmessage = (e: MessageEvent) => finish(e.data) + worker.onerror = () => { + // Worker script failed to load/run — fall back to a main-thread search. + clearTimeout(timer) + worker.terminate() + searchFiles(entries, pattern, opts).then(resolve) + } + worker.postMessage({ + files: entries.map((e) => ({ name: e.name, file: e.file })), + pattern, + flags: opts.flags, + pathFilter: opts.pathFilter, + maxHits: opts.maxHits + }) + }) +} + +export class FileReadError extends Error { + constructor( + public fileName: string, + message: string + ) { + super(message) + this.name = 'FileReadError' + } +} + +/** + * Max characters buffered for a single line while streaming. A newline-less file + * (e.g. minified JS) would otherwise accumulate wholesale in `buffer`; past this + * cap excess characters are dropped (the line's intact prefix is preserved) — + * harmless for search, which only tests/echoes a prefix far smaller than this. + */ +const MAX_LINE_BUFFER_CHARS = 1_000_000 + +/** + * Stream a file and invoke `onLine` for each line (1-based). A trailing newline + * does not produce an empty final line. `onLine` returns false to stop early. + * A trailing '\r' (CRLF) is stripped before the callback. Overlong lines are + * passed with at least their first MAX_LINE_BUFFER_CHARS characters intact; + * content beyond the cap may be dropped. + */ +async function streamLines( + file: Blob, + onLine: (line: string, lineNo: number) => boolean +): Promise { + const reader = file.stream().getReader() + const decoder = new TextDecoder('utf-8') + let buffer = '' + let lineNo = 0 + try { + while (true) { + const { done, value } = await reader.read() + if (done) { + buffer += decoder.decode() + break + } + buffer += decoder.decode(value as Uint8Array, { stream: true }) + let start = 0 + let nlIdx: number + while ((nlIdx = buffer.indexOf('\n', start)) !== -1) { + let line = buffer.slice(start, nlIdx) + if (line.endsWith('\r')) line = line.slice(0, -1) + start = nlIdx + 1 + lineNo++ + if (!onLine(line, lineNo)) return + } + buffer = buffer.slice(start) + // The remainder holds no newline — cap how much of an overlong line we keep. + // Dropped characters are line content only, so newline detection and line + // numbering in later chunks are unaffected. + if (buffer.length > MAX_LINE_BUFFER_CHARS) { + buffer = buffer.slice(0, MAX_LINE_BUFFER_CHARS) + } + } + if (buffer.length > 0) { + let line = buffer + if (line.endsWith('\r')) line = line.slice(0, -1) + lineNo++ + onLine(line, lineNo) + } + } finally { + reader.releaseLock() + } +} + +/** + * Sniff the first bytes of a file to decide whether it is text (UTF-8 decodable, + * no NUL bytes). Used to reject binary files at attach time. + */ +export async function isTextFile(file: Blob, sampleBytes = 8192): Promise { + if (file.size === 0) return true + const slice = file.slice(0, Math.min(sampleBytes, file.size)) + const buf = new Uint8Array(await slice.arrayBuffer()) + for (let i = 0; i < buf.length; i++) { + if (buf[i] === 0) return false // NUL byte → binary + } + try { + // `fatal` throws on invalid UTF-8. We may cut a multibyte char at the sample + // boundary, so only treat it as binary if the error is not at the very end. + new TextDecoder('utf-8', { fatal: true }).decode(buf) + return true + } catch { + // Could be a truncated trailing multibyte sequence — retry on a trimmed buffer. + if (buf.length >= 4) { + try { + new TextDecoder('utf-8', { fatal: true }).decode(buf.slice(0, buf.length - 3)) + return true + } catch { + return false + } + } + return false + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts new file mode 100644 index 0000000000..0bd45153af --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts @@ -0,0 +1,66 @@ +import { describe, expect, it, vi } from 'vitest' + +// '../shared' transitively imports monaco (CSS) which the node test env can't load. +// fileTools only needs createToolDef from it (called at module load), so stub it. +vi.mock('../shared', () => ({ + createToolDef: (_schema: unknown, name: string, description: string) => ({ name, description }) +})) + +import { searchFilesTool } from './fileTools' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Minimal store stub: searchFilesTool's empty-ready path only reads count/readyFiles/list. */ +function fakeStore(rows: Array>): AttachedFilesStore { + const files = rows as AttachedFile[] + return { + get count() { + return files.length + }, + readyFiles: () => files.filter((f) => f.status === 'ready' && !f.isFolderRoot), + list: () => files + } as unknown as AttachedFilesStore +} + +async function runSearch(store: AttachedFilesStore): Promise { + const res = await searchFilesTool.fn({ + args: { pattern: 'x' }, + helpers: { attachedFiles: store }, + toolId: 't', + toolCallbacks: { setToolStatus: () => {} } + } as any) + return res as string +} + +describe('search_files — attachments present but nothing readable', () => { + it('reports no searchable text for an empty/binary-only linked folder (only ready placeholders)', async () => { + // An empty/all-binary folder leaves a single `ready` placeholder row, filtered out of readyFiles(). + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'ready', isFolderRoot: true }])) + expect(msg).toMatch(/no searchable text files/i) + expect(msg).not.toMatch(/indexed/i) + }) + + it('tells the user to restore access when a restored folder is locked', async () => { + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'locked', isFolderRoot: true }])) + expect(msg).toMatch(/restore access/i) + }) + + it('tells the user to re-link when files are unavailable', async () => { + const msg = await runSearch(fakeStore([{ name: 'gone.txt', status: 'unavailable' }])) + expect(msg).toMatch(/re-link/i) + }) + + it('still reports indexing while a file is genuinely indexing', async () => { + const msg = await runSearch(fakeStore([{ name: 'a.txt', status: 'indexing' }])) + expect(msg).toMatch(/still being indexed/i) + }) + + it('prefers the indexing message when an indexing file coexists with an empty folder', async () => { + const msg = await runSearch( + fakeStore([ + { name: 'proj', status: 'ready', isFolderRoot: true }, + { name: 'a.txt', status: 'indexing' } + ]) + ) + expect(msg).toMatch(/still being indexed/i) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.ts new file mode 100644 index 0000000000..982b858154 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.ts @@ -0,0 +1,226 @@ +/** + * AI tools and system-prompt roster for files attached to the GLOBAL chat. + * + * The model is made aware of attached files via a metadata-only roster appended to + * the system message (see `appendAttachedFilesRoster`). Their contents are NEVER + * inlined — the model pulls only the slices it needs through these two read-only + * tools, which stream from disk via ./fileEngine. + */ +import { z } from 'zod' +import type { ChatCompletionSystemMessageParam } from 'openai/resources/chat/completions.mjs' +import { createToolDef, type Tool } from '../shared' +import { + readFile, + searchFilesInWorker, + numberLines, + FileReadError, + type SearchHit +} from './fileEngine' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Slice of the GLOBAL tool helpers that exposes the attached-files store. */ +export interface AttachedFilesHelper { + attachedFiles?: AttachedFilesStore +} + +function storeFrom(helpers: unknown): AttachedFilesStore | undefined { + return (helpers as AttachedFilesHelper | undefined)?.attachedFiles +} + +/** + * For a specifically requested attached file, a message describing why it can't be read / + * searched yet (still indexing, locked, unavailable, errored) or that it isn't attached — + * or undefined when it's `ready`. Shared by read_file and search_files so both report the + * same accurate status instead of search_files claiming a non-ready file isn't attached. + */ +function notReadyMessage(store: AttachedFilesStore, file: string): string | undefined { + const entry = store.get(file) + if (entry?.status === 'ready') return undefined + if (entry?.status === 'indexing') + return `File "${file}" is still being indexed. Try again shortly.` + if (entry?.status === 'locked') + return `File "${file}" is locked after a reload. Ask the user to restore access (send a message, or click "Restore access").` + if (entry?.status === 'unavailable') + return `File "${file}" is no longer available (moved, deleted, or its local copy was evicted). Ask the user to re-link it.` + if (entry?.status === 'error') + return `File "${file}" failed to load: ${entry.error ?? 'unknown error'}.` + const names = store + .list() + .map((f) => f.name) + .join(', ') + return `No attached file named "${file}". Attached files: ${names || '(none)'}.` +} + +/** + * When attachments exist but none expose a readable target (`readyFiles()` is empty), + * explain the actual reason instead of always claiming files are still indexing. Empty + * or binary-only linked folders leave only `ready` placeholder rows (filtered out of + * `readyFiles`), while a locked/unavailable restore surfaces those statuses on the rows. + */ +function noReadyFilesMessage(store: AttachedFilesStore): string { + const statuses = new Set(store.list().map((f) => f.status)) + if (statuses.has('indexing')) return 'Attached files are still being indexed. Try again shortly.' + if (statuses.has('locked')) + return 'The attached files are locked after a reload. Ask the user to restore access (send a message, or click "Restore access").' + if (statuses.has('unavailable')) + return 'The attached files are no longer available (moved, deleted, or their local copies were evicted). Ask the user to re-link them.' + if (statuses.has('error')) return 'The attached files failed to load.' + return 'No searchable text files are attached (a linked folder may be empty or contain only non-text files).' +} + +function humanSize(bytes: number): string { + if (bytes < 1024) return `${bytes} B` + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB` + return `${(bytes / (1024 * 1024)).toFixed(1)} MB` +} + +const searchFilesSchema = z.object({ + pattern: z.string().describe('JavaScript regular expression to search for.'), + file: z + .string() + .optional() + .describe( + 'Optional exact filename (as listed under "Attached files") to restrict the search to. Omit to search across all attached files.' + ), + ignore_case: z.boolean().optional().describe('Case-insensitive matching. Defaults to false.') +}) + +const searchFilesToolDef = createToolDef( + searchFilesSchema, + 'search_files', + 'Search the user-attached files with a regular expression and return matching lines with their line numbers. Use this to locate content before reading a specific window with read_file.' +) + +export const searchFilesTool: Tool<{}> = { + def: searchFilesToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = searchFilesSchema.parse(args) + // Validate a specifically requested file against the full store first, so a non-ready + // target reports its real status (indexing/locked/…) instead of "not attached". + if (parsed.file) { + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + } + const ready = store.readyFiles() + if (ready.length === 0) { + return noReadyFilesMessage(store) + } + toolCallbacks.setToolStatus(toolId, { + content: `Searching attached files for /${parsed.pattern}/...` + }) + + // Run in a Worker so a pathological model-supplied regex can't freeze the tab. + const result = await searchFilesInWorker(ready, parsed.pattern, { + flags: parsed.ignore_case ? 'i' : '', + pathFilter: parsed.file + }) + if (result.error) { + return `Error: ${result.error}` + } + const scope = parsed.file ? `"${parsed.file}"` : `${ready.length} file(s)` + if (result.hits.length === 0) { + return `No matches for /${parsed.pattern}/ in ${scope}.` + } + const body = result.hits.map((h: SearchHit) => `${h.file}:${h.line}: ${h.text}`).join('\n') + const header = `Found ${result.hits.length} match(es) in ${scope}:` + const footer = result.truncated + ? '\n\n(Stopped at the result limit — refine your pattern or pass a `file` to narrow the search.)' + : '' + return `${header}\n${body}${footer}` + } +} + +const readFileSchema = z.object({ + file: z.string().describe('Exact filename to read, as listed under "Attached files".'), + start_line: z.number().int().optional().describe('1-based first line to read. Defaults to 1.'), + end_line: z + .number() + .int() + .optional() + .describe('1-based last line to read. The window is capped at 200 lines.') +}) + +const readFileToolDef = createToolDef( + readFileSchema, + 'read_file', + 'Read a bounded window of lines from a user-attached file. Returns each line prefixed with its 1-based number (``) plus a pagination note. Files are not in context, so use this to inspect their contents.' +) + +export const readFileTool: Tool<{}> = { + def: readFileToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = readFileSchema.parse(args) + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + const entry = store.get(parsed.file)! + toolCallbacks.setToolStatus(toolId, { content: `Reading "${parsed.file}"...` }) + + try { + const res = await readFile(entry, { + startLine: parsed.start_line, + endLine: parsed.end_line + }) + return res.text ? `${res.note}\n\n${numberLines(res.text, res.startLine)}` : res.note + } catch (e) { + if (e instanceof FileReadError) { + return `Could not read "${parsed.file}": ${e.message}. The file may have been moved or deleted since it was attached.` + } + return `Error reading "${parsed.file}": ${e instanceof Error ? e.message : String(e)}` + } + } +} + +export const fileTools: Tool<{}>[] = [searchFilesTool, readFileTool] + +function rosterLine(f: AttachedFile): string { + if (f.status === 'indexing') return `- ${f.name} (indexing…)` + if (f.status === 'locked') return `- ${f.name} (locked — needs the user to restore access)` + if (f.status === 'unavailable') return `- ${f.name} (unavailable)` + if (f.status === 'error') return `- ${f.name} (failed to load)` + return `- ${f.name} — ${f.lineCount} lines, ${humanSize(f.size)}` +} + +/** Build the `## Attached files` system-prompt section (metadata only, never content). */ +export function buildAttachedFilesRoster(store: AttachedFilesStore): string { + const lines: string[] = [] + for (const folder of store.folders) { + // A locked/unavailable folder has no readable children — one line for the whole folder. + if (folder.status === 'locked') { + lines.push(`- ${folder.name} (locked — needs the user to restore access)`) + } else if (folder.status === 'unavailable') { + lines.push(`- ${folder.name} (unavailable)`) + } else { + lines.push(...folder.files.map(rosterLine)) + } + } + lines.push(...store.standalone.map(rosterLine)) + if (lines.length === 0) return '' + return [ + '## Attached files', + 'The user has attached the following files to this conversation. Their contents are NOT included here.', + 'Use the `search_files` tool to find content with a regex, and `read_file` to read a bounded window of lines.', + '', + lines.join('\n') + ].join('\n') +} + +/** + * Return a copy of the system message with the attached-files roster appended. + * Always derives from the provided base so the roster never accumulates across turns. + */ +export function appendAttachedFilesRoster( + base: ChatCompletionSystemMessageParam, + store: AttachedFilesStore +): ChatCompletionSystemMessageParam { + const roster = buildAttachedFilesRoster(store) + if (!roster || typeof base.content !== 'string') return base + return { ...base, content: `${base.content}\n\n${roster}` } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts new file mode 100644 index 0000000000..e4536b6eac --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts @@ -0,0 +1,49 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { hasFileSystemAccess, isIgnoredPath, pickDirectory } from './fsAccess' + +describe('hasFileSystemAccess', () => { + it('is false when the File System Access API is absent (node / Firefox / Safari today)', () => { + // The test env exposes none of showOpenFilePicker / showDirectoryPicker / + // DataTransferItem.getAsFileSystemHandle, so the gate must report false. + // The positive path (all three present) is exercised via browser verification. + expect(hasFileSystemAccess()).toBe(false) + }) +}) + +describe('pickDirectory', () => { + afterEach(() => { + delete (window as { showDirectoryPicker?: unknown }).showDirectoryPicker + }) + + it('returns undefined when the user dismisses the picker (AbortError)', async () => { + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('aborted', 'AbortError') + } + await expect(pickDirectory()).resolves.toBeUndefined() + }) + + it('rethrows any non-abort failure instead of silently no-oping', async () => { + // e.g. a policy that blocks the File System Access API, or a lost user-activation. + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('blocked by policy', 'SecurityError') + } + await expect(pickDirectory()).rejects.toThrow(/blocked by policy/) + }) +}) + +describe('isIgnoredPath', () => { + it('keeps normal source paths', () => { + expect(isIgnoredPath('myproj/src/app.ts')).toBe(false) + expect(isIgnoredPath('README.md')).toBe(false) + }) + it('skips ignored directories', () => { + expect(isIgnoredPath('myproj/node_modules/lib/index.js')).toBe(true) + expect(isIgnoredPath('myproj/dist/bundle.js')).toBe(true) + expect(isIgnoredPath('a/target/x')).toBe(true) + }) + it('skips dotfiles and dotdirs', () => { + expect(isIgnoredPath('myproj/.env')).toBe(true) + expect(isIgnoredPath('myproj/.git/config')).toBe(true) + expect(isIgnoredPath('.DS_Store')).toBe(true) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts new file mode 100644 index 0000000000..2c741c0988 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts @@ -0,0 +1,192 @@ +/** + * Thin wrappers over the File System Access API, used when available so linked + * files/folders can be re-read live after a reload (re-grantable handles). + * Capability is feature-detected (never browser-sniffed): the day Firefox/Safari + * ship the API, the handle path lights up automatically. + */ +const IGNORED_DIRS = new Set([ + 'node_modules', + 'dist', + 'build', + 'out', + 'target', + 'vendor', + 'coverage', + '__pycache__', + '.git', + '.svelte-kit', + '.next', + '.nuxt', + '.venv', + 'venv', + '.idea', + '.vscode', + '.turbo', + '.cache' +]) + +function isIgnoredSegment(name: string): boolean { + return name.startsWith('.') || IGNORED_DIRS.has(name) +} + +/** True if any segment of a relative path is a dotfile/dotdir or an ignored directory. */ +export function isIgnoredPath(path: string): boolean { + return path.split('/').some(isIgnoredSegment) +} + +type FSWindow = Window & { + showDirectoryPicker?: (opts?: { + mode?: 'read' | 'readwrite' + }) => Promise +} + +type FSDataTransferItem = DataTransferItem & { + getAsFileSystemHandle?: () => Promise +} + +/** + * True when the File System Access API needed for FOLDER linking is usable: + * the directory picker plus drag-drop handles. (Files never use the API — they're + * always snapshotted — so showOpenFilePicker is intentionally not required.) + */ +export function hasFileSystemAccess(): boolean { + return ( + typeof window !== 'undefined' && + 'showDirectoryPicker' in window && + typeof DataTransferItem !== 'undefined' && + 'getAsFileSystemHandle' in DataTransferItem.prototype + ) +} + +/** + * Open the directory picker. Returns undefined if the user dismisses it. + * Any other failure (a policy that blocks the File System Access API, a lost + * user-activation, etc.) is rethrown — swallowing it makes the picker silently + * never open, which is indistinguishable from a no-op and impossible to debug. + */ +export async function pickDirectory(): Promise { + const w = window as FSWindow + if (!w.showDirectoryPicker) return undefined + try { + return await w.showDirectoryPicker({ mode: 'read' }) + } catch (e) { + // AbortError means the user dismissed the dialog (and, under browser automation, + // that CDP intercepted the chooser) — a no-op, not a failure. + if (e instanceof DOMException && e.name === 'AbortError') return undefined + throw e + } +} + +/** + * Resolve File System Access handles from a drop's items. The `getAsFileSystemHandle` + * calls are kicked off synchronously (items are only valid during the drop event); + * the returned promise resolves the handles. + */ +export function handlesFromDataTransfer(dt: DataTransfer): Promise { + const pending = Array.from(dt.items) + .filter((it) => it.kind === 'file') + .map((it) => (it as FSDataTransferItem).getAsFileSystemHandle?.() ?? Promise.resolve(null)) + return Promise.all(pending).then((handles) => handles.filter((h): h is FileSystemHandle => !!h)) +} + +export function isFileHandle(h: FileSystemHandle): h is FileSystemFileHandle { + return h.kind === 'file' +} +export function isDirectoryHandle(h: FileSystemHandle): h is FileSystemDirectoryHandle { + return h.kind === 'directory' +} + +/** + * Recursively read a directory handle into a flat list of files with relative paths, + * skipping junk (dotfiles/dotdirs, node_modules, …). No file-count cap — the browser's + * memory/quota are the only limit. Used on link and on live re-enumeration after a reload. + */ +export async function enumerateDir( + dir: FileSystemDirectoryHandle +): Promise<{ file: File; path: string }[]> { + const out: { file: File; path: string }[] = [] + + async function walk(handle: FileSystemDirectoryHandle, prefix: string): Promise { + // @ts-ignore - values() is an async iterator in the File System Access API + for await (const entry of handle.values() as AsyncIterable) { + const path = `${prefix}/${entry.name}` + if (isIgnoredPath(path)) continue + if (isFileHandle(entry)) { + out.push({ file: await entry.getFile(), path }) + } else if (isDirectoryHandle(entry)) { + await walk(entry, path) + } + } + } + + await walk(dir, dir.name) + return out +} + +/** + * Recursively read dropped files AND folders via the legacy `webkitGetAsEntry` API — + * the fallback for browsers without the File System Access API (Firefox/Safari). Folder + * contents are snapshotted into the browser (no live handle). Each result `path` is + * folder-relative (`folder/sub/file` for a dropped folder, bare name for a loose file), + * junk paths skipped, no file-count cap. + * + * `webkitGetAsEntry()` is only valid synchronously during the drop event, so this MUST be + * called from the drop handler — its `.map(...)` runs before the first `await`, capturing + * the entries while the items are still live. + */ +export async function readDroppedEntries( + items: DataTransferItem[] +): Promise<{ file: File; path: string }[]> { + const roots = items + .map((it) => it.webkitGetAsEntry?.() ?? null) + .filter((e): e is FileSystemEntry => !!e) + const out: { file: File; path: string }[] = [] + for (const root of roots) await walkDropEntry(root, out) + return out +} + +async function walkDropEntry( + entry: FileSystemEntry, + out: { file: File; path: string }[] +): Promise { + const path = entry.fullPath.replace(/^\//, '') + if (isIgnoredPath(path)) return + if (entry.isFile) { + const fileEntry = entry as FileSystemFileEntry + const file = await new Promise((res, rej) => fileEntry.file(res, rej)) + out.push({ file, path }) + } else if (entry.isDirectory) { + const reader = (entry as FileSystemDirectoryEntry).createReader() + // readEntries yields in batches and returns [] once exhausted — loop until empty. + while (true) { + const batch = await new Promise((res, rej) => reader.readEntries(res, rej)) + if (batch.length === 0) break + for (const child of batch) await walkDropEntry(child, out) + } + } +} + +/** queryPermission without a user gesture; 'granted' | 'prompt' | 'denied'. Never rejects. */ +export async function queryReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - queryPermission is part of the File System Access API + return (await handle.queryPermission?.({ mode: 'read' })) ?? 'prompt' + } catch { + return 'prompt' + } +} + +/** + * requestPermission — MUST be called within a user gesture. Never rejects: the spec + * rejects with SecurityError when user activation is missing (e.g. a second prompt + * after the first consumed the gesture) — that maps to 'denied' here so callers can + * treat it as "still locked" instead of blowing up the send path. + */ +export async function requestReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - requestPermission is part of the File System Access API + return (await handle.requestPermission?.({ mode: 'read' })) ?? 'denied' + } catch { + return 'denied' + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/searchWorker.ts b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts new file mode 100644 index 0000000000..68960801da --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts @@ -0,0 +1,38 @@ +/** + * Web Worker that runs `search_files` off the main thread. + * + * The regex is model-supplied and `RegExp.prototype.test()` can't be interrupted, so a + * catastrophic-backtracking pattern (e.g. /^(a+)+$/) would otherwise hang the whole tab. + * Running it here lets the caller `terminate()` this worker on a timeout instead. The + * matching itself reuses `searchFiles` from the engine (single source of truth). + */ +import { searchFiles, type FileEntry } from './fileEngine' + +interface SearchRequest { + files: { name: string; file: Blob }[] + pattern: string + flags?: string + pathFilter?: string + maxHits?: number +} + +self.onmessage = async (e: MessageEvent) => { + const { files, pattern, flags, pathFilter, maxHits } = e.data + // searchFiles only reads `name` + `file` (it streams); the index fields are unused here. + const entries: FileEntry[] = files.map((f) => ({ + name: f.name, + file: f.file, + lineIndex: [], + lineCount: 0 + })) + try { + const result = await searchFiles(entries, pattern, { flags, pathFilter, maxHits }) + ;(self as unknown as Worker).postMessage(result) + } catch (err) { + ;(self as unknown as Worker).postMessage({ + hits: [], + truncated: false, + error: err instanceof Error ? err.message : String(err) + }) + } +} diff --git a/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte b/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte index 37db9a90ba..b2496bbcda 100644 --- a/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte +++ b/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte @@ -135,7 +135,8 @@ await acceptPendingFlowEditsIfEnabled() }, getFlowInputsSchema: async () => { - return flowStore.val.schema ?? {} + const s = flowStore.val.schema ?? {} + return { type: 'object', properties: {}, required: [], ...s } }, updateExprsToSet: (id: string, inputTransforms: Record) => { diff --git a/frontend/src/lib/components/copilot/chat/flow/openFlow.json b/frontend/src/lib/components/copilot/chat/flow/openFlow.json index 0309aeaf91..c7ede4c0b5 100644 --- a/frontend/src/lib/components/copilot/chat/flow/openFlow.json +++ b/frontend/src/lib/components/copilot/chat/flow/openFlow.json @@ -1 +1 @@ -{"openapi":"3.0.3","info":{"version":"1.716.0","title":"OpenFlow Spec","contact":{"name":"Ruben Fiszel","email":"ruben@windmill.dev","url":"https://windmill.dev"},"license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0.html"}},"paths":{},"externalDocs":{"description":"documentation portal","url":"https://windmill.dev"},"components":{"schemas":{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution — modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind"}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type"}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw",null]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw",null]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}]},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}}} \ No newline at end of file +{"openapi":"3.0.3","info":{"version":"1.716.0","title":"OpenFlow Spec","contact":{"name":"Ruben Fiszel","email":"ruben@windmill.dev","url":"https://windmill.dev"},"license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0.html"}},"paths":{},"externalDocs":{"description":"documentation portal","url":"https://windmill.dev"},"components":{"schemas":{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution — modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind"}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type"}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw",null]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw",null]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}]},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}}} \ No newline at end of file diff --git a/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts b/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts index 21e6735ab1..3e0b8fcf69 100644 --- a/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts +++ b/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts @@ -1,6 +1,6 @@ import { z } from "zod" -export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { +export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { const schemas = [z.intersection(z.object({ "tool_type": z.literal("flowmodule") }), z.lazy(() => flowModuleValueSchema)).describe("A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module"), z.object({ "tool_type": z.literal("mcp"), "resource_path": z.string().describe("Path to the MCP resource/server configuration"), "include_tools": z.array(z.string()).describe("Whitelist of specific tools to include from this MCP server").optional(), "exclude_tools": z.array(z.string()).describe("Blacklist of tools to exclude from this MCP server").optional() }).describe("Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers"), z.object({ "tool_type": z.literal("websearch") }).describe("A tool implemented as a websearch tool. The AI can call this like any other websearch tool")]; const errors = schemas.reduce( (errors, schema) => @@ -20,7 +20,7 @@ export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "i }).describe("The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference") }).describe("A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool")).describe("Array of tools the agent can use. The agent decides which tools to call based on the task"), "type": z.literal("aiagent"), "omit_output_from_conversation": z.boolean().describe("If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled.").default(false), "parallel": z.boolean().describe("If true, the agent can execute multiple tool calls in parallel").optional() }).describe("AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task")]).describe("The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type") -export const flowModuleSchema = z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { +export const flowModuleSchema = z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { const schemas = [z.intersection(z.object({ "tool_type": z.literal("flowmodule") }), z.lazy(() => flowModuleValueSchema)).describe("A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module"), z.object({ "tool_type": z.literal("mcp"), "resource_path": z.string().describe("Path to the MCP resource/server configuration"), "include_tools": z.array(z.string()).describe("Whitelist of specific tools to include from this MCP server").optional(), "exclude_tools": z.array(z.string()).describe("Blacklist of tools to exclude from this MCP server").optional() }).describe("Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers"), z.object({ "tool_type": z.literal("websearch") }).describe("A tool implemented as a websearch tool. The AI can call this like any other websearch tool")]; const errors = schemas.reduce( (errors, schema) => diff --git a/frontend/src/lib/components/copilot/chat/flow/providerKind.test.ts b/frontend/src/lib/components/copilot/chat/flow/providerKind.test.ts new file mode 100644 index 0000000000..ab520762ec --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/flow/providerKind.test.ts @@ -0,0 +1,41 @@ +import { describe, expect, it } from 'vitest' +import { flowModulesSchema } from './openFlowZod.gen' + +// Guards against the generated copilot flow Zod schema (openFlowZod.gen.ts) +// drifting from the AIProviderKind enum in openflow.openapi.yaml. A missing +// provider kind here silently rejects AI-generated flow edits for that provider +// in the copilot flow-editing path (validateFlowModules -> flowModulesSchema). +function aiAgentModuleWithProviderKind(kind: string) { + return { + id: 'agent', + value: { + type: 'aiagent', + tools: [], + input_transforms: { + provider: { + type: 'static', + value: { kind, resource: '$res:u/admin/foundry', model: 'gpt-4o' } + }, + user_message: { type: 'static', value: 'hello' }, + output_type: { type: 'static', value: 'text' } + } + } + } +} + +describe('copilot flow module validation - AI agent provider kind', () => { + it('accepts azure_foundry (and the existing azure_openai baseline)', () => { + expect( + flowModulesSchema.safeParse([aiAgentModuleWithProviderKind('azure_openai')]).success + ).toBe(true) + expect( + flowModulesSchema.safeParse([aiAgentModuleWithProviderKind('azure_foundry')]).success + ).toBe(true) + }) + + it('still rejects an unknown provider kind (enum is actually enforced)', () => { + expect( + flowModulesSchema.safeParse([aiAgentModuleWithProviderKind('not_a_real_provider')]).success + ).toBe(false) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/global/core.test.ts b/frontend/src/lib/components/copilot/chat/global/core.test.ts index 50ad2547f0..2438302a00 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.test.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.test.ts @@ -36,7 +36,7 @@ const { backendDrafts, serverTimestamps, failingWrites, failingReads } = vi.hois // concurrent writer advancing the row; otherwise empty, so the conflict // branch in `updateDraft` stays inert for every pre-existing test. serverTimestamps: new Map(), - // Keys whose `updateDraft` / `getDraftForUser` throw a non-404 (network/5xx); + // Keys whose `updateDraft` / draft reads throw a non-404 (network/5xx); // only set by the error-handling tests, empty otherwise. failingWrites: new Set(), failingReads: new Set() @@ -64,6 +64,9 @@ vi.mock('$lib/gen', async () => { getScriptByPath: vi.fn(async () => { throw new Error('getScriptByPath mock not configured') }), + getScriptByHash: vi.fn(async () => { + throw new Error('getScriptByHash mock not configured') + }), queryHubScripts: vi.fn(async () => []), getHubScriptContentByPath: vi.fn(async () => ''), listScripts: vi.fn(async () => []) @@ -119,6 +122,9 @@ vi.mock('$lib/gen', async () => { getFlowByPath: vi.fn(async () => { throw new Error('getFlowByPath mock not configured') }), + getFlowVersion: vi.fn(async () => { + throw new Error('getFlowVersion mock not configured') + }), getFlowLatestVersion: vi.fn(async () => ({ id: 1 })), listFlows: vi.fn(async () => []) }), @@ -126,13 +132,17 @@ vi.mock('$lib/gen', async () => { existsSchedule: vi.fn(async () => false), getSchedule: vi.fn(async () => { throw new Error('getSchedule mock not configured') - }) + }), + createSchedule: vi.fn(async () => 'created'), + updateSchedule: vi.fn(async () => 'updated') }), HttpTriggerService: wrapService(actual.HttpTriggerService, { existsHttpTrigger: vi.fn(async () => false), getHttpTrigger: vi.fn(async () => { throw new Error('getHttpTrigger mock not configured') - }) + }), + createHttpTrigger: vi.fn(async () => 'created'), + updateHttpTrigger: vi.fn(async () => 'updated') }), AppService: wrapService(actual.AppService, { existsApp: vi.fn(async () => false), @@ -141,13 +151,18 @@ vi.mock('$lib/gen', async () => { getAppByPath: vi.fn(async () => { throw new Error('getAppByPath mock not configured') }), + getAppByVersion: vi.fn(async () => { + throw new Error('getAppByVersion mock not configured') + }), listApps: vi.fn(async () => []) }), ResourceService: wrapService(actual.ResourceService, { existsResource: vi.fn(async () => false), getResource: vi.fn(async () => { throw new Error('getResource mock not configured') - }) + }), + createResource: vi.fn(async () => 'created'), + updateResource: vi.fn(async () => 'updated') }), VariableService: wrapService(actual.VariableService, { existsVariable: vi.fn(async () => false), @@ -157,6 +172,9 @@ vi.mock('$lib/gen', async () => { createVariable: vi.fn(async () => 'created'), updateVariable: vi.fn(async () => 'updated') }), + FolderService: wrapService(actual.FolderService, { + createFolder: vi.fn(async () => 'created') + }), DraftService: wrapService(actual.DraftService, { updateDraft: vi.fn(async ({ kind, path, requestBody }: any) => { const key = `${kind}:${path}` @@ -178,14 +196,27 @@ vi.mock('$lib/gen', async () => { return { status: 'saved', current_timestamp: '2026-06-15T00:00:00Z' } }), getDraftForUser: vi.fn(async ({ kind, path }: any) => { + // The real endpoint rejects drawer kinds up front (drafts for + // schedule/trigger/resource/variable are private to their owner) — + // mirror it so a caller regressing to this route for those kinds + // fails in tests the same way it does against the backend. + if (!['script', 'flow', 'app', 'raw_app'].includes(kind)) + throw Object.assign(new Error('drafts for this item kind are private to their owner'), { + status: 404 + }) const key = `${kind}:${path}` if (failingReads.has(key)) throw Object.assign(new Error('server error'), { status: 500 }) - // 404-shaped (status) like the real ApiError, so the adapter's - // narrowed catch treats it as "no draft" rather than re-throwing. if (!backendDrafts.has(key)) throw Object.assign(new Error('no draft for that owner at that path'), { status: 404 }) return { value: backendDrafts.get(key), created_at: '2026-06-15T00:00:00Z' } }), + getOwnDraft: vi.fn(async ({ kind, path }: any) => { + const key = `${kind}:${path}` + if (failingReads.has(key)) throw Object.assign(new Error('server error'), { status: 500 }) + // The real endpoint returns 200 with null when the user has no draft. + if (!backendDrafts.has(key)) return null + return { value: backendDrafts.get(key), created_at: '2026-06-15T00:00:00Z' } + }), listDrafts: vi.fn(async () => Array.from(backendDrafts.entries()).map(([key, value]) => { const idx = key.indexOf(':') @@ -209,6 +240,13 @@ vi.mock('./rawAppBundlerBridge', () => ({ })) })) +vi.mock('$lib/infer', async () => ({ + ...(await vi.importActual('$lib/infer')), + // Avoid the wasm parser in unit tests: the script deploy path infers the arg + // schema but tolerates failure, and these tests don't assert on the schema. + inferArgs: vi.fn(async () => {}) +})) + import { globalTools, globalToolsFor, @@ -233,6 +271,7 @@ import { bundleRawAppDraft } from './rawAppBundlerBridge' import { AppService, FlowService, + FolderService, HttpTriggerService, JobService, ResourceService, @@ -240,6 +279,8 @@ import { ScriptService, VariableService } from '$lib/gen' +import { userStore } from '$lib/stores' +import { get } from 'svelte/store' import type { Tool, ToolCallbacks } from '../shared' const WORKSPACE = 'global-core-test' @@ -605,6 +646,178 @@ describe('global AI tools', () => { expect(VariableService.updateVariable).not.toHaveBeenCalled() }) + it('deploys every field of a script draft (not just content/summary)', async () => { + // The deploy delegates to the shared deployer, which reads the full persisted + // draft via getScriptByPath(getDraft) and deploys all of it. Config fields + // (tag/priority/schema/description/concurrency) were previously dropped, + // sourced from the deployed version instead. + seedBackendDraft( + 'script', + 'f/scripts/full', + { path: 'f/scripts/full', content: 'export async function main() {}', language: 'bun' }, + { workspace: WORKSPACE } + ) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ + hash: 1234, + path: 'f/scripts/full', + summary: 'Full script', + description: 'desc', + content: 'export async function main() {}', + schema: { foo: 'bar' }, + language: 'bun', + kind: 'script', + tag: 'custom-tag', + priority: 7, + concurrent_limit: 3, + draft_only: true + } as any) + + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/full' }) + + expect(ScriptService.createScript).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'f/scripts/full', + content: 'export async function main() {}', + summary: 'Full script', + description: 'desc', + schema: { foo: 'bar' }, + language: 'bun', + tag: 'custom-tag', + priority: 7, + concurrent_limit: 3, + parent_hash: 1234 + }) + }) + // Editor-only / server-managed draft keys must not leak into the deploy body. + const calls = vi.mocked(ScriptService.createScript).mock.calls + const body = calls[calls.length - 1][0].requestBody as any + expect(body.draft_only).toBeUndefined() + }) + + it('deploys every config field of a flow draft via createFlow', async () => { + seedBackendDraft( + 'flow', + 'f/flows/full', + { summary: 'Full flow', description: 'flow desc', value: { modules: [] }, schema: {} }, + { workspace: WORKSPACE } + ) + vi.mocked(FlowService.getFlowByPath).mockResolvedValueOnce({ + path: 'f/flows/full', + summary: 'Full flow', + description: 'flow desc', + value: { modules: [] }, + schema: { x: 1 }, + tag: 'flow-tag', + dedicated_worker: true + } as any) + + await callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/full' }) + + // No deployed flow row (existsFlowByPath defaults to false) → create. + expect(FlowService.createFlow).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'f/flows/full', + summary: 'Full flow', + description: 'flow desc', + value: { modules: [] }, + schema: { x: 1 }, + tag: 'flow-tag', + dedicated_worker: true + }) + }) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + }) + + it('deploys an editor draft_only script at its chosen path, not its synthetic storage key', async () => { + // A new script created in the editor lives at a synthetic `u/{user}/draft_{uuid}` + // storage key while its chosen path is in the draft value. The chat addresses + // it by the chosen (display) path; deploy must resolve to the storage key so the + // shared deployer can read the draft via getScriptByPath, then deploy at the + // chosen path. Reading at the chosen path would 404. + const storageKey = 'u/admin/draft_abc123' + const chosenPath = 'f/team/chosen_path' + seedBackendDraft( + 'script', + storageKey, + { + path: chosenPath, + summary: 'New script', + description: '', + content: 'export async function main() {}', + schema: {}, + is_template: false, + language: 'bun', + kind: 'script' + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'script', + storagePath: storageKey, + effectivePath: chosenPath + }) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ + path: chosenPath, + summary: 'New script', + description: '', + content: 'export async function main() {}', + schema: {}, + language: 'bun', + kind: 'script' + } as any) + + const flushSpy = vi.spyOn(UserDraftDbSyncer, 'flush') + + await callGlobalTool('deploy_workspace_item', { type: 'script', path: chosenPath }) + + // Any pending editor autosave is flushed at the storage key before delegating, + // so the shared deployer reads the latest value, not a stale persisted draft. + expect(flushSpy).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, itemKind: 'script', path: storageKey }) + ) + // The draft is read at the STORAGE key (the chosen path would 404)… + expect(ScriptService.getScriptByPath).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, path: storageKey, getDraft: true }) + ) + // …and deployed at the chosen path. + expect(ScriptService.createScript).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ path: chosenPath }) + }) + }) + + it('aborts deploy when the pre-deploy draft flush hit a conflict', async () => { + // flush() resolves even when the save recorded a conflict; deploy must abort + // rather than publish the stale persisted draft. + seedBackendDraft( + 'script', + 'f/scripts/conflicted', + { + path: 'f/scripts/conflicted', + summary: '', + description: '', + content: 'export async function main() {}', + schema: {}, + is_template: false, + language: 'bun', + kind: 'script' + }, + { workspace: WORKSPACE } + ) + const conflictSpy = vi + .spyOn(UserDraftDbSyncer, 'getConflict') + .mockReturnValue({ conflict: { serverTimestamp: '2026', localLastSync: null } } as any) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/conflicted' }) + ).rejects.toThrow(/conflicting/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + conflictSpy.mockRestore() + }) + it('writes script drafts into UserDraft', async () => { const content = 'export async function main() {\n\treturn "hello"\n}' @@ -969,6 +1182,170 @@ describe('global AI tools', () => { expect(draft).not.toHaveProperty('override') }) + // Schedule drafts (like all drawer kinds) are private to their owner, so the + // cross-user draft route 404s on them. Reading them back must go through the + // own-draft route, else a freshly written schedule draft is listed but can + // never be read or deployed. + it('reads and deploys a schedule draft written by the chat', async () => { + await callGlobalTool('write_schedule', { + path: 'u/admin/test_schedule_greet', + schedule: '0 0 9 * * *', + timezone: 'UTC', + script_path: 'f/scripts/greet', + is_flow: false, + args: {} + }) + + const readRaw = await callGlobalTool('read_workspace_item', { + type: 'schedule', + path: 'u/admin/test_schedule_greet' + }) + expect(JSON.parse(readRaw)).toMatchObject({ + type: 'schedule', + path: 'u/admin/test_schedule_greet', + isDraft: true + }) + + await callGlobalTool('deploy_workspace_item', { + type: 'schedule', + path: 'u/admin/test_schedule_greet' + }) + expect(ScheduleService.createSchedule).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'u/admin/test_schedule_greet', + schedule: '0 0 9 * * *', + script_path: 'f/scripts/greet' + }) + }) + // The draft is consumed by the deploy. + expect( + getBackendDraft('trigger_schedule', 'u/admin/test_schedule_greet', { + workspace: WORKSPACE + }) + ).toBeUndefined() + }) + + // Same private-owner read path as schedules, for the trigger drawer kinds. + it('reads and deploys a trigger draft written by the chat', async () => { + await callGlobalTool('write_trigger', { + kind: 'http', + config: { + path: 'u/admin/fresh_route', + script_path: 'f/scripts/handler', + is_flow: false, + route_path: 'api/fresh', + http_method: 'get', + authentication_method: 'none', + is_static_website: false + } + }) + + const readRaw = await callGlobalTool('read_workspace_item', { + type: 'trigger', + trigger_kind: 'http', + path: 'u/admin/fresh_route' + }) + expect(JSON.parse(readRaw)).toMatchObject({ + type: 'trigger', + triggerKind: 'http', + path: 'u/admin/fresh_route', + isDraft: true + }) + + await callGlobalTool('deploy_workspace_item', { + type: 'trigger', + trigger_kind: 'http', + path: 'u/admin/fresh_route' + }) + expect(HttpTriggerService.createHttpTrigger).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'u/admin/fresh_route', + route_path: 'api/fresh', + script_path: 'f/scripts/handler' + }) + }) + expect( + getBackendDraft('trigger_http', 'u/admin/fresh_route', { workspace: WORKSPACE }) + ).toBeUndefined() + }) + + // Same private-owner read path as schedules, for the resource drawer kind. + it('reads and deploys a resource draft written by the chat', async () => { + await callGlobalTool('write_resource', { + path: 'u/admin/fresh_db', + value: { host: 'db.example.com', port: 5432 }, + resource_type: 'postgresql', + description: 'fresh database' + }) + + const readRaw = await callGlobalTool('read_workspace_item', { + type: 'resource', + path: 'u/admin/fresh_db' + }) + expect(JSON.parse(readRaw)).toMatchObject({ + type: 'resource', + path: 'u/admin/fresh_db', + isDraft: true + }) + + await callGlobalTool('deploy_workspace_item', { + type: 'resource', + path: 'u/admin/fresh_db' + }) + expect(ResourceService.createResource).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'u/admin/fresh_db', + resource_type: 'postgresql', + value: { host: 'db.example.com', port: 5432 } + }) + }) + expect( + getBackendDraft('resource', 'u/admin/fresh_db', { workspace: WORKSPACE }) + ).toBeUndefined() + }) + + // Same private-owner read path as schedules, for the variable drawer kind. + // Secret variables deploy through the ephemeral in-memory value instead + // (see the ephemeral-value tests above); this pins the plain-value cycle. + it('reads and deploys a non-secret variable draft written by the chat', async () => { + await callGlobalTool('write_variable', { + path: 'u/admin/fresh_config', + value: 'plain-value', + is_secret: false, + description: 'fresh config' + }) + + const readRaw = await callGlobalTool('read_workspace_item', { + type: 'variable', + path: 'u/admin/fresh_config' + }) + expect(JSON.parse(readRaw)).toMatchObject({ + type: 'variable', + path: 'u/admin/fresh_config', + isDraft: true + }) + + await callGlobalTool('deploy_workspace_item', { + type: 'variable', + path: 'u/admin/fresh_config' + }) + expect(VariableService.createVariable).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'u/admin/fresh_config', + value: 'plain-value', + is_secret: false, + description: 'fresh config' + }) + }) + expect( + getBackendDraft('variable', 'u/admin/fresh_config', { workspace: WORKSPACE }) + ).toBeUndefined() + }) + it('requires trigger_kind when discarding a trigger draft', async () => { await expect( callGlobalTool('discard_local_draft', { @@ -1009,6 +1386,260 @@ describe('global AI tools', () => { }) }) + describe('stale-draft deploy guard and rebase', () => { + // The suite's beforeEach only clears mock calls (not implementations), so + // restore the script-service mocks these tests override back to their factory + // defaults; otherwise a persistent resolved value leaks into later tests. + afterEach(() => { + vi.mocked(ScriptService.existsScriptByPath).mockResolvedValue(false) + vi.mocked(ScriptService.getScriptByPath).mockImplementation(async () => { + throw new Error('getScriptByPath mock not configured') + }) + vi.mocked(ScriptService.getScriptByHash).mockImplementation(async () => { + throw new Error('getScriptByHash mock not configured') + }) + vi.mocked(FlowService.existsFlowByPath).mockResolvedValue(false) + vi.mocked(FlowService.getFlowByPath).mockImplementation(async () => { + throw new Error('getFlowByPath mock not configured') + }) + vi.mocked(FlowService.getFlowVersion).mockImplementation(async () => { + throw new Error('getFlowVersion mock not configured') + }) + vi.mocked(FlowService.getFlowLatestVersion).mockResolvedValue({ id: 1 } as any) + vi.mocked(AppService.existsApp).mockResolvedValue(false) + vi.mocked(AppService.getAppByPath).mockImplementation(async () => { + throw new Error('getAppByPath mock not configured') + }) + vi.mocked(AppService.getAppByVersion).mockImplementation(async () => { + throw new Error('getAppByVersion mock not configured') + }) + }) + + function seedStaleScriptDraft(path: string, parentHash: string, content = 'draft content') { + seedBackendDraft('script', path, { + path, + summary: 's', + description: '', + content, + language: 'bun', + kind: 'script', + parent_hash: parentHash, + schema: {} + }) + } + + function mockDeployedScript(path: string, hash: string, content = 'latest deployed') { + vi.mocked(ScriptService.existsScriptByPath).mockResolvedValue(true) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValue({ + path, + hash, + content, + language: 'bun', + summary: 's' + } as any) + } + + it('blocks deploying a script draft started from an older deployed version', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash') + mockDeployedScript('f/scripts/stale', 'new-hash') + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + }) + + it('deploys a stale script draft when force is set', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash') + mockDeployedScript('f/scripts/stale', 'new-hash') + + const result = JSON.parse( + await callGlobalTool('deploy_workspace_item', { + type: 'script', + path: 'f/scripts/stale', + force: true + }) + ) + expect(result.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + it('deploys a script draft that is based on the current deployed head', async () => { + seedStaleScriptDraft('f/scripts/fresh', 'head-hash') + mockDeployedScript('f/scripts/fresh', 'head-hash') + + const result = JSON.parse( + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/fresh' }) + ) + expect(result.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + it('rebase_draft discards the stale draft and surfaces the changes to replay', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash', 'base content\nmy added line\n') + mockDeployedScript('f/scripts/stale', 'new-hash', 'latest deployed content\n') + vi.mocked(ScriptService.getScriptByHash).mockResolvedValue({ + hash: 'base-hash', + content: 'base content\n', + language: 'bun' + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'script', path: 'f/scripts/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_hash).toBe('new-hash') + // The diff surfaces the draft's own change over its fork base. + expect(result.your_changes).toContain('my added line') + + // The stale draft is discarded (not reset), so a premature deploy fails + // cleanly rather than silently shipping the latest unchanged. + expect(getBackendDraft('script', 'f/scripts/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + + // Re-applying re-bases onto the current head; the deploy then passes. + await callGlobalTool('write_script', { + path: 'f/scripts/stale', + summary: 's', + language: 'bun', + content: 'latest deployed content\nmy added line\n' + }) + const deploy = JSON.parse( + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ) + expect(deploy.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + function seedStaleFlowDraft(path: string, versionId: number, modules: any[] = []) { + seedBackendDraft('flow', path, { + path, + summary: 'f', + description: '', + version_id: versionId, + value: { modules }, + schema: {} + }) + } + + function mockDeployedFlow(path: string, versionId: number) { + vi.mocked(FlowService.existsFlowByPath).mockResolvedValue(true) + vi.mocked(FlowService.getFlowByPath).mockResolvedValue({ + path, + summary: 'f', + version_id: versionId, + value: { modules: [] }, + schema: {} + } as any) + } + + it('blocks deploying a flow draft started from an older deployed version', async () => { + seedStaleFlowDraft('f/flows/stale', 1) + mockDeployedFlow('f/flows/stale', 2) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + expect(FlowService.createFlow).not.toHaveBeenCalled() + }) + + it('rebase_draft discards the stale flow draft and surfaces the changes to replay', async () => { + seedStaleFlowDraft('f/flows/stale', 1, [{ id: 'a', value: { type: 'identity' } }]) + mockDeployedFlow('f/flows/stale', 2) + vi.mocked(FlowService.getFlowVersion).mockResolvedValue({ + value: { modules: [] } + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'flow', path: 'f/flows/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_version).toBe(2) + expect(result.your_changes).toContain('identity') + + // The stale draft is discarded, so a premature deploy fails cleanly. + expect(getBackendDraft('flow', 'f/flows/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + }) + + function seedStaleAppDraft(path: string, parentVersion: number, file = 'old') { + seedBackendDraft('raw_app', path, { + summary: 'a', + files: { '/index.tsx': file }, + runnables: {}, + data: { tables: [] }, + parent_version: parentVersion + }) + } + + function mockDeployedApp(path: string, versionId: number, file = 'latest') { + vi.mocked(AppService.existsApp).mockResolvedValue(true) + vi.mocked(AppService.getAppByPath).mockResolvedValue({ + path, + summary: 'a', + versions: [versionId], + value: { files: { '/index.tsx': file }, runnables: {}, data: { tables: [] } }, + policy: { execution_mode: 'publisher' } + } as any) + } + + it('grafts the fork-base version onto a new app draft and keeps it through the save whitelist', async () => { + // No draft yet: the first app edit projects the deployed app into a draft. + // This exercises the runtime path types can't catch — the graft in + // appSourceToDraftValue AND survival through normalizeAppDraftValue's whitelist. + mockDeployedApp('f/apps/fresh', 5) + + await callGlobalTool('write_app_file', { + path: 'f/apps/fresh', + file_path: '/src/New.tsx', + content: 'export default function New() { return null }' + }) + + const draft = getBackendDraft('raw_app', 'f/apps/fresh', { workspace: WORKSPACE }) + expect(draft.parent_version).toBe(5) + }) + + it('blocks deploying an app draft started from an older deployed version', async () => { + seedStaleAppDraft('f/apps/stale', 1) + mockDeployedApp('f/apps/stale', 2) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(AppService.createAppRaw).not.toHaveBeenCalled() + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + + it('rebase_draft discards the stale app draft and surfaces the changes to replay', async () => { + seedStaleAppDraft('f/apps/stale', 1, 'my-change') + mockDeployedApp('f/apps/stale', 2, 'latest-deployed') + vi.mocked(AppService.getAppByVersion).mockResolvedValue({ + value: { files: { '/index.tsx': 'base' }, runnables: {}, data: { tables: [] } } + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'app', path: 'f/apps/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_version).toBe(2) + expect(result.your_changes).toContain('my-change') + + // The stale draft is discarded, so a premature deploy fails cleanly. + expect(getBackendDraft('raw_app', 'f/apps/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + }) + it('preserves existing flow metadata and seeds freshness on first flow write', async () => { vi.mocked(FlowService.existsFlowByPath).mockResolvedValueOnce(true) vi.mocked(FlowService.getFlowLatestVersion).mockResolvedValueOnce({ id: 42 } as any) @@ -1383,9 +2014,7 @@ describe('global AI tools', () => { file_path: '/min.tsx' }) - expect(result).toContain( - 'lines 1-3 of 3, truncated to the first 50000 of 90002 chars.' - ) + expect(result).toContain('lines 1-3 of 3, truncated to the first 50000 of 90002 chars.') expect(result).toContain('the file is likely minified') expect(result.split('\n\n')[1]).toHaveLength(50_000) }) @@ -1401,9 +2030,7 @@ describe('global AI tools', () => { file_path: '/generated.js' }) - expect(result).toContain( - 'lines 1-1 of 1, truncated to the first 50000 of 60000 chars.' - ) + expect(result).toContain('lines 1-1 of 1, truncated to the first 50000 of 60000 chars.') expect(result).toContain('re-read with a smaller limit') expect(result.split('\n\n')[1]).toBe('x'.repeat(50_000)) }) @@ -1488,8 +2115,7 @@ describe('global AI tools', () => { versions: [5], value: { files: { - '/lib/aggregations.ts': - 'export function computeRevenue(o) {\n return o.unitPrice\n}\n', + '/lib/aggregations.ts': 'export function computeRevenue(o) {\n return o.unitPrice\n}\n', '/components/SummaryPanel.tsx': 'import { computeRevenue } from "../lib/aggregations"\nconst total = computeRevenue(order)\n', '/components/OrdersTable.tsx': 'const r = computeRevenue(row)\n// renders revenue\n', @@ -1787,6 +2413,9 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + // getAppByPath resolves with no draft_path → deploy at the item's own path. + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + const raw = await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/report', @@ -1832,6 +2461,93 @@ describe('global AI tools', () => { }) }) + it('deploys an editor raw app draft at its draft_path, not its synthetic storage key', async () => { + // An editor-created draft_only raw app lives at a synthetic storage key with + // its chosen path in `draft_path`; deploy must resolve to the storage key, + // read draft_path, and create the app there — not at the synthetic key. + const storageKey = 'u/admin/draft_app999' + const chosenPath = 'f/team/chosen_app' + seedBackendDraft( + 'raw_app', + storageKey, + { + summary: 'Editor app', + files: { '/App.tsx': 'export default () => null' }, + runnables: {}, + data: { tables: [] }, + draft_path: chosenPath + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'raw_app', + storagePath: storageKey, + effectivePath: chosenPath + }) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ + draft: { draft_path: chosenPath } + } as any) + const flushSpy = vi.spyOn(UserDraftDbSyncer, 'flush') + + await callGlobalTool('deploy_workspace_item', { type: 'app', path: chosenPath }) + + // The draft is flushed at the storage key before the draft_path read, so a + // not-yet-saved editor rename isn't read stale. + expect(flushSpy).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, itemKind: 'raw_app', path: storageKey }) + ) + // draft_path is read from the backend draft at the storage key… + expect(AppService.getAppByPath).toHaveBeenCalledWith( + expect.objectContaining({ + workspace: WORKSPACE, + path: storageKey, + getDraft: true, + rawApp: true + }) + ) + // …and the app is created at the chosen path, not the synthetic key. + expect(AppService.createAppRaw).toHaveBeenCalledWith( + expect.objectContaining({ + formData: expect.objectContaining({ + app: expect.objectContaining({ path: chosenPath }) + }) + }) + ) + }) + + it('aborts a raw app deploy when the draft_path lookup fails (non-404)', async () => { + // A real lookup failure (network/5xx) must abort, not silently fall back to the + // storage path and deploy there. Only a 404 justifies the storage-path fallback. + seedBackendDraft( + 'raw_app', + 'u/admin/draft_appfail', + { + summary: 'Editor app', + files: { '/App.tsx': 'export default () => null' }, + runnables: {}, + data: { tables: [] }, + draft_path: 'f/team/chosen_app' + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'raw_app', + storagePath: 'u/admin/draft_appfail', + effectivePath: 'f/team/chosen_app' + }) + vi.mocked(AppService.getAppByPath).mockRejectedValueOnce( + Object.assign(new Error('server error'), { status: 500 }) + ) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/team/chosen_app' }) + ).rejects.toThrow() + expect(AppService.createAppRaw).not.toHaveBeenCalled() + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + it('deploys an existing raw app draft by bundling files and updating the raw app', async () => { vi.mocked(AppService.existsApp).mockResolvedValueOnce(true) seedBackendDraft( @@ -1848,6 +2564,8 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/report' @@ -1876,6 +2594,40 @@ describe('global AI tools', () => { expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() }) + it('forwards preserve_on_behalf_of when the deployed policy carries an on_behalf_of', async () => { + // Without the flag the backend resets the policy's on_behalf_of to the + // deploying user; this chat path has no on-behalf-of selector, so it must + // preserve whatever the carried policy already holds. + vi.mocked(AppService.existsApp).mockResolvedValueOnce(true) + seedBackendDraft( + 'raw_app', + 'f/apps/obo', + { + summary: 'On-behalf app', + files: { '/index.tsx': 'console.log("obo")' }, + runnables: {}, + data: { tables: [] }, + policy: { + execution_mode: 'publisher', + on_behalf_of: 'u/alice', + on_behalf_of_email: 'alice@windmill.dev' + } + }, + { workspace: WORKSPACE } + ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + + await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/obo' }) + + expect(AppService.updateAppRaw).toHaveBeenCalledWith( + expect.objectContaining({ + formData: expect.objectContaining({ + app: expect.objectContaining({ preserve_on_behalf_of: true }) + }) + }) + ) + }) + it('notifies the session preview (as raw_app) after deploying a raw app', async () => { const onDeployed = vi.fn() setDeployedInSessionHandler(onDeployed) @@ -1892,6 +2644,8 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + await callGlobalTool( 'deploy_workspace_item', { type: 'app', path: 'f/apps/report' }, @@ -2372,7 +3126,7 @@ describe('global AI tools', () => { const callbacks: ToolCallbacks = { setToolStatus: vi.fn(), removeToolStatus: vi.fn(), - requestUserQuestion: vi.fn(async (_toolId, question) => question.choices[1]) + requestUserQuestion: vi.fn(async (_toolId, question) => [question.choices[1]]) } const raw = await callGlobalTool( @@ -2398,7 +3152,45 @@ describe('global AI tools', () => { content: 'User answered question: python3', isLoading: false, result: 'python3', - userQuestion: expect.objectContaining({ selectedChoice: 'python3' }) + userQuestion: expect.objectContaining({ selectedChoices: ['python3'] }) + }) + ) + }) + + it('returns a newline-bulleted list when several answers are selected', async () => { + const callbacks: ToolCallbacks = { + setToolStatus: vi.fn(), + removeToolStatus: vi.fn(), + requestUserQuestion: vi.fn(async (_toolId, question) => [ + question.choices[0], + question.choices[2] + ]) + } + + const raw = await callGlobalTool( + 'askUserQuestion', + { + question: 'Which languages should be supported?', + choices: ['bun', 'python3', 'go'], + multiSelect: true + }, + callbacks + ) + + // Model-facing return stays newline-bulleted; the header readback is a + // compact comma list. + expect(raw).toBe('- bun\n- go') + expect(callbacks.requestUserQuestion).toHaveBeenCalledWith( + 'test-askUserQuestion', + expect.objectContaining({ multiSelect: true }) + ) + expect(callbacks.setToolStatus).toHaveBeenLastCalledWith( + 'test-askUserQuestion', + expect.objectContaining({ + content: 'User answered question: bun, go', + isLoading: false, + result: '- bun\n- go', + userQuestion: expect.objectContaining({ selectedChoices: ['bun', 'go'] }) }) ) }) @@ -2408,7 +3200,7 @@ describe('global AI tools', () => { const callbacks: ToolCallbacks = { setToolStatus: vi.fn(), removeToolStatus: vi.fn(), - requestUserQuestion: vi.fn(async (_toolId, question) => question.choices[9]) + requestUserQuestion: vi.fn(async (_toolId, question) => [question.choices[9]]) } const raw = await callGlobalTool( @@ -2453,7 +3245,7 @@ describe('global AI tools', () => { const callbacks: ToolCallbacks = { setToolStatus: vi.fn(), removeToolStatus: vi.fn(), - requestUserQuestion: vi.fn(async () => 'use deno instead') + requestUserQuestion: vi.fn(async () => ['use deno instead']) } const raw = await callGlobalTool( @@ -2471,12 +3263,58 @@ describe('global AI tools', () => { expect.objectContaining({ content: 'User answered question: use deno instead', result: 'use deno instead', - userQuestion: expect.objectContaining({ selectedChoice: 'use deno instead' }) + userQuestion: expect.objectContaining({ selectedChoices: ['use deno instead'] }) }) ) }) }) +describe('folder tools', () => { + beforeEach(() => { + vi.clearAllMocks() + userStore.set(undefined) + }) + afterEach(() => { + userStore.set(undefined) + }) + + it('create_folder requires confirmation', () => { + const tool = getGlobalTool('create_folder') + expect(tool.requiresConfirmation).toBe(true) + expect(tool.confirmationMessage).toBe('Create folder') + }) + + it('create_folder creates the folder and reflects it in the path context', async () => { + userStore.set({ username: 'bob', is_admin: false, folders: ['existing'] } as any) + const raw = await callGlobalTool('create_folder', { name: 'analytics', summary: 'team data' }) + + expect(vi.mocked(FolderService.createFolder)).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: { name: 'analytics', summary: 'team data' } + }) + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(true) + expect(parsed.message).toContain('f/analytics') + expect((get(userStore) as any)?.folders).toContain('analytics') + }) + + it('create_folder rejects an invalid name without calling the API', async () => { + const raw = await callGlobalTool('create_folder', { name: 'bad name!' }) + expect(vi.mocked(FolderService.createFolder)).not.toHaveBeenCalled() + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(false) + expect(parsed.error).toContain('alphanumeric') + }) + + it('create_folder surfaces a backend error (e.g. name conflict)', async () => { + vi.mocked(FolderService.createFolder).mockRejectedValueOnce(new Error('Folder already exists')) + const raw = await callGlobalTool('create_folder', { name: 'taken' }) + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(false) + expect(parsed.error).toContain('Folder already exists') + }) +}) + describe('prepareGlobalSystemMessage', () => { it('keeps global chat draft instructions concise and user-facing', () => { const message = prepareGlobalSystemMessage() @@ -2496,6 +3334,104 @@ describe('prepareGlobalSystemMessage', () => { expect(content).not.toContain('frontend AI draft store') }) + it('honors user-supplied shared folder paths without asking first', () => { + const content = prepareGlobalSystemMessage(undefined, { + user: { username: 'admin', is_admin: true, folders: ['evals'] } + }).content as string + + expect(content).toContain( + 'If the user supplies a fully qualified `f//...` path, use that exact path' + ) + expect(content).toContain('Do not ask for folder confirmation') + expect(content).toContain('substitute a `u/admin/...` path unless a tool rejects it') + }) + + it('tells the model to create a folder only when the user explicitly asks', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).toContain( + 'create one with `create_folder` only when the user explicitly asks for a new folder' + ) + }) + + describe('folder guidance', () => { + const guidanceOf = (user: { + username: string + is_admin?: boolean + folders?: string[] + folders_read?: string[] + }) => prepareGlobalSystemMessage(undefined, { user }).content as string + + it('lists the writable folders for a non-admin', () => { + const content = guidanceOf({ + username: 'bob', + is_admin: false, + folders: ['marketing', 'data_engineering'], + folders_read: ['marketing', 'data_engineering'] + }) + expect(content).toContain( + 'Folders you can write to in this workspace: `f/marketing`, `f/data_engineering`.' + ) + expect(content).not.toContain('You can see but NOT write to') + }) + + it('flags read-only folders a non-admin cannot write to', () => { + const content = guidanceOf({ + username: 'bob', + is_admin: false, + folders: ['team_a'], + folders_read: ['team_a', 'team_b'] + }) + expect(content).toContain('Folders you can write to in this workspace: `f/team_a`.') + expect(content).toContain( + 'You can see but NOT write to: `f/team_b` — never create or deploy items there.' + ) + }) + + it('points a non-admin with no writable folders at the personal scope', () => { + const content = guidanceOf({ username: 'bob', is_admin: false, folders: [] }) + expect(content).toContain( + 'You have no shared folders you can write to in this workspace, so use `u/bob/`.' + ) + }) + + it('gives an admin permission-agnostic guidance with a non-exhaustive hint', () => { + const content = guidanceOf({ + username: 'admin', + is_admin: true, + folders: ['marketing', 'data_engineering'] + }) + expect(content).toContain('As a workspace admin you can write to any existing folder.') + expect(content).toContain( + 'Folders here include `f/marketing`, `f/data_engineering` (you can also write to others not listed).' + ) + expect(content).toContain( + 'If the user names a folder, use it; if they explicitly ask for a new folder, create it with `create_folder`; otherwise ask them which folder to use rather than guessing or creating one unprompted.' + ) + expect(content).not.toContain('Folders you can write to in this workspace') + }) + + it('omits the folder hint for an admin with no associated folders', () => { + const content = guidanceOf({ username: 'admin', is_admin: true, folders: [] }) + expect(content).toContain( + '- As a workspace admin you can write to any existing folder. If the user names a folder, use it; if they explicitly ask for a new folder, create it with `create_folder`; otherwise ask them which folder to use rather than guessing or creating one unprompted.' + ) + expect(content).not.toContain('Folders here include') + }) + + it('caps the folder list and notes the remainder', () => { + const folders = Array.from({ length: 45 }, (_, i) => `f${i}`) + const content = guidanceOf({ username: 'bob', is_admin: false, folders }) + expect(content).toContain('(+5 more)') + }) + + it('emits no folder guidance when no user is available', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).not.toContain('Folders you can write to in this workspace') + expect(content).not.toContain('As a workspace admin you can write to any existing folder') + expect(content).not.toContain('You have no shared folders you can write to') + }) + }) + it('exposes separate tools for discarding drafts and deleting workspace items', () => { const discard = getGlobalTool('discard_local_draft') const deleteItem = getGlobalTool('delete_workspace_item') @@ -2668,6 +3604,156 @@ describe('session-only preview tools gating', () => { expect(on).toContain('get_app_runtime_logs') expect(on).toContain('list_app_runs') }) + + // The instruction headers are matched by their distinctive parenthetical so the + // guidance bullet (which references both block names) doesn't false-positive. + const WS_HEADER = 'WORKSPACE INSTRUCTIONS (configured by a workspace admin' + const USER_HEADER = "USER INSTRUCTIONS (this user's personal instructions" + + it('renders only the workspace block when given workspace instructions', () => { + const content = prepareGlobalSystemMessage({ workspace: 'Always be terse.' }).content as string + expect(content).toContain(WS_HEADER) + expect(content).toContain('Always be terse.') + expect(content).not.toContain(USER_HEADER) + }) + + it('renders the user block with the edit-tool mention when given user instructions', () => { + const content = prepareGlobalSystemMessage({ user: 'Prefer Bun for new scripts.' }) + .content as string + expect(content).toContain(USER_HEADER) + expect(content).toContain('update_user_instructions') + expect(content).toContain('Prefer Bun for new scripts.') + expect(content).not.toContain(WS_HEADER) + }) + + it('renders the workspace block before the user block when both are present', () => { + const content = prepareGlobalSystemMessage({ workspace: 'WS rule.', user: 'User rule.' }) + .content as string + expect(content).toContain(WS_HEADER) + expect(content.indexOf(USER_HEADER)).toBeGreaterThan(content.indexOf(WS_HEADER)) + }) + + it('omits both instruction headers when none are provided', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).not.toContain(WS_HEADER) + expect(content).not.toContain(USER_HEADER) + }) +}) + +describe('update_user_instructions', () => { + function makeHelpers(initial = '') { + let value = initial + return { + getUserInstructions: () => value, + setUserInstructions: vi.fn((v: string) => { + value = v + }) + } + } + + it('appends to empty instructions', async () => { + const helpers = makeHelpers('') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Prefer Bun for new scripts.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Prefer Bun for new scripts.') + expect(res).toContain('Added a personal instruction') + }) + + it('appends to existing instructions joined by a blank line', async () => { + const helpers = makeHelpers('Existing rule.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Another rule.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Existing rule.\n\nAnother rule.') + }) + + it('returns only a short confirmation, not the resulting instructions', async () => { + const helpers = makeHelpers('Existing rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Another rule.' }, + toolCallbacks, + helpers + ) + expect(res).not.toContain('Existing rule.') + expect(res).not.toContain('Another rule.') + }) + + it('replaces an exact match', async () => { + const helpers = makeHelpers('Prefer Bun.\n\nUse tabs.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'Prefer Bun.', new_string: 'Prefer Deno.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Prefer Deno.\n\nUse tabs.') + }) + + it('removes the matched text when new_string is empty', async () => { + const helpers = makeHelpers('Keep this.\n\nDrop this.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: '\n\nDrop this.', new_string: '' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Keep this.') + }) + + it('clears all instructions when the whole text is replaced with empty', async () => { + const helpers = makeHelpers('Only rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'Only rule.', new_string: '' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('') + expect(res).toContain('Cleared your personal instructions') + }) + + it('errors without writing when old_string is not found, and echoes the current text for recovery', async () => { + const helpers = makeHelpers('Existing rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'missing', new_string: 'x' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).not.toHaveBeenCalled() + expect(res).toContain('not found') + expect(res).toContain('Existing rule.') + }) + + it('rejects a result over the length cap without writing', async () => { + const helpers = makeHelpers('') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'a'.repeat(5001) }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).not.toHaveBeenCalled() + expect(res).toContain('over the 5000') + }) + + it('fails gracefully when the context does not provide instruction helpers', async () => { + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'x' }, + toolCallbacks, + {} + ) + expect(res).toContain('cannot modify user instructions') + }) }) describe('prepareGlobalUserMessage', () => { @@ -2705,15 +3791,23 @@ describe('prepareGlobalUserMessage', () => { path: 'f/flows/reporting', title: 'f/flows/reporting', summary: 'Reporting flow' + }, + { + type: 'workspace_app', + path: 'f/apps/dashboard', + title: 'f/apps/dashboard', + summary: 'Dashboard raw app' } ]) expect(message.content).toContain('## SELECTED CONTEXT') expect(message.content).toContain('- type: script, path: f/scripts/report') expect(message.content).toContain('- type: flow, path: f/flows/reporting') + expect(message.content).toContain('- type: raw_app, path: f/apps/dashboard') expect(message.content).toContain('## INSTRUCTIONS:\nUpdate these items') expect(message.content).not.toContain('Report script') expect(message.content).not.toContain('Reporting flow') + expect(message.content).not.toContain('Dashboard raw app') }) it('omits selected context section when no workspace item is selected', () => { diff --git a/frontend/src/lib/components/copilot/chat/global/core.ts b/frontend/src/lib/components/copilot/chat/global/core.ts index 90925255d2..f26e3a670d 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.ts @@ -2,6 +2,7 @@ import { AppService, AzureTriggerService, FlowService, + FolderService, GcpTriggerService, HttpTriggerService, JobService, @@ -14,8 +15,10 @@ import { ScriptService, SqsTriggerService, VariableService, - WebsocketTriggerService + WebsocketTriggerService, + WorkspaceService } from '$lib/gen' +import { createTwoFilesPatch } from 'diff' import { $ScriptLang } from '$lib/gen/schemas.gen' import type { AppWithLastVersion, @@ -53,6 +56,7 @@ import { createInlineScriptSession } from '../flow/inlineScriptsUtils' import { getDatatableSdkReference, getFlowPrompt, + getPipelinePrompt, getRawAppPrompt, getResourcePrompt, getScriptPrompt @@ -76,6 +80,8 @@ import { import { searchDocsTool, readDocsPageTool } from '../docs/core' import type { ContextElement } from '../context' import { getDatatableTools } from '../datatableTools' +import { fileTools } from '../files/fileTools' +import type { AttachedFilesStore } from '../files/attachedFiles.svelte' import { UserDraft } from '$lib/userDraft.svelte' import { emptySchema } from '$lib/utils' import { inferArgs } from '$lib/infer' @@ -97,16 +103,42 @@ import { type WorkspaceItem, type WorkspaceItemType } from './workspaceItems' -import { buildFlowDeployRequestBody, buildScriptDeployRequestBody } from './deployRequests' -import { userStore } from '$lib/stores' +import { + userStore, + superadmin, + enterpriseLicense, + userWorkspaces, + workspaceStore +} from '$lib/stores' import { get } from 'svelte/store' +import { deployDraft as deployDraftToWorkspace } from '$lib/utils_draft_deploy' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { bundleRawAppDraft } from './rawAppBundlerBridge' +import { + buildRunsUrl, + buildSchedulesUrl, + buildVariablesUrl, + buildResourcesUrl, + buildAssetsUrl, + buildAuditLogsUrl, + buildWorkspaceSettingsUrl, + buildFoldersUrl, + buildGroupsUrl, + buildTriggersUrl, + WORKSPACE_SETTINGS_TABS +} from './pageNavigation' +import { + pageHref, + TRIGGER_PAGES, + type TriggerKind as PageTriggerKind +} from '$lib/components/sessions/previewRouter' import { clearEphemeralSecretVariableDraftValue, deleteGlobalDraft, getEphemeralSecretVariableDraftValue, getGlobalDraft, getGlobalDraftStoragePath, + itemKindFor, listGlobalDrafts, persistGlobalDraft, readGlobalDraftValue, @@ -132,7 +164,9 @@ const INSTRUCTION_SUBJECTS = [ ] as const satisfies readonly WorkspaceItemType[] // `datatable` is not a workspace item type, but the model can request the // datatable SDK reference (the wmill.datatable() runnable API) the same way. -const INSTRUCTION_SUBJECTS_EXTRA = ['datatable'] as const +// `pipeline` likewise isn't an item type — a data pipeline is a set of +// annotated scripts in a folder, so it gets authoring guidance, not a CRUD type. +const INSTRUCTION_SUBJECTS_EXTRA = ['datatable', 'pipeline'] as const const ALL_INSTRUCTION_SUBJECTS = [...INSTRUCTION_SUBJECTS, ...INSTRUCTION_SUBJECTS_EXTRA] as const const MAX_LIST_LIMIT = 100 type ActiveGlobalEditorType = Extract @@ -165,7 +199,7 @@ const scriptLangSchema = z.enum($ScriptLang.enum) const getInstructionsSchema = z.object({ subject: instructionSubjectSchema.describe( - 'What to get authoring instructions for: a workspace item type (script, flow, resource, app) or "datatable" for the wmill.datatable() SQL SDK used inside runnables. Schedules, triggers, and variables don\'t need instructions — their tool schemas describe everything.' + 'What to get authoring instructions for: a workspace item type (script, flow, resource, app), "pipeline" for building a data pipeline (a DAG of annotated scripts wired by storage assets — NOT a flow), or "datatable" for the wmill.datatable() SQL SDK used inside runnables. Schedules, triggers, and variables don\'t need instructions — their tool schemas describe everything.' ), language: scriptLangSchema .optional() @@ -183,7 +217,52 @@ const askUserQuestionSchema = z.object({ .array(z.string().min(1).describe('Proposed answer text shown to the user and returned as-is.')) .min(2) .max(10) - .describe('Two to ten mutually exclusive proposed answer strings.') + .describe('Two to ten proposed answer strings.'), + multiSelect: z + .boolean() + .optional() + .describe( + 'When true, the user may select several answers; use only when the choices can genuinely co-apply (not mutually exclusive). Defaults to single-select.' + ) +}) + +// Matches the per-mode cap enforced by the prompt-settings UI (AIPromptsModal) and +// the backend workspace prompt (MAX_CUSTOM_PROMPT_LENGTH). +export const MAX_USER_INSTRUCTIONS_LENGTH = 5000 + +const updateUserInstructionsSchema = z.object({ + operation: z + .enum(['append', 'replace']) + .describe( + 'What to do with your personal Global instructions. \'append\' adds a new instruction to the end (use for "remember this" / "always do X"). \'replace\' performs an exact find-and-replace to edit or remove existing text (use for "change X" / "stop doing Y"); set new_string to "" to remove.' + ), + text: z + .string() + .min(1) + .optional() + .describe( + "Required when operation is 'append': the instruction to add. Ignored for 'replace'." + ), + old_string: z + .string() + .min(1) + .optional() + .describe( + "Required when operation is 'replace': exact text to find in your current personal instructions. Ignored for 'append'." + ), + new_string: z + .string() + .optional() + .describe( + "Required when operation is 'replace': replacement text. Use an empty string to delete the matched text. Ignored for 'append'." + ), + replace_all: z + .boolean() + .optional() + .default(false) + .describe( + "For operation 'replace': when true, replace every exact match; when false, old_string must match exactly once." + ) }) const listWorkspaceItemsSchema = z.object({ @@ -370,6 +449,10 @@ const getJobLogsSchema = z.object({ id: z.string().describe('The UUID of the job to fetch logs for.') }) +const cancelJobSchema = z.object({ + id: z.string().describe('The UUID of the job to cancel.') +}) + const listRunsSchema = z.object({ path: z.string().optional().describe('Filter to runs of this exact script or flow path.'), created_by: z.string().optional().describe('Filter by the username that started the run.'), @@ -413,7 +496,20 @@ const deployWorkspaceItemSchema = z.object({ deployment_message: z .string() .optional() - .describe('Optional deployment message recorded with the change.') + .describe('Optional deployment message recorded with the change.'), + force: z + .boolean() + .optional() + .describe( + 'Deploy even if the draft was started from an older deployed version, overwriting the version deployed since. Defaults to false; prefer calling rebase_draft first to keep the newer changes.' + ) +}) + +const rebaseDraftSchema = z.object({ + type: itemTypeSchema, + path: z + .string() + .describe('Workspace path of the draft to rebase onto the latest deployed version.') }) const editScriptSchema = z.object({ @@ -451,9 +547,32 @@ const testRunArgsSchema = z .optional() .describe('Arguments to pass to the runnable. Omit or pass null when no arguments are needed.') +const backgroundArgSchema = z + .boolean() + .optional() + .describe( + 'Run in the background without waiting. Set true for jobs you expect to be long (deploys, backfills, big queries) — you will be notified when it finishes. Leave unset for normal runs, which wait briefly and only background automatically if slow.' + ) + +const waitSecondsArgSchema = z + .number() + .optional() + .describe( + 'How many seconds to wait for the job in-turn before it detaches into the background jobs tray. Defaults to 15. Raise it (capped at 120) for a job you expect to finish in, say, 30–60s and want the result in this same turn. Ignored when background is true. Do not use this to poll — larger values just hold the turn longer.' + ) + +/** Translate the model's `wait_seconds` into executeTestRun's `detachAfterMs` + * (the value is clamped to MAX_DETACH_AFTER_MS there). `undefined` keeps the + * default inline budget; negatives are floored to 0 (immediate detach). */ +function waitSecondsToDetachMs(waitSeconds: number | undefined): number | undefined { + return waitSeconds == null ? undefined : Math.max(0, waitSeconds) * 1000 +} + const testRunScriptSchema = z.object({ path: z.string().describe('Workspace path of the script to test.'), - args: testRunArgsSchema + args: testRunArgsSchema, + background: backgroundArgSchema, + wait_seconds: waitSecondsArgSchema }) const testRunScriptToolDef = createToolDef( @@ -465,7 +584,9 @@ const testRunScriptToolDef = createToolDef( const testRunFlowSchema = z.object({ path: z.string().describe('Workspace path of the flow to test.'), - args: testRunArgsSchema + args: testRunArgsSchema, + background: backgroundArgSchema, + wait_seconds: waitSecondsArgSchema }) const testRunFlowToolDef = createToolDef( @@ -478,7 +599,9 @@ const testRunFlowToolDef = createToolDef( const testRunStepSchema = z.object({ path: z.string().describe('Workspace path of the flow containing the step to test.'), stepId: z.string().describe('The id of the step/module to test.'), - args: testRunArgsSchema + args: testRunArgsSchema, + background: backgroundArgSchema, + wait_seconds: waitSecondsArgSchema }) const testRunStepToolDef = createToolDef( @@ -622,15 +745,30 @@ const deleteAppRunnableSchema = z.object({ const openPreviewSchema = z.object({ kind: z - .enum(['script', 'flow', 'raw_app']) + .enum(['script', 'flow', 'raw_app', 'pipeline']) .describe( - 'Item kind to preview. Use "raw_app" for code-based apps (created via init_app). The legacy drag-and-drop app builder ("app") is not previewable in the session panel — don\'t pass it.' + 'Item kind to preview. Use "raw_app" for code-based apps (created via init_app). Use "pipeline" to show the data-pipeline graph for a folder — here `path` is the folder name, not an item path. The legacy drag-and-drop app builder ("app") is not previewable in the session panel — don\'t pass it.' ), - path: z.string().describe('Workspace path of the item to preview.') + path: z + .string() + .describe('Workspace path of the item to preview, or the folder name when kind is "pipeline".') }) const getPreviewStatusSchema = z.object({}) +const closePageSchema = z.object({ + all: z + .boolean() + .optional() + .describe('Close every open preview tab, clearing the side panel. Ignores `match` when true.'), + match: z + .string() + .optional() + .describe( + 'Close the preview tab(s) whose page name or item path contains this text (case-insensitive), e.g. "runs", "schedules", or a script path. Call get_preview_status first if unsure what is open.' + ) +}) + type SessionToolResult = { aiResult: string uiMessage: string @@ -678,22 +816,87 @@ const initAppSchema = z.object({ ) }) +// Mirrors the backend VALID_FOLDER_NAME check so an invalid name fails before the +// network round-trip (the server enforces the same rule). +const VALID_FOLDER_NAME = /^[a-zA-Z_0-9-]+$/ + +const createFolderSchema = z.object({ + name: z + .string() + .describe( + 'Folder name — letters, digits, underscores or hyphens only. The folder becomes addressable as `f//`.' + ), + summary: z.string().optional().describe('Optional human-readable description of the folder.') +}) + +type FolderPromptContext = { folders: string[]; foldersRead: string[]; isAdmin: boolean } + +// Renders the folders the current user can act on into the system prompt so the +// model can pick an `f//...` path without a discovery round-trip (there +// is no folder-listing tool). For a non-admin, `folders` (from whoami) is exactly +// the writable set, so read-only folders are listed separately as off-limits. +// Admins bypass folder ACLs and can write anywhere, but `folders` only carries +// their explicitly-permissioned subset, so for admins it is offered as a +// non-exhaustive hint alongside permission-agnostic guidance (the complete set +// needs a folder-listing tool — follow-up). +// Capped so a folder-heavy workspace can't dominate the prompt. +function buildFolderGuidance(username: string, ctx?: FolderPromptContext): string { + if (!ctx) return '' + const MAX = 40 + const writable = ctx.folders ?? [] + const fmt = (names: string[]) => { + const shown = names + .slice(0, MAX) + .map((n) => `\`f/${n}\``) + .join(', ') + return names.length > MAX ? `${shown} (+${names.length - MAX} more)` : shown + } + if (ctx.isAdmin) { + const known = + writable.length > 0 + ? ` Folders here include ${fmt(writable)} (you can also write to others not listed).` + : '' + return `- As a workspace admin you can write to any existing folder.${known} If the user names a folder, use it; if they explicitly ask for a new folder, create it with \`create_folder\`; otherwise ask them which folder to use rather than guessing or creating one unprompted.` + } + const readOnly = (ctx.foldersRead ?? []).filter((f) => !writable.includes(f)) + const lines: string[] = [] + if (writable.length > 0) { + lines.push( + `- Folders you can write to in this workspace: ${fmt(writable)}. For shared/team work, pick the one whose purpose matches the request; if none clearly fits, ask which folder to use (askUserQuestion) rather than inventing a path. Use \`create_folder\` only when the user explicitly asks for a new folder.` + ) + } else { + lines.push( + `- You have no shared folders you can write to in this workspace, so use \`u/${username}/\`. If the user explicitly asks for a shared folder, create one with \`create_folder\` (you become an owner); otherwise ask before placing shared work rather than inventing an \`f//...\` path.` + ) + } + if (readOnly.length > 0) { + lines.push( + `- You can see but NOT write to: ${fmt(readOnly)} — never create or deploy items there.` + ) + } + return lines.join('\n') +} + const buildGlobalSystemPrompt = ( username: string, - previewTools: boolean -) => `You are Windmill's global workspace assistant. + previewTools: boolean, + folderCtx?: FolderPromptContext, + skills: AiSkillListItem[] = [] +) => { + const folderGuidance = buildFolderGuidance(username, folderCtx) + const folderGuidanceBlock = folderGuidance ? `\n${folderGuidance}` : '' + return `You are Windmill's global workspace assistant. The current user's workspace username is "${username}". Use tools to inspect workspace items and create per-user drafts (saved server-side, visible only to this user — not deployed) for scripts, flows, schedules, triggers, resources, variables, and raw apps. Path conventions: -- Every workspace path has exactly three segments and starts with one of two namespaces: - - \`u/${username}/\` — the current user's personal scope. Default for ad-hoc, exploratory, or scratch work. - - \`f//\` — a shared folder scope. The folder must already exist; bare \`f/\` is INVALID and will fail. -- When the user gives a bare name without a namespace prefix (e.g. "create a flow called myflow"), default to \`u/${username}/\`. Do NOT invent \`f/\` — that is a structurally invalid path. -- If the request implies shared / team work but doesn't name a specific folder (e.g. "the marketing flow"), ask which folder to use rather than guessing. Call \`list_workspace_items\` with \`type: ['folder']\` (or rely on the user's hint) before assuming a folder exists. -- Only use an \`f//\` path when the user explicitly named the folder or you confirmed it exists. +- A workspace path starts with one of two namespaces; its trailing may itself contain "/", so a path has three or more segments: + - \`u/${username}/\` — your personal scope. Default for ad-hoc, exploratory, or scratch work. + - \`f//\` — a shared folder scope; the must already exist (a bare \`f/\` with no folder segment is INVALID and will fail). +- If the user supplies a fully qualified \`f//...\` path, use that exact path; they have already chosen the folder. Do not ask for folder confirmation or substitute a \`u/${username}/...\` path unless a tool rejects it. +- Default a bare name with no namespace prefix (e.g. "create a flow called myflow") to \`u/${username}/\`. Never invent an \`f//...\` path for a folder that does not exist; create one with \`create_folder\` only when the user explicitly asks for a new folder.${folderGuidanceBlock} Rules: - Draft tools create or update drafts only; they do not deploy or mutate deployed workspace items. @@ -704,17 +907,22 @@ Rules: - Variable values are never readable. For secrets, create a secret variable and reference it from resources as "$var:path/to/variable". - Use search_resource_types before write_resource. - Use get_instructions before writing scripts, flows, resources, or apps. For scripts, pass the target language. +- A "data pipeline" is NOT a flow: it is a DAG of independent scripts in one folder, wired by storage assets (DuckLake/data tables/S3) and triggers via top-of-file \`pipeline\` / \`on \` annotation comments written in each script's comment syntax (\`--\` for SQL, \`#\` for Python/Bash, \`//\` for TS — a \`//\` line in a SQL node is a syntax error). When the user asks for a data pipeline (or to ingest/transform/materialize data across steps), call get_instructions with subject "pipeline" and build annotated script drafts — do not build a flow. - After creating or editing a script or flow draft, run test_run_script, test_run_flow, or test_run_step with representative args before reporting that it works. These tools prefer drafts, so testing does not require deployment. - Use list_runs to find recent runs (optionally filtered by path, creator, label, or status), then get_job_logs with a returned id to inspect a specific run's logs — without starting a new test run. -- When a required decision is ambiguous, use askUserQuestion with two to ten clear proposed answer strings instead of guessing. The user can also type a custom answer when none of the proposed answers fit. +- Use open_page to show a workspace page with filters applied — Runs, Schedules, Variables, Resources, Assets, Audit logs, or Workspace settings on a specific tab (e.g. "open the failed runs of f/foo/bar", "open the schedule for X", "open the git sync settings"). Only the pages listed for this user in the tool are available; don't offer pages that aren't listed. Don't use it as a substitute for list_runs when you just need the data yourself. +- When a required decision is ambiguous, use askUserQuestion with two to ten clear proposed answer strings instead of guessing. The user can also type a custom answer when none of the proposed answers fit. Set multiSelect: true only when the answers can genuinely co-apply and the user may pick several (not mutually exclusive). +- When the user asks you to remember a lasting preference, always/never do something, or change/stop a behavior going forward, call update_user_instructions to persist it. It edits only the USER INSTRUCTIONS block (not WORKSPACE INSTRUCTIONS). Keep each instruction concise; do not use it for one-off requests scoped to the current task. - Keep context targeted.${ - previewTools - ? ` + previewTools + ? ` - After writing or substantially editing a script / flow / app draft, show it via open_preview(kind, path) so the user sees the editor and live preview right next to the chat. First check whether it is already shown: if unsure, call get_preview_status. Only call open_preview (or offer to) when no preview is open or it is showing a different item — don't re-open a preview already showing the item you just edited. +- Building a data pipeline: call open_preview(kind="pipeline", path="") as the FIRST step, before creating any node — this opens the pipeline editor the user reviews in. path is the folder, not an item; an empty or not-yet-created folder is fine (create_folder first if needed, then open it). Opening it registers build_pipeline_node / edit_pipeline_node — use ONLY those to add or change pipeline nodes, never write_script for a pipeline node — they apply directly as unsaved drafts on the canvas (no separate accept/reject step) that the user reviews and deploys. Do not write pipeline scripts without first opening the editor. - When debugging a running raw app, call get_app_runtime_logs to read the live preview's browser console output. It needs the raw app preview open (open_preview kind="raw_app"). -- get_app_runtime_logs only shows the app's browser console. For the server-side logs of a backend runnable the app invoked (a backend. call), call list_app_runs to get that run's job_id from the live preview, then get_job_logs with it. Use this when a backend call errors or returns something unexpected.` - : '' -} +- get_app_runtime_logs only shows the app's browser console. For the server-side logs of a backend runnable the app invoked (a backend. call), call list_app_runs to get that run's job_id from the live preview, then get_job_logs with it. Use this when a backend call errors or returns something unexpected. +- open_page opens its page as a tab in the side-panel preview next to the chat — the only way to show one of these pages there (open_preview only handles editable items). Changing filters on a page already open updates that same tab; only pass new_tab when the user explicitly asks for a separate tab.` + : '' + } Documentation: - Use search_docs to look up how a Windmill feature works in the official documentation (a flag, concept, function, or "does Windmill support X") instead of guessing about product behavior. It returns matching doc snippets with their Source URL; call read_docs_page with a Source URL to read the full page (or a section, if it returns headings). Cite the Source URL when you rely on it. @@ -740,7 +948,17 @@ Data Tables: - Use list_datatables to discover the available datatables and their tables. Reuse an existing table rather than creating a duplicate. If list_datatables reports none, this is a blocking prerequisite — tell the user to set up a datatable in their workspace settings and stop; do not assume a "main" datatable exists or call exec_datatable_sql. - Use get_datatable_table_schema only when you need a table's column names/types; list_datatables is enough for table-list or availability summaries. - Use exec_datatable_sql to explore data, run queries, mutate rows, or change schema (CREATE/ALTER/DROP). Creating a table is a normal CREATE TABLE statement — it appears in list_datatables afterward, with no registration step. -- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) — it returns only that language so you get just what you need.` +- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) — it returns only that language so you get just what you need.${ + skills.length > 0 + ? ` + +Skills: +- Skills are reusable instruction sets curated for this workspace, each covering a specific kind of task. The available skills are listed below by name and description. +- When a user's request matches a skill's description, call read_skill with its exact name to load the full instructions BEFORE acting, then follow them. +${skills.map((s) => `- ${s.name}: ${s.description}`).join('\n')}` + : '' + }` +} const DEFAULT_LIST_TYPES = ['script', 'flow'] as const satisfies readonly WorkspaceItemType[] @@ -1476,6 +1694,10 @@ Datatables are workspace-scoped managed PostgreSQL databases. In chat, explore a ${getDatatableSdkReference(lang)}` } +function getPipelineInstructions(): string { + return getPipelinePrompt() +} + function getInstructions(subject: InstructionSubject, language?: ScriptLang): string { switch (subject) { case 'script': @@ -1488,15 +1710,384 @@ function getInstructions(subject: InstructionSubject, language?: ScriptLang): st return getAppInstructions() case 'datatable': return getDatatableInstructions(language) + case 'pipeline': + return getPipelineInstructions() + } +} + +export type AiSkillListItem = { name: string; description: string } + +/** Fetch the workspace's AI skills (name + description) for the global system prompt. */ +export async function loadWorkspaceSkills(workspace: string): Promise { + if (!workspace) return [] + try { + return await WorkspaceService.listAiSkills({ workspace }) + } catch (e) { + console.error('Failed to load AI skills', e) + return [] + } +} + +const readSkillSchema = z.object({ + name: z + .string() + .describe('The exact skill name as listed in the Skills section of the system prompt.') +}) + +export const readSkillTool: Tool<{}> = { + def: createToolDef( + readSkillSchema, + 'read_skill', + 'Load the full instructions for a workspace AI skill by name. Skills are listed in the system prompt under "Skills"; call this before acting on a task a skill covers, then follow its instructions.' + ), + fn: async ({ args, workspace, toolId, toolCallbacks }) => { + const parsed = readSkillSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Reading skill "${parsed.name}"...` }) + try { + const skill = await WorkspaceService.getAiSkill({ workspace, name: parsed.name }) + toolCallbacks.setToolStatus(toolId, { content: `Read skill "${parsed.name}"` }) + return `Skill: ${skill.name}\nDescription: ${skill.description}\n\nInstructions:\n${skill.instructions}` + } catch (e) { + const msg = e instanceof Error ? e.message : String(e) + toolCallbacks.setToolStatus(toolId, { + content: `Error reading skill "${parsed.name}"`, + error: msg + }) + return `Failed to read skill "${parsed.name}": ${msg}. Check the name against the Skills list in the system prompt.` + } + } +} + +const OPEN_PAGE_NAMES = [ + 'runs', + 'schedules', + 'variables', + 'resources', + 'assets', + 'audit_logs', + 'folders', + 'groups', + 'triggers', + 'workspace_settings' +] as const +type OpenPageName = (typeof OPEN_PAGE_NAMES)[number] + +const OPEN_PAGE_LABELS: Record = { + runs: 'Runs', + schedules: 'Schedules', + variables: 'Variables', + resources: 'Resources', + assets: 'Assets', + audit_logs: 'Audit logs', + folders: 'Folders', + groups: 'Groups', + triggers: 'Triggers', + workspace_settings: 'Workspace settings' +} + +// Trigger kinds available given the workspace's license — the EE-gated kinds +// (kafka/nats/sqs/gcp/azure) are only offered with an enterprise license. +function allowedTriggerKinds(): PageTriggerKind[] { + const ee = !!get(enterpriseLicense) + return (Object.keys(TRIGGER_PAGES) as PageTriggerKind[]).filter((k) => ee || !TRIGGER_PAGES[k].ee) +} + +// Which pages the current user can actually reach — mirrors the sidebar's gating. +// Operators see exactly the pages enabled in the operating workspace's operator_settings +// (the same source OperatorMenu gates on); a missing/false flag means no access, and +// operators are never admins so workspace_settings is always excluded. Non-operators +// get every page except workspace_settings, which is admin/superadmin only. `workspaceId` +// is the chat's operating workspace (a session targets its own, possibly forked workspace); +// it defaults to the navigation workspace for the global side-panel chat. The tool only +// ever advertises, and only ever acts on, pages in this set. +function allowedOpenPages(workspaceId: string | undefined = get(workspaceStore)): OpenPageName[] { + const u = get(userStore) + const isAdmin = !!u?.is_admin || !!get(superadmin) + if (u?.operator) { + const settings = get(userWorkspaces).find((w) => w.id === workspaceId)?.operator_settings + return OPEN_PAGE_NAMES.filter( + (p) => + p !== 'workspace_settings' && settings?.[p as keyof NonNullable] === true + ) + } + const allowed = new Set([ + 'runs', + 'assets', + 'schedules', + 'variables', + 'resources', + 'audit_logs', + 'folders', + 'groups', + 'triggers' + ]) + if (isAdmin) allowed.add('workspace_settings') + return OPEN_PAGE_NAMES.filter((p) => allowed.has(p)) +} + +// One flat object (not a discriminated union): `page` selects the target and the +// per-page fields are optional. Top-level `type: object` is what Anthropic's +// input_schema requires; a top-level oneOf would be rejected. Each per-page URL builder +// drops any key that isn't one of its page's real query params, so a field that doesn't +// apply to the chosen page is harmless. This full schema is used to PARSE tool args; the +// advertised schema (what the model sees) is narrowed per-user in `setSchema`. +const openPageFullSchema = z.object({ + page: z + .enum([ + 'runs', + 'schedules', + 'variables', + 'resources', + 'assets', + 'audit_logs', + 'folders', + 'groups', + 'triggers', + 'workspace_settings' + ]) + .describe('Which page to open'), + path: z + .string() + .optional() + .describe( + 'Runs/Schedules/Variables/Resources/Assets: the script, flow or item path to filter by' + ), + status: z + .enum(['running', 'success', 'failure', 'canceled', 'waiting', 'suspended']) + .optional() + .describe('Runs: filter by job execution status'), + schedule_path: z + .string() + .optional() + .describe( + 'Runs: only runs triggered by this schedule path. Schedules: filter by this exact schedule path.' + ), + job_kinds: z + .enum(['all', 'runs', 'dependencies', 'previews', 'deploymentcallbacks']) + .optional() + .describe('Runs: filter by job category (defaults to top-level runs)'), + user: z.string().optional().describe('Runs: filter by the user who created the job'), + open: z + .string() + .optional() + .describe( + 'Schedules/Triggers: exact schedule or trigger path to open in the edit drawer, e.g. f/foo/my_schedule' + ), + summary: z + .string() + .optional() + .describe('Schedules: search text matched against schedule summaries'), + trigger_kind: z + .enum([...(Object.keys(TRIGGER_PAGES) as [PageTriggerKind, ...PageTriggerKind[]])]) + .optional() + .describe('Triggers: which trigger kind page to open (http, websocket, postgres, kafka, ...)'), + resource_type: z + .string() + .optional() + .describe('Resources: filter by resource type, e.g. postgres'), + owner: z + .string() + .optional() + .describe('Variables/Resources: filter by owner, e.g. u/alice or f/team'), + username: z.string().optional().describe('Audit logs: filter by the acting username'), + operation: z.string().optional().describe('Audit logs: filter by operation, e.g. jobs.run'), + resource: z.string().optional().describe('Audit logs: filter by the affected resource path'), + tab: z + .enum([...WORKSPACE_SETTINGS_TABS] as [string, ...string[]]) + .optional() + .describe('Workspace settings: which settings tab to open'), + new_tab: z + .boolean() + .optional() + .describe( + 'Open in a NEW preview tab instead of updating the tab already showing this page. Only set true when the user explicitly asks for a new or separate tab; by default changing filters reuses the existing tab.' + ) +}) + +type OpenPageArgs = z.infer + +// Which pages each optional field applies to — drives narrowing the advertised schema +// so the model isn't shown fields for pages it can't open. +const OPEN_PAGE_FIELD_PAGES: Record = { + path: ['runs', 'schedules', 'variables', 'resources', 'assets'], + status: ['runs'], + schedule_path: ['runs', 'schedules'], + job_kinds: ['runs'], + user: ['runs'], + open: ['schedules', 'triggers'], + summary: ['schedules'], + trigger_kind: ['triggers'], + resource_type: ['resources'], + owner: ['variables', 'resources'], + username: ['audit_logs'], + operation: ['audit_logs'], + resource: ['audit_logs'], + tab: ['workspace_settings'] +} + +// The model-facing schema for the given allowed pages: the `page` enum plus only the +// fields relevant to those pages (reusing the full schema's field definitions). The +// `trigger_kind` enum is narrowed to the license-available kinds. +function buildOpenPageDefSchema( + pages: readonly OpenPageName[], + triggerKinds: readonly PageTriggerKind[] +): z.ZodTypeAny { + const full = openPageFullSchema.shape as Record + // z.enum() rejects an empty list, and a user with no reachable pages (e.g. an operator + // with every operator_settings flag off) yields exactly that. Fall back to a plain + // string so schema-building can't throw; the handler still fails closed on every page. + const shape: Record = { + page: pages.length + ? z.enum([...pages] as [OpenPageName, ...OpenPageName[]]).describe('Which page to open') + : z.string().describe('No pages are available to you in this workspace') + } + for (const [field, fieldPages] of Object.entries(OPEN_PAGE_FIELD_PAGES)) { + if (!fieldPages.some((p) => pages.includes(p))) continue + shape[field] = + field === 'trigger_kind' + ? z + .enum([...triggerKinds] as [string, ...string[]]) + .optional() + .describe('Triggers: which trigger kind page to open') + : full[field] + } + shape.new_tab = full.new_tab + return z.object(shape) +} + +const OPEN_PAGE_DESCRIPTION = + 'Open a Windmill page with filters applied — Runs, Schedules, Variables, Resources, Assets, Audit logs, Folders, Groups, Triggers (by kind), or Workspace settings (on a specific tab). Inside an AI session it opens as a tab in the side-panel preview next to the chat; elsewhere it offers a clickable link. Use after surfacing something the user likely wants to inspect (e.g. "show me the failed runs of X", "open the schedule for Y", "open the git sync settings", "open the kafka triggers"). This is the only way to show one of these pages in the session preview — open_preview only handles editable items (scripts, flows, raw apps, pipelines). Only pages listed for this user are available; do not offer others.' + +function buildOpenPageUrl(page: OpenPageName, a: OpenPageArgs): string { + switch (page) { + case 'runs': + return buildRunsUrl({ + status: a.status, + path: a.path, + schedule_path: a.schedule_path, + job_kinds: a.job_kinds, + user: a.user + }) + case 'schedules': + return buildSchedulesUrl({ + open: a.open, + filters: { path: a.path, schedule_path: a.schedule_path, summary: a.summary } + }) + case 'variables': + return buildVariablesUrl({ path: a.path, owner: a.owner }) + case 'resources': + return buildResourcesUrl({ path: a.path, resource_type: a.resource_type, owner: a.owner }) + case 'assets': + return buildAssetsUrl({ path: a.path }) + case 'audit_logs': + return buildAuditLogsUrl({ + username: a.username, + operation: a.operation, + resource: a.resource + }) + case 'folders': + return buildFoldersUrl() + case 'groups': + return buildGroupsUrl() + case 'triggers': + return buildTriggersUrl({ + // Default to HTTP routes when the model names no kind. + trigger_kind: (a.trigger_kind as PageTriggerKind | undefined) ?? 'http', + open: a.open + }) + case 'workspace_settings': + return buildWorkspaceSettingsUrl({ tab: a.tab }) + } +} + +// Human-readable one-liner for the tool status/chip: the applied query params (and any +// hash target), or "all " when unfiltered. +function summarizeOpenPage(url: string, page: OpenPageName): string { + const u = new URL(url, 'http://x') + const parts: string[] = [] + u.searchParams.forEach((v, k) => parts.push(`${k}=${v}`)) + if (u.hash) parts.push(u.hash.slice(1)) + return parts.length ? parts.join(', ') : `all ${OPEN_PAGE_LABELS[page].toLowerCase()}` +} + +export const openPageTool: Tool<{}> = { + def: createToolDef( + buildOpenPageDefSchema(allowedOpenPages(), allowedTriggerKinds()), + 'open_page', + OPEN_PAGE_DESCRIPTION + ), + // Keep the row expanded so the link chip (attached below as an action) is visible + // without the user having to expand the tool call. + showDetails: true, + autoCollapseDetails: false, + // Re-narrow the advertised `page` enum to this user's permissions each iteration, so + // the model never sees (or suggests) a page the user can't reach. Gates on the chat's + // operating workspace (the session's, when different from the navigation workspace). + setSchema: async function (helpers) { + this.def = createToolDef( + buildOpenPageDefSchema( + allowedOpenPages(operatingWorkspaceFromHelpers(helpers)), + allowedTriggerKinds() + ), + 'open_page', + OPEN_PAGE_DESCRIPTION + ) + }, + fn: async (ctx) => { + const { args, toolId, toolCallbacks } = ctx + const parsed = openPageFullSchema.parse(args) + const page = parsed.page as OpenPageName + const workspaceId = operatingWorkspaceFromHelpers(ctx.helpers) + // Defense in depth: never act on a page the user can't reach, even if the model + // requests one outside the advertised enum. + if (!allowedOpenPages(workspaceId).includes(page)) { + return `You don't have access to the ${OPEN_PAGE_LABELS[page] ?? page} page in this workspace.` + } + // Same fail-closed check for trigger_kind, which is narrowed to license-available + // kinds in the advertised schema: a model ignoring that narrowing must not get an + // EE-only trigger route built on a non-EE instance. + const triggerKind = parsed.trigger_kind as PageTriggerKind | undefined + if (page === 'triggers' && triggerKind && !allowedTriggerKinds().includes(triggerKind)) { + return `${TRIGGER_PAGES[triggerKind].label} aren't available on this instance.` + } + const url = buildOpenPageUrl(page, parsed) + const pageLabel = OPEN_PAGE_LABELS[page] + const summary = summarizeOpenPage(url, page) + + // In a session, show the page as a preview tab alongside the chat (the primary + // UX). By default a filter change reuses the tab already showing this page; + // new_tab forces a separate tab. openPagePreview returns undefined when there + // is no active session, in which case we offer a link chip instead. + const previewResult = openPagePreview({ + sessionId: sessionIdFromCtx(ctx), + href: pageHref(url), + label: pageLabel, + newTab: parsed.new_tab ?? false + }) + if (previewResult) { + toolCallbacks.setToolStatus(toolId, { content: `Opened ${pageLabel} preview: ${summary}` }) + return previewResult + } + + // Outside a session there is no preview panel — offer a clickable link the user + // controls. (We deliberately don't navigate in place: a same-route goto would + // not re-sync the page's URL-driven filter state, so the change wouldn't land.) + toolCallbacks.setToolStatus(toolId, { + content: `Prepared a link to ${pageLabel}: ${summary}`, + actions: [{ id: `open-page:${page}:${url}`, type: 'navigate', label: summary, url, page }] + }) + return `Offered the user a link to the ${pageLabel} page (${summary}). They can click it to open.` } } export const globalTools: Tool<{}>[] = [ + readSkillTool, + openPageTool, { def: createToolDef( getInstructionsSchema, 'get_instructions', - 'Get authoring guidance for scripts, flows, resources, apps, or the datatable SQL SDK (wmill.datatable()) used inside runnables.' + 'Get authoring guidance for scripts, flows, data pipelines, resources, apps, or the datatable SQL SDK (wmill.datatable()) used inside runnables.' ), fn: async ({ args, toolId, toolCallbacks }) => { const parsed = getInstructionsSchema.parse(args) @@ -1521,7 +2112,8 @@ export const globalTools: Tool<{}>[] = [ const parsed = askUserQuestionSchema.parse(args) const userQuestion = { question: parsed.question, - choices: parsed.choices + choices: parsed.choices, + multiSelect: parsed.multiSelect } toolCallbacks.setToolStatus(toolId, { @@ -1541,8 +2133,8 @@ export const globalTools: Tool<{}>[] = [ return JSON.stringify({ success: false, error: message }) } - const selectedChoice = await toolCallbacks.requestUserQuestion(toolId, userQuestion) - if (!selectedChoice) { + const selected = await toolCallbacks.requestUserQuestion(toolId, userQuestion) + if (!selected?.length) { const message = 'Question cancelled by user' toolCallbacks.setToolStatus(toolId, { content: message, @@ -1553,16 +2145,99 @@ export const globalTools: Tool<{}>[] = [ return JSON.stringify({ success: false, error: message }) } + // Model-facing answer: bare string for one pick (preserves the single-select + // contract, even when multiSelect was set), newline-bulleted list for several. + // Comma-joining is avoided here so a choice that itself contains a comma + // ("Yes, immediately") stays unambiguous to the model reading it back. + const answerText = + selected.length === 1 ? selected[0] : selected.map((c) => `- ${c}`).join('\n') + // The collapsed tool-header is a human glance, not model input, so the picks + // read as a compact comma list there instead of a stacked bullet list. + const answerSummary = selected.join(', ') + toolCallbacks.setToolStatus(toolId, { - content: `User answered question: ${selectedChoice}`, + content: `User answered question: ${answerSummary}`, userQuestion: { ...userQuestion, - selectedChoice + selectedChoices: selected }, - result: selectedChoice, + result: answerText, isLoading: false }) - return selectedChoice + return answerText + } + }, + { + def: createToolDef( + updateUserInstructionsSchema, + 'update_user_instructions', + 'Modify your own persistent personal instructions for the Global assistant. Use when the user asks you to remember a preference, always/never do something, or change/stop a behavior. Edits only the user-level instructions (the USER INSTRUCTIONS block, not workspace-level); changes persist in this browser and take effect on your next message.' + ), + showDetails: true, + fn: async ({ args, toolId, toolCallbacks, helpers }) => { + const parsed = updateUserInstructionsSchema.parse(args) + const h = helpers as GlobalToolHelpers + if (!h?.getUserInstructions || !h?.setUserInstructions) { + const message = 'This chat context cannot modify user instructions.' + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + + const current = h.getUserInstructions() + let next: string + if (parsed.operation === 'append') { + const text = parsed.text?.trim() + if (!text) { + const message = "operation 'append' requires a non-empty text." + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + next = current.trim() ? `${current.trim()}\n\n${text}` : text + } else { + if (parsed.old_string === undefined || parsed.new_string === undefined) { + const message = "operation 'replace' requires old_string and new_string." + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + try { + next = findAndReplace( + current, + parsed.old_string, + parsed.new_string, + parsed.replace_all ?? false, + 'personal instructions' + ).trim() + } catch (e) { + const detail = e instanceof Error ? e.message : String(e) + // Echo the current text on this rare recovery path so the model can rebuild a + // correct old_string. The system prompt's USER INSTRUCTIONS block can be stale + // within a turn that already made a successful edit, but `current` is always live. + const hint = current.trim() + ? `Your current personal instructions are:\n${current.trim()}` + : "You have no personal instructions yet; use operation 'append' to add one." + const message = `${detail} ${hint}` + toolCallbacks.setToolStatus(toolId, { content: detail, error: detail }) + return message + } + } + + if (next.length > MAX_USER_INSTRUCTIONS_LENGTH) { + const message = `Resulting instructions would be ${next.length} characters, over the ${MAX_USER_INSTRUCTIONS_LENGTH} limit. Make them more concise.` + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + + h.setUserInstructions(next) + + const summary = next + ? parsed.operation === 'append' + ? 'Added a personal instruction' + : 'Updated your personal instructions' + : 'Cleared your personal instructions' + toolCallbacks.setToolStatus(toolId, { content: summary, result: summary }) + // Return only a short confirmation. The updated text is re-injected into the system + // prompt on the next iteration, so echoing it back here would waste context. + return `${summary}. It takes effect from your next message and is editable in the Global chat settings.` } }, { @@ -1636,6 +2311,46 @@ export const globalTools: Tool<{}>[] = [ return JSON.stringify(serializeWorkspaceItemForRead(item), null, 2) } }, + { + def: createToolDef( + createFolderSchema, + 'create_folder', + 'Create a new shared folder (addressable as f//) in the workspace. The current user is added as an owner.' + ), + requiresConfirmation: true, + confirmationMessage: 'Create folder', + showDetails: true, + fn: async ({ args, workspace, toolId, toolCallbacks }) => { + const parsed = createFolderSchema.parse(args) + if (!VALID_FOLDER_NAME.test(parsed.name)) { + const error = + 'Folder name can only contain alphanumeric characters, underscores, and hyphens.' + toolCallbacks.setToolStatus(toolId, { content: error, error }) + return JSON.stringify({ success: false, error }) + } + toolCallbacks.setToolStatus(toolId, { content: `Creating folder \`f/${parsed.name}\`...` }) + try { + await FolderService.createFolder({ + workspace, + requestBody: { name: parsed.name, summary: parsed.summary } + }) + // Reflect the new folder in the path-convention context for the rest of this + // session, matching FolderPicker's local update (avoids userStore.set()). + const user = get(userStore) + if (user) { + if (!user.folders) user.folders = [] + if (!user.folders.includes(parsed.name)) user.folders.push(parsed.name) + } + const message = `Created folder \`f/${parsed.name}\`. You can now write items to \`f/${parsed.name}/\`.` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ success: true, message }) + } catch (e) { + const error = e instanceof Error ? e.message : String(e) + toolCallbacks.setToolStatus(toolId, { content: `Error: ${error}`, error }) + return JSON.stringify({ success: false, error }) + } + } + }, { def: createToolDef(writeScriptSchema, 'write_script', 'Create or overwrite a draft script.'), showDetails: true, @@ -1824,6 +2539,33 @@ export const globalTools: Tool<{}>[] = [ return result } }, + { + def: createToolDef( + cancelJobSchema, + 'cancel_job', + 'Cancel a running or queued job by its id (e.g. a background test run you started that is no longer needed).' + ), + showDetails: true, + fn: async ({ args, workspace, toolId, toolCallbacks }) => { + const parsed = cancelJobSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Canceling job ${parsed.id}...` }) + try { + await JobService.cancelQueuedJob({ workspace, id: parsed.id, requestBody: {} }) + } catch (e) { + const msg = e instanceof Error ? e.message : 'Unknown error' + toolCallbacks.setToolStatus(toolId, { + content: `Could not cancel job ${parsed.id}`, + error: msg + }) + return `Failed to cancel job ${parsed.id}: ${msg}. It may have already finished.` + } + // The tray's background poller will pick up the canceled state (updating + // its status + Job snapshot together); a bare status patch here would leave + // the badge stale and stop the poller. Just report to the model. + toolCallbacks.setToolStatus(toolId, { content: `Canceled job ${parsed.id}` }) + return `Job ${parsed.id} was canceled.` + } + }, { def: createToolDef( deployWorkspaceItemSchema, @@ -1840,6 +2582,20 @@ export const globalTools: Tool<{}>[] = [ return deployDraft(parsed, { ...ctx, sessionId: sessionIdFromCtx(ctx) }) } }, + { + def: createToolDef( + rebaseDraftSchema, + 'rebase_draft', + 'Discard a stale script, flow, or app draft and return your changes as a diff to re-apply on the latest deployed version. Use when deploy_workspace_item reports the draft was started from an older deployed version.', + { strict: false } + ), + showDetails: true, + showFade: true, + fn: async (ctx) => { + const parsed = rebaseDraftSchema.parse(ctx.args) + return rebaseDraft(parsed, ctx) + } + }, { def: createToolDef( deleteWorkspaceItemSchema, @@ -2074,6 +2830,17 @@ export const globalTools: Tool<{}>[] = [ ), fn: async (ctx) => getSessionPreviewStatus(sessionIdFromCtx(ctx)) }, + { + def: createToolDef( + closePageSchema, + 'close_page', + 'Close one or more preview tabs in the side panel of this AI session. Pass `match` to close the tab(s) whose page name or item path contains that text, or `all: true` to clear the panel. Use this when the user asks to close/dismiss a tab they no longer need. Only works inside a session.' + ), + fn: async (ctx) => { + const parsed = closePageSchema.parse(ctx.args) + return closeSessionPreviewTabs(parsed, sessionIdFromCtx(ctx)) + } + }, { def: createToolDef( getRuntimeLogsSchema, @@ -2112,7 +2879,9 @@ export const globalTools: Tool<{}>[] = [ } }, // Workspace-scoped datatable tools (unrestricted: no whitelist, no creation policy) - ...getDatatableTools() + ...getDatatableTools(), + // Read-only tools over files the user attached to the conversation + ...fileTools ] // Tools that only make sense inside an AI session (they drive the session's @@ -2121,6 +2890,7 @@ export const globalTools: Tool<{}>[] = [ export const SESSION_PREVIEW_TOOL_NAMES = new Set([ 'open_preview', 'get_preview_status', + 'close_page', 'get_app_runtime_logs', 'list_app_runs' ]) @@ -2160,12 +2930,26 @@ export type SessionToolHelpers = { sessionId?: string } export type GlobalToolHelpers = SessionToolHelpers & { testActiveFlow?: (args?: Record) => Promise + attachedFiles?: AttachedFilesStore + // Read/write the user-level Global instructions. `setUserInstructions` persists the + // value and rebuilds the system message so the change applies on the next chat-loop + // iteration. Backed by the update_user_instructions tool. + getUserInstructions?: () => string + setUserInstructions?: (instructions: string) => void + // The workspace this chat actually operates on — a session chat targets its own + // (possibly forked) workspace while $workspaceStore stays on the navigation workspace, + // so permission gating (open_page) must read this, not the global store. + operatingWorkspace?: string } function sessionIdFromCtx(ctx: { helpers?: unknown }): string | undefined { return (ctx.helpers as GlobalToolHelpers | undefined)?.sessionId } +function operatingWorkspaceFromHelpers(helpers: unknown): string | undefined { + return (helpers as GlobalToolHelpers | undefined)?.operatingWorkspace +} + function activeFlowTestFromCtx( ctx: { workspace: string; helpers?: unknown }, path: string @@ -2179,7 +2963,7 @@ function activeFlowTestFromCtx( export type OpenPreviewHandler = (req: { sessionId: string | undefined - kind: 'script' | 'flow' | 'raw_app' + kind: 'script' | 'flow' | 'raw_app' | 'pipeline' path: string }) => string @@ -2190,7 +2974,7 @@ export function setOpenPreviewHandler(handler: OpenPreviewHandler | undefined): } function openSessionPreview( - args: { kind: 'script' | 'flow' | 'raw_app'; path: string }, + args: { kind: 'script' | 'flow' | 'raw_app' | 'pipeline'; path: string }, sessionId: string | undefined ) { if (!openPreviewHandler) { @@ -2199,6 +2983,34 @@ function openSessionPreview( return openPreviewHandler({ ...args, sessionId }) } +// Opens a workspace *page* (Runs, Schedules, …) as a page tab in the session's +// side-panel preview, so open_page can show it next to the chat instead of +// navigating the whole browser. Registered by the session runtime. Returns a +// status string when opened in a session, or undefined when there is no active +// session — signalling open_page to fall back to a link chip / direct navigation. +export type OpenPagePreviewHandler = (req: { + sessionId: string | undefined + href: string + label: string + // Force a separate tab instead of reusing the tab already showing this page. + newTab: boolean +}) => string | undefined + +let openPagePreviewHandler: OpenPagePreviewHandler | undefined + +export function setOpenPagePreviewHandler(handler: OpenPagePreviewHandler | undefined): void { + openPagePreviewHandler = handler +} + +function openPagePreview(req: { + sessionId: string | undefined + href: string + label: string + newTab: boolean +}): string | undefined { + return openPagePreviewHandler?.(req) +} + // Companion to `open_preview`: lets the assistant query the current preview // state (open? which item?) so it can avoid re-opening a preview already // showing the item it just edited. Registered by the session runtime @@ -2218,6 +3030,35 @@ function getSessionPreviewStatus(sessionId: string | undefined): string { return getPreviewStatusHandler(sessionId) } +// Closes preview tabs in the calling session's side panel. Registered by the +// session runtime, which owns the tab model. Returns a status string the model +// relays to the user. `all` clears the panel; otherwise `match` is a +// case-insensitive substring tested against each tab's page label / item path. +export type ClosePreviewTabsHandler = (req: { + sessionId: string | undefined + all: boolean + match: string | undefined +}) => string + +let closePreviewTabsHandler: ClosePreviewTabsHandler | undefined + +export function setClosePreviewTabsHandler(handler: ClosePreviewTabsHandler | undefined): void { + closePreviewTabsHandler = handler +} + +function closeSessionPreviewTabs( + args: { all?: boolean; match?: string }, + sessionId: string | undefined +): string { + if (!closePreviewTabsHandler) { + return 'Error: close_page is only available inside an AI session.' + } + if (!args.all && !args.match?.trim()) { + return 'Nothing to close: pass `match` with the tab to close, or `all: true` to close every tab.' + } + return closePreviewTabsHandler({ sessionId, all: args.all ?? false, match: args.match }) +} + export type GetRuntimeLogsHandler = (req: { sessionId: string | undefined limit: number @@ -2453,6 +3294,7 @@ function finishAppDraftWrite( ): string { const failure = draftWriteFailure(result, ctx) if (failure) return failure + ctx.toolCallbacks.onItemModified?.(result.itemKind, result.storagePath) const { content, message } = onSaved() ctx.toolCallbacks.setToolStatus(ctx.toolId, { content, result: 'Saved as draft' }) return JSON.stringify({ success: true, message }, null, 2) @@ -2465,6 +3307,7 @@ function finishDraftWrite( ): string { const failure = draftWriteFailure(result, ctx) if (failure) return failure + ctx.toolCallbacks.onItemModified?.(result.itemKind, result.storagePath) const stored = result.item const verb = existed ? 'Updated' : 'Created' // Don't echo the flow value back: the model just sent it in the write call, @@ -2500,7 +3343,7 @@ function finishDraftWrite( type WriteSpec = { probe: (workspace: string, path: string) => Promise fetchDeployed: (workspace: string, path: string) => Promise - buildDraft: (base: T | undefined, args: A, path: string) => T + buildDraft: (base: T | undefined, args: A, path: string) => T | Promise beforePersist?: (workspace: string, args: A) => void } @@ -2523,7 +3366,7 @@ async function writeDraft( existed = true } - const draft = spec.buildDraft(base, args, path) + const draft = await spec.buildDraft(base, args, path) spec.beforePersist?.(workspace, args) const result = await persistGlobalDraft(workspace, type, path, draft, { @@ -2547,8 +3390,8 @@ const SCRIPT_SPEC: WriteSpec = { const existing = await ScriptService.getScriptByPath({ workspace, path }) return { ...(existing as unknown as NewScript), parent_hash: existing.hash } }, - buildDraft: (base, args, path) => - base + buildDraft: async (base, args, path) => { + const draft: NewScript = base ? { ...structuredClone(base), path, @@ -2566,6 +3409,18 @@ const SCRIPT_SPEC: WriteSpec = { language: args.language, kind: 'script' } + // Infer the arg schema from the content at save time, like the editor does, + // so the persisted draft is the single source of truth at deploy. Keep the + // previous schema (or empty) on failure rather than blanking it. + try { + const schema = emptySchema() + await inferArgs(draft.language, draft.content, schema) + draft.schema = schema + } catch (e) { + console.error('Failed to infer script schema before saving draft', e) + } + return draft + } } function writeScriptDraft(args: ScriptDraftArgs, ctx: WriteDraftCtx): Promise { @@ -2898,7 +3753,10 @@ async function testRunScriptByPath( toolCallbacks, toolId, startMessage: `Running test for script "${args.path}"...`, - contextName: 'script' + contextName: 'script', + background: args.background, + detachAfterMs: waitSecondsToDetachMs(args.wait_seconds), + label: args.path }) } @@ -2932,7 +3790,10 @@ async function testRunFlowByPath( toolCallbacks, toolId, startMessage: `Starting flow test run for "${args.path}"...`, - contextName: 'flow' + contextName: 'flow', + background: args.background, + detachAfterMs: waitSecondsToDetachMs(args.wait_seconds), + label: args.path }) } @@ -2952,7 +3813,10 @@ async function testRunFlowByPath( toolCallbacks, toolId, startMessage: `Starting flow test run for "${args.path}"...`, - contextName: 'flow' + contextName: 'flow', + background: args.background, + detachAfterMs: waitSecondsToDetachMs(args.wait_seconds), + label: args.path }) } @@ -2972,6 +3836,8 @@ async function testRunFlowStepByPath( workspace, toolCallbacks, toolId, + background: args.background, + detachAfterMs: waitSecondsToDetachMs(args.wait_seconds), loadScript: loadScriptForFlowStep, loadFlowPreviewValue: loadDraftFlowPreviewValue }) @@ -3253,7 +4119,9 @@ async function searchApp( const header = `${totalMatchCount} match${ totalMatchCount === 1 ? '' : 'es' } in ${fileCount} file${fileCount === 1 ? '' : 's'}${ - truncated ? ` (showing the first ${maxMatches}; narrow with file_glob or a more specific query)` : '' + truncated + ? ` (showing the first ${maxMatches}; narrow with file_glob or a more specific query)` + : '' }` const out: string[] = [header] @@ -3558,6 +4426,16 @@ async function discardLocalDraft( await deleteGlobalDraft(workspace, type, path, triggerKind) + // The chat's touch on the item is undone — drop it from the mask so a + // pre-existing deployed item doesn't keep reading as this chat's edit. + const discardedKind = itemKindFor(type, triggerKind) + if (discardedKind) { + toolCallbacks.onItemDiscarded?.( + discardedKind, + getGlobalDraftStoragePath(workspace, type, path, triggerKind) + ) + } + toolCallbacks.setToolStatus(toolId, { content: `Discarded ${type} "${path}" draft`, result: 'Draft discarded' @@ -3575,17 +4453,310 @@ async function discardLocalDraft( ) } +// A draft started from an older deploy would silently overwrite whatever was +// deployed since. Block the deploy and point the model at rebase_draft, unless it +// explicitly forces the overwrite. `base`/`head` undefined ⇒ can't tell ⇒ allow. +function assertDraftBasedOnLatest( + type: WorkspaceItemType, + path: string, + base: string | number | undefined, + head: string | number | undefined, + force: boolean | undefined +): void { + if (force || base == null || head == null || base === head) return + throw new Error( + `This ${type} draft "${path}" was started from an older deployed version (forked from ${base}, ` + + `latest is ${head}). Deploying now would overwrite the version deployed since. Call rebase_draft to ` + + `discard the stale draft and get your changes back as a diff, then re-apply them (the new draft ` + + `re-bases onto the latest version) and deploy. To deploy as-is and replace the newer version, call ` + + `deploy_workspace_item again with force: true.` + ) +} + +// Discard a stale draft and return its own changes (vs the fork base) as a diff so +// the model can re-apply them on the latest deploy. Discarding rather than resetting +// keeps the base pointer honest (the next write re-bases on the current head) and +// fails safe: a premature deploy hits "no draft" instead of silently shipping the +// latest unchanged. +async function rebaseDraft( + args: { type: WorkspaceItemType; path: string }, + ctx: WriteDraftCtx +): Promise { + switch (args.type) { + case 'script': + return rebaseScriptDraft(args.path, ctx) + case 'flow': + return rebaseFlowDraft(args.path, ctx) + case 'app': + return rebaseAppDraft(args.path, ctx) + default: + throw new Error('rebase_draft currently supports scripts, flows, and apps.') + } +} + +async function rebaseScriptDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'script', path) + if (!draft || typeof draft.value !== 'string' || !draft.language) { + throw new Error(`No script draft found for "${path}".`) + } + if (!(await ScriptService.existsScriptByPath({ workspace, path }))) { + throw new Error(`Script "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const latest = await ScriptService.getScriptByPath({ workspace, path }) + const baseHash = draft.parentHash + if (baseHash && baseHash === latest.hash) { + const message = `Draft "${path}" is already based on the latest deployed version (${latest.hash}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ success: true, alreadyLatest: true, latest_hash: latest.hash, message }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // Capture the draft's own changes (vs its fork base) BEFORE discarding — this + // diff is the only clean record of what to replay. Best-effort: if the base + // version is gone, diff against empty so the full draft is surfaced. + let baseContent = '' + if (baseHash) { + try { + baseContent = (await ScriptService.getScriptByHash({ workspace, hash: baseHash })).content + } catch (e) { + console.error(`rebase_draft: could not fetch base version ${baseHash} for "${path}"`, e) + } + } + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + baseContent, + draft.value, + '', + '' + ) + + // Discard the stale draft rather than resetting it to latest: the next write + // re-bases on the current head, and a premature deploy fails cleanly ("no + // draft") instead of silently shipping the latest unchanged and losing the work. + await deleteGlobalDraft(workspace, 'script', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a diff against the ` + + `version you forked from). Re-apply them with edit_script / write_script — the new draft will be ` + + `based on the latest deployed version (hash ${latest.hash}) — then deploy.`, + latest_hash: latest.hash, + your_changes: yourChanges + }, + null, + 2 + ) +} + +async function rebaseFlowDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'flow', path) + if (!draft || draft.value === undefined || typeof draft.value === 'string') { + throw new Error(`No flow draft found for "${path}".`) + } + if (!(await FlowService.existsFlowByPath({ workspace, path }))) { + throw new Error(`Flow "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const latest = await FlowService.getFlowByPath({ workspace, path }) + const baseVersion = draft.parentVersionId + if (baseVersion != null && baseVersion === latest.version_id) { + const message = `Draft "${path}" is already based on the latest deployed version (${latest.version_id}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ + success: true, + alreadyLatest: true, + latest_version: latest.version_id, + message + }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // The draft's own changes vs its fork-base flow value, as a JSON diff for the + // model to replay. Best-effort: skip if the base version can't be fetched. + let baseValue: unknown = {} + if (baseVersion != null) { + try { + baseValue = (await FlowService.getFlowVersion({ workspace, version: baseVersion })).value + } catch (e) { + console.error( + `rebase_draft: could not fetch base flow version ${baseVersion} for "${path}"`, + e + ) + } + } + const oursValue = (draft.value as FlowDraftValue).value + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + JSON.stringify(baseValue, null, 2), + JSON.stringify(oursValue, null, 2), + '', + '' + ) + + // Discard the stale draft (see rebaseScriptDraft): the next write re-bases on + // the current head, and a premature deploy fails cleanly instead of shipping + // the latest unchanged. + await deleteGlobalDraft(workspace, 'flow', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a JSON diff against ` + + `the version you forked from). Re-apply them with the flow edit tools — the new draft will be ` + + `based on the latest deployed version (version ${latest.version_id}) — then deploy.`, + latest_version: latest.version_id, + your_changes: yourChanges + }, + null, + 2 + ) +} + +async function rebaseAppDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'app', path) + if (!draft || !draft.value || typeof draft.value === 'string' || !('files' in draft.value)) { + throw new Error(`No app draft found for "${path}".`) + } + if (!(await AppService.existsApp({ workspace, path }))) { + throw new Error(`App "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const deployed = await AppService.getAppByPath({ workspace, path }) + const headVersion = deployed.versions?.[deployed.versions.length - 1] + const baseVersion = draft.parentVersionId + if (baseVersion != null && baseVersion === headVersion) { + const message = `Draft "${path}" is already based on the latest deployed version (${headVersion}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ + success: true, + alreadyLatest: true, + latest_version: headVersion, + message + }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // The draft's own changes vs its fork-base app source, as a JSON diff for the + // model to replay. Best-effort: skip if the base version can't be fetched. + const oursValue = draft.value as AppDraftValue + let baseSource: Pick = { + files: {}, + runnables: {}, + data: undefined + } + if (baseVersion != null) { + try { + const baseApp = await AppService.getAppByVersion({ workspace, id: baseVersion }) + const base = appSourceToDraftValue(baseApp, baseApp) + baseSource = { files: base.files, runnables: base.runnables, data: base.data } + } catch (e) { + console.error( + `rebase_draft: could not fetch base app version ${baseVersion} for "${path}"`, + e + ) + } + } + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + JSON.stringify(baseSource, null, 2), + JSON.stringify( + { files: oursValue.files, runnables: oursValue.runnables, data: oursValue.data }, + null, + 2 + ), + '', + '' + ) + + // Discard the stale draft (see rebaseScriptDraft): the next write re-projects + // the deployed app into a fresh draft (re-pinning parent_version to the head), + // and a premature deploy fails cleanly instead of shipping the latest unchanged. + await deleteGlobalDraft(workspace, 'app', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a JSON diff against ` + + `the version you forked from). Re-apply them with the app edit tools — the new draft will be based ` + + `on the latest deployed version (version ${headVersion}) — then deploy.`, + latest_version: headVersion, + your_changes: yourChanges + }, + null, + 2 + ) +} + +// Flush a draft's pending editor autosave, then verify it actually landed before +// the caller re-reads the persisted draft. `flush()` resolves even when the save +// recorded a conflict (server has a newer version) or failed (network/5xx) — it +// does not throw — so without this check a deploy could publish a stale/conflicting +// draft. Abort with a clear message instead. +async function flushDraftOrThrow( + query: Parameters[0], + label: string +): Promise { + await UserDraftDbSyncer.flush(query) + if (UserDraftDbSyncer.getConflict(query).conflict) { + throw new Error( + `Cannot deploy ${label}: the draft has a conflicting newer version on the server. Open it in the editor and resolve the conflict first.` + ) + } + const { state, failureMessage } = UserDraftDbSyncer.getState(query) + if (state === 'failed') { + throw new Error( + `Cannot deploy ${label}: saving the latest draft failed (${failureMessage ?? 'unknown error'}). Retry once the draft saves.` + ) + } +} + async function deployDraft( args: { type: WorkspaceItemType path: string trigger_kind?: TriggerKind deployment_message?: string + force?: boolean }, ctx: WriteDraftCtx ): Promise { const { workspace, toolId, toolCallbacks, sessionId } = ctx - const { type, path, trigger_kind: triggerKind, deployment_message: deploymentMessage } = args + const { + type, + path, + trigger_kind: triggerKind, + deployment_message: deploymentMessage, + force + } = args if (type === 'trigger' && !triggerKind) { throw new Error('trigger_kind is required when deploying a trigger.') @@ -3604,175 +4775,263 @@ async function deployDraft( }) let actions: ToolDisplayAction[] | undefined + // Where the deploy actually lands — the app branch can resolve a different + // target from the draft's own path fields; the mask rename below must track it. + let deployedPath = path - switch (type) { - case 'script': { + if (type === 'script' || type === 'flow') { + // Promote the full persisted draft via the shared deploy module — the same + // "promote a draft to deployed" code the compare page's Review & Deploy uses. + // It deploys every field of the draft; the previous local builders dropped + // most config fields (tag, priority, schema, description, concurrency…), + // reading them from the already-deployed version instead. The other kinds + // below already deploy their draft value directly, so only script/flow need + // this. Scripts always create (with parent_hash); a flow on a deployed item + // updates, a draft-only flow (no flow row) is created. + // Address the draft by its STORAGE path: a draft_only item created in the + // editor lives at a synthetic `u/{user}/draft_{uuid}` key while its chosen + // path is held in the draft value. The shared deployer reads the draft via + // getScriptByPath/getFlowByPath at the path we pass (then deploys at the + // draft's own `path`), so passing the display/chosen path would 404. For a + // draft on a deployed item the storage path is just the item path. + const storagePath = getGlobalDraftStoragePath(workspace, type, path, triggerKind) + // The shared deployer re-reads the persisted DB draft, but an open editor's + // edit may still be parked in a debounced/disabled autosave. Flush it first so + // we deploy the latest value (not a stale persisted one) — and so the + // post-deploy draft delete doesn't drop an unsaved edit. `flush` always saves + // the parked value (like Ctrl/Cmd+S), since the user explicitly asked to deploy. + await flushDraftOrThrow({ workspace, itemKind: type, path: storagePath }, `${type} "${path}"`) + // Stale-draft guard: block when the draft was forked from an older deploy than + // the current head (unless force), pointing the model at rebase_draft. + if (type === 'script') { const existing = (await ScriptService.existsScriptByPath({ workspace, path })) ? await ScriptService.getScriptByPath({ workspace, path }) : undefined - const requestBody = buildScriptDeployRequestBody(path, draft, existing, deploymentMessage) - // Infer the arg schema from the content so it matches the code, like the editor does. - try { - const schema = emptySchema() - await inferArgs(requestBody.language, requestBody.content, schema) - requestBody.schema = schema - } catch (e) { - console.error('Failed to infer script schema before deploy', e) - } - await ScriptService.createScript({ workspace, requestBody }) - break - } - case 'flow': { - const flowDraft = draft.value as FlowDraftValue + assertDraftBasedOnLatest('script', path, draft.parentHash, existing?.hash, force) + } else { const existing = (await FlowService.existsFlowByPath({ workspace, path })) ? await FlowService.getFlowByPath({ workspace, path }) : undefined - const requestBody = buildFlowDeployRequestBody( - path, - draft.summary, - flowDraft, - existing, - deploymentMessage - ) - if (existing) { - await FlowService.updateFlow({ workspace, path, requestBody }) - } else { - await FlowService.createFlow({ workspace, requestBody }) - } - break + assertDraftBasedOnLatest('flow', path, draft.parentVersionId, existing?.version_id, force) } - case 'schedule': { - const requestBody = draft.value as any - if (await ScheduleService.existsSchedule({ workspace, path })) { - await ScheduleService.updateSchedule({ workspace, path, requestBody }) - } else { - await ScheduleService.createSchedule({ workspace, requestBody }) - } - actions = [createOpenScheduleAction(path, requestBody.is_flow ? 'flow' : 'script')] - break + const draftOnly = + type === 'flow' + ? !(await FlowService.existsFlowByPath({ workspace, path: storagePath })) + : false + const result = await deployDraftToWorkspace(type, storagePath, workspace, { + draftOnly, + deploymentMessage + }) + if (!result.success) { + throw new Error(result.error ?? `Failed to deploy ${type} "${path}".`) } - case 'trigger': { - const service = triggerServices[triggerKind!] - const requestBody = draft.value as { is_flow?: boolean } - if (await service.exists({ workspace, path })) { - await service.update({ workspace, path, requestBody }) - } else { - await service.create({ workspace, requestBody }) - } - actions = [ - createOpenTriggerAction(triggerKind!, path, requestBody.is_flow ? 'flow' : 'script') - ] - break - } - case 'resource': { - const requestBody = draft.value as any - if (await ResourceService.existsResource({ workspace, path })) { - await ResourceService.updateResource({ workspace, path, requestBody }) - } else { - await ResourceService.createResource({ workspace, requestBody }) - } - actions = [createOpenResourceAction(path)] - break - } - case 'variable': { - const requestBody = buildVariableDeployRequestBody( - workspace, - path, - draft.value as CreateVariable - ) - if (await VariableService.existsVariable({ workspace, path })) { - await VariableService.updateVariable({ workspace, path, requestBody }) - } else { - await VariableService.createVariable({ workspace, requestBody }) - } - actions = [createOpenVariableAction(path)] - break - } - case 'app': { - const appDraft = draft.value as AppDraftValue - const appValue: AppDraftValue = { - ...appDraft, - files: { ...(appDraft.files ?? {}) }, - runnables: { ...(appDraft.runnables ?? {}) }, - data: appDraft.data ?? { ...DEFAULT_RAW_APP_DATA } - } - await recomputeAppPolicy(appValue) - const policy = appValue.policy - if (!policy) { - throw new Error(`Draft app "${path}" has no policy to deploy.`) - } - - toolCallbacks.setToolStatus(toolId, { - content: `Bundling app "${path}"...` - }) - const bundle = await bundleRawAppDraft({ - workspace, - files: appValue.files, - onLog: (delta) => { - const lines = delta - .split('\n') - .map((line) => line.trim()) - .filter(Boolean) - const latest = lines[lines.length - 1] - if (latest) { - toolCallbacks.setToolStatus(toolId, { - content: `Bundling app "${path}"... ${latest}` - }) - } + } else { + switch (type) { + case 'schedule': { + const requestBody = draft.value as any + if (await ScheduleService.existsSchedule({ workspace, path })) { + await ScheduleService.updateSchedule({ workspace, path, requestBody }) + } else { + await ScheduleService.createSchedule({ workspace, requestBody }) } - }) - - toolCallbacks.setToolStatus(toolId, { - content: `Deploying app "${path}"...` - }) - const rawAppValue = { - files: appValue.files, - runnables: appValue.runnables, - data: appValue.data ?? { ...DEFAULT_RAW_APP_DATA } + actions = [createOpenScheduleAction(path, requestBody.is_flow ? 'flow' : 'script')] + break } - const summary = appValue.summary ?? draft.summary ?? '' - if (await AppService.existsApp({ workspace, path })) { - // Omit custom_path on update for now. The backend preserves it when absent, while - // sending it requires admin privileges; this chat deploy path does not yet mirror - // the raw app editor's user/admin-specific custom_path handling. - await AppService.updateAppRaw({ + case 'trigger': { + const service = triggerServices[triggerKind!] + const requestBody = draft.value as { is_flow?: boolean } + if (await service.exists({ workspace, path })) { + await service.update({ workspace, path, requestBody }) + } else { + await service.create({ workspace, requestBody }) + } + actions = [ + createOpenTriggerAction(triggerKind!, path, requestBody.is_flow ? 'flow' : 'script') + ] + break + } + case 'resource': { + const requestBody = draft.value as any + if (await ResourceService.existsResource({ workspace, path })) { + await ResourceService.updateResource({ workspace, path, requestBody }) + } else { + await ResourceService.createResource({ workspace, requestBody }) + } + actions = [createOpenResourceAction(path)] + break + } + case 'variable': { + // The chat keeps secret draft values only in memory (the DB draft + // stores `''`); buildVariableDeployRequestBody re-injects the ephemeral + // secret, so this can't go through the DB-reading shared deployer. + const requestBody = buildVariableDeployRequestBody( workspace, path, - formData: { - app: { - path, - value: rawAppValue, - summary, - policy, - deployment_message: deploymentMessage - }, - js: bundle.js, - css: bundle.css - } - }) - } else { - await AppService.createAppRaw({ - workspace, - formData: { - app: { - path, - value: rawAppValue, - summary, - policy, - deployment_message: deploymentMessage, - custom_path: appValue.custom_path - }, - js: bundle.js, - css: bundle.css - } - }) + draft.value as CreateVariable + ) + if (await VariableService.existsVariable({ workspace, path })) { + await VariableService.updateVariable({ workspace, path, requestBody }) + } else { + await VariableService.createVariable({ workspace, requestBody }) + } + actions = [createOpenVariableAction(path)] + break + } + case 'app': { + // Raw apps store a flat AppDraftValue (files/runnables at top level), + // not the deployed app's nested `value` shape the shared raw-app + // deployer reads, so they deploy through the chat's own bundle path. + const appDraft = draft.value as AppDraftValue + // Stale-draft guard: only fetch the deployed head when the draft records + // a fork base to compare against (pre-feature drafts have none). + if (draft.parentVersionId != null) { + const deployedApp = (await AppService.existsApp({ workspace, path })) + ? await AppService.getAppByPath({ workspace, path }) + : undefined + assertDraftBasedOnLatest( + 'app', + path, + draft.parentVersionId, + deployedApp?.versions?.[deployedApp.versions.length - 1], + force + ) + } + const appValue: AppDraftValue = { + ...appDraft, + files: { ...(appDraft.files ?? {}) }, + runnables: { ...(appDraft.runnables ?? {}) }, + data: appDraft.data ?? { ...DEFAULT_RAW_APP_DATA } + } + await recomputeAppPolicy(appValue) + const policy = appValue.policy + if (!policy) { + throw new Error(`Draft app "${path}" has no policy to deploy.`) + } + + toolCallbacks.setToolStatus(toolId, { + content: `Bundling app "${path}"...` + }) + const bundle = await bundleRawAppDraft({ + workspace, + files: appValue.files, + onLog: (delta) => { + const lines = delta + .split('\n') + .map((line) => line.trim()) + .filter(Boolean) + const latest = lines[lines.length - 1] + if (latest) { + toolCallbacks.setToolStatus(toolId, { + content: `Bundling app "${path}"... ${latest}` + }) + } + } + }) + + toolCallbacks.setToolStatus(toolId, { + content: `Deploying app "${path}"...` + }) + const rawAppValue = { + files: appValue.files, + runnables: appValue.runnables, + data: appValue.data ?? { ...DEFAULT_RAW_APP_DATA } + } + const summary = appValue.summary ?? draft.summary ?? '' + // Deploy at the draft's chosen path. A draft_only raw app created in the + // editor lives at a synthetic `u/{user}/draft_{uuid}` storage key with its + // chosen path in the raw_app draft's `draft_path`; the chat's AppDraftValue + // doesn't carry it, so read it from the backend draft. For a chat-created app + // (real path, no draft_path) or a draft on a deployed app, the storage path + // is the deploy path. Same storage-path resolution as script/flow. + const storagePath = getGlobalDraftStoragePath(workspace, 'app', path) + // `draft_path` is read from the persisted backend draft below, but an + // editor rename may still be parked in a debounced/disabled autosave. + // Flush first (like script/flow) so we read the latest chosen path. + await flushDraftOrThrow( + { workspace, itemKind: 'raw_app', path: storagePath }, + `app "${path}"` + ) + let targetPath = storagePath + try { + const row = (await AppService.getAppByPath({ + workspace, + path: storagePath, + getDraft: true, + rawApp: true + })) as { draft?: { draft_path?: string; path?: string }; draft_path?: string } + targetPath = row?.draft?.draft_path ?? row?.draft?.path ?? row?.draft_path ?? storagePath + } catch (e) { + // Only a missing item (404) justifies falling back to the storage path; + // a real lookup failure (network/5xx) must abort rather than silently + // deploy to the wrong path. + if ((e as { status?: number } | null | undefined)?.status === 404) { + targetPath = storagePath + } else { + throw e + } + } + deployedPath = targetPath + if (await AppService.existsApp({ workspace, path: targetPath })) { + // Omit custom_path on update for now. The backend preserves it when absent, while + // sending it requires admin privileges; this chat deploy path does not yet mirror + // the raw app editor's user/admin-specific custom_path handling. + await AppService.updateAppRaw({ + workspace, + path: targetPath, + formData: { + app: { + path: targetPath, + value: rawAppValue, + summary, + policy, + deployment_message: deploymentMessage, + // Preserve the policy's on_behalf_of: this chat deploy path has no + // on-behalf-of selector, so without the flag the backend resets it to + // the deploying user (gated server-side by can_preserve_on_behalf_of). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined + }, + js: bundle.js, + css: bundle.css + } + }) + } else { + await AppService.createAppRaw({ + workspace, + formData: { + app: { + path: targetPath, + value: rawAppValue, + summary, + policy, + deployment_message: deploymentMessage, + custom_path: appValue.custom_path, + // Preserve the policy's on_behalf_of (see update branch above). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined + }, + js: bundle.js, + css: bundle.css + } + }) + } + break } - break } } await deleteGlobalDraft(workspace, type, path, triggerKind, { preserveLiveDraft: true }) + // Move the chat's mask entry to the deployed path: a draft-only item's + // synthetic storage key never exists deployed, so the entry would otherwise + // stop matching anything after the draft is gone. + const deployedKind = itemKindFor(type, triggerKind) + if (deployedKind) { + toolCallbacks.onItemDeployed?.( + deployedKind, + getGlobalDraftStoragePath(workspace, type, path, triggerKind), + deployedPath + ) + } + // Reload the session preview if it's open on the deployed item. Map the // deploy type to the preview kind — a raw app deploys under 'app' but the // preview addresses it as 'raw_app'; non-previewable types map to undefined. @@ -3843,6 +5102,17 @@ async function deleteWorkspaceItem( await deleteGlobalDraft(workspace, type, path, triggerKind) + // Record the deletion in the chat's modified-items mask. In a fork this leaves a + // reviewable "removed" diff vs the parent that stays scoped to this chat. Keyed + // by the same (itemKind, storagePath) as writes so it joins the draft/fork lists. + const deletedKind = itemKindFor(type, triggerKind) + if (deletedKind) { + toolCallbacks.onItemModified?.( + deletedKind, + getGlobalDraftStoragePath(workspace, type, path, triggerKind) + ) + } + toolCallbacks.setToolStatus(toolId, { content: `Deleted ${type} "${path}"`, result: 'Deleted' @@ -3861,13 +5131,36 @@ async function deleteWorkspaceItem( } export function prepareGlobalSystemMessage( - customPrompt?: string, - opts?: { previewTools?: boolean } + instructions?: { workspace?: string; user?: string }, + opts?: { + previewTools?: boolean + // Identity the path-convention guidance is built from. Production omits it + // (read from userStore); callers that must not touch the process-global + // store (the eval harness) pass it explicitly instead. + user?: { username: string; is_admin?: boolean; folders?: string[]; folders_read?: string[] } + skills?: AiSkillListItem[] + } ): ChatCompletionSystemMessageParam { - const username = get(userStore)?.username ?? '' - let content = buildGlobalSystemPrompt(username, opts?.previewTools ?? false) - if (customPrompt?.trim()) { - content = `${content}\n\nUSER GIVEN INSTRUCTIONS:\n${customPrompt.trim()}` + const user = opts?.user ?? get(userStore) + const username = user?.username ?? '' + const folderCtx: FolderPromptContext | undefined = user + ? { + folders: user.folders ?? [], + foldersRead: user.folders_read ?? user.folders ?? [], + isAdmin: user.is_admin ?? false + } + : undefined + let content = buildGlobalSystemPrompt( + username, + opts?.previewTools ?? false, + folderCtx, + opts?.skills ?? [] + ) + if (instructions?.workspace?.trim()) { + content = `${content}\n\nWORKSPACE INSTRUCTIONS (configured by a workspace admin, shared by everyone in this workspace — you cannot modify these):\n${instructions.workspace.trim()}` + } + if (instructions?.user?.trim()) { + content = `${content}\n\nUSER INSTRUCTIONS (this user's personal instructions — update them with the update_user_instructions tool when the user asks you to remember, change, or stop something):\n${instructions.user.trim()}` } return { @@ -3893,7 +5186,10 @@ export function prepareGlobalUserMessage( options: GlobalUserMessageOptions = {} ): ChatCompletionUserMessageParam { const selectedWorkspaceItems = selectedContext.filter( - (context) => context.type === 'workspace_script' || context.type === 'workspace_flow' + (context) => + context.type === 'workspace_script' || + context.type === 'workspace_flow' || + context.type === 'workspace_app' ) const activeEditor = options.activeEditor ?? @@ -3910,7 +5206,13 @@ export function prepareGlobalUserMessage( if (selectedWorkspaceItems.length > 0) { content += '## SELECTED CONTEXT\n' for (const context of selectedWorkspaceItems) { - content += `- type: ${context.type === 'workspace_script' ? 'script' : 'flow'}, path: ${context.path}\n` + const itemType = + context.type === 'workspace_script' + ? 'script' + : context.type === 'workspace_flow' + ? 'flow' + : 'raw_app' + content += `- type: ${itemType}, path: ${context.path}\n` } content += '\n' } diff --git a/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts b/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts deleted file mode 100644 index 61a6827cbf..0000000000 --- a/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts +++ /dev/null @@ -1,190 +0,0 @@ -import { describe, expect, it } from 'vitest' -import type { Flow, NewScript, Script } from '$lib/gen/types.gen' -import { buildFlowDeployRequestBody, buildScriptDeployRequestBody } from './deployRequests' -import type { WorkspaceItem } from './workspaceItems' - -describe('global AI deploy request builders', () => { - it('preserves existing script metadata while replacing draft-controlled fields', () => { - const existing = { - hash: 'parent-hash', - path: 'f/demo/script', - summary: 'existing summary', - description: 'existing description', - content: 'old content', - schema: { properties: { name: { type: 'string' } } }, - is_template: true, - language: 'bun', - kind: 'script', - tag: 'node', - envs: ['ENV_A'], - concurrent_limit: 3, - concurrency_time_window_s: 60, - concurrency_key: 'key', - debounce_key: 'debounce', - debounce_delay_s: 5, - debounce_args_to_accumulate: ['ids'], - max_total_debouncing_time: 120, - max_total_debounces_amount: 4, - cache_ttl: 30, - cache_ignore_s3_path: true, - dedicated_worker: true, - ws_error_handler_muted: true, - priority: 9, - restart_unless_cancelled: true, - timeout: 300, - delete_after_secs: 600, - visible_to_runner_only: true, - auto_kind: 'script', - codebase: 'repo', - has_preprocessor: true, - on_behalf_of_email: 'deployer@example.com', - assets: [{ path: 's3://bucket/key', kind: 's3object' }], - modules: { 'helper.ts': { content: 'export const helper = 1', language: 'bun' } }, - labels: ['prod'], - lock: 'stale lock' - } as unknown as Script & Partial - const draft: WorkspaceItem = { - type: 'script', - path: 'f/demo/script', - summary: 'draft summary', - language: 'bun', - value: 'new content', - isDraft: true - } - - const requestBody = buildScriptDeployRequestBody('f/demo/script', draft, existing, 'ai deploy') - - expect(requestBody).toMatchObject({ - path: 'f/demo/script', - parent_hash: 'parent-hash', - summary: 'draft summary', - description: 'existing description', - content: 'new content', - schema: existing.schema, - tag: 'node', - envs: ['ENV_A'], - concurrent_limit: 3, - concurrency_time_window_s: 60, - concurrency_key: 'key', - debounce_key: 'debounce', - debounce_delay_s: 5, - debounce_args_to_accumulate: ['ids'], - max_total_debouncing_time: 120, - max_total_debounces_amount: 4, - cache_ttl: 30, - cache_ignore_s3_path: true, - dedicated_worker: true, - ws_error_handler_muted: true, - priority: 9, - restart_unless_cancelled: true, - timeout: 300, - delete_after_secs: 600, - visible_to_runner_only: true, - auto_kind: 'script', - codebase: 'repo', - has_preprocessor: true, - on_behalf_of_email: 'deployer@example.com', - preserve_on_behalf_of: true, - assets: existing.assets, - modules: existing.modules, - labels: ['prod'], - deployment_message: 'ai deploy' - }) - expect(requestBody.lock).toBeUndefined() - }) - - it('preserves existing flow metadata and uses draft value/schema overrides', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - value: { modules: [] }, - schema: { required: ['name'] }, - tag: 'python', - ws_error_handler_muted: true, - priority: 7, - dedicated_worker: true, - timeout: 60, - visible_to_runner_only: true, - on_behalf_of_email: 'deployer@example.com', - labels: ['critical'] - } as unknown as Flow - const draftValue = { - value: { modules: [{ id: 'step', value: { type: 'identity' } }] }, - schema: { properties: { name: { type: 'string' } } }, - groups: [{ start_id: 'step', end_id: 'step', summary: 'Group' }] - } - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - undefined, - draftValue as any, - existing, - 'ai deploy' - ) - - expect(requestBody).toMatchObject({ - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - schema: draftValue.schema, - tag: 'python', - ws_error_handler_muted: true, - priority: 7, - dedicated_worker: true, - timeout: 60, - visible_to_runner_only: true, - on_behalf_of_email: 'deployer@example.com', - preserve_on_behalf_of: true, - labels: ['critical'], - deployment_message: 'ai deploy' - }) - expect(requestBody.value.modules).toHaveLength(1) - expect(requestBody.value.groups).toEqual(draftValue.groups) - }) - - it('deploys a draft-set flow description, overriding the existing one', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - value: { modules: [] }, - schema: {} - } as unknown as Flow - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - undefined, - { - value: { modules: [] }, - schema: null, - groups: null, - description: 'draft-set description' - } as any, - existing, - undefined - ) - - expect(requestBody.description).toBe('draft-set description') - }) - - it('falls back to existing flow schema when the draft has no schema', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - value: { modules: [] }, - schema: { properties: { existing: { type: 'boolean' } } } - } as unknown as Flow - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - 'draft summary', - { value: { modules: [] }, schema: null, groups: null }, - existing, - undefined - ) - - expect(requestBody.summary).toBe('draft summary') - expect(requestBody.schema).toBe(existing.schema) - }) -}) diff --git a/frontend/src/lib/components/copilot/chat/global/deployRequests.ts b/frontend/src/lib/components/copilot/chat/global/deployRequests.ts deleted file mode 100644 index 5fe44566c9..0000000000 --- a/frontend/src/lib/components/copilot/chat/global/deployRequests.ts +++ /dev/null @@ -1,101 +0,0 @@ -import type { Flow, NewScript, OpenFlowWPath, Script } from '$lib/gen/types.gen' -import type { FlowDraftValue, WorkspaceItem } from './workspaceItems' - -type ScriptWithDeployMetadata = Script & Partial> - -export type FlowDeployRequestBody = OpenFlowWPath & { - deployment_message?: string -} - -function preserveOnBehalfOf(email: string | undefined): true | undefined { - return email ? true : undefined -} - -export function buildScriptDeployRequestBody( - path: string, - draft: WorkspaceItem, - existing: Script | undefined, - deploymentMessage: string | undefined -): NewScript { - if (typeof draft.value !== 'string' || !draft.language) { - throw new Error(`Draft script "${path}" is missing content or language.`) - } - - const existingWithMetadata = existing as ScriptWithDeployMetadata | undefined - - return { - path, - summary: draft.summary ?? existing?.summary ?? '', - description: existing?.description ?? '', - content: draft.value, - parent_hash: existing?.hash, - schema: existing?.schema, - is_template: existing?.is_template, - language: draft.language, - kind: existing?.kind, - tag: existing?.tag, - envs: existing?.envs, - concurrent_limit: existing?.concurrent_limit, - concurrency_time_window_s: existing?.concurrency_time_window_s, - debounce_key: existing?.debounce_key, - debounce_delay_s: existing?.debounce_delay_s, - debounce_args_to_accumulate: existing?.debounce_args_to_accumulate, - max_total_debouncing_time: existing?.max_total_debouncing_time, - max_total_debounces_amount: existing?.max_total_debounces_amount, - cache_ttl: existing?.cache_ttl, - cache_ignore_s3_path: existingWithMetadata?.cache_ignore_s3_path, - dedicated_worker: existing?.dedicated_worker, - ws_error_handler_muted: existing?.ws_error_handler_muted, - priority: existing?.priority, - restart_unless_cancelled: existing?.restart_unless_cancelled, - timeout: existing?.timeout, - delete_after_secs: existing?.delete_after_secs, - deployment_message: deploymentMessage, - concurrency_key: existing?.concurrency_key, - visible_to_runner_only: existing?.visible_to_runner_only, - auto_kind: existing?.auto_kind, - codebase: existing?.codebase, - has_preprocessor: existing?.has_preprocessor, - on_behalf_of_email: existing?.on_behalf_of_email, - preserve_on_behalf_of: preserveOnBehalfOf(existing?.on_behalf_of_email), - assets: existingWithMetadata?.assets, - modules: existing?.modules, - labels: existing?.labels - } -} - -function flowValueWithDraftGroups(flowDraft: FlowDraftValue): FlowDraftValue['value'] { - if (flowDraft.groups === undefined) { - return flowDraft.value - } - return { - ...flowDraft.value, - groups: flowDraft.groups ?? undefined - } -} - -export function buildFlowDeployRequestBody( - path: string, - draftSummary: string | undefined, - flowDraft: FlowDraftValue, - existing: Flow | undefined, - deploymentMessage: string | undefined -): FlowDeployRequestBody { - return { - path, - summary: draftSummary ?? existing?.summary ?? '', - description: flowDraft.description ?? existing?.description ?? '', - value: flowValueWithDraftGroups(flowDraft), - schema: flowDraft.schema ?? existing?.schema ?? {}, - tag: existing?.tag, - ws_error_handler_muted: existing?.ws_error_handler_muted, - priority: existing?.priority, - dedicated_worker: existing?.dedicated_worker, - timeout: existing?.timeout, - visible_to_runner_only: existing?.visible_to_runner_only, - on_behalf_of_email: existing?.on_behalf_of_email, - preserve_on_behalf_of: preserveOnBehalfOf(existing?.on_behalf_of_email), - labels: existing?.labels, - deployment_message: deploymentMessage - } -} diff --git a/frontend/src/lib/components/copilot/chat/global/gate.ts b/frontend/src/lib/components/copilot/chat/global/gate.ts index 664f06ff39..df8035de87 100644 --- a/frontend/src/lib/components/copilot/chat/global/gate.ts +++ b/frontend/src/lib/components/copilot/chat/global/gate.ts @@ -11,8 +11,8 @@ * When the mode is ready to ship to everyone, replace every call to * `isGlobalAiEnabled()` with `true` and delete this file. The references are * intentionally narrow (chat mode visibility, custom prompt settings, the - * `change_mode` tool enum, and the `/global_drafts` dev route) so the rip-out - * is a small grep. + * `change_mode` tool enum, the AI skills workspace settings tab, and the + * `/global_drafts` dev route) so the rip-out is a small grep. */ const STORAGE_KEY = 'wm_dev_global_ai' diff --git a/frontend/src/lib/components/copilot/chat/global/pageNavigation.test.ts b/frontend/src/lib/components/copilot/chat/global/pageNavigation.test.ts new file mode 100644 index 0000000000..5f7a84ee1c --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/global/pageNavigation.test.ts @@ -0,0 +1,66 @@ +import { describe, it, expect } from 'vitest' +import { + buildWorkspaceSettingsUrl, + buildAuditLogsUrl, + buildResourcesUrl, + buildVariablesUrl, + buildTriggersUrl, + buildFoldersUrl +} from './pageNavigation' + +function parse(appPath: string): URL { + return new URL(appPath, 'http://x') +} + +describe('pageNavigation builders', () => { + it('opens workspace settings on a specific tab', () => { + const u = parse(buildWorkspaceSettingsUrl({ tab: 'git_sync' })) + expect(u.pathname).toBe('/workspace_settings') + expect(u.searchParams.get('tab')).toBe('git_sync') + }) + + it('opens workspace settings with no tab', () => { + const u = parse(buildWorkspaceSettingsUrl({})) + expect(u.pathname).toBe('/workspace_settings') + expect(u.search).toBe('') + }) + + it('audit logs allow-lists its own keys and drops others', () => { + const u = parse( + buildAuditLogsUrl({ username: 'admin', operation: 'jobs.run', status: 'failure' }) + ) + expect(u.pathname).toBe('/audit_logs') + expect(u.searchParams.get('username')).toBe('admin') + expect(u.searchParams.get('operation')).toBe('jobs.run') + expect(u.searchParams.has('status')).toBe(false) // not an audit-logs key + }) + + it('resources keeps resource_type + path, drops runs-only keys', () => { + const u = parse( + buildResourcesUrl({ resource_type: 'postgres', path: 'f/x', status: 'failure' }) + ) + expect(u.searchParams.get('resource_type')).toBe('postgres') + expect(u.searchParams.get('path')).toBe('f/x') + expect(u.searchParams.has('status')).toBe(false) + }) + + it('variables keeps path + owner only', () => { + const u = parse(buildVariablesUrl({ path: 'f/x', owner: 'u/alice', resource_type: 'postgres' })) + expect(u.searchParams.get('path')).toBe('f/x') + expect(u.searchParams.get('owner')).toBe('u/alice') + expect(u.searchParams.has('resource_type')).toBe(false) + }) + + it('triggers route to the kind page, opening a specific trigger via hash', () => { + expect(parse(buildTriggersUrl({ trigger_kind: 'kafka' })).pathname).toBe('/kafka_triggers') + const u = parse(buildTriggersUrl({ trigger_kind: 'http', open: 'f/a/b' })) + expect(u.pathname).toBe('/routes') + expect(u.hash).toBe('#f/a/b') + }) + + it('folders opens the list page with no params', () => { + const u = parse(buildFoldersUrl()) + expect(u.pathname).toBe('/folders') + expect(u.search).toBe('') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/global/pageNavigation.ts b/frontend/src/lib/components/copilot/chat/global/pageNavigation.ts new file mode 100644 index 0000000000..ea783e19e7 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/global/pageNavigation.ts @@ -0,0 +1,133 @@ +import { buildFilterUrl } from '$lib/navigation' +import { buildRunsFilterSearchbarSchema } from '$lib/components/runs/runsFilter' +import { buildSchedulesFilterSchema } from '$lib/components/schedules/schedulesFilter' +import { TRIGGER_PAGES, type TriggerKind } from '$lib/components/sessions/previewRouter' + +// In-app paths for the deep-linkable preview pages the AI chat can open. +export const RUNS_PATH = '/runs' +export const SCHEDULES_PATH = '/schedules' +export const VARIABLES_PATH = '/variables' +export const RESOURCES_PATH = '/resources' +export const ASSETS_PATH = '/assets' +export const AUDIT_LOGS_PATH = '/audit_logs' +export const WORKSPACE_SETTINGS_PATH = '/workspace_settings' +export const FOLDERS_PATH = '/folders' +export const GROUPS_PATH = '/groups' + +// Selectable tabs on the Workspace settings page (the `?tab=` query param). Mirrors the +// union in routes/(root)/(logged)/workspace_settings/+page.svelte. +export const WORKSPACE_SETTINGS_TABS = [ + 'users', + 'slack', + 'teams', + 'premium', + 'general', + 'webhook', + 'deploy_to', + 'dev_workspace', + 'error_handler', + 'success_handler', + 'critical_alerts', + 'ai', + 'windmill_data_tables', + 'windmill_lfs', + 'volume_storage', + 'ducklake', + 'git_sync', + 'default_app', + 'native_triggers', + 'encryption', + 'dependencies', + 'rulesets', + 'shared_ui' +] as const + +// Valid query-param keys are derived from the real filter schemas (option arrays are +// irrelevant to the key set), so a renamed filter key propagates here for free. +const RUNS_FILTER_KEYS = Object.keys( + buildRunsFilterSearchbarSchema({ + paths: [], + usernames: [], + folders: [], + jobTriggerKinds: [], + isSuperAdminOrDevops: false, + isAdminsWorkspace: false + }) +) +const SCHEDULES_FILTER_KEYS = Object.keys( + buildSchedulesFilterSchema({ paths: [], scriptPaths: [] }) +) + +/** Deep-link to the Runs page with the given filters (keys must match `runsFilter`). */ +export function buildRunsUrl(filters: Record): string { + return buildFilterUrl(RUNS_PATH, filters, { validKeys: RUNS_FILTER_KEYS }) +} + +/** + * Deep-link to the Schedules page. When `open` is set, the schedule at that exact path + * is opened in the edit drawer via the `#` hash the page already handles. + */ +export function buildSchedulesUrl({ + open, + filters +}: { + open?: string + filters?: Record +}): string { + return buildFilterUrl(SCHEDULES_PATH, filters ?? {}, { + validKeys: SCHEDULES_FILTER_KEYS, + hash: open + }) +} + +// The remaining pages expose a curated subset of each page's real query params (not the +// full filter schema), so the allow-list is the exact set of keys the builder emits — +// these names match the query params the pages read (variablesFilter/resourcesFilter/ +// assetsFilter and audit_logs/+page.svelte). +export function buildVariablesUrl(filters: Record): string { + return buildFilterUrl(VARIABLES_PATH, filters, { validKeys: ['path', 'owner'] }) +} + +export function buildResourcesUrl(filters: Record): string { + return buildFilterUrl(RESOURCES_PATH, filters, { + validKeys: ['path', 'resource_type', 'owner'] + }) +} + +export function buildAssetsUrl(filters: Record): string { + return buildFilterUrl(ASSETS_PATH, filters, { validKeys: ['path'] }) +} + +export function buildAuditLogsUrl(filters: Record): string { + return buildFilterUrl(AUDIT_LOGS_PATH, filters, { + validKeys: ['username', 'operation', 'resource'] + }) +} + +/** Deep-link to the Workspace settings page, optionally on a specific `?tab=`. */ +export function buildWorkspaceSettingsUrl({ tab }: { tab?: string }): string { + return buildFilterUrl(WORKSPACE_SETTINGS_PATH, tab ? { tab } : {}) +} + +/** Folders and Groups list pages have no query filters — just open them. */ +export function buildFoldersUrl(): string { + return FOLDERS_PATH +} + +export function buildGroupsUrl(): string { + return GROUPS_PATH +} + +/** + * Deep-link to a trigger list page (by kind). When `open` is set, the trigger at that + * exact path is opened in the edit drawer via the `#` hash the page handles. + */ +export function buildTriggersUrl({ + trigger_kind, + open +}: { + trigger_kind: TriggerKind + open?: string +}): string { + return buildFilterUrl(TRIGGER_PAGES[trigger_kind].path, {}, { hash: open }) +} diff --git a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts index 009050342e..342717f26e 100644 --- a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts +++ b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts @@ -1,7 +1,5 @@ import type { Flow, NewSchedule, NewScript } from '$lib/gen/types.gen' import { DraftService } from '$lib/gen' -import { get } from 'svelte/store' -import { userStore } from '$lib/stores' import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { DEFAULT_DATA as DEFAULT_RAW_APP_DATA } from '$lib/components/raw_apps/dataTableRefUtils' import { UserDraft, type UserDraftEntry, type UserDraftItemKind } from '$lib/userDraft.svelte' @@ -66,7 +64,10 @@ function normalizeAppDraftValue(value: AppDraftValue): AppDraftValue { runnables: { ...(value.runnables ?? {}) }, data: value.data ?? { ...DEFAULT_RAW_APP_DATA }, policy: value.policy === undefined ? undefined : clone(value.policy), - custom_path: value.custom_path + custom_path: value.custom_path, + // Carry the fork-base version through the whitelist — it is dropped on every + // save otherwise, which would defeat the stale-draft check. + parent_version: value.parent_version } } @@ -105,7 +106,7 @@ function clearEphemeralSecretVariableDraftValues(workspace: string): void { secretVariableDraftValues.delete(workspace) } -function itemKindFor( +export function itemKindFor( type: WorkspaceItemType, triggerKind?: TriggerKind ): UserDraftItemKind | undefined { @@ -135,6 +136,7 @@ function scriptDraftToWorkspaceItem(path: string, draft: NewScript): WorkspaceIt summary: draft.summary, language: draft.language, value: draft.content, + parentHash: draft.parent_hash, isDraft: true } } @@ -144,6 +146,10 @@ function flowDraftToWorkspaceItem(path: string, draft: Flow): WorkspaceItem { type: 'flow', path, summary: draft.summary, + // The persisted flow draft carries `version_id` (the deployed head it was + // forked from, pinned at fork by writeDraft/the editor) — the flow analog + // of a script's parent_hash. + parentVersionId: draft.version_id, value: { value: draft.value, schema: draft.schema ?? null, @@ -160,6 +166,7 @@ function appDraftToWorkspaceItem(path: string, draft: AppDraftValue): WorkspaceI type: 'app', path, summary: value.summary, + parentVersionId: value.parent_version, value, isDraft: true } @@ -327,29 +334,26 @@ function getGlobalDraftSlot( } // Current user's persisted draft value (+ records the sync baseline so a later -// save detects external conflicts). undefined on 404 (no draft at that path). +// save detects external conflicts). undefined when no draft exists at that path. +// Uses `getOwnDraft` (not `getDraftForUser`): the latter rejects drawer kinds +// (schedule/trigger/resource/variable drafts are private to their owner), which +// would make those drafts write-only here — listed but never readable/deployable. +// Errors (403/500/network) MUST propagate: swallowing one would make the write +// merge fall through to the deployed item instead of the user's in-progress +// draft, silently overwriting their draft-only changes. async function fetchBackendDraftValue( workspace: string, itemKind: UserDraftItemKind, storagePath: string ): Promise { - try { - const resp = await DraftService.getDraftForUser({ - workspace, - kind: itemKind as any, - path: storagePath, - username: get(userStore)?.username - }) - UserDraftDbSyncer.recordRemoteSync({ workspace, itemKind, path: storagePath }, resp.created_at) - return resp.value ?? undefined - } catch (e) { - // 404 = no draft for this owner at that path (the intended empty case). - // Anything else (403/500/network) MUST propagate: swallowing it would make - // the write merge fall through to the deployed item instead of the user's - // in-progress draft, silently overwriting their draft-only changes. - if ((e as { status?: number } | null | undefined)?.status === 404) return undefined - throw e - } + const resp = await DraftService.getOwnDraft({ + workspace, + kind: itemKind, + path: storagePath + }) + if (!resp) return undefined + UserDraftDbSyncer.recordRemoteSync({ workspace, itemKind, path: storagePath }, resp.created_at) + return resp.value ?? undefined } // Draft VALUE for a write merge: cell-if-present (the user's freshest in-tab @@ -368,10 +372,25 @@ export async function readGlobalDraftValue( return (await fetchBackendDraftValue(workspace, itemKind, storagePath)) as V | undefined } +// `itemKind` + `storagePath` are the canonical identity of the persisted draft +// (NOT item.path, which is the friendly display path). Callers use them to record +// the chat's modified-items mask. export type DraftPersistResult = - | { status: 'saved'; item: WorkspaceItem } - | { status: 'conflict'; item: WorkspaceItem; serverTimestamp?: string } - | { status: 'error'; item: WorkspaceItem; message: string } + | { status: 'saved'; item: WorkspaceItem; itemKind: UserDraftItemKind; storagePath: string } + | { + status: 'conflict' + item: WorkspaceItem + itemKind: UserDraftItemKind + storagePath: string + serverTimestamp?: string + } + | { + status: 'error' + item: WorkspaceItem + itemKind: UserDraftItemKind + storagePath: string + message: string + } // Persist a built draft value. `UserDraft.seed` reflects it into an open editor's // cell WITHOUT a double-POST (no-ops if no cell; its seedNextWrite suppresses the @@ -408,14 +427,20 @@ export async function persistGlobalDraft( // the chat "saved" while the DB-backed source of truth was never updated. const saveState = UserDraftDbSyncer.getState({ workspace, itemKind, path: storagePath }) if (saveState.state === 'failed') { - return { status: 'error', item, message: saveState.failureMessage ?? 'Draft save failed' } + return { + status: 'error', + item, + itemKind, + storagePath, + message: saveState.failureMessage ?? 'Draft save failed' + } } const conflict = opts.force ? undefined : UserDraftDbSyncer.getConflict({ workspace, itemKind, path: storagePath }).conflict return conflict - ? { status: 'conflict', item, serverTimestamp: conflict.serverTimestamp } - : { status: 'saved', item } + ? { status: 'conflict', item, itemKind, storagePath, serverTimestamp: conflict.serverTimestamp } + : { status: 'saved', item, itemKind, storagePath } } export async function getGlobalDraft( diff --git a/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts b/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts index 09bbc12da5..a8196a5054 100644 --- a/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts +++ b/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts @@ -72,6 +72,9 @@ export type AppDraftValue = { data?: any policy?: Policy custom_path?: string + // Fork base: the deployed app version this draft was started from, pinned at + // fork. The app analog of a script's parent_hash / a flow's version_id. + parent_version?: number } export type ResourceDraftState = { @@ -99,6 +102,11 @@ export type WorkspaceItem = { summary?: string language?: ScriptLang triggerKind?: TriggerKind + // Fork base: the deployed version this draft was started from, compared against + // the current deployed head to detect a stale draft. `parentHash` for scripts + // (script hash), `parentVersionId` for flows (flow_version id). + parentHash?: string + parentVersionId?: number value?: | string | FlowDraftValue diff --git a/frontend/src/lib/components/copilot/chat/mention.test.ts b/frontend/src/lib/components/copilot/chat/mention.test.ts new file mode 100644 index 0000000000..8d379ce3b4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from 'vitest' +import { MENTION_RE, mentionTitle, formatMention } from './mention' + +describe('formatMention', () => { + it('leaves a simple name bare', () => { + expect(formatMention('app.ts')).toBe('@app.ts') + expect(formatMention('proj/sub/a.ts')).toBe('@proj/sub/a.ts') + }) + it('brackets a name containing whitespace', () => { + expect(formatMention('my file.txt')).toBe('@[my file.txt]') + expect(formatMention('my folder/a b.ts')).toBe('@[my folder/a b.ts]') + }) + it('brackets names with HTML-sensitive chars, parens, brackets', () => { + expect(formatMention('R&D notes.md')).toBe('@[R&D notes.md]') + expect(formatMention('a.txt')).toBe('@[a.txt]') + expect(formatMention('report(final).csv')).toBe('@[report(final).csv]') + }) +}) + +describe('mentionTitle', () => { + it('strips the @ from a bare mention', () => { + expect(mentionTitle('@app.ts')).toBe('app.ts') + }) + it('strips the @[ ] from a bracketed mention', () => { + expect(mentionTitle('@[my file.txt]')).toBe('my file.txt') + }) +}) + +describe('MENTION_RE', () => { + it('captures a bracketed (spaced) mention whole alongside bare ones', () => { + const tokens = [...'see @app.ts and @[my file.txt] ok'.matchAll(MENTION_RE)].map((m) => m[0]) + expect(tokens).toEqual(['@app.ts', '@[my file.txt]']) + expect(tokens.map(mentionTitle)).toEqual(['app.ts', 'my file.txt']) + }) + it('round-trips formatMention → MENTION_RE → mentionTitle for a spaced name', () => { + const name = 'my notes (v2).md' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips a name containing both whitespace and a closing bracket', () => { + const name = 'notes ] draft.md' + expect(formatMention(name)).toBe('@[notes \\] draft.md]') + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(m[0]).toBe('@[notes \\] draft.md]') + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips an HTML-sensitive name (highlighter handles HTML-escaping separately)', () => { + const name = 'a & c].txt' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/mention.ts b/frontend/src/lib/components/copilot/chat/mention.ts new file mode 100644 index 0000000000..65a2b3b7c6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.ts @@ -0,0 +1,31 @@ +/** + * `@mention` formatting shared between the chat input (which inserts mentions) and the + * textarea highlighter (which parses them) so the two never disagree. + * + * A simple name is inserted bare (`@app.ts`); a name containing whitespace is bracketed + * (`@[my file.txt]`) so it's captured whole instead of truncating at the first space. + */ + +/** + * Matches a mention token: a bracketed `@[name with spaces]` (where `\]` and `\\` are + * escaped, so a `]` inside the name doesn't end the token early) first, then a bare `@name`. + */ +export const MENTION_RE = /@\[(?:\\.|[^\]\\\r\n])*\]|@[\w/.\-\[\]]+/g + +/** The title of a mention token (`@name` or `@[name]`), brackets stripped and unescaped. */ +export function mentionTitle(token: string): string { + if (token.startsWith('@[') && token.endsWith(']')) { + return token.slice(2, -1).replace(/\\(.)/g, '$1') + } + return token.slice(1) +} + +/** Chars the bare `@name` regex matches without truncating; anything else needs brackets. */ +const BARE_SAFE = /^[\w/.\-]+$/ + +/** Format a name as a mention token. A bare `@name` only survives for simple names; anything + * with whitespace, HTML-sensitive chars (`< > &`), brackets, parens, etc. is bracketed (with + * `\` and `]` escaped) so the token is captured whole and round-trips through the parser. */ +export function formatMention(name: string): string { + return BARE_SAFE.test(name) ? `@${name}` : `@[${name.replace(/[\\\]]/g, '\\$&')}]` +} diff --git a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts index dfe3eb7565..1600d8fe9a 100644 --- a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts +++ b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts @@ -129,9 +129,17 @@ export class AIChatEditorHandler { const deletedChange = group.changes[0] const addedChange = group.changes[1] if (deletedChange.type === 'deleted' && addedChange.type === 'added_block') { - applyChange(this.editor, deletedChange) - addedChange.position.afterLineNumber = deletedChange.range.startLine - 1 - applyChange(this.editor, addedChange) + this.editor.executeEdits('chat', [ + { + range: { + startLineNumber: deletedChange.range.startLine, + startColumn: 1, + endLineNumber: deletedChange.range.endLine + 1, + endColumn: 0 + }, + text: addedChange.value + '\n' + } + ]) } else { throw new Error('Invalid group') } @@ -284,7 +292,7 @@ export class AIChatEditorHandler { }) if (!opts?.applyAll) { - ;({ collection, ids } = await displayVisualChanges( + ; ({ collection, ids } = await displayVisualChanges( 'editor-windmill-chat-style', this.editor, changes, diff --git a/frontend/src/lib/components/copilot/chat/openai-responses.ts b/frontend/src/lib/components/copilot/chat/openai-responses.ts index eb1dd2b2d5..426e1e76e2 100644 --- a/frontend/src/lib/components/copilot/chat/openai-responses.ts +++ b/frontend/src/lib/components/copilot/chat/openai-responses.ts @@ -454,12 +454,14 @@ export async function getNonStreamingOpenAIResponsesCompletion( workspace?: string resourcePath?: string forceModelProvider?: AIProviderModel + maxTokensCap?: number } ): Promise { const { provider, config } = getProviderAndCompletionConfig({ messages, stream: false, - forceModelProvider: options?.forceModelProvider + forceModelProvider: options?.forceModelProvider, + maxTokensCap: options?.maxTokensCap }) const { instructions, input } = convertMessagesToResponsesInput(messages) diff --git a/frontend/src/lib/components/copilot/chat/pipeline/core.test.ts b/frontend/src/lib/components/copilot/chat/pipeline/core.test.ts new file mode 100644 index 0000000000..4f3e31986d --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/pipeline/core.test.ts @@ -0,0 +1,171 @@ +import { describe, it, expect, vi } from 'vitest' + +// `../shared` transitively pulls in the monaco editor (and its CSS), which the +// node test environment can't load — mirror the sibling chat tests' stub. +vi.mock('monaco-editor', () => ({ editor: {} })) + +import { + pipelineTools, + getPipelinePromptSection, + type PipelineAIChatHelpers, + type PipelineContext +} from './core' +import type { ToolCallbacks } from '../shared' + +function toolByName(name: string) { + const tool = pipelineTools.find((t) => t.def.function.name === name) + if (!tool) throw new Error(`tool ${name} not found`) + return tool +} + +function noopCallbacks(): ToolCallbacks { + return { setToolStatus: () => {}, removeToolStatus: () => {} } +} + +const sampleContext: PipelineContext = { + folder: 'analytics', + mode: 'edit', + nodes: [ + { + path: 'f/analytics/orders', + language: 'bun', + unsaved: true, + writes: ['ducklake://main/orders'], + reads: [], + triggers: ['schedule'] + } + ], + assets: ['ducklake://main/orders'] +} + +function makeHelpers(overrides: Partial = {}): { + helpers: { pipeline: PipelineAIChatHelpers } + calls: Record +} { + const calls: Record = {} + const record = + (name: string, ret?: any) => + (...args: any[]) => { + ;(calls[name] ??= []).push(args) + return ret + } + const pipeline: PipelineAIChatHelpers = { + getPipelineContext: () => sampleContext, + getNodeBody: async (path: string) => { + calls.getNodeBody = [...(calls.getNodeBody ?? []), [path]] + return { language: 'bun', content: 'export async function main() { return 1 }' } + }, + proposeNode: async (input) => { + calls.proposeNode = [...(calls.proposeNode ?? []), [input]] + return { path: input.path } + }, + editNode: async (path, content) => { + calls.editNode = [...(calls.editNode ?? []), [path, content]] + }, + removeProposedNode: record('removeProposedNode'), + testNode: async () => 'job-123', + ...overrides + } + return { helpers: { pipeline }, calls } +} + +describe('pipeline tools', () => { + it('exposes the expected tool surface', () => { + expect(pipelineTools.map((t) => t.def.function.name).sort()).toEqual([ + 'build_pipeline_node', + 'edit_pipeline_node', + 'get_pipeline_graph', + 'read_pipeline_node', + 'remove_pipeline_node', + 'test_pipeline_node' + ]) + }) + + it('get_pipeline_graph returns the live context as JSON', async () => { + const { helpers } = makeHelpers() + const out = await toolByName('get_pipeline_graph').fn({ + args: {}, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + expect(JSON.parse(out)).toMatchObject({ folder: 'analytics' }) + }) + + it('build_pipeline_node forwards to proposeNode and does not deploy', async () => { + const { helpers, calls } = makeHelpers() + const out = await toolByName('build_pipeline_node').fn({ + args: { + path: 'f/analytics/clean', + language: 'bun', + content: '// pipeline\nexport async function main() {}', + output_kind: 'ducklake' + }, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + expect(calls.proposeNode?.[0]?.[0]).toMatchObject({ + path: 'f/analytics/clean', + language: 'bun', + outputKind: 'ducklake' + }) + expect(out).toContain('not deployed') + }) + + it('edit_pipeline_node reads then applies an exact find/replace', async () => { + const { helpers, calls } = makeHelpers({ + getNodeBody: async () => ({ language: 'bun', content: 'const x = 1\nconst y = 2\n' }) + }) + await toolByName('edit_pipeline_node').fn({ + args: { path: 'f/analytics/orders', old_string: 'const x = 1', new_string: 'const x = 42' }, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + expect(calls.editNode?.[0]?.[1]).toContain('const x = 42') + }) + + it('edit_pipeline_node surfaces a clear error when old_string is absent', async () => { + const { helpers } = makeHelpers({ + getNodeBody: async () => ({ language: 'bun', content: 'const x = 1\n' }) + }) + await expect( + toolByName('edit_pipeline_node').fn({ + args: { path: 'f/analytics/orders', old_string: 'NOT THERE', new_string: 'x' }, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + ).rejects.toThrow(/was not found/) + }) + + it('mutation tools fail clearly when no pipeline editor is registered', async () => { + await expect( + toolByName('build_pipeline_node').fn({ + args: { path: 'f/a/b', language: 'bun', content: 'x' }, + workspace: 'w', + helpers: {}, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + ).rejects.toThrow(/No pipeline editor is open/) + }) + + it('test_pipeline_node requires confirmation', () => { + expect(toolByName('test_pipeline_node').requiresConfirmation).toBe(true) + }) +}) + +describe('getPipelinePromptSection', () => { + it('names the active folder and the direct-draft workflow', () => { + const section = getPipelinePromptSection(sampleContext) + expect(section).toContain('/pipeline/analytics') + expect(section).toContain('build_pipeline_node') + expect(section).toContain('directly as unsaved drafts') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/pipeline/core.ts b/frontend/src/lib/components/copilot/chat/pipeline/core.ts new file mode 100644 index 0000000000..86fe9eb634 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/pipeline/core.ts @@ -0,0 +1,328 @@ +import { z } from 'zod' +import { $ScriptLang } from '$lib/gen/schemas.gen' +import type { ScriptLang } from '$lib/gen' +import { createToolDef, executeTestRun, findAndReplace, type Tool } from '../shared' +import type { PipelineOutputKind } from '$lib/components/assets/AssetGraph/pipelineTemplates' + +// ============================================================================ +// Pipeline AI chat tools. +// +// These tools extend the GLOBAL chat mode when the user is on a /pipeline/ +// editor (the page registers `PipelineAIChatHelpers` on the AIChatManager). They +// let the model read the live pipeline graph and BUILD/EDIT pipeline nodes +// (scripts annotated with `// pipeline`). Mutations don't deploy: they apply +// directly as an unsaved DRAFT on the canvas — the same way the flow/script +// editor applies AI edits — which the user then deploys. There is no separate +// approve/reject step: the draft IS the change. +// +// The pipeline tools are added on top of the full global tool set, so docs +// search, datatable SQL, and workspace-item tools are already available +// alongside them — this file only carries the pipeline-graph-specific surface. +// ============================================================================ + +/** Compact, model-facing summary of one node in the pipeline graph. */ +export type PipelineNodeSummary = { + path: string + language?: ScriptLang + /** Has an unsaved local edit (draft) not yet deployed. */ + unsaved: boolean + summary?: string + /** Asset URIs this node writes (its outputs). */ + writes: string[] + /** Asset URIs this node reads (its inputs). */ + reads: string[] + /** Declared `// on ` execution-DAG bindings (asset URIs or native kinds). */ + triggers: string[] +} + +/** Compact, model-facing snapshot of the whole pipeline graph. */ +export type PipelineContext = { + folder: string + mode: 'view' | 'edit' + nodes: PipelineNodeSummary[] + /** All storage assets referenced by the graph, as URIs. */ + assets: string[] +} + +/** + * Bridge the pipeline page registers on the AIChatManager. Reads expose the live + * graph; writes apply directly as unsaved drafts (never deploy). Kept intentionally + * small — the page owns the draft Map and canvas rendering. + */ +export interface PipelineAIChatHelpers { + getPipelineContext: () => PipelineContext + /** Read a node's source (the in-flight draft body if one exists, else deployed). */ + getNodeBody: (path: string) => Promise<{ language: ScriptLang; content: string } | undefined> + /** Create a brand-new pipeline node as an unsaved draft on the canvas. */ + proposeNode: (input: { + path: string + language: ScriptLang + content: string + outputKind?: PipelineOutputKind + }) => Promise<{ path: string }> + /** Replace an existing node's body, applied as an unsaved draft. */ + editNode: (path: string, content: string) => Promise + /** Discard the unsaved draft at a path (undo a build_pipeline_node). */ + removeProposedNode: (path: string) => Promise + /** Preview-run a node (draft body preferred). Returns the started job id. */ + testNode: (path: string, args?: Record) => Promise +} + +/** Helper bag the pipeline tools receive from the manager in global mode. */ +export type PipelineToolHelpers = { pipeline?: PipelineAIChatHelpers } + +function requirePipeline(helpers: PipelineToolHelpers): PipelineAIChatHelpers { + if (!helpers?.pipeline) { + throw new Error( + 'No pipeline editor is open. Pipeline tools only work on a /pipeline/ page in edit mode.' + ) + } + return helpers.pipeline +} + +const scriptLangSchema = z.enum($ScriptLang.enum) + +const outputKindSchema = z + .enum(['none', 'datatable', 'ducklake', 'materialize', 's3_parquet', 's3_object']) + .describe( + 'Kind of output asset this node materializes, used to seed the output edge on the canvas before the body is parsed: "materialize"/"ducklake" → a DuckLake table, "datatable" → a Postgres data table, "s3_parquet"/"s3_object" → an S3 file, "none" → side-effect only. Defaults to none.' + ) + +// ---------------------------------------------------------------------------- +// Read tools +// ---------------------------------------------------------------------------- + +const getPipelineGraphSchema = z.object({}) + +const getPipelineGraphToolDef = createToolDef( + getPipelineGraphSchema, + 'get_pipeline_graph', + "Read the live pipeline graph for the open /pipeline/ editor: its nodes (scripts), each node's language, asset reads/writes, declared triggers, and whether it has an unsaved draft edit. Call this before building or editing nodes so you reuse existing assets/paths and understand the current DAG." +) + +const readPipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the pipeline node (script) to read.') +}) + +const readPipelineNodeToolDef = createToolDef( + readPipelineNodeSchema, + 'read_pipeline_node', + 'Read the full source of one pipeline node (its in-flight draft body if it has unsaved edits, otherwise the deployed body). Use before edit_pipeline_node so edits target the exact current text.' +) + +// ---------------------------------------------------------------------------- +// Mutation tools (apply directly as unsaved drafts; never deploy) +// ---------------------------------------------------------------------------- + +const buildPipelineNodeSchema = z.object({ + path: z + .string() + .describe( + "Workspace path for the new node, e.g. f//. Use the open pipeline's folder. Must not collide with an existing node." + ), + language: scriptLangSchema.describe( + 'Script language. SQL-shaped data work uses duckdb (DuckLake/S3) or postgresql (data tables); bun/python3 for general transforms.' + ), + content: z + .string() + .describe( + "Full script source. Start it with the `pipeline` annotation as a top-of-file comment in the LANGUAGE'S comment syntax — `-- pipeline` for SQL (duckdb/postgresql), `# pipeline` for python3/bash, `// pipeline` for bun/TS — to mark it a pipeline member; declare inputs the same way (e.g. `-- on `), and write outputs via the wmill SDK / SQL so the lineage edges are inferred. A `// pipeline` line in a SQL node is a syntax error. Read existing node bodies first to match conventions." + ), + output_kind: outputKindSchema.optional() +}) + +const buildPipelineNodeToolDef = createToolDef( + buildPipelineNodeSchema, + 'build_pipeline_node', + 'Build a NEW pipeline node. It is applied directly as an unsaved draft on the canvas (a dashed node wired by its parsed asset reads/writes) — it does NOT deploy; the user deploys it. Prefer this over editing for new scripts.', + { strict: false } +) + +const editPipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the node to edit.'), + old_string: z.string().min(1).describe("Exact text to find in the node's current source."), + new_string: z.string().describe('Replacement text.'), + replace_all: z + .boolean() + .optional() + .default(false) + .describe( + 'When true, replace every exact match. When false, old_string must match exactly once.' + ) +}) + +const editPipelineNodeToolDef = createToolDef( + editPipelineNodeSchema, + 'edit_pipeline_node', + 'Edit an existing pipeline node by exact find/replace. The result is applied directly as an unsaved draft (does NOT deploy). Call read_pipeline_node first to get the exact current text.', + { strict: false } +) + +const removePipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the node whose unsaved draft should be discarded.') +}) + +const removePipelineNodeToolDef = createToolDef( + removePipelineNodeSchema, + 'remove_pipeline_node', + 'Discard the unsaved draft at a path (undo a build_pipeline_node). Only affects the in-flight draft — to delete a deployed node, ask the user to do it on the canvas.' +) + +const testPipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the node to preview-run.'), + args: z + .record(z.string(), z.any()) + .nullable() + .optional() + .describe('Arguments to pass to the script. Omit or pass null when none are needed.') +}) + +const testPipelineNodeToolDef = createToolDef( + testPipelineNodeSchema, + 'test_pipeline_node', + 'Preview-run one pipeline node (using its draft body when unsaved) and return the result/logs, without deploying. Requires user confirmation before it runs.', + { strict: false } +) + +// ---------------------------------------------------------------------------- +// Tool set +// ---------------------------------------------------------------------------- + +export const pipelineTools: Tool[] = [ + { + def: getPipelineGraphToolDef, + fn: async ({ helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + toolCallbacks.setToolStatus(toolId, { content: 'Reading pipeline graph...' }) + const ctx = pipeline.getPipelineContext() + toolCallbacks.setToolStatus(toolId, { + content: `Read pipeline graph (${ctx.nodes.length} node${ctx.nodes.length === 1 ? '' : 's'})`, + result: 'Success' + }) + return JSON.stringify(ctx, null, 2) + } + }, + { + def: readPipelineNodeToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path } = readPipelineNodeSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Reading node '${path}'...` }) + const node = await pipeline.getNodeBody(path) + if (!node) { + return `No pipeline node found at '${path}'. Call get_pipeline_graph to list the available nodes.` + } + toolCallbacks.setToolStatus(toolId, { content: `Read node '${path}'`, result: 'Success' }) + return JSON.stringify({ path, language: node.language, content: node.content }) + } + }, + { + def: buildPipelineNodeToolDef, + streamArguments: true, + showDetails: true, + showFade: true, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path, language, content, output_kind } = buildPipelineNodeSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Building node '${path}'...` }) + await pipeline.proposeNode({ + path, + language: language as ScriptLang, + content, + outputKind: output_kind as PipelineOutputKind | undefined + }) + toolCallbacks.setToolStatus(toolId, { + content: `Added draft node '${path}'`, + result: 'Success' + }) + return `Pipeline node '${path}' added as an unsaved draft on the canvas. It is not deployed — the user deploys it.` + } + }, + { + def: editPipelineNodeToolDef, + streamArguments: true, + showDetails: true, + showFade: true, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path, old_string, new_string, replace_all } = editPipelineNodeSchema.parse(args) + const node = await pipeline.getNodeBody(path) + if (!node) { + return `No pipeline node found at '${path}'. Call get_pipeline_graph to list the available nodes.` + } + toolCallbacks.setToolStatus(toolId, { content: `Editing node '${path}'...` }) + const updated = findAndReplace( + node.content, + old_string, + new_string, + replace_all ?? false, + 'node source' + ) + await pipeline.editNode(path, updated) + toolCallbacks.setToolStatus(toolId, { + content: `Edited draft '${path}'`, + result: 'Success' + }) + return `Pipeline node '${path}' updated as an unsaved draft on the canvas (not deployed).` + } + }, + { + def: removePipelineNodeToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path } = removePipelineNodeSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Discarding draft '${path}'...` }) + await pipeline.removeProposedNode(path) + toolCallbacks.setToolStatus(toolId, { + content: `Discarded draft '${path}'`, + result: 'Success' + }) + return `Discarded the unsaved draft at '${path}'.` + } + }, + { + def: testPipelineNodeToolDef, + requiresConfirmation: true, + confirmationMessage: 'Run pipeline node', + showDetails: true, + autoCollapseDetails: false, + fn: async ({ args, workspace, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path, args: runArgs } = testPipelineNodeSchema.parse(args) + return executeTestRun({ + jobStarter: async () => { + const jobId = await pipeline.testNode(path, runArgs ?? undefined) + if (!jobId) { + throw new Error(`Could not start a run for node '${path}'.`) + } + return jobId + }, + workspace, + toolCallbacks, + toolId, + startMessage: `Starting run of '${path}'...`, + contextName: 'script' + }) + } + } +] + +/** + * Pipeline-specific guidance appended to the global system prompt when a + * /pipeline editor is open. Describes the annotation model and the direct-draft + * workflow so the model uses the pipeline tools rather than the generic + * write_script draft tools. + */ +export function getPipelinePromptSection(ctx: PipelineContext): string { + return ` + +Data Pipeline editor (ACTIVE): +- The user has the /pipeline/${ctx.folder} editor open. A pipeline is a DAG of scripts (nodes) connected by storage assets (DuckLake tables, data tables, S3 objects, volumes, resources) and execution triggers. +- Annotations are top-of-file comments in the NODE'S OWN comment syntax: \`--\` for SQL (duckdb/postgresql), \`#\` for python3/bash, \`//\` for bun/TS. The \`//\` shown below is the TS form — translate it (a \`// pipeline\` line in a SQL node is a syntax error that won't deploy). +- A script becomes a pipeline node when its source starts with the \`// pipeline\` annotation. Declare execution-DAG inputs with \`// on \` (e.g. \`// on ducklake://main/orders\`). Outputs are inferred from what the body writes (wmill SDK calls / SQL CREATE TABLE / writeS3File); declare a managed output with \`// materialize \`. Optional badges: \`// partitioned \`, \`// freshness \`, \`// tag \`, \`// retry [delay]\`, \`// data_test ...\`. +- \`materialize\` (the managed output): \`// materialize \` means the runtime writes the node's output table FOR you — write the body as a single SELECT and the runtime wraps it in the create/replace, so do NOT also write your own CREATE TABLE / INSERT. IMPORTANT: \`// materialize\` is **DuckDB-only** and its target MUST be a DuckLake table (\`ducklake:///
\`) — deploy rejects it on any other language or target. For a \`python3\`/\`bun\`/\`postgresql\` node, do NOT use \`// materialize\`; write the output via the SDK instead (e.g. \`wmill.writeS3File(...)\`, a \`CREATE TABLE\` in postgresql, or \`wmill.databaseUrlFromResource\`/ducklake helpers) and let the output be inferred. Reach for \`duckdb\` when a node should materialize a DuckLake table. Write strategy: with no option it REPLACES the whole table each run (full refresh; the only mode whose output columns may change); \`// materialize append\` INSERT-appends rows (incremental); \`// materialize key=\` merges/upserts on \`\`. \`// materialize manual \` opts OUT of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. \`materialize\` is paired with partitioning for incremental pipelines: a \`// partitioned \` node runs once per partition (append/merge into a fixed-schema table), and the \`{partition}\` token — usable in any asset URI AND in the body SQL — is substituted with the current partition's IDENTITY string at run time. To filter the source to the active slice on a time grain, use the runtime-injected macro: \`WHERE wm_partition() = {partition}\`. \`wm_partition(ts)\` buckets a timestamp with the exact identity format the runtime used (daily/hourly/weekly/monthly), so it always matches and you never hand-write a \`strftime\` format. Do NOT write \`= TIMESTAMP {partition}\`: the identity string is not a valid timestamp literal for hourly/weekly/monthly and errors at runtime. For \`dynamic\` partitioning the identity is your caller-supplied key (not a timestamp, no macro), so filter on it directly: \`WHERE = {partition}\`. \`materialize\` is an output DECLARATION on the node — it is not a command; there is no "materialize run". +- Use get_pipeline_graph to see the current nodes/assets/triggers, and read_pipeline_node before editing one. +- Build new nodes with build_pipeline_node and edit existing ones with edit_pipeline_node. These apply directly as unsaved drafts on the canvas (like the flow/script editor applies AI edits) — they DO NOT deploy. There is no separate Accept/Reject step. Prefer these over the generic write_script/edit_script draft tools while a pipeline is open. +- Reuse existing asset paths from the graph when wiring a downstream node to an upstream one (read the upstream's write asset, then \`// on\` that same URI). +- Only deploy when the user explicitly asks; the user deploys drafts from the canvas.` +} diff --git a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte index 3709eda53c..2be52a32d2 100644 --- a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte @@ -1,96 +1,22 @@ -
-
- -
-
+ diff --git a/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte new file mode 100644 index 0000000000..dda1892baa --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte @@ -0,0 +1,199 @@ + + +{#if showSvg} +
+
+
+
+ + {@html svg} +
+
+ + + {#snippet settings()} +
+
+ {/snippet} +
+ {#if expanded} +
+ + {@html svg} +
+ {/if} +
+
+{:else} + +
{code}
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/shared.test.ts b/frontend/src/lib/components/copilot/chat/shared.test.ts index 56b6bc4d2b..1b77e060c4 100644 --- a/frontend/src/lib/components/copilot/chat/shared.test.ts +++ b/frontend/src/lib/components/copilot/chat/shared.test.ts @@ -17,7 +17,7 @@ vi.mock('$lib/components/flows/flowTree', () => ({ vi.mock('$lib/gen', () => ({ ScriptService: {}, FlowService: {}, - JobService: {}, + JobService: { getJob: vi.fn() }, ScheduleService: { previewSchedule: vi.fn(), createSchedule: vi.fn() @@ -53,6 +53,12 @@ vi.mock('@leeoniya/ufuzzy', () => ({ } })) +// deriveChatJobStatus's scheduled branch calls forLater; stub it deterministically +// (a real one pulls in stores/db-clock drift) — "later" = >5s in the future. +vi.mock('$lib/forLater', () => ({ + forLater: (scheduled: string | number | Date) => new Date(scheduled).getTime() > Date.now() + 5000 +})) + describe('createToolDef', () => { it('builds the create_trigger schema without top-level composition', async () => { const { createToolDef } = await import('./shared') @@ -136,14 +142,22 @@ describe('buildContextString', () => { path: 'f/flows/reporting', title: 'f/flows/reporting', summary: 'Reporting flow' + }, + { + type: 'workspace_app', + path: 'f/apps/dashboard', + title: 'f/apps/dashboard', + summary: 'Dashboard raw app' } ]) expect(context).toContain('SELECTED WORKSPACE ITEMS:') expect(context).toContain('- type: script, path: f/scripts/report') expect(context).toContain('- type: flow, path: f/flows/reporting') + expect(context).toContain('- type: raw_app, path: f/apps/dashboard') expect(context).not.toContain('Report script') expect(context).not.toContain('Reporting flow') + expect(context).not.toContain('Dashboard raw app') expect(context).not.toContain('Code:') expect(context).not.toContain('Value:') }) @@ -637,7 +651,18 @@ describe('isActiveUserQuestion', () => { expect(isActiveUserQuestion(toolMessage())).toBe(true) }) - it('is false once a choice has been selected', async () => { + it('is false once choices have been selected', async () => { + const { isActiveUserQuestion } = await import('./shared') + expect( + isActiveUserQuestion( + toolMessage({ + userQuestion: { question: 'Pick one', choices: ['a', 'b'], selectedChoices: ['a'] } + }) + ) + ).toBe(false) + }) + + it('is false once a legacy scalar selectedChoice is present', async () => { const { isActiveUserQuestion } = await import('./shared') expect( isActiveUserQuestion( @@ -648,6 +673,17 @@ describe('isActiveUserQuestion', () => { ).toBe(false) }) + it('stays active when selectedChoices is present but empty', async () => { + const { isActiveUserQuestion } = await import('./shared') + expect( + isActiveUserQuestion( + toolMessage({ + userQuestion: { question: 'Pick one', choices: ['a', 'b'], selectedChoices: [] } + }) + ) + ).toBe(true) + }) + it('is false when the question was canceled', async () => { const { isActiveUserQuestion } = await import('./shared') expect( @@ -681,3 +717,184 @@ describe('isActiveUserQuestion', () => { expect(isActiveUserQuestion(assistantMessage)).toBe(false) }) }) + +describe('pollJobCompletion detach', () => { + function makeCallbacks() { + return { + setToolStatus: vi.fn(), + removeToolStatus: vi.fn(), + onJobStatus: vi.fn() + } + } + + it('detaches immediately (no polling) when detachAfterMs is 0', async () => { + const { pollJobCompletion } = await import('./shared') + const { JobService } = await import('$lib/gen') + const getJob = vi.mocked(JobService.getJob) + getJob.mockReset() + const cbs = makeCallbacks() + + const outcome = await pollJobCompletion('job1', 'w', 'tool1', cbs as any, { detachAfterMs: 0 }) + + expect(outcome).toBe('detached') + expect(getJob).not.toHaveBeenCalled() + }) + + it('detaches after the inline budget when the job is still running', async () => { + vi.useFakeTimers() + try { + const { pollJobCompletion } = await import('./shared') + const { JobService } = await import('$lib/gen') + const getJob = vi.mocked(JobService.getJob) + getJob.mockReset() + getJob.mockResolvedValue({ type: 'QueuedJob', running: true } as any) + const cbs = makeCallbacks() + + // detachAfterMs 2000 → 2 polls at 1s each, then detach. + const promise = pollJobCompletion('job1', 'w', 'tool1', cbs as any, { detachAfterMs: 2000 }) + await vi.advanceTimersByTimeAsync(2000) + + expect(await promise).toBe('detached') + // Status is reported as running during the wait (alongside the trimmed + // Job snapshot that feeds JobStatusIcon). + expect(cbs.onJobStatus).toHaveBeenCalledWith( + 'job1', + expect.objectContaining({ status: 'running' }) + ) + } finally { + vi.useRealTimers() + } + }) + + it('returns the completed job when it finishes within the inline budget', async () => { + vi.useFakeTimers() + try { + const { pollJobCompletion } = await import('./shared') + const { JobService } = await import('$lib/gen') + const getJob = vi.mocked(JobService.getJob) + getJob.mockReset() + const completed = { type: 'CompletedJob', success: true, result: 42 } + getJob.mockResolvedValue(completed as any) + const cbs = makeCallbacks() + + const promise = pollJobCompletion('job1', 'w', 'tool1', cbs as any, { detachAfterMs: 15000 }) + await vi.advanceTimersByTimeAsync(1000) + + expect(await promise).toBe(completed) + } finally { + vi.useRealTimers() + } + }) + + it('legacy mode (no detach) throws a timeout error when the job never completes', async () => { + vi.useFakeTimers() + try { + const { pollJobCompletion } = await import('./shared') + const { JobService } = await import('$lib/gen') + const getJob = vi.mocked(JobService.getJob) + getJob.mockReset() + getJob.mockResolvedValue({ type: 'QueuedJob', running: true } as any) + const cbs = makeCallbacks() + + const promise = pollJobCompletion('job1', 'w', 'tool1', cbs as any) + const assertion = expect(promise).rejects.toThrow('timed out') + await vi.advanceTimersByTimeAsync(60000) + await assertion + expect(cbs.setToolStatus).toHaveBeenCalledWith( + 'tool1', + expect.objectContaining({ error: expect.any(String) }) + ) + } finally { + vi.useRealTimers() + } + }) +}) + +describe('deriveChatJobStatus', () => { + // CompletedJob is discriminated by the presence of a `success` key; the branch + // order deliberately mirrors JobStatusIcon so the badge and scalar never drift. + it('maps a canceled completed job to canceled (canceled wins over success=false)', async () => { + const { deriveChatJobStatus } = await import('./shared') + expect(deriveChatJobStatus({ success: false, canceled: true } as any)).toBe('canceled') + }) + + it('maps a successful completed job to success', async () => { + const { deriveChatJobStatus } = await import('./shared') + expect(deriveChatJobStatus({ success: true, canceled: false } as any)).toBe('success') + }) + + it('maps a non-canceled failed completed job to failure', async () => { + const { deriveChatJobStatus } = await import('./shared') + expect(deriveChatJobStatus({ success: false, canceled: false } as any)).toBe('failure') + }) + + it('maps a running suspended queued job to suspended', async () => { + const { deriveChatJobStatus } = await import('./shared') + expect(deriveChatJobStatus({ running: true, suspend: 1 } as any)).toBe('suspended') + }) + + it('maps a running queued job to running', async () => { + const { deriveChatJobStatus } = await import('./shared') + expect(deriveChatJobStatus({ running: true } as any)).toBe('running') + }) + + it('maps a future-scheduled queued job to scheduled', async () => { + const { deriveChatJobStatus } = await import('./shared') + const future = new Date(Date.now() + 3_600_000).toISOString() + expect(deriveChatJobStatus({ running: false, scheduled_for: future } as any)).toBe('scheduled') + }) + + it('maps a plain (non-running, non-scheduled) queued job to queued', async () => { + const { deriveChatJobStatus } = await import('./shared') + expect(deriveChatJobStatus({ running: false } as any)).toBe('queued') + }) +}) + +describe('trimJob', () => { + const HEAVY = ['logs', 'args', 'result', 'raw_code', 'raw_flow', 'flow_status'] + + it('preserves the ABSENCE of a success key on a queued job (JobStatusIcon in-operator invariant)', async () => { + const { trimJob } = await import('./shared') + const queued = { + id: 'j1', + running: true, + logs: 'x', + args: {}, + result: 1, + raw_code: 'c', + raw_flow: {}, + flow_status: {} + } + const trimmed = trimJob(queued as any) + // The load-bearing invariant: a running/queued job must NOT gain a `success` + // key, or deriveChatJobStatus/JobStatusIcon would misread it as completed. + expect('success' in trimmed).toBe(false) + expect(trimmed.running).toBe(true) + expect(trimmed.id).toBe('j1') + }) + + it('deletes the six heavy fields but keeps the status-discriminant scalar', async () => { + const { trimJob } = await import('./shared') + const job = { + id: 'j1', + success: true, + logs: 'x', + args: { a: 1 }, + result: [1], + raw_code: 'c', + raw_flow: { modules: [] }, + flow_status: { step: 0 } + } + const trimmed = trimJob(job as any) as Record + for (const k of HEAVY) expect(k in trimmed).toBe(false) + expect('success' in trimmed).toBe(true) + expect(trimmed.success).toBe(true) + }) + + it('does not mutate the input job', async () => { + const { trimJob } = await import('./shared') + const job = { id: 'j1', success: true, result: 42 } + trimJob(job as any) + expect(job.result).toBe(42) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/shared.ts b/frontend/src/lib/components/copilot/chat/shared.ts index aecea1ddfc..dd69139e60 100644 --- a/frontend/src/lib/components/copilot/chat/shared.ts +++ b/frontend/src/lib/components/copilot/chat/shared.ts @@ -3,6 +3,7 @@ import type { ChatCompletionMessageFunctionToolCall, ChatCompletionMessageParam } from 'openai/resources/chat/completions.mjs' +import type { UserDraftItemKind } from '$lib/gen' /** * Special module IDs used throughout the flow system @@ -27,6 +28,7 @@ import { ScriptService, FlowService, JobService, + type Job, type CompletedJob, type FlowValue, type FlowModule, @@ -36,6 +38,7 @@ import { } from '$lib/gen' import uFuzzy from '@leeoniya/ufuzzy' import { emptyString } from '$lib/utils' +import { forLater } from '$lib/forLater' import { scriptLangToEditorLang } from '$lib/scripts' import { getCurrentModel } from '$lib/aiStore' import { type editor as meditor } from 'monaco-editor' @@ -422,6 +425,11 @@ export function buildContextString(selectedContext: ContextElement[]): string { workspaceItemsContext = 'SELECTED WORKSPACE ITEMS:\n' } workspaceItemsContext += `- type: flow, path: ${context.path}\n` + } else if (context.type === 'workspace_app') { + if (!workspaceItemsContext) { + workspaceItemsContext = 'SELECTED WORKSPACE ITEMS:\n' + } + workspaceItemsContext += `- type: raw_app, path: ${context.path}\n` } } @@ -484,15 +492,36 @@ export type CreatedResourceAction = { triggerKind?: CreatedResourceTriggerKind } -export type ToolDisplayAction = CreatedResourceAction +// A clickable chip that deep-links the user to an in-app page (e.g. Runs filtered to +// a script's failures). Used for cross-page navigation from the chat; the handler is +// registered by a top-level layout and calls `goto(url)`. +export type NavigateAction = { + id: string + type: 'navigate' + label: string + url: string + // Which page the chip opens (runs, schedules, variables, …); drives its icon/title. + page: string +} + +export type ToolDisplayAction = CreatedResourceAction | NavigateAction export type UserQuestionDisplay = { question: string choices: string[] - selectedChoice?: string + multiSelect?: boolean + selectedChoices?: string[] // canonical answer (new code writes only this) + selectedChoice?: string // legacy/read-only: pre-multiselect persisted history canceled?: boolean } +// The single place that understands the legacy answer shape: new code writes +// selectedChoices, but history persisted before multi-select only has the +// scalar selectedChoice. Read answers through this so both shapes resolve. +export function answeredChoices(q: UserQuestionDisplay): string[] | undefined { + return q.selectedChoices ?? (q.selectedChoice ? [q.selectedChoice] : undefined) +} + export type ToolDisplayMessage = { role: 'tool' tool_call_id: string @@ -553,11 +582,23 @@ export function isActiveUserQuestion(message: DisplayMessage | undefined): boole message.userQuestion && message.isLoading && !message.error && - !message.userQuestion.selectedChoice && + !answeredChoices(message.userQuestion)?.length && !message.userQuestion.canceled ) } +// Fires after every tool call resolves, with the tool name. Lets a host (e.g. +// the sessions page) react to mutating tools — refreshing previews — without +// the tool layer knowing about the UI. Single slot; the consumer filters by name +// and reads the tool args (e.g. the mutated item's `path`) to scope its refresh. +let toolCompletionListener: ((toolName: string, args: any) => void) | undefined + +export function setToolCompletionListener( + fn: ((toolName: string, args: any) => void) | undefined +): void { + toolCompletionListener = fn +} + async function callTool({ tools, functionName, @@ -581,7 +622,9 @@ async function callTool({ `Unknown tool call: ${functionName}. Probably not in the correct mode, use the change_mode tool to switch to the correct mode.` ) } - return tool.fn({ args, workspace, helpers, toolCallbacks, toolId }) + const result = await tool.fn({ args, workspace, helpers, toolCallbacks, toolId }) + toolCompletionListener?.(functionName, args) + return result } type MaybePromise = T | Promise @@ -740,9 +783,100 @@ export interface Tool { showFade?: boolean } +/** Status of a job the chat started and tracks in the jobs tray. Mirrors the + * runs page: `suspended` = a flow step waiting for approval, `scheduled` = a run + * scheduled for later. Kept in lockstep with `ChatJob.job` (see below). */ +export type ChatJobStatus = + | 'queued' + | 'running' + | 'suspended' + | 'scheduled' + | 'success' + | 'failure' + | 'canceled' + +/** Serializable identity of a tool's terminal result formatter, stored on a ChatJob + * so a detached job that survives a reload can still reconstruct the shaped result + * its launching tool would have produced (see formatChatJobCompletion). A closure + * can't be persisted to IndexedDB; this discriminant can. */ +export type ChatJobResultFormat = { kind: 'datatable'; datatableName: string } + +/** A job the chat started and is tracking. Rendered in the jobs tray, persisted + * with the chat, and advanced by a single background poller on the manager. */ +export type ChatJob = { + jobId: string + /** Pairs with the ToolDisplayMessage card that launched it. */ + toolCallId: string + kind: 'script' | 'flow' + /** Path or step label shown in the tray row. */ + label: string + workspace: string + createdAt: number + status: ChatJobStatus + durationMs?: number + /** True once it left the inline wait and is polled in the background. */ + detached: boolean + /** Notify-only: whether its completion has been surfaced to the model yet. */ + reported: boolean + /** Trimmed snapshot of the last fetched Job (heavy fields stripped, see + * `trimJob`), fed to `` so the tray badge matches the runs page + * exactly. Always written together with `status` from the SAME job so the two + * can't drift. Undefined only before the first fetch. */ + job?: Job + /** Set by tools that shape their result (e.g. exec_datatable_sql). Persisted, so + * a detached job that finishes after a reload still reports through the tool's + * result contract rather than generic job output. */ + resultFormat?: ChatJobResultFormat +} + +/** Derive the tray status from a fetched Job. Deliberately mirrors the branch + * order of JobStatusIcon.svelte so the scalar status and the badge never + * disagree. */ +export function deriveChatJobStatus(job: Job): ChatJobStatus { + if ('success' in job) { + return job.canceled ? 'canceled' : job.success ? 'success' : 'failure' + } + // QueuedJob + if (job.running && job.suspend) return 'suspended' + if (job.running) return 'running' + if (job.scheduled_for && forLater(job.scheduled_for)) return 'scheduled' + return 'queued' +} + +/** Strip the heavy fields from a fetched Job before storing it on a ChatJob (the + * tray only needs the status-discriminant scalars JobStatusIcon reads). + * + * MUST clone + delete — never rebuild as an object literal. JobStatusIcon + * discriminates with the `in` operator (`'success' in job`), which tests KEY + * PRESENCE, not truthiness. A literal that always carries a `success` key would + * make every running/queued job misrender as a completed (failed) job. */ +export function trimJob(job: Job): Job { + const trimmed = { ...job } as Record + delete trimmed.logs + delete trimmed.args + delete trimmed.result + delete trimmed.raw_code + delete trimmed.raw_flow + delete trimmed.flow_status + return trimmed as unknown as Job +} + +/** The subset supplied when a job first starts; the manager fills in the rest. */ +export type ChatJobInit = Pick< + ChatJob, + 'jobId' | 'toolCallId' | 'kind' | 'label' | 'workspace' | 'resultFormat' +> + export interface ToolCallbacks { setToolStatus: (id: string, metadata?: Partial) => void removeToolStatus: (id: string) => void + /** Job-tracking hooks, wired only by the global/sessions chat (mode === GLOBAL). + * Their presence is what enables detach-into-background in executeTestRun; when + * absent (in-editor script/flow/pipeline chats), test runs stay blocking with a + * 60s cap. */ + onJobStarted?: (job: ChatJobInit) => void + onJobStatus?: (jobId: string, update: Partial) => void + onJobDetached?: (jobId: string) => void /** Streamed reasoning/thinking deltas, rendered as a collapsible block in the chat. */ onReasoningDelta?: (token: string) => void /** Fired when the model starts reasoning — drives a "Thinking" indicator even when @@ -753,7 +887,16 @@ export interface ToolCallbacks { requestUserQuestion?: ( toolId: string, question: UserQuestionDisplay - ) => Promise + ) => Promise + /** Records a workspace item the tool call created/edited/deleted, by its + * canonical (itemKind, storagePath). Session chats wire this to accumulate the + * chat's modified-items mask; the global side-panel chat omits it (no-op). */ + onItemModified?: (itemKind: UserDraftItemKind, storagePath: string) => void + /** A tool deployed a draft: the mask entry moves from the draft's storage path + * to the deployed path (they differ for synthetic draft-only storage keys). */ + onItemDeployed?: (itemKind: UserDraftItemKind, storagePath: string, deployedPath: string) => void + /** A tool discarded a draft: the chat's touch on the item is undone. */ + onItemDiscarded?: (itemKind: UserDraftItemKind, storagePath: string) => void } export function createToolDef( @@ -936,14 +1079,20 @@ export async function buildSchemaForTool( throw new Error(`Invalid flow inputs schema: ${invalidProperties.join(', ')}`) } - toolDef.function.parameters = { ...schema, additionalProperties: false } + // Anthropic requires input_schema.type to be present; flows with no inputs + // can produce a sparse schema (e.g. { order: [] }) lacking it. + toolDef.function.parameters = { type: 'object', ...schema, additionalProperties: false } // recursively normalize provider-incompatible schema fragments normalizeToolParameterSchema(toolDef.function.parameters) // OPEN AI models don't support strict mode well with schema with complex properties, so we disable it const model = getCurrentModel() - if (model.provider === 'openai' || model.provider === 'azure_openai') { + if ( + model.provider === 'openai' || + model.provider === 'azure_openai' || + model.provider === 'azure_foundry' + ) { toolDef.function.strict = false } return true @@ -968,6 +1117,16 @@ const MAX_RESULT_LENGTH = 12000 const MAX_LOG_LENGTH = 4000 export const MAX_RUNNABLE_CONTENT_LENGTH = 20000 +/** How long a test run is awaited inline before it detaches into the background + * (global/sessions chat only). Quick runs finish well inside this; slow ones are + * handed to the background poller so the chat loop is freed. */ +export const DETACH_AFTER_MS = 15000 + +/** Upper bound on a model-requested inline wait. Beyond this, backgrounding is + * almost always better than holding the chat turn, so we clamp rather than let + * the model block the loop for minutes. */ +export const MAX_DETACH_AFTER_MS = 120000 + export interface TestRunConfig { jobStarter: () => Promise workspace: string @@ -975,17 +1134,56 @@ export interface TestRunConfig { toolId: string startMessage?: string contextName: 'script' | 'flow' + /** Detach immediately instead of waiting the inline budget (the model's opt-in). */ + background?: boolean + /** Overrides the inline wait budget (ms) before the job detaches into the tray. + * The model's opt-in for jobs it expects to take a bit longer than the 15s + * default but still wants to await in-turn. Ignored when `background` is set + * (that detaches immediately). Clamped to MAX_DETACH_AFTER_MS. */ + detachAfterMs?: number + /** Human label for the jobs tray row (path / step id). Defaults to the job id. */ + label?: string + /** Overrides the default "…test started, waiting for completion" status while the + * job runs inline (e.g. an SQL tool shows "SQL running…"). */ + runningMessage?: string + /** Custom terminal formatting for the INLINE completion path (callers whose + * result isn't a plain test-run summary, e.g. exec_datatable_sql shaping rows). + * Returns the string handed to the model plus the tool-card patch. When omitted, + * the default summary is used. For the DETACHED/rehydrated path, supply + * `resultFormat` too so the completion can be reconstructed without this closure. */ + formatCompletion?: BackgroundJobFormatter + /** Serializable twin of `formatCompletion`, stored on the ChatJob so a detached + * job that finishes after a reload still reports through the tool's result + * contract (see AIChatManager.#onBackgroundJobComplete). */ + resultFormat?: ChatJobResultFormat } -// Common job polling function +/** Terminal formatter a tool supplies so its result keeps the same model-visible + * contract whether the job finishes inline or completes after detaching into the + * background. */ +export type BackgroundJobFormatter = (job: CompletedJob) => { + llmText: string + card: Partial +} + +// Common job polling function. +// +// Two modes, selected by whether `detachAfterMs` is provided: +// - Blocking (undefined): poll up to 60×1s, then set a timeout error and throw. +// Used by in-editor chats, which have no jobs tray to hand off to. +// - Detach (a number): poll only for that inline budget; if the job is still +// running when it elapses, resolve `'detached'` instead of throwing so the +// caller can background the job. `0` detaches without polling at all. export async function pollJobCompletion( jobId: string, workspace: string, toolId: string, - toolCallbacks: ToolCallbacks -): Promise { + toolCallbacks: ToolCallbacks, + options?: { detachAfterMs?: number } +): Promise { + const detachEnabled = options?.detachAfterMs !== undefined + const maxAttempts = detachEnabled ? Math.ceil((options?.detachAfterMs ?? 0) / 1000) : 60 let attempts = 0 - const maxAttempts = 60 let job: CompletedJob | null = null while (attempts < maxAttempts) { @@ -1004,14 +1202,22 @@ export async function pollJobCompletion( job = fetchedJob break } + // Keep the tray's status + Job snapshot fresh during the inline wait. + toolCallbacks.onJobStatus?.(jobId, { + status: deriveChatJobStatus(fetchedJob), + job: trimJob(fetchedJob) + }) } catch (error) { - if (attempts >= maxAttempts) { + if (!detachEnabled && attempts >= maxAttempts) { throw error } } } if (!job) { + if (detachEnabled) { + return 'detached' + } toolCallbacks.setToolStatus(toolId, { content: 'Test timed out', error: 'Execution timed out or failed to complete' @@ -1093,8 +1299,62 @@ export async function buildTestRunArgs( return parsedArgs } +// The string handed back to the model when a job is backgrounded. It carries the +// job id so the model can pull status/logs on demand (get_job_logs / list_runs), +// and tells it the completion will be reported later (notify-only wake). +function backgroundedSummary(jobId: string, label: string): string { + return ( + `Job ${jobId} for "${label}" is taking a while and is now running in the background — ` + + `the chat is free to continue and you'll be told when it finishes. ` + + `To inspect it now, call get_job_logs with id="${jobId}" (or list_runs); ` + + `to stop it, call cancel_job with id="${jobId}".` + ) +} + +// Tool-card status patch for a completed background job. Mirrors the inline +// terminal branch of executeTestRun so a job that finished in the background +// fills its card the same way one that finished inline does. +export function completedJobToolStatus(job: CompletedJob): Partial { + // A canceled job isn't a `success`, but it isn't a failure either — the user + // stopped it — so don't dress the card as an error. + if (job.canceled) { + return { content: 'Background job canceled', logs: formatLogs(job.logs) } + } + return { + content: `Background job ${job.success ? 'completed successfully' : 'failed'}`, + result: formatResult(job.result), + logs: formatLogs(job.logs), + ...(job.success ? {} : { error: getErrorMessage(job.result) }) + } +} + +// Short completion note handed to the model on its next turn (notify-only wake). +// Carries the id so the model can pull full logs via get_job_logs on demand. +export function backgroundJobCompletionNote( + jobId: string, + label: string, + job: CompletedJob, + // When the launching tool supplied a formatter (e.g. exec_datatable_sql), pass its + // `llmText` here so the notify-only note carries the same shaped result the inline + // path would have returned — row-capped, friendly errors — instead of the raw job + // result. Omitted → the generic 2000-char result head. + formattedResult?: string +): string { + const status = job.success ? 'succeeded' : 'FAILED' + const resultHead = formattedResult ?? formatResult(job.result).slice(0, 2000) + return ( + `Background job ${jobId} for "${label}" ${status}.\n` + + `Result: ${resultHead}\n` + + `(For full logs call get_job_logs with id="${jobId}".)` + ) +} + // Main execution function for test runs export async function executeTestRun(config: TestRunConfig): Promise { + // Detach-into-background is enabled only when the host wired the job hooks + // (global/sessions chat). Otherwise this stays a blocking call. + const detachEnabled = !!config.toolCallbacks.onJobStarted + const label = config.label ?? config.contextName try { config.toolCallbacks.setToolStatus(config.toolId, { content: config.startMessage || `Starting ${config.contextName} test...` @@ -1104,17 +1364,58 @@ export async function executeTestRun(config: TestRunConfig): Promise { const contextName = config.contextName.charAt(0).toUpperCase() + config.contextName.slice(1) - config.toolCallbacks.setToolStatus(config.toolId, { - content: `${contextName} test started, waiting for completion...` + // Register the job so the tray shows it from the moment it is queued. Carry the + // serializable resultFormat so a job that later detaches (and may outlive a + // reload) can reconstruct the same model-visible contract this inline path + // applies below. + config.toolCallbacks.onJobStarted?.({ + jobId, + toolCallId: config.toolId, + kind: config.contextName, + label, + workspace: config.workspace, + resultFormat: config.resultFormat }) - const job = await pollJobCompletion( + config.toolCallbacks.setToolStatus(config.toolId, { + content: config.runningMessage ?? `${contextName} test started, waiting for completion...` + }) + + const outcome = await pollJobCompletion( jobId, config.workspace, config.toolId, - config.toolCallbacks + config.toolCallbacks, + detachEnabled + ? { + detachAfterMs: config.background + ? 0 + : Math.min(config.detachAfterMs ?? DETACH_AFTER_MS, MAX_DETACH_AFTER_MS) + } + : undefined ) + if (outcome === 'detached') { + config.toolCallbacks.onJobDetached?.(jobId) + config.toolCallbacks.setToolStatus(config.toolId, { + content: `${contextName} test running in background (job ${jobId})` + }) + return backgroundedSummary(jobId, label) + } + + const job = outcome + config.toolCallbacks.onJobStatus?.(jobId, { + status: deriveChatJobStatus(job), + durationMs: job.duration_ms, + job: trimJob(job) + }) + + if (config.formatCompletion) { + const { llmText, card } = config.formatCompletion(job) + config.toolCallbacks.setToolStatus(config.toolId, card) + return llmText + } + config.toolCallbacks.setToolStatus(config.toolId, { content: `${contextName} test ${job.success ? 'completed successfully' : 'failed'}`, result: formatResult(job.result), @@ -1147,6 +1448,10 @@ export type FlowStepTestRunConfig = { workspace: string toolCallbacks: ToolCallbacks toolId: string + background?: boolean + /** Inline wait budget (ms) before the step job detaches into the tray; forwarded + * to executeTestRun. Ignored when `background` is set. */ + detachAfterMs?: number loadScript?: FlowStepScriptLoader loadFlowPreviewValue?: FlowStepPreviewLoader } @@ -1188,6 +1493,8 @@ export async function executeFlowStepTestRun({ workspace, toolCallbacks, toolId, + background, + detachAfterMs, loadScript = loadDeployedScriptForFlowStep, loadFlowPreviewValue }: FlowStepTestRunConfig): Promise { @@ -1221,7 +1528,10 @@ export async function executeFlowStepTestRun({ toolCallbacks, toolId, startMessage: `Starting test run of step "${stepId}"...`, - contextName: 'script' + contextName: 'script', + label: `step ${stepId}`, + background, + detachAfterMs }) } @@ -1242,7 +1552,10 @@ export async function executeFlowStepTestRun({ toolCallbacks, toolId, startMessage: `Starting test run of script step "${stepId}"...`, - contextName: 'script' + contextName: 'script', + label: `step ${stepId}`, + background, + detachAfterMs }) } @@ -1263,7 +1576,10 @@ export async function executeFlowStepTestRun({ toolCallbacks, toolId, startMessage: `Starting test run of draft flow step "${stepId}"...`, - contextName: 'flow' + contextName: 'flow', + label: `step ${stepId}`, + background, + detachAfterMs }) } @@ -1278,7 +1594,10 @@ export async function executeFlowStepTestRun({ toolCallbacks, toolId, startMessage: `Starting test run of flow step "${stepId}"...`, - contextName: 'flow' + contextName: 'flow', + label: `step ${stepId}`, + background, + detachAfterMs }) } diff --git a/frontend/src/lib/components/copilot/chat/toolCallArguments.test.ts b/frontend/src/lib/components/copilot/chat/toolCallArguments.test.ts new file mode 100644 index 0000000000..53246bc50e --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/toolCallArguments.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from 'vitest' +import type { ChatCompletionMessageParam } from 'openai/resources/chat/completions.mjs' +import { hasValidToolCallArguments, sanitizeToolCallArguments } from './toolCallArguments' + +describe('hasValidToolCallArguments', () => { + it('accepts empty or valid JSON arguments', () => { + expect(hasValidToolCallArguments(undefined)).toBe(true) + expect(hasValidToolCallArguments('')).toBe(true) + expect(hasValidToolCallArguments('{}')).toBe(true) + expect(hasValidToolCallArguments('{"path": "u/admin/app", "content": "x"}')).toBe(true) + }) + + it('rejects arguments truncated mid-stream', () => { + expect(hasValidToolCallArguments('{"path": "u/admin/app", "old_string": "setMess')).toBe(false) + expect(hasValidToolCallArguments('{"path": "u/admin/app"')).toBe(false) + }) +}) + +describe('sanitizeToolCallArguments', () => { + const poisoned: ChatCompletionMessageParam = { + role: 'assistant', + tool_calls: [ + { + id: 'call_bad', + type: 'function', + function: { name: 'patch_app_file', arguments: '{"path": "u/x", "old_string": "trunc' } + }, + { + id: 'call_ok', + type: 'function', + function: { name: 'read_file', arguments: '{"file": "a.txt"}' } + } + ] + } + + it('replaces only unparseable arguments with {}', () => { + const [sanitized] = sanitizeToolCallArguments([poisoned]) as any[] + expect(sanitized.tool_calls[0].function.arguments).toBe('{}') + expect(sanitized.tool_calls[1].function.arguments).toBe('{"file": "a.txt"}') + // The stored history object is not mutated + expect((poisoned as any).tool_calls[0].function.arguments).toContain('trunc') + }) + + it('rewrites empty arguments to {} so replayed history stays parseable', () => { + const emptyArgs: ChatCompletionMessageParam = { + role: 'assistant', + tool_calls: [{ id: 'c1', type: 'function', function: { name: 'list_files', arguments: '' } }] + } + const [sanitized] = sanitizeToolCallArguments([emptyArgs]) as any[] + expect(sanitized.tool_calls[0].function.arguments).toBe('{}') + }) + + it('returns untouched messages by reference', () => { + const user: ChatCompletionMessageParam = { role: 'user', content: 'hi' } + const validAssistant: ChatCompletionMessageParam = { + role: 'assistant', + tool_calls: [{ id: 'c1', type: 'function', function: { name: 'read_file', arguments: '{}' } }] + } + const result = sanitizeToolCallArguments([user, validAssistant]) + expect(result[0]).toBe(user) + expect(result[1]).toBe(validAssistant) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/toolCallArguments.ts b/frontend/src/lib/components/copilot/chat/toolCallArguments.ts new file mode 100644 index 0000000000..997186a611 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/toolCallArguments.ts @@ -0,0 +1,58 @@ +import type { ChatCompletionMessageParam } from 'openai/resources/chat/completions.mjs' + +/** + * A tool call whose `arguments` string is not valid JSON (typically a stream + * cut mid-arguments) must never reach a provider: the whole request is + * rejected, and once persisted in the session history every follow-up request + * fails the same way. + * + * Empty arguments count as valid: some providers stream '' for no-arg tool + * calls, and flagging those would wrongly report the tool as not executed. + * Callers persisting arguments must normalize '' to '{}' themselves. + */ +export function hasValidToolCallArguments(args: string | undefined): boolean { + if (!args) { + return true + } + try { + JSON.parse(args) + return true + } catch { + return false + } +} + +// '' is valid per hasValidToolCallArguments but JSON.parse('') still throws +// provider-side, so replaying history additionally requires non-empty args. +function isReplayableArguments(args: string | undefined): boolean { + return !!args && hasValidToolCallArguments(args) +} + +/** + * Replaces unparseable or empty assistant tool_call arguments with '{}' in the + * outgoing copy of the history, so a session whose persisted history contains + * a truncated tool call recovers instead of failing every request. The paired + * tool result already tells the model the call failed. + */ +export function sanitizeToolCallArguments( + messages: ChatCompletionMessageParam[] +): ChatCompletionMessageParam[] { + return messages.map((m) => { + if ( + m.role !== 'assistant' || + !m.tool_calls?.some( + (t) => t.type === 'function' && !isReplayableArguments(t.function.arguments) + ) + ) { + return m + } + return { + ...m, + tool_calls: m.tool_calls.map((t) => + t.type === 'function' && !isReplayableArguments(t.function.arguments) + ? { ...t, function: { ...t.function, arguments: '{}' } } + : t + ) + } + }) +} diff --git a/frontend/src/lib/components/copilot/chat/typewriterReveal.test.ts b/frontend/src/lib/components/copilot/chat/typewriterReveal.test.ts new file mode 100644 index 0000000000..b99048c360 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/typewriterReveal.test.ts @@ -0,0 +1,200 @@ +import { describe, expect, it } from 'vitest' +import { TypewriterReveal, type TypewriterRevealOptions } from './typewriterReveal' + +// A fake clock + manual scheduler modelling requestAnimationFrame: each frame() +// advances the clock by `dt` ms, then fires whatever callback is currently +// scheduled (which may reschedule the next). This drives the pacing +// deterministically without a browser. +class FakeScheduler { + t = 0 + scheduleCount = 0 + private queue = new Map void>() + private id = 0 + + now = () => this.t + schedule = (cb: () => void) => { + this.scheduleCount++ + const h = ++this.id + this.queue.set(h, cb) + return h + } + cancel = (h: unknown) => { + this.queue.delete(h as number) + } + pending() { + return this.queue.size + } + frame(dt: number) { + this.t += dt + const cbs = [...this.queue.values()] + this.queue.clear() + cbs.forEach((cb) => cb()) + } + frames(count: number, dt = 16) { + for (let i = 0; i < count; i++) this.frame(dt) + } +} + +function makeReveal(sched: FakeScheduler, opts: Partial = {}) { + const chunks: string[] = [] + const reveal = new TypewriterReveal({ + onReveal: (c) => chunks.push(c), + now: sched.now, + schedule: sched.schedule, + cancel: sched.cancel, + ...opts + }) + return { reveal, chunks, revealed: () => chunks.join('') } +} + +// No lone surrogate at any string end (a split pair would leave one). +function hasLoneSurrogate(s: string): boolean { + for (let i = 0; i < s.length; i++) { + const c = s.charCodeAt(i) + if (c >= 0xd800 && c <= 0xdbff) { + const next = s.charCodeAt(i + 1) + if (!(next >= 0xdc00 && next <= 0xdfff)) return true + i++ + } else if (c >= 0xdc00 && c <= 0xdfff) { + return true + } + } + return false +} + +describe('TypewriterReveal', () => { + it('reveals gradually — a burst is not fully painted on the first frame', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched) + const text = 'x'.repeat(300) + reveal.push(text) + sched.frame(16) + expect(revealed().length).toBeGreaterThan(0) + expect(revealed().length).toBeLessThan(text.length) + }) + + it('preserves text exactly after flush (no loss, no duplication)', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched) + const parts = ['Here are ', 'some names: ', 'Charles, George, ', 'Alfred, Harold.'] + parts.forEach((p) => reveal.push(p)) + sched.frames(3) // reveal only part of it + reveal.flush() + expect(revealed()).toBe(parts.join('')) + }) + + it('flushes repeatedly across tool boundaries without loss or duplication', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched) + // Each segment is a message that ends in a flush (as onMessageEnd does at a + // tool-call boundary); the buffer is compacted between them. + const segments = ['first reply', 'second reply', 'third reply'] + segments.forEach((seg) => { + reveal.push(seg) + sched.frames(2) // partially reveal + reveal.flush() + }) + expect(revealed()).toBe(segments.join('')) + }) + + it('reset() drops un-revealed backlog', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched) + const text = 'y'.repeat(300) + reveal.push(text) + sched.frame(16) // reveal a prefix + const afterOneFrame = revealed() + expect(afterOneFrame.length).toBeLessThan(text.length) + expect(text.startsWith(afterOneFrame)).toBe(true) + reveal.reset() + sched.frames(20) + expect(revealed()).toBe(afterOneFrame) // nothing further reached onReveal + }) + + it('fast-path: a backlog above the cap is revealed whole in one emit', () => { + const sched = new FakeScheduler() + const { reveal, chunks, revealed } = makeReveal(sched, { maxBacklogChars: 1500 }) + const text = 'z'.repeat(2000) + reveal.push(text) + sched.frame(16) + expect(revealed()).toBe(text) + expect(chunks.length).toBe(1) // dumped, not stretched + }) + + it('never splits a surrogate pair across chunks', () => { + const sched = new FakeScheduler() + const { reveal, chunks, revealed } = makeReveal(sched, { smoothingMs: 5000 }) // force ~1 char/frame + const text = 'a😀b👨‍👩‍👧c' + reveal.push(text) + sched.frames(60) + reveal.flush() + expect(revealed()).toBe(text) + chunks.forEach((c) => expect(hasLoneSurrogate(c)).toBe(false)) + }) + + it('instant mode reveals synchronously and never schedules', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched, { instant: true }) + reveal.push('hello ') + reveal.push('world') + expect(revealed()).toBe('hello world') + expect(sched.scheduleCount).toBe(0) + }) + + it('steady-state backlog converges to ~arrivalRate × smoothingMs (no runaway, no stall)', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched, { smoothingMs: 500 }) + const perFrame = 10 // chars pushed each 16ms frame → 0.625 chars/ms + let pushed = 0 + for (let i = 0; i < 200; i++) { + reveal.push('c'.repeat(perFrame)) + pushed += perFrame + sched.frame(16) + } + const backlog = pushed - revealed().length + // Target ≈ 0.625 * 500 = ~312. Assert it neither ran away nor drained to zero. + expect(backlog).toBeGreaterThan(50) + expect(backlog).toBeLessThan(900) + }) + + it('emit frequency stays bounded by the throttle', () => { + const sched = new FakeScheduler() + const minEmitIntervalMs = 33 + const { reveal, chunks } = makeReveal(sched, { minEmitIntervalMs }) + const frames = 60 + const dt = 16 + for (let i = 0; i < frames; i++) { + reveal.push('c'.repeat(10)) + sched.frame(dt) + } + const windowMs = frames * dt + expect(chunks.length).toBeLessThanOrEqual(Math.ceil(windowMs / minEmitIntervalMs) + 2) + }) + + it('suspends when fully revealed and resumes on the next push', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched) + reveal.push('short') + sched.frames(30) + expect(revealed()).toBe('short') + expect(sched.pending()).toBe(0) // idle: no live frame scheduled + reveal.push(' more') + expect(sched.pending()).toBe(1) // restarted + sched.frames(30) + expect(revealed()).toBe('short more') + }) + + it('clamps a large elapsed time (backgrounded-tab resume) instead of blasting the backlog', () => { + const sched = new FakeScheduler() + const { reveal, revealed } = makeReveal(sched, { smoothingMs: 500, maxCatchupMs: 100 }) + const text = 'q'.repeat(300) + reveal.push(text) + sched.frame(16) // first (nominal) emit; lastEmit now set + const before = revealed().length + sched.frame(5000) // loop kept running but the frame fired seconds late + const revealedInBigFrame = revealed().length - before + // Without the clamp, rate × 5000ms would reveal the entire backlog at once. + expect(revealed().length).toBeLessThan(text.length) + expect(revealedInBigFrame).toBeLessThan(text.length / 2) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/typewriterReveal.ts b/frontend/src/lib/components/copilot/chat/typewriterReveal.ts new file mode 100644 index 0000000000..a4c71212f1 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/typewriterReveal.ts @@ -0,0 +1,204 @@ +// Perceived-smoothness layer for streamed assistant text. +// +// Providers (notably Anthropic for some model/tier combos) deliver text in +// coarse bursts — tens of tokens batched into one delta every ~450 ms — so the +// raw stream reads as freeze→jump→freeze. This module decouples *display* from +// *arrival*: pushed text lands in a non-reactive buffer, and a paint loop +// reveals a slice at a time through `onReveal`, so the bursts read as continuous +// typing. It is deliberately free of Svelte — the only coupling to reactive +// state is the `onReveal` callback — so the pacing is unit-testable with an +// injected clock and scheduler. + +type Schedule = (cb: () => void) => unknown +type Cancel = (handle: unknown) => void + +export interface TypewriterRevealOptions { + /** Called with each revealed slice; the owner appends it to reactive state. */ + onReveal: (chunk: string) => void + /** Reveal synchronously on push with no pacing (reduced-motion / SSR). */ + instant?: boolean + /** Target lag between arrival and display, in ms. The one meaningful knob. */ + smoothingMs?: number + /** Backlog at/above which the whole buffer is dumped in one emit (fast path). */ + maxBacklogChars?: number + /** Minimum gap between emits, in ms — caps downstream re-parse/reflow frequency. */ + minEmitIntervalMs?: number + /** Upper bound on the per-emit elapsed time, so a backgrounded-tab resume + * catches up over several emits instead of one large jump. */ + maxCatchupMs?: number + // Injectables for tests: + now?: () => number + schedule?: Schedule + cancel?: Cancel +} + +const defaultNow: () => number = + typeof performance !== 'undefined' && typeof performance.now === 'function' + ? () => performance.now() + : () => Date.now() + +const hasRAF = typeof requestAnimationFrame !== 'undefined' +const defaultSchedule: Schedule = hasRAF + ? (cb) => requestAnimationFrame(cb) + : (cb) => setTimeout(cb, 16) +const defaultCancel: Cancel = hasRAF + ? (h) => cancelAnimationFrame(h as number) + : (h) => clearTimeout(h as ReturnType) + +export class TypewriterReveal { + private readonly onReveal: (chunk: string) => void + private readonly instant: boolean + private readonly smoothingMs: number + private readonly maxBacklogChars: number + private readonly minEmitIntervalMs: number + private readonly maxCatchupMs: number + private readonly now: () => number + private readonly schedule: Schedule + private readonly cancel: Cancel + + // A stable buffer + an index into it: reveal advances `revealed` (O(1) per + // emit, no re-split). Everything before `revealed` has been emitted. + private buffer = '' + private revealed = 0 + private lastEmit: number | null = null + private handle: unknown = null + private running = false + + constructor(opts: TypewriterRevealOptions) { + this.onReveal = opts.onReveal + this.instant = opts.instant ?? false + this.smoothingMs = opts.smoothingMs ?? 500 + this.maxBacklogChars = opts.maxBacklogChars ?? 1500 + this.minEmitIntervalMs = opts.minEmitIntervalMs ?? 33 + this.maxCatchupMs = opts.maxCatchupMs ?? 100 + this.now = opts.now ?? defaultNow + this.schedule = opts.schedule ?? defaultSchedule + this.cancel = opts.cancel ?? defaultCancel + } + + /** Enqueue received text. In instant mode it is revealed synchronously. */ + push(text: string): void { + if (!text) return + if (this.instant) { + this.onReveal(text) + return + } + this.buffer += text + this.ensureRunning() + } + + /** Reveal everything still buffered now and stop. Call before reading the + * owner's reactive state into committed state, so the read sees the full text. */ + flush(): void { + this.stop() + if (this.instant) return + if (this.revealed < this.buffer.length) { + this.onReveal(this.buffer.slice(this.revealed)) + } + // Everything is revealed now, so drop the buffer rather than carry an + // ever-growing turn's worth of text: onMessageEnd fires flush() at every + // tool-call boundary, and without this the buffer would keep every prior + // segment until the turn's reset(). The next push starts a fresh buffer. + this.buffer = '' + this.revealed = 0 + } + + /** Drop un-revealed backlog and stop. Call at turn boundaries. */ + reset(): void { + this.stop() + this.buffer = '' + this.revealed = 0 + this.lastEmit = null + } + + private ensureRunning(): void { + if (this.running) return + this.running = true + // Re-anchor on resume from idle: a long gap since the last emit must not + // count as elapsed reveal time (the maxCatchupMs clamp only covers a + // still-running loop whose frame fired late). + this.lastEmit = null + this.handle = this.schedule(this.tick) + } + + private stop(): void { + if (this.handle != null) { + this.cancel(this.handle) + this.handle = null + } + this.running = false + } + + private tick = (): void => { + this.handle = null + const t = this.now() + const backlog = this.buffer.length - this.revealed + if (backlog <= 0) { + this.running = false + return + } + + if (backlog >= this.maxBacklogChars) { + // Fast path: a cached/non-streaming reply dumped as one big delta shows + // instantly instead of being stretched. Steady streaming settles well + // under the cap, so smoothing only ever applies to genuinely bursty input. + this.emit(backlog) + this.lastEmit = t + } else { + const first = this.lastEmit === null + // First emit after (re)start reveals a small nominal slice one frame + // after arrival, keeping first paint essentially immediate. + const sinceLast = first ? this.minEmitIntervalMs : t - this.lastEmit! + if (sinceLast < this.minEmitIntervalMs) { + // Throttle: too soon since the last emit — wait another frame. + this.handle = this.schedule(this.tick) + return + } + const elapsedMs = Math.min(sinceLast, this.maxCatchupMs) + const rate = backlog / this.smoothingMs // chars per ms; grows with backlog + const n = Math.min(backlog, Math.max(1, Math.floor(rate * elapsedMs))) + const emitted = this.emit(n) + this.lastEmit = t + if (emitted === 0) { + // Nothing revealable yet (a lone trailing high surrogate awaiting its + // low half). Suspend; the next push restarts the loop. + this.running = false + return + } + } + + if (this.revealed < this.buffer.length) { + this.handle = this.schedule(this.tick) + } else { + this.running = false + } + } + + // Reveal up to `n` chars from `revealed`, never splitting a surrogate pair. + // Returns the number of chars actually emitted (0 only when the buffer ends on + // a lone high surrogate whose low half hasn't arrived). + private emit(n: number): number { + let end = Math.min(this.revealed + n, this.buffer.length) + if (end < this.buffer.length) { + const c = this.buffer.charCodeAt(end) + // Landed on a low surrogate → cut before its high half. + if (c >= 0xdc00 && c <= 0xdfff) end -= 1 + } + if (end <= this.revealed) { + // A floor-1 slice landed inside a pair; take the whole pair so we still + // make progress rather than stalling on the same boundary each tick. + end = Math.min(this.revealed + 2, this.buffer.length) + } + if (end === this.buffer.length && end - 1 >= this.revealed) { + // Hold back a lone trailing high surrogate: its low half may still be + // streaming in, and revealing it alone would emit a broken code unit. + const last = this.buffer.charCodeAt(end - 1) + if (last >= 0xd800 && last <= 0xdbff) end -= 1 + } + if (end <= this.revealed) return 0 + const chunk = this.buffer.slice(this.revealed, end) + this.revealed = end + this.onReveal(chunk) + return chunk.length + } +} diff --git a/frontend/src/lib/components/copilot/chat/workspaceItems.svelte.ts b/frontend/src/lib/components/copilot/chat/workspaceItems.svelte.ts index bc46d644df..8a7932b8d9 100644 --- a/frontend/src/lib/components/copilot/chat/workspaceItems.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/workspaceItems.svelte.ts @@ -20,7 +20,7 @@ import { itemHref as offboardingItemHref } from '$lib/components/offboarding-uti import { findAndReplace } from 'mdast-util-find-and-replace' import { visit } from 'unist-util-visit' import type { Root, InlineCode, Link } from 'mdast' -import type { ToolDisplayAction } from './shared' +import type { CreatedResourceAction, ToolDisplayAction } from './shared' export type WindmillItemKind = | 'script' @@ -67,9 +67,9 @@ export const WINDMILL_PATH_REGEX = */ const WINDMILL_PATH_EXACT_REGEX = /^[uf]\/[A-Za-z0-9_.\-]+\/[A-Za-z0-9_./\-]*[A-Za-z0-9_\-]$/ -function workspaceItemTriggerKind(kind: WindmillItemKind): ToolDisplayAction['triggerKind'] { +function workspaceItemTriggerKind(kind: WindmillItemKind): CreatedResourceAction['triggerKind'] { if (!kind.endsWith('_trigger')) return undefined - return kind.slice(0, -'_trigger'.length) as ToolDisplayAction['triggerKind'] + return kind.slice(0, -'_trigger'.length) as CreatedResourceAction['triggerKind'] } /** diff --git a/frontend/src/lib/components/copilot/lib.anthropicRouting.test.ts b/frontend/src/lib/components/copilot/lib.anthropicRouting.test.ts new file mode 100644 index 0000000000..a701c5b47b --- /dev/null +++ b/frontend/src/lib/components/copilot/lib.anthropicRouting.test.ts @@ -0,0 +1,250 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import type { AIProviderModel } from '$lib/gen' +import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs' + +// getCurrentModel/getMetadataModel are read per call, so a hoisted holder lets +// each test point the routing at a different provider/model. +const h = vi.hoisted(() => ({ currentModel: undefined as AIProviderModel | undefined })) + +vi.mock('monaco-editor', () => ({ editor: {} })) + +vi.mock('$lib/stores', () => ({ + workspaceStore: { subscribe: () => () => undefined } +})) + +vi.mock('$lib/components/flows/flowTree', () => ({ + findModuleInModules: () => undefined +})) + +vi.mock('$lib/gen', () => ({ + OpenAPI: { BASE: '/api', TOKEN: undefined }, + ResourceService: {}, + ScriptService: {}, + FlowService: {}, + JobService: {}, + ScheduleService: {}, + HttpTriggerService: {}, + WebsocketTriggerService: {}, + KafkaTriggerService: {}, + NatsTriggerService: {}, + PostgresTriggerService: {}, + MqttTriggerService: {}, + SqsTriggerService: {}, + GcpTriggerService: {}, + AzureTriggerService: {} +})) + +vi.mock('$lib/utils', () => ({ + emptyString: (value: string | undefined | null) => !value, + generateRandomString: () => 'generated_id' +})) + +vi.mock('$lib/scripts', () => ({ + scriptLangToEditorLang: (language: string) => language +})) + +vi.mock('$lib/aiStore', () => ({ + getCurrentModel: () => h.currentModel, + getMetadataModel: () => h.currentModel, + copilotInfo: { + subscribe: (run: (value: unknown) => void) => { + run({}) + return () => undefined + } + } +})) + +vi.mock('@leeoniya/ufuzzy', () => ({ + default: class { + search() { + return [[], [], []] + } + } +})) + +function streamOf(chunks: unknown[]): any { + return (async function* () { + for (const chunk of chunks) { + yield chunk + } + })() +} + +function textDelta(text: string) { + return { type: 'content_block_delta', delta: { type: 'text_delta', text } } +} + +const messages: ChatCompletionMessageParam[] = [{ role: 'user', content: 'hi' }] + +let anthropicCreate: ReturnType +let anthropicStream: ReturnType +let openaiCreate: ReturnType +let openaiResponsesCreate: ReturnType + +async function setupClients() { + const { workspaceAIClients } = await import('./lib') + + anthropicCreate = vi.fn().mockResolvedValue({ + content: [ + { type: 'text', text: 'Hel' }, + { type: 'thinking', thinking: 'ignored' }, + { type: 'text', text: 'lo' } + ] + }) + anthropicStream = vi + .fn() + .mockReturnValue( + streamOf([ + { type: 'message_start' }, + textDelta('Hel'), + { type: 'content_block_delta', delta: { type: 'input_json_delta', partial_json: '{' } }, + textDelta('lo'), + { type: 'message_stop' } + ]) + ) + openaiCreate = vi.fn().mockResolvedValue({ choices: [{ message: { content: 'openai text' } }] }) + openaiResponsesCreate = vi.fn().mockResolvedValue({ output_text: 'responses text' }) + + vi.spyOn(workspaceAIClients, 'getAnthropicClient').mockReturnValue({ + messages: { create: anthropicCreate, stream: anthropicStream } + } as any) + vi.spyOn(workspaceAIClients, 'getOpenaiClient').mockReturnValue({ + chat: { completions: { create: openaiCreate } }, + responses: { create: openaiResponsesCreate } + } as any) +} + +beforeEach(async () => { + await setupClients() +}) + +afterEach(() => { + vi.restoreAllMocks() + h.currentModel = undefined +}) + +describe('Anthropic Messages API routing', () => { + it('getNonStreamingCompletion routes Foundry Claude through the Anthropic client', async () => { + const { getNonStreamingCompletion } = await import('./lib') + h.currentModel = { provider: 'azure_foundry', model: 'claude-sonnet-5' } + + const response = await getNonStreamingCompletion(messages, new AbortController()) + + expect(anthropicCreate).toHaveBeenCalledTimes(1) + expect(openaiCreate).not.toHaveBeenCalled() + // text blocks concatenated, non-text blocks dropped + expect(response).toBe('Hello') + + const headers = anthropicCreate.mock.calls[0][1].headers + // X-Provider must carry the real provider so the backend resolves Foundry + // credentials/URL; the SDK header selects the Messages API path. + expect(headers['X-Provider']).toBe('azure_foundry') + expect(headers['X-Anthropic-SDK']).toBe('true') + }) + + it('getNonStreamingCompletion routes native Anthropic through the Anthropic client', async () => { + const { getNonStreamingCompletion } = await import('./lib') + h.currentModel = { provider: 'anthropic', model: 'claude-opus-4-8' } + + await getNonStreamingCompletion(messages, new AbortController()) + + expect(anthropicCreate).toHaveBeenCalledTimes(1) + expect(anthropicCreate.mock.calls[0][1].headers['X-Provider']).toBe('anthropic') + }) + + it('getNonStreamingCompletion keeps non-Claude Foundry models on the OpenAI path', async () => { + const { getNonStreamingCompletion } = await import('./lib') + h.currentModel = { provider: 'azure_foundry', model: 'gpt-4o' } + + await getNonStreamingCompletion(messages, new AbortController()) + + expect(anthropicCreate).not.toHaveBeenCalled() + expect(openaiCreate).toHaveBeenCalledTimes(1) + }) + + it('getCompletion adapts the Anthropic stream into OpenAI text chunks', async () => { + const { getCompletion, getResponseFromEvent } = await import('./lib') + h.currentModel = { provider: 'azure_foundry', model: 'claude-sonnet-5' } + + const completion = await getCompletion(messages, new AbortController()) + + let text = '' + let chunks = 0 + for await (const part of completion) { + chunks++ + text += getResponseFromEvent(part) + } + + expect(anthropicStream).toHaveBeenCalledTimes(1) + // only the two text deltas surface; message_start/stop and input_json are dropped + expect(chunks).toBe(2) + expect(text).toBe('Hello') + }) + + it('testKey routes Foundry Claude through the Anthropic client', async () => { + const { testKey } = await import('./lib') + + await testKey({ + resourcePath: 'u/admin/foundry', + model: 'claude-sonnet-5', + abortController: new AbortController(), + messages, + aiProvider: 'azure_foundry' + }) + + expect(anthropicCreate).toHaveBeenCalledTimes(1) + const headers = anthropicCreate.mock.calls[0][1].headers + expect(headers['X-Provider']).toBe('azure_foundry') + expect(headers['X-Resource-Path']).toBe('u/admin/foundry') + }) + + it('caps max_tokens for metadata completions so the Anthropic SDK stays non-streaming', async () => { + const { getNonStreamingCompletion, getNonStreamingMetadataCompletion, METADATA_MAX_TOKENS } = + await import('./lib') + // claude-sonnet defaults to 64000 max_tokens; the Anthropic SDK refuses a + // non-streaming request that large (>10min worst case), which silently broke + // session auto-rename and the other metadata generators. + h.currentModel = { provider: 'anthropic', model: 'claude-sonnet-4-6' } + + await getNonStreamingCompletion(messages, new AbortController()) + expect(anthropicCreate.mock.calls[0][0].max_tokens).toBe(64000) + + anthropicCreate.mockClear() + await getNonStreamingMetadataCompletion(messages, new AbortController()) + expect(anthropicCreate.mock.calls[0][0].max_tokens).toBe(METADATA_MAX_TOKENS) + expect(METADATA_MAX_TOKENS).toBeLessThanOrEqual(21333) + }) + + it('caps max_output_tokens for metadata completions on the OpenAI Responses path', async () => { + const { getNonStreamingCompletion, getNonStreamingMetadataCompletion, METADATA_MAX_TOKENS } = + await import('./lib') + // OpenAI/Azure non-streaming routes through the Responses API; the cap must + // reach it too, not just the Anthropic and chat.completions paths. + h.currentModel = { provider: 'openai', model: 'gpt-4o' } + + await getNonStreamingCompletion(messages, new AbortController()) + expect(openaiResponsesCreate).toHaveBeenCalledTimes(1) + expect(openaiResponsesCreate.mock.calls[0][0].max_output_tokens).toBe(16384) + + openaiResponsesCreate.mockClear() + await getNonStreamingMetadataCompletion(messages, new AbortController()) + expect(openaiResponsesCreate.mock.calls[0][0].max_output_tokens).toBe(METADATA_MAX_TOKENS) + }) + + it('getFimCompletion no-ops for Anthropic Messages API models', async () => { + const { getFimCompletion } = await import('./lib') + const fetchSpy = vi.spyOn(globalThis, 'fetch') + + for (const provider of ['anthropic', 'azure_foundry'] as const) { + const result = await getFimCompletion( + 'prefix', + 'suffix', + { provider, model: 'claude-sonnet-5' }, + new AbortController() + ) + expect(result).toBeUndefined() + } + // no autocomplete request should be issued for these models + expect(fetchSpy).not.toHaveBeenCalled() + }) +}) diff --git a/frontend/src/lib/components/copilot/lib.toolCalls.test.ts b/frontend/src/lib/components/copilot/lib.toolCalls.test.ts new file mode 100644 index 0000000000..36e667f57d --- /dev/null +++ b/frontend/src/lib/components/copilot/lib.toolCalls.test.ts @@ -0,0 +1,167 @@ +import { describe, expect, it, vi } from 'vitest' +import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs' + +vi.mock('monaco-editor', () => ({ + editor: {} +})) + +vi.mock('$lib/stores', () => ({ + workspaceStore: { subscribe: () => () => undefined } +})) + +vi.mock('$lib/components/flows/flowTree', () => ({ + findModuleInModules: () => undefined +})) + +vi.mock('$lib/gen', () => ({ + OpenAPI: {}, + ResourceService: {}, + ScriptService: {}, + FlowService: {}, + JobService: {}, + ScheduleService: {}, + HttpTriggerService: {}, + WebsocketTriggerService: {}, + KafkaTriggerService: {}, + NatsTriggerService: {}, + PostgresTriggerService: {}, + MqttTriggerService: {}, + SqsTriggerService: {}, + GcpTriggerService: {}, + AzureTriggerService: {} +})) + +vi.mock('$lib/utils', () => ({ + emptyString: (value: string | undefined | null) => !value, + generateRandomString: () => 'generated_id' +})) + +vi.mock('$lib/scripts', () => ({ + scriptLangToEditorLang: (language: string) => language +})) + +vi.mock('$lib/aiStore', () => ({ + getCurrentModel: () => undefined, + getMetadataModel: () => undefined, + copilotInfo: { + subscribe: (run: (value: unknown) => void) => { + run({}) + return () => undefined + } + } +})) + +vi.mock('@leeoniya/ufuzzy', () => ({ + default: class { + search() { + return [[], [], []] + } + } +})) + +function streamOf(chunks: unknown[]): any { + return (async function* () { + for (const chunk of chunks) { + yield chunk + } + })() +} + +function toolCallChunk(delta: Record) { + return { choices: [{ delta: { tool_calls: [{ index: 0, ...delta }] } }] } +} + +function createCallbacks() { + return { + onNewToken: vi.fn(), + onMessageEnd: vi.fn(), + setToolStatus: vi.fn(), + removeToolStatus: vi.fn() + } +} + +function createTool(fn = vi.fn().mockResolvedValue('tool ok')) { + return { + def: { + type: 'function' as const, + function: { name: 'patch_app_file', parameters: { type: 'object' } } + }, + fn + } +} + +describe('parseOpenAICompletion tool call arguments', () => { + it('does not execute nor persist a tool call whose streamed arguments are truncated', async () => { + const { parseOpenAICompletion } = await import('./lib') + const fn = vi.fn() + const callbacks = createCallbacks() + const messages: ChatCompletionMessageParam[] = [] + const addedMessages: ChatCompletionMessageParam[] = [] + + const result = await parseOpenAICompletion( + streamOf([ + toolCallChunk({ + id: 'call_1', + function: { name: 'patch_app_file', arguments: '{"path": "u/admin/app", ' } + }), + // The stream ends mid-arguments (e.g. output token limit or dropped connection) + toolCallChunk({ function: { arguments: '"old_string": "setMessages(prev' } }) + ]), + callbacks, + messages, + addedMessages, + [createTool(fn)] as any, + {}, + undefined, + { workspace: 'test' } + ) + + expect(fn).not.toHaveBeenCalled() + expect(result.shouldContinue).toBe(true) + + const assistant = messages.find((m) => m.role === 'assistant') as any + // Persisting the truncated arguments string would make every follow-up + // request fail provider-side JSON parsing, bricking the session. + expect(assistant.tool_calls[0].function.arguments).toBe('{}') + + const toolResult = messages.find((m) => m.role === 'tool') as any + expect(toolResult.tool_call_id).toBe('call_1') + expect(toolResult.content).toContain('NOT executed') + expect(callbacks.setToolStatus).toHaveBeenCalledWith( + 'call_1', + expect.objectContaining({ error: expect.stringContaining('invalid or truncated') }) + ) + expect(addedMessages).toEqual(messages) + }) + + it('executes a tool call with valid streamed arguments and keeps them verbatim', async () => { + const { parseOpenAICompletion } = await import('./lib') + const fn = vi.fn().mockResolvedValue('tool ok') + const messages: ChatCompletionMessageParam[] = [] + + const result = await parseOpenAICompletion( + streamOf([ + toolCallChunk({ + id: 'call_1', + function: { name: 'patch_app_file', arguments: '{"path": ' } + }), + toolCallChunk({ function: { arguments: '"u/admin/app"}' } }) + ]), + createCallbacks(), + messages, + [], + [createTool(fn)] as any, + {}, + undefined, + { workspace: 'test' } + ) + + expect(fn).toHaveBeenCalledWith(expect.objectContaining({ args: { path: 'u/admin/app' } })) + expect(result.shouldContinue).toBe(true) + + const assistant = messages.find((m) => m.role === 'assistant') as any + expect(assistant.tool_calls[0].function.arguments).toBe('{"path": "u/admin/app"}') + const toolResult = messages.find((m) => m.role === 'tool') as any + expect(toolResult.content).toBe('tool ok') + }) +}) diff --git a/frontend/src/lib/components/copilot/lib.ts b/frontend/src/lib/components/copilot/lib.ts index ffa4e5db24..88afa58b7d 100644 --- a/frontend/src/lib/components/copilot/lib.ts +++ b/frontend/src/lib/components/copilot/lib.ts @@ -14,10 +14,11 @@ import Anthropic from '@anthropic-ai/sdk' import { get, type Writable } from 'svelte/store' import { OpenAPI, ResourceService, type Script } from '../../gen' import { EDIT_CONFIG, FIX_CONFIG, GEN_CONFIG } from './prompts' -import { requiresMaxCompletionTokens } from './modelConfig' +import { requiresMaxCompletionTokens, usesAnthropicMessagesApi } from './modelConfig' import { applyReasoningToConfig } from './reasoningRegistry' import { formatResourceTypes } from './utils' import { processToolCall, type Tool, type ToolCallbacks } from './chat/shared' +import { hasValidToolCallArguments } from './chat/toolCallArguments' import { getNonStreamingOpenAIResponsesCompletion, getOpenAIResponsesCompletionStream @@ -62,22 +63,10 @@ export const AI_PROVIDERS: Record = { label: 'OpenAI', defaultModels: OPENAI_MODELS }, - azure_openai: { - label: 'Azure OpenAI', - defaultModels: OPENAI_MODELS - }, anthropic: { label: 'Anthropic', defaultModels: ['claude-sonnet-4-6', 'claude-3-5-haiku-latest'] }, - mistral: { - label: 'Mistral', - defaultModels: ['codestral-latest'] - }, - deepseek: { - label: 'DeepSeek', - defaultModels: ['deepseek-v4-pro', 'deepseek-chat', 'deepseek-reasoner'] - }, googleai: { label: 'Google AI', defaultModels: [ @@ -89,6 +78,29 @@ export const AI_PROVIDERS: Record = { 'gemini-3.1-flash-lite' ] }, + azure_openai: { + label: 'Azure OpenAI', + defaultModels: OPENAI_MODELS + }, + azure_foundry: { + label: 'Azure AI Foundry', + defaultModels: [ + 'gpt-4o', + 'gpt-4o-mini', + 'DeepSeek-R1', + 'Llama-3.3-70B-Instruct', + 'Phi-4', + 'Mistral-Large-2411' + ] + }, + mistral: { + label: 'Mistral', + defaultModels: ['codestral-latest'] + }, + deepseek: { + label: 'DeepSeek', + defaultModels: ['deepseek-v4-pro', 'deepseek-chat', 'deepseek-reasoner'] + }, groq: { label: 'Groq', defaultModels: ['llama-3.3-70b-versatile', 'llama-3.1-8b-instant'] @@ -267,7 +279,10 @@ export async function fetchAvailableModels( export function getModelMaxTokens(provider: AIProvider, model: string) { if (model.includes('gpt-5')) { return 128000 - } else if ((provider === 'azure_openai' || provider === 'openai') && model.startsWith('o')) { + } else if ( + (provider === 'azure_openai' || provider === 'openai' || provider === 'azure_foundry') && + model.startsWith('o') + ) { return 100000 } else if ( model.includes('claude-sonnet') || @@ -287,11 +302,12 @@ export function getModelMaxTokens(provider: AIProvider, model: string) { return 8192 } - -function getModelSpecificConfig( - modelProvider: AIProviderModel, - tools?: OpenAI.Chat.Completions.ChatCompletionTool[] -) { +// Resolves the completion token cap for a model: the workspace's per-model +// override when set, otherwise the built-in default. Shared by the OpenAI and +// Anthropic request paths so both honor the same limit. `cap` bounds the result +// (used by short metadata completions, see METADATA_MAX_TOKENS) — a hard ceiling +// that wins over both the workspace override and the default. +function resolveMaxTokens(modelProvider: AIProviderModel, cap?: number): number { const defaultMaxTokens = getModelMaxTokens(modelProvider.provider, modelProvider.model) const modelKey = `${modelProvider.provider}:${modelProvider.model}` let customMaxTokensStore: Record | undefined @@ -300,9 +316,27 @@ function getModelSpecificConfig( } catch { // copilotInfo store may not be initialized in vitest } - const maxTokens = customMaxTokensStore?.[modelKey] ?? defaultMaxTokens + const resolved = customMaxTokensStore?.[modelKey] ?? defaultMaxTokens + return cap !== undefined ? Math.min(resolved, cap) : resolved +} + +// Token cap for metadata completions (session titles, cron/predicate/step-input +// generation). These outputs are short, and the Anthropic SDK refuses a +// non-streaming request whose max_tokens implies a >10-minute worst case +// (60min × max_tokens / 128000): the model defaults (sonnet/haiku 64000, opus +// 32000) all trip it. Capping keeps every provider's metadata call non-streaming. +export const METADATA_MAX_TOKENS = 4096 + +function getModelSpecificConfig( + modelProvider: AIProviderModel, + tools?: OpenAI.Chat.Completions.ChatCompletionTool[], + maxTokensCap?: number +) { + const maxTokens = resolveMaxTokens(modelProvider, maxTokensCap) if ( - (modelProvider.provider === 'openai' || modelProvider.provider === 'azure_openai') && + (modelProvider.provider === 'openai' || + modelProvider.provider === 'azure_openai' || + modelProvider.provider === 'azure_foundry') && requiresMaxCompletionTokens(modelProvider.model) ) { return { @@ -352,6 +386,7 @@ const DEFAULT_COMPLETION_CONFIG: ChatCompletionCreateParams = { export const PROVIDER_COMPLETION_CONFIG_MAP: Record = { openai: DEFAULT_COMPLETION_CONFIG, azure_openai: DEFAULT_COMPLETION_CONFIG, + azure_foundry: DEFAULT_COMPLETION_CONFIG, groq: DEFAULT_COMPLETION_CONFIG, openrouter: DEFAULT_COMPLETION_CONFIG, togetherai: DEFAULT_COMPLETION_CONFIG, @@ -449,19 +484,10 @@ export async function testKey({ throw new Error('Missing a model to test') } - // Use Anthropic SDK for Anthropic provider - if (aiProvider === 'anthropic') { - await testAnthropicKey({ - apiKey, - workspace, - resourcePath, - model: modelToTest, - abortController, - messages - }) - return - } - + // getNonStreamingCompletion routes Anthropic-Messages-API models (native + // Anthropic and Claude on Azure Foundry) through the Anthropic SDK and + // everything else through OpenAI chat completions, so the test exercises the + // same request shape the feature actually sends. await getNonStreamingCompletion(messages, abortController, { apiKey, workspace, @@ -473,25 +499,37 @@ export async function testKey({ }) } -async function testAnthropicKey({ - apiKey, - workspace, - resourcePath, - model, - abortController, - messages -}: { +// Providers served through the Anthropic Messages API (native Anthropic, and +// Claude deployments on Azure Foundry) require the Anthropic SDK request shape: +// OpenAI chat-completions requests fail against them because the proxy forwards +// the body verbatim and, for Foundry, rewrites the URL to the /anthropic/v1 +// surface that only serves /messages. This centralizes the client/header/message +// setup so every completion entry point routes them the same way the chat does. +interface AnthropicCompletionParams { + messages: ChatCompletionMessageParam[] + modelProvider: AIProviderModel + abortController: AbortController apiKey?: string workspace?: string resourcePath?: string - model: string - abortController: AbortController - messages: ChatCompletionMessageParam[] -}) { + maxTokensCap?: number +} + +function buildAnthropicProxyRequest({ + messages, + modelProvider, + apiKey, + workspace, + resourcePath, + maxTokensCap +}: Omit) { const { system, messages: anthropicMessages } = convertOpenAIToAnthropicMessages(messages) + // X-Provider must be the real provider (e.g. azure_foundry) so the backend + // resolves the right credentials and Anthropic URL; the SDK headers tell it to + // route through the Anthropic Messages API. const headers: Record = { - 'X-Provider': 'anthropic', + 'X-Provider': modelProvider.provider, 'anthropic-version': '2023-06-01', 'X-Anthropic-SDK': 'true' } @@ -502,24 +540,65 @@ async function testAnthropicKey({ headers['X-API-Key'] = apiKey } - const anthropicClient = apiKey + const client = apiKey ? createAnthropicProxyClient(getAiProxyBaseURL()) : workspace ? workspaceAIClients.createAnthropicClient(workspace) : workspaceAIClients.getAnthropicClient() - await anthropicClient.messages.create( - { - model, - max_tokens: 100, - messages: anthropicMessages, - ...(system && { system }) - }, - { - signal: abortController.signal, - headers + const body = { + model: modelProvider.model, + max_tokens: resolveMaxTokens(modelProvider, maxTokensCap), + messages: anthropicMessages, + ...(system && { system }) + } + + return { client, headers, body } +} + +async function getAnthropicNonStreamingCompletion({ + abortController, + ...params +}: AnthropicCompletionParams): Promise { + const { client, headers, body } = buildAnthropicProxyRequest(params) + + const message = await client.messages.create(body, { + signal: abortController.signal, + headers + }) + + return message.content.map((block) => (block.type === 'text' ? block.text : '')).join('') +} + +// Adapts an Anthropic Messages stream into the OpenAI ChatCompletionChunk shape +// the completion consumers already iterate, so they need no Anthropic-specific +// handling. Only text deltas are surfaced (these paths don't use tool calls). +function getAnthropicStreamingCompletion({ + abortController, + ...params +}: AnthropicCompletionParams): Stream { + const { client, headers, body } = buildAnthropicProxyRequest(params) + + const stream = client.messages.stream(body, { + signal: abortController.signal, + headers + }) + + async function* toOpenAIChunks(): AsyncGenerator { + for await (const event of stream) { + if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') { + yield { + id: '', + object: 'chat.completion.chunk', + created: 0, + model: params.modelProvider.model, + choices: [{ index: 0, delta: { content: event.delta.text }, finish_reason: null }] + } + } } - ) + } + + return toOpenAIChunks() as unknown as Stream } interface BaseOptions { @@ -711,12 +790,14 @@ export function getProviderAndCompletionConfig({ messages, stream, tools, - forceModelProvider + forceModelProvider, + maxTokensCap }: { messages: ChatCompletionMessageParam[] stream: K tools?: OpenAI.Chat.Completions.ChatCompletionTool[] forceModelProvider?: AIProviderModel + maxTokensCap?: number }): { provider: AIProvider config: K extends true @@ -730,7 +811,7 @@ export function getProviderAndCompletionConfig({ provider: modelProvider.provider, config: { ...providerConfig, - ...getModelSpecificConfig(modelProvider, tools), + ...getModelSpecificConfig(modelProvider, tools, maxTokensCap), messages: processedMessages, stream } as any @@ -745,13 +826,29 @@ export async function getNonStreamingCompletion( resourcePath?: string // testing resource path passed as a header to the backend proxy workspace?: string // use a specific workspace proxy when testing a workspace resource forceModelProvider?: AIProviderModel + maxTokensCap?: number // hard ceiling on output tokens (see METADATA_MAX_TOKENS) } ) { + const modelProvider = options?.forceModelProvider ?? getCurrentModel() + + if (usesAnthropicMessagesApi(modelProvider.provider, modelProvider.model)) { + return getAnthropicNonStreamingCompletion({ + messages, + modelProvider, + abortController, + apiKey: options?.apiKey, + workspace: options?.workspace, + resourcePath: options?.resourcePath, + maxTokensCap: options?.maxTokensCap + }) + } + let response: string | undefined = '' const { provider, config } = getProviderAndCompletionConfig({ messages, stream: false, - forceModelProvider: options?.forceModelProvider + forceModelProvider: options?.forceModelProvider, + maxTokensCap: options?.maxTokensCap }) // Use Responses API for OpenAI and Azure OpenAI @@ -808,7 +905,8 @@ export async function getNonStreamingMetadataCompletion( abortController: AbortController ) { return getNonStreamingCompletion(messages, abortController, { - forceModelProvider: getMetadataModel() + forceModelProvider: getMetadataModel(), + maxTokensCap: METADATA_MAX_TOKENS }) } @@ -820,6 +918,14 @@ export async function getFimCompletion( providerModel: AIProviderModel, abortController: AbortController ): Promise { + // The Anthropic Messages API has no fill-in-the-middle endpoint, and Foundry + // Claude deployments don't expose the OpenAI-compatible completions surface the + // FIM proxy targets. Skip autocomplete for these models rather than issuing a + // request that can't succeed. + if (usesAnthropicMessagesApi(providerModel.provider, providerModel.model)) { + return undefined + } + const fetchOptions: { signal: AbortSignal headers: Record @@ -882,6 +988,12 @@ export async function getCompletion( reasoningEffort?: string } ): Promise> { + const modelProvider = options?.forceModelProvider ?? getCurrentModel() + + if (usesAnthropicMessagesApi(modelProvider.provider, modelProvider.model)) { + return getAnthropicStreamingCompletion({ messages, modelProvider, abortController }) + } + const { provider, config } = getProviderAndCompletionConfig({ messages, stream: true, @@ -906,7 +1018,10 @@ export async function getCompletion( // Use Completions API for other providers const client = options?.openaiClient ?? workspaceAIClients.getOpenaiClient() const completionConfig = applyReasoningToConfig( - (provider === 'openai' || provider === 'azure_openai' || provider === 'googleai') && + (provider === 'openai' || + provider === 'azure_openai' || + provider === 'azure_foundry' || + provider === 'googleai') && config.stream ? { ...config, @@ -1094,11 +1209,14 @@ export async function parseOpenAICompletion( } if (toolCalls.length > 0) { + const invalidToolCallIds = new Set( + toolCalls.filter((t) => !hasValidToolCallArguments(t.function.arguments)).map((t) => t.id) + ) const normalizedToolCalls = toolCalls.map((t) => ({ ...t, function: { ...t.function, - arguments: t.function.arguments || '{}' + arguments: invalidToolCallIds.has(t.id) ? '{}' : t.function.arguments || '{}' } })) const toAdd = buildAssistantToolCallMessage({ @@ -1113,6 +1231,22 @@ export async function parseOpenAICompletion( messages.push(toAdd) addedMessages.push(toAdd) for (const toolCall of toolCalls) { + if (invalidToolCallIds.has(toolCall.id)) { + callbacks.setToolStatus(toolCall.id, { + isLoading: false, + isStreamingArguments: false, + error: 'Tool call arguments were invalid or truncated' + }) + const messageToAdd = { + role: 'tool' as const, + tool_call_id: toolCall.id, + content: + 'The tool call arguments were invalid or truncated JSON, so the tool was NOT executed. Retry the call; if the arguments were long, split the work into several smaller calls.' + } + messages.push(messageToAdd) + addedMessages.push(messageToAdd) + continue + } const messageToAdd = await processToolCall({ tools, toolCall, diff --git a/frontend/src/lib/components/copilot/modelConfig.test.ts b/frontend/src/lib/components/copilot/modelConfig.test.ts new file mode 100644 index 0000000000..1788d9c365 --- /dev/null +++ b/frontend/src/lib/components/copilot/modelConfig.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from 'vitest' +import { usesAnthropicMessagesApi } from './modelConfig' + +describe('usesAnthropicMessagesApi', () => { + it('routes the native Anthropic provider through the Messages API', () => { + expect(usesAnthropicMessagesApi('anthropic', 'claude-sonnet-5')).toBe(true) + }) + + it('routes Azure Foundry Claude deployments through the Messages API', () => { + expect(usesAnthropicMessagesApi('azure_foundry', 'claude-sonnet-5')).toBe(true) + expect(usesAnthropicMessagesApi('azure_foundry', 'Claude-Opus-4-8')).toBe(true) + }) + + it('keeps other Azure Foundry models on the OpenAI-compatible path', () => { + expect(usesAnthropicMessagesApi('azure_foundry', 'gpt-4o')).toBe(false) + expect(usesAnthropicMessagesApi('azure_foundry', 'DeepSeek-R1')).toBe(false) + }) + + it('does not affect other providers', () => { + expect(usesAnthropicMessagesApi('openai', 'gpt-4o')).toBe(false) + expect(usesAnthropicMessagesApi('azure_openai', 'gpt-4o')).toBe(false) + }) +}) diff --git a/frontend/src/lib/components/copilot/modelConfig.ts b/frontend/src/lib/components/copilot/modelConfig.ts index 4ab08ce02c..af0dd942b2 100644 --- a/frontend/src/lib/components/copilot/modelConfig.ts +++ b/frontend/src/lib/components/copilot/modelConfig.ts @@ -1,3 +1,17 @@ +import type { AIProvider } from '$lib/gen' + +// Azure AI Foundry fronts multiple model families under one resource. Claude +// deployments are served only through the Anthropic Messages API, so the chat must +// route them like the native Anthropic provider (Anthropic SDK, message format) +// rather than the OpenAI-compatible surface used for the rest of Foundry's catalog. +// Mirrors the backend `AIProvider::is_anthropic_model`. +export function usesAnthropicMessagesApi(provider: AIProvider, model: string): boolean { + return ( + provider === 'anthropic' || + (provider === 'azure_foundry' && model.toLowerCase().startsWith('claude')) + ) +} + // gpt-5+ and o-series reasoning models reject the legacy `max_tokens` field on // the OpenAI/Azure Chat Completions API and require `max_completion_tokens` // instead. The check strips any provider prefix (e.g. OpenRouter's "openai/o3") diff --git a/frontend/src/lib/components/copilot/reasoningRegistry.test.ts b/frontend/src/lib/components/copilot/reasoningRegistry.test.ts index 42f0006a1d..2b27ac9d7b 100644 --- a/frontend/src/lib/components/copilot/reasoningRegistry.test.ts +++ b/frontend/src/lib/components/copilot/reasoningRegistry.test.ts @@ -243,6 +243,36 @@ describe('supportsReasoning (static registry)', () => { }) }) +describe('Azure AI Foundry reasoning follows the model family', () => { + it('treats Foundry Claude deployments like the Anthropic provider', () => { + // Live-verified: Foundry Claude accepts the adaptive-thinking effort ladder. + expect(supportsReasoning('azure_foundry', 'claude-sonnet-5')).toBe(true) + expect(supportsReasoning('azure_foundry', 'claude-opus-4-8')).toBe(true) + expect(getReasoningCapability('azure_foundry', 'claude-opus-4-8').levels).toEqual([ + 'low', + 'medium', + 'high', + 'xhigh', + 'max' + ]) + // Off is achieved by omission (Foundry rejects effort 'none'), like Anthropic. + expect(getReasoningCapability('azure_foundry', 'claude-sonnet-5').canDisable).toBe(true) + expect( + resolveRequestReasoning({ + provider: 'azure_foundry', + model: 'claude-sonnet-5', + reasoning: REASONING_OFF + }) + ).toBeUndefined() + }) + + it('treats Foundry OpenAI deployments like the OpenAI provider', () => { + expect(supportsReasoning('azure_foundry', 'gpt-5.1')).toBe(true) + expect(supportsReasoning('azure_foundry', 'gpt-4o')).toBe(false) + expect(supportsReasoning('azure_foundry', 'DeepSeek-R1')).toBe(false) + }) +}) + describe('resolveEffectiveReasoning', () => { it('defaults capable models to high when unset', () => { expect(resolveEffectiveReasoning({ provider: 'anthropic', model: 'claude-sonnet-4-6' })).toBe( diff --git a/frontend/src/lib/components/copilot/reasoningRegistry.ts b/frontend/src/lib/components/copilot/reasoningRegistry.ts index e1ce2467f2..519c1a4021 100644 --- a/frontend/src/lib/components/copilot/reasoningRegistry.ts +++ b/frontend/src/lib/components/copilot/reasoningRegistry.ts @@ -1,4 +1,5 @@ import type { AIProvider, AIProviderModel } from '$lib/gen' +import { usesAnthropicMessagesApi } from './modelConfig' /** * Reasoning effort is provider/model-specific. We never normalize a single @@ -38,6 +39,20 @@ function baseModelId(model: string): string { return normalized.split('/').pop() ?? normalized } +/** + * Azure AI Foundry hosts multiple model families under one provider, so reasoning + * support follows the underlying model rather than the provider: Claude deployments + * reason like the native Anthropic provider (adaptive thinking + `output_config.effort`), + * everything else (gpt-5 / o-series) like OpenAI. Resolving to the owning family here + * lets the rest of the registry keep its per-family logic unchanged. + */ +function reasoningProviderFamily(provider: AIProvider, model: string): AIProvider { + if (provider === 'azure_foundry') { + return usesAnthropicMessagesApi(provider, model) ? 'anthropic' : 'openai' + } + return provider +} + /** * Suggested effort levels per provider, sourced from each provider SDK's own * vocabulary. @@ -143,14 +158,16 @@ function anthropicReasoningLevels(model: string): ReasoningEffort[] { function supportsReasoningStatic(provider: AIProvider, model: string): boolean { const m = model.toLowerCase() const base = baseModelId(model) - switch (provider) { + switch (reasoningProviderFamily(provider, model)) { case 'anthropic': // Bedrock serves the same Claude models under prefixed ids // (e.g. `us.anthropic.claude-opus-4-6-v1`), so match on the full string. case 'aws_bedrock': // 4.6+ only: Opus 4.5 rejects adaptive thinking (and, on Bedrock, // the whole output_config surface) — live-verified hard 400. - return /claude-opus-4-(6|7|8)/.test(m) || /claude-sonnet-4-6/.test(m) || m.includes('fable') + return ( + /claude-opus-4-(6|7|8)/.test(m) || /claude-sonnet-(4-6|5)/.test(m) || m.includes('fable') + ) case 'openai': case 'azure_openai': return base.startsWith('gpt-5') || /^o\d/.test(base) @@ -204,16 +221,17 @@ export function getReasoningCapability(provider: AIProvider, model: string): Rea if (!supported) { return { supported: false, levels: [], canDisable: false } } + const family = reasoningProviderFamily(provider, bareModel) const levels = - provider === 'anthropic' || provider === 'aws_bedrock' + family === 'anthropic' || family === 'aws_bedrock' ? anthropicReasoningLevels(bareModel) - : provider === 'googleai' + : family === 'googleai' ? geminiReasoningLevels(bareModel) - : provider === 'openai' || provider === 'azure_openai' + : family === 'openai' || family === 'azure_openai' ? openaiReasoningLevels(bareModel) - : provider === 'openrouter' + : family === 'openrouter' ? openrouterReasoningLevels(bareModel) - : (PROVIDER_REASONING_LEVELS[provider] ?? ['low', 'medium', 'high']) + : (PROVIDER_REASONING_LEVELS[family] ?? ['low', 'medium', 'high']) return { supported, levels, canDisable: canDisableReasoning(provider, bareModel) } } @@ -226,7 +244,7 @@ export function getReasoningCapability(provider: AIProvider, model: string): Rea function canDisableReasoning(provider: AIProvider, model: string): boolean { const m = model.toLowerCase() const base = baseModelId(model) - switch (provider) { + switch (reasoningProviderFamily(provider, model)) { case 'anthropic': case 'aws_bedrock': // Claude 4.6+ only think when asked, so omission is a real off — @@ -301,8 +319,8 @@ export const DEEPSEEK_OFF_SENTINEL: ReasoningEffort = 'none' * model that reasons *by default* — omitting the field would silently keep * the default-on behavior. Undefined means omission is the correct off. */ -function explicitOffToken(provider: AIProvider, model: string): ReasoningEffort | undefined { - switch (provider) { +export function explicitOffToken(provider: AIProvider, model: string): ReasoningEffort | undefined { + switch (reasoningProviderFamily(provider, model)) { case 'googleai': // Gemini 2.5/3 think by default (dynamic budget / level). The backend // proxy maps 'none' to off on Flash, or the floor on Pro (only diff --git a/frontend/src/lib/components/custom_ui.ts b/frontend/src/lib/components/custom_ui.ts index 68ecf43a57..a04c9ea4b5 100644 --- a/frontend/src/lib/components/custom_ui.ts +++ b/frontend/src/lib/components/custom_ui.ts @@ -94,6 +94,11 @@ export type PreviewPanelUi = { // "what happened the last time this ran". No-op when a test is already // in progress (we don't clobber a live run). loadLastRunOnMount?: boolean + // Pipeline-only: when set, the Test split's caret popover gains a "Run + // downstream up to…" entry that calls this, letting the user bound the + // cascade from the script they're editing. Wired by the pipeline details + // pane only when the open script is a valid bounded-run start. + onBoundedRun?: () => void } export type EditorBarUi = { diff --git a/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts b/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts index c23a864a2b..59345c6ae7 100644 --- a/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts +++ b/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts @@ -108,7 +108,10 @@ export interface DbManagerUriState { selectedSchema: string | undefined selectedTable: string | undefined readonly open: boolean - openDrawer: (nInput: DbInput) => void + /** Workspace the drawer's DB operations run against — the acting workspace of + * the editor that opened it, else the navigation workspace. */ + workspace: string | undefined + openDrawer: (nInput: DbInput, workspace?: string) => void closeDrawer: () => void } @@ -148,7 +151,12 @@ export function useDbManagerUriState(): DbManagerUriState { params.dbm = buildDbm(p) } - function openDrawer(nInput: DbInput) { + // Not URL-persisted: the drawer defaults back to the nav workspace on reload, + // which is the correct fallback outside the session that opened it. + let workspace = $state(undefined) + + function openDrawer(nInput: DbInput, ws?: string) { + workspace = ws if (nInput.type === 'database') { const isDatatable = nInput.resourcePath.startsWith('datatable://') params.dbm = buildDbm({ @@ -203,6 +211,12 @@ export function useDbManagerUriState(): DbManagerUriState { get open() { return !!input }, + get workspace() { + return workspace + }, + set workspace(v: string | undefined) { + workspace = v + }, openDrawer, closeDrawer } diff --git a/frontend/src/lib/components/dbOps.ts b/frontend/src/lib/components/dbOps.ts index 9ef67e4f4d..21a6386478 100644 --- a/frontend/src/lib/components/dbOps.ts +++ b/frontend/src/lib/components/dbOps.ts @@ -1,5 +1,6 @@ import { getLanguageByResourceType, + ColumnIdentity, type ColumnDef, type TableMetadata } from './apps/components/display/dbtable/utils' @@ -8,6 +9,8 @@ import type { DBSchema, SQLSchema } from '$lib/stores' import { stringifySchema } from './copilot/lib' import type { DbInput, DbType } from './dbTypes' import { assert } from '$lib/utils' +import { WorkspaceService } from '$lib/gen' +import { pendingMigrations } from './workspaceSettings/datatableMigrationUtils' import { buildTableEditorValues, type TableEditorValues @@ -45,12 +48,20 @@ export function dbTableOpsWithPreviewScripts({ input, tableKey, colDefs, - workspace + workspace, + whereClause, + version }: { input: DbInput tableKey: string colDefs: ColumnDef[] workspace: string + // Optional raw SQL predicate AND-ed into the read queries (count + rows). + // Caller-trusted — build it with escaped values. + whereClause?: string + // DuckLake time-travel: when set, reads are pinned to this catalog snapshot + // via `AT (VERSION => n)` (DuckDB/ducklake only). Read-only by nature. + version?: number }): IDbTableOps { const dbType = getDbType(input) const language = getLanguageByResourceType(dbType) @@ -67,7 +78,12 @@ export function dbTableOpsWithPreviewScripts({ tableKey, colDefs, getCount: async ({ quicksearch }) => { - const content = makeMarker('COUNT', { table: tableKey, columnDefs: colDefs }) + const content = makeMarker('COUNT', { + table: tableKey, + columnDefs: colDefs, + ...(whereClause ? { whereClause } : {}), + ...(version != undefined ? { version } : {}) + }) const result = await runScriptAndPollResult({ workspace, requestBody: { args: { ...dbArg, quicksearch }, language, content } @@ -79,7 +95,9 @@ export function dbTableOpsWithPreviewScripts({ const content = makeMarker('SELECT', { table: tableKey, columnDefs: colDefs, - fixPgIntTypes: true + fixPgIntTypes: true, + ...(whereClause ? { whereClause } : {}), + ...(version != undefined ? { version } : {}) }) let items = (await runScriptAndPollResult({ workspace, @@ -122,11 +140,100 @@ export function dbTableOpsWithPreviewScripts({ } } +export type DucklakeSnapshot = { + snapshot_id: number + // DuckLake returns this as microseconds-since-epoch serialized as a string + // (TIMESTAMP); callers must convert before formatting. + snapshot_time: string | number +} + +/** + * Column metadata of a ducklake table *at a specific snapshot*. The catalog's + * `information_schema` only reflects the current schema, so a time-travel read + * pinned to an older version must enumerate the columns that existed *then* — + * otherwise a column added in a later snapshot would break the `SELECT … AT + * (VERSION => n)`. `DESCRIBE SELECT * FROM … AT (VERSION => n)` gives exactly + * that. Returns minimal `ColumnDef`s (field + datatype) — enough for the + * read-only preview's SELECT/COUNT and grid headers. + */ +export async function fetchDucklakeColumnsAtVersion({ + workspace, + ducklake, + tableKey, + version +}: { + workspace: string + ducklake: string + tableKey: string + version: number +}): Promise { + // Quote each identifier part (schema.table) so a dotted/odd table name can't + // break the statement, and double single-quotes in the catalog name so it + // can't break out of the ATTACH string literal (mirrors the backend's + // `escape_sql_literal`). `version` is a number — injection-safe. + const quoted = tableKey + .split('.') + .map((p) => `"${p.replace(/"/g, '""')}"`) + .join('.') + const ducklakeLit = ducklake.replace(/'/g, "''") + const content = + `ATTACH 'ducklake://${ducklakeLit}' AS __dlv__; USE __dlv__; ` + + `DESCRIBE SELECT * FROM ${quoted} AT (VERSION => ${version});` + const rows = (await runScriptAndPollResult({ + workspace, + requestBody: { args: {}, language: 'duckdb', content } + })) as { column_name: string; column_type: string }[] + if (!Array.isArray(rows)) return [] + return rows.map((r) => ({ + field: r.column_name, + datatype: r.column_type, + defaultvalue: '', + isprimarykey: false, + isidentity: ColumnIdentity.No, + isnullable: 'YES' as const, + isenum: false + })) +} + +/** + * List a ducklake table's time-travel history, newest first. DuckLake snapshots + * are catalog-wide commits; passing `table` (schema-qualified, e.g. + * `main.events_daily`) scopes the list to snapshots where the table exists — + * otherwise an `AT (VERSION => n)` read could target a version predating the + * table's creation and error. Runs the `DUCKLAKE_SNAPSHOTS` marker as a duckdb + * preview job (server-side SQL build + ATTACH), so no raw SQL is constructed in + * the client. + */ +export async function fetchDucklakeSnapshots({ + workspace, + ducklake, + table +}: { + workspace: string + ducklake: string + table?: string +}): Promise { + const content = `-- WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS ${JSON.stringify({ + ducklake, + ...(table ? { table } : {}) + })}` + const rows = await runScriptAndPollResult({ + workspace, + requestBody: { args: {}, language: 'duckdb', content } + }) + return Array.isArray(rows) ? (rows as DucklakeSnapshot[]) : [] +} + export type IDbSchemaOps = { onDelete: (params: { tableKey: string; schema?: string }) => Promise onCreate: (params: { values: TableEditorValues; schema?: string }) => Promise previewCreateSql: (params: { values: TableEditorValues; schema?: string }) => Promise - onAlter: (params: { values: AlterTableValues; schema?: string }) => Promise + onAlter: (params: { + values: AlterTableValues + /** Reverse diff (new → old), used to generate the down migration. */ + reverse?: AlterTableValues + schema?: string + }) => Promise previewAlterSql: (params: { values: AlterTableValues; schema?: string }) => Promise onCreateSchema: (params: { schema: string }) => Promise onDeleteSchema: (params: { schema: string }) => Promise @@ -137,30 +244,146 @@ export type IDbSchemaOps = { }) => Promise } +/** Thrown by a schema op when the user declines the out-of-order run warning. + * Callers should treat it as a silent cancel (no error toast). */ +export class MigrationRunCancelled extends Error { + constructor() { + super('Migration run cancelled') + this.name = 'MigrationRunCancelled' + } +} + export function dbSchemaOpsWithPreviewScripts({ workspace, - input + input, + confirmRunOutOfOrder }: { workspace: string input: DbInput + /** Asked before running a just-created migration ahead of `pendingCount` + * still-pending earlier ones. Return false to abort (throws MigrationRunCancelled). */ + confirmRunOutOfOrder?: (pendingCount: number) => Promise }): IDbSchemaOps { const dbType = getDbType(input) const dbArg = getDatabaseArg(input) const language = getLanguageByResourceType(dbType) const ducklake = input.type === 'ducklake' ? input.ducklake : undefined + // When managing a data table, schema changes are recorded as migrations + // instead of being run ad-hoc, so the manager stays the source of truth. + const datatableName = + input.type === 'database' && input.resourcePath.startsWith('datatable://') + ? input.resourcePath.slice('datatable://'.length) + : undefined + function makeMarker(op: string, payload: Record): string { if (ducklake) payload.ducklake = ducklake return `-- WM_INTERNAL_DB_${op} ${JSON.stringify(payload)}` } + // Auto-generated migration name, e.g. `create_customers`. The server allocates + // a unique timestamp (bumping on collision), so the name itself need not be unique. + function migrationName(op: string, target: string): string { + const safe = target.replace(/[^a-zA-Z0-9_-]+/g, '_').replace(/^_+|_+$/g, '') + return safe ? `${op}_${safe}` : op + } + + // A dropped SERIAL column reports its default as `nextval()` of an owned + // sequence that is dropped along with the column. When the down re-adds such a + // column, recreate it as its SERIAL type so a fresh sequence is created + // instead of referencing the gone one. + const SERIAL_FOR: Record = { + BIGINT: 'BIGSERIAL', + INT8: 'BIGSERIAL', + INTEGER: 'SERIAL', + INT: 'SERIAL', + INT4: 'SERIAL', + SMALLINT: 'SMALLSERIAL', + INT2: 'SMALLSERIAL' + } + function reverseSerialFix(reverse: AlterTableValues): AlterTableValues { + return { + ...reverse, + operations: reverse.operations.map((op) => { + if (op.kind !== 'addColumn' || !/nextval\s*\(/i.test(op.column.defaultValue ?? '')) { + return op + } + const serial = SERIAL_FOR[(op.column.datatype ?? '').toUpperCase()] + return serial + ? { ...op, column: { ...op.column, datatype: serial, defaultValue: undefined } } + : op + }) + } + } + + // Frame a single (or multi-) statement body in an explicit, `;`-terminated + // transaction, matching the data table migration convention. Some expanded + // markers (e.g. ALTER TABLE) already come wrapped in their own transaction, so + // avoid nesting BEGIN/COMMIT in that case. + function wrapMigration(sql: string): string { + const t = sql.trim() + if (/^BEGIN\b/i.test(t)) return t + return `BEGIN;\n\n${t.endsWith(';') ? t : `${t};`}\n\nEND;` + } + + // Apply a DDL marker. For a data table that has migrations enabled this + // creates a migration and runs it (rolling the record back if the run fails); + // otherwise it runs ad-hoc via the internal-db job as before. `downContent`, + // when provided, is expanded into the migration's down SQL (Postgres only). + async function applyDdl(migName: string, content: string, downContent?: string): Promise { + // A DDL edit on a migrations-enabled data table must be captured as a + // migration. Don't swallow a status-check failure by defaulting to ad-hoc: + // that would run the change untracked (schema drift) — exactly what this + // feature prevents. Let the error propagate (fail closed); only fall back to + // ad-hoc when there's no data table, or `enabled === false` is returned. + const status = datatableName + ? await WorkspaceService.getDatatableMigrationsStatus({ workspace, datatableName }) + : undefined + if (!datatableName || !status?.enabled) { + await runScriptAndPollResult({ workspace, requestBody: { args: dbArg, content, language } }) + return + } + // The new migration gets the highest timestamp, so any still-pending + // migration is earlier: running only this one applies it out of order. + // Warn like the row-level Run action does (skipped if no confirm hook). + if (confirmRunOutOfOrder) { + const pending = pendingMigrations(status.migrations).length + if (pending > 0 && !(await confirmRunOutOfOrder(pending))) { + throw new MigrationRunCancelled() + } + } + const codeUp = wrapMigration(await expandMarker(workspace, language, content)) + // Down migrations are only generated for Postgres for now. + let codeDown: string | undefined + if (downContent && dbType === 'postgresql') { + const downSql = (await expandMarker(workspace, language, downContent)).trim() + if (downSql) codeDown = wrapMigration(downSql) + } + const created = await WorkspaceService.createDatatableMigration({ + workspace, + datatableName, + requestBody: { name: migName, code_up: codeUp, ...(codeDown ? { code_down: codeDown } : {}) } + }) + try { + await WorkspaceService.runDatatableMigrations({ + workspace, + datatableName, + only: created.timestamp + }) + } catch (e) { + await WorkspaceService.deleteDatatableMigration({ + workspace, + datatableName, + timestamp: created.timestamp + }).catch(() => {}) + throw e + } + } + return { onDelete: async ({ tableKey, schema }) => { const content = makeMarker('DROP_TABLE', { table: tableKey, schema }) - await runScriptAndPollResult({ - workspace, - requestBody: { args: { ...dbArg }, language, content } - }) + await applyDdl(migrationName('drop', tableKey), content) }, onCreate: async ({ values, schema }) => { const content = makeMarker('CREATE_TABLE', { @@ -169,10 +392,8 @@ export function dbSchemaOpsWithPreviewScripts({ foreignKeys: values.foreignKeys, schema }) - await runScriptAndPollResult({ - workspace, - requestBody: { args: dbArg, content, language } - }) + const downContent = makeMarker('DROP_TABLE', { table: values.name, schema }) + await applyDdl(migrationName('create', values.name), content, downContent) }, previewCreateSql: async ({ values, schema }) => { const content = makeMarker('CREATE_TABLE', { @@ -183,16 +404,21 @@ export function dbSchemaOpsWithPreviewScripts({ }) return expandMarker(workspace, language, content) }, - onAlter: async ({ values, schema }) => { + onAlter: async ({ values, reverse, schema }) => { const content = makeMarker('ALTER_TABLE', { name: values.name, operations: values.operations, schema }) - await runScriptAndPollResult({ - workspace, - requestBody: { args: dbArg, content, language } - }) + // The down is the same alter run in the opposite direction. + const downContent = reverse + ? makeMarker('ALTER_TABLE', { + name: reverse.name, + operations: reverseSerialFix(reverse).operations, + schema + }) + : undefined + await applyDdl(migrationName('alter', values.name), content, downContent) }, previewAlterSql: async ({ values, schema }) => { const content = makeMarker('ALTER_TABLE', { @@ -204,17 +430,13 @@ export function dbSchemaOpsWithPreviewScripts({ }, onCreateSchema: async ({ schema }) => { const content = makeMarker('CREATE_SCHEMA', { schema }) - await runScriptAndPollResult({ - workspace, - requestBody: { args: { ...dbArg }, language, content } - }) + const downContent = makeMarker('DROP_SCHEMA', { schema }) + await applyDdl(migrationName('create_schema', schema), content, downContent) }, onDeleteSchema: async ({ schema }) => { const content = makeMarker('DROP_SCHEMA', { schema }) - await runScriptAndPollResult({ - workspace, - requestBody: { args: { ...dbArg }, language, content } - }) + const downContent = makeMarker('CREATE_SCHEMA', { schema }) + await applyDdl(migrationName('drop_schema', schema), content, downContent) }, onFetchTableEditorDefinition: async ({ table, schema, colDefs }) => { let foreignKeys: import('./apps/components/display/dbtable/tableEditor').TableEditorForeignKey[] = diff --git a/frontend/src/lib/components/diffEditorTypes.ts b/frontend/src/lib/components/diffEditorTypes.ts index df93edce31..029c96d3ca 100644 --- a/frontend/src/lib/components/diffEditorTypes.ts +++ b/frontend/src/lib/components/diffEditorTypes.ts @@ -5,3 +5,8 @@ export interface ButtonProp { color?: ButtonType.Color onClick: () => void } + +// Below this editor width, side-by-side is too cramped and the diff falls back +// to the unified/inline view. Shared so consumers (e.g. WorkspaceDiffDrawer's +// view toggle) can mirror the same threshold instead of duplicating the number. +export const SIDE_BY_SIDE_MIN_WIDTH = 700 diff --git a/frontend/src/lib/components/flow_builder.ts b/frontend/src/lib/components/flow_builder.ts index 1fe043707b..ce848c6f83 100644 --- a/frontend/src/lib/components/flow_builder.ts +++ b/frontend/src/lib/components/flow_builder.ts @@ -24,6 +24,9 @@ export type FlowBuilderProps = { disabledFlowInputs?: boolean savedPrimarySchedule?: ScheduleTrigger | undefined // used to set the primary schedule in the legacy primaryScheduleStore version?: number | undefined + /** flow_version the draft was forked from; when set, the deploy-time staleness + * check compares it (not the load-time head `version`) against the latest. */ + draftBaseVersion?: number | undefined draftTriggersFromUrl?: Trigger[] | undefined selectedTriggerIndexFromUrl?: number | undefined children?: import('svelte').Snippet @@ -59,4 +62,8 @@ export type FlowBuilderProps = { // Fired whenever a test run is started from the flow editor, with the // preview job id. Used by whitelabel embedders to track test jobs. onTestJob?: (e: { jobId: string }) => void + // Condensed top bar: smaller (sm) buttons, a shorter bar, and the + // EditorHeader's path/breadcrumb row dropped (summary only). Used by the + // session preview to save vertical room. + condensedHeader?: boolean } diff --git a/frontend/src/lib/components/flows/FlowAssetsHandler.svelte b/frontend/src/lib/components/flows/FlowAssetsHandler.svelte index e753fc2892..c216022520 100644 --- a/frontend/src/lib/components/flows/FlowAssetsHandler.svelte +++ b/frontend/src/lib/components/flows/FlowAssetsHandler.svelte @@ -7,6 +7,7 @@ let s = $state({ val: { selectedAsset: undefined, + workspace: undefined, s3FilePicker: undefined, resourceEditorDrawer: undefined, resourceMetadataCache: {}, @@ -61,7 +62,13 @@ } = $props() const flowGraphAssetsCtx = getContext('FlowGraphAssetContext') - const { selectionManager } = getContext('FlowEditorContext') || {} + const { selectionManager, opWorkspace } = getContext('FlowEditorContext') || {} + let opWs = $derived(opWorkspace?.() ?? $workspaceStore) + // Expose the acting workspace to the asset explore controls (ExploreAssetButton + // reads it from this context; the DB manager / S3 picker act on it). + $effect(() => { + if (flowGraphAssetsCtx) flowGraphAssetsCtx.val.workspace = opWs + }) let selectedId = $derived(selectionManager?.getSelectedId()) let allModules = $derived(getAllModules(modules)) @@ -79,7 +86,7 @@ let truncatedPath = asset.path.split('?table=')[0] if (truncatedPath in resMetadataCache) continue resMetadataCache[truncatedPath] = undefined // avoid fetching multiple times because of async - ResourceService.getResource({ path: truncatedPath, workspace: $workspaceStore! }) + ResourceService.getResource({ path: truncatedPath, workspace: opWs! }) .then((r) => (resMetadataCache[truncatedPath] = { resource_type: r.resource_type })) .catch((err) => console.error("Couldn't fetch resource", truncatedPath, err)) } @@ -88,7 +95,7 @@ // Fetch transitive assets (path scripts and flows) $effect(() => { - if (!$workspaceStore || !flowGraphAssetsCtx || !enablePathScriptAndFlowAssets) return + if (!opWs || !flowGraphAssetsCtx || !enablePathScriptAndFlowAssets) return let usages: { path: string; kind: AssetUsageKind }[] = [] let modIds: string[] = [] for (const mod of allModules) { @@ -101,7 +108,7 @@ } if (usages.length) { AssetService.listAssetsByUsage({ - workspace: $workspaceStore, + workspace: opWs, requestBody: { usages } }).then((result) => { result.forEach((assets, idx) => { @@ -182,6 +189,6 @@ {#if flowGraphAssetsCtx} - - + + {/if} diff --git a/frontend/src/lib/components/flows/FlowEditor.svelte b/frontend/src/lib/components/flows/FlowEditor.svelte index ea32afdc52..a484675f83 100644 --- a/frontend/src/lib/components/flows/FlowEditor.svelte +++ b/frontend/src/lib/components/flows/FlowEditor.svelte @@ -2,6 +2,7 @@ import { Pane, Splitpanes } from 'svelte-splitpanes' import FlowEditorPanel from './content/FlowEditorPanel.svelte' import FlowModuleSchemaMap from './map/FlowModuleSchemaMap.svelte' + import type { OpenInSessionSource } from '$lib/components/sessions/OpenInSessionButton.svelte' import WindmillIcon from '../icons/WindmillIcon.svelte' import { Skeleton } from '../common' import { getContext, onDestroy, onMount, setContext } from 'svelte' @@ -52,6 +53,7 @@ aiChatOpen?: boolean showFlowAiButton?: boolean toggleAiChat?: () => void + sessionOpen?: OpenInSessionSource localModuleStates?: Record testModuleStates?: ModulesTestStates isOwner?: boolean @@ -90,6 +92,7 @@ aiChatOpen, showFlowAiButton, toggleAiChat, + sessionOpen, isOwner, onTestFlow, isRunning, @@ -207,6 +210,7 @@ {aiChatOpen} {showFlowAiButton} {toggleAiChat} + {sessionOpen} {isOwner} {onTestFlow} {isRunning} diff --git a/frontend/src/lib/components/flows/FlowHistoryInner.svelte b/frontend/src/lib/components/flows/FlowHistoryInner.svelte index 10415c970b..1f906c1e45 100644 --- a/frontend/src/lib/components/flows/FlowHistoryInner.svelte +++ b/frontend/src/lib/components/flows/FlowHistoryInner.svelte @@ -9,6 +9,8 @@ import { Skeleton } from '$lib/components/common' import Button from '../common/button/Button.svelte' import { ArrowRight, Loader2, Pencil, X } from 'lucide-svelte' + import { getContext } from 'svelte' + import type { FlowEditorContext } from './types' interface Props { path: string @@ -17,6 +19,10 @@ } let { path, allowFork = false, onHistoryRestore }: Props = $props() + + const flowEditorContext = getContext('FlowEditorContext') + let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) + let loading: boolean = $state(false) let versions: FlowVersion[] = $state([]) @@ -29,7 +35,7 @@ async function loadFlow(version: number) { selected = await FlowService.getFlowVersion({ - workspace: $workspaceStore!, + workspace: opWs!, version }) } @@ -37,7 +43,7 @@ async function loadVersions() { loading = true versions = await FlowService.getFlowHistory({ - workspace: $workspaceStore!, + workspace: opWs!, path: path }) loading = false @@ -52,7 +58,7 @@ return } await FlowService.updateFlowHistory({ - workspace: $workspaceStore!, + workspace: opWs!, version, requestBody: { deployment_msg: deploymentMsgUpdate! @@ -66,7 +72,7 @@ async function restoreVersion(flow: Flow | undefined) { if (!flow) return await FlowService.updateFlow({ - workspace: $workspaceStore!, + workspace: opWs!, requestBody: { ...flow, path diff --git a/frontend/src/lib/components/flows/common/FlowCardHeader.svelte b/frontend/src/lib/components/flows/common/FlowCardHeader.svelte index a5ff48053e..2640fbbba9 100644 --- a/frontend/src/lib/components/flows/common/FlowCardHeader.svelte +++ b/frontend/src/lib/components/flows/common/FlowCardHeader.svelte @@ -16,7 +16,8 @@ import { ScriptService, type FlowModuleValue, type PathScript } from '$lib/gen' import { hubBaseUrlStore, workspaceStore } from '$lib/stores' import { Flag, Lock, RefreshCw, Unlock } from 'lucide-svelte' - import { createEventDispatcher, untrack } from 'svelte' + import { createEventDispatcher, getContext, untrack } from 'svelte' + import type { FlowEditorContext } from '../types' import { twMerge } from 'tailwind-merge' import { getToolNameError } from '$lib/components/graph/renderers/nodes/AIToolNode.svelte' import { DEFAULT_HUB_BASE_URL, PRIVATE_HUB_MIN_VERSION } from '$lib/hub' @@ -47,6 +48,9 @@ let latestHash: string | undefined = $state(undefined) + const flowEditorContext = getContext('FlowEditorContext') + let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) + // Extract version_id from hub path (format: hub/{version_id}/{app}/{summary}) let hubVersionId = $derived( flowModuleValue?.type === 'script' && flowModuleValue.path?.startsWith('hub/') @@ -55,7 +59,7 @@ ) function getCachedKey(path: string) { - return `${$workspaceStore}-${path}` + return `${opWs}-${path}` } function getCachedValues(path: string) { const key = getCachedKey(path) @@ -68,7 +72,7 @@ async function loadLatestHash(value: PathScript) { let script = await ScriptService.getScriptByPath({ - workspace: $workspaceStore!, + workspace: opWs!, path: value.path }) const key = getCachedKey(value.path) diff --git a/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte b/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte index 0ce0843bbf..4478d6e06f 100644 --- a/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte +++ b/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte @@ -7,15 +7,19 @@ import { ExternalLink, Loader2 } from 'lucide-svelte' import Button from '$lib/components/common/button/Button.svelte' import { emptySchema, type StateStore } from '$lib/utils' - import { createEventDispatcher } from 'svelte' + import { createEventDispatcher, getContext } from 'svelte' import { fade } from 'svelte/transition' import { initFlow } from '$lib/components/flows/flowStore.svelte' import type { FlowState } from '$lib/components/flows/flowState' + import type { FlowEditorContext } from '../types' let flowEditorDrawer: Drawer | undefined = $state() const dispatch = createEventDispatcher() + const flowEditorContext = getContext('FlowEditorContext') + let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) + export async function openDrawer(path: string, cb: () => void): Promise { flowPath = path flow = undefined @@ -25,7 +29,7 @@ try { const backendFlow = await FlowService.getFlowByPath({ - workspace: $workspaceStore!, + workspace: opWs!, path }) @@ -86,6 +90,7 @@ {flowStore} {flowStateStore} initialPath={flowPath} + autosaveWorkspace={opWs} newFlow={false} selectedId="settings-metadata" loading={false} diff --git a/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte b/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte index 3b6b867abe..e0f9a7d319 100644 --- a/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte +++ b/frontend/src/lib/components/flows/content/FlowEnvironmentVariables.svelte @@ -32,7 +32,8 @@ let { noEditor }: Props = $props() - const { flowStore } = getContext('FlowEditorContext') + const { flowStore, opWorkspace } = getContext('FlowEditorContext') + let opWs = $derived(opWorkspace?.() ?? $workspaceStore) if (!flowStore.val.value.flow_env) { flowStore.val.value.flow_env = {} @@ -244,8 +245,10 @@ Flow envs can be referenced in any flow step input using the syntax{' '} flow_env.VARIABLE_NAME or flow_env["VARIABLE_NAME"]. These variables are available in the property picker and can be used in JavaScript expressions and - input bindings. String values can link to workspace variables using the button. Resource type references workspace resources resolved at runtime. + input bindings. String values can link to workspace variables using the button. Resource type references workspace resources resolved at runtime. {#if flowEnvEntries.length === 0} @@ -303,6 +306,7 @@ {:else if entry.type === 'json'}
@@ -320,8 +324,7 @@ - updateEnvValue(entry.key, e.currentTarget.value, 'string')} + oninput={(e) => updateEnvValue(entry.key, e.currentTarget.value, 'string')} disabled={noEditor} class="input w-full" placeholder="Variable value" @@ -346,8 +349,7 @@ Linked to variable {entry.value.slice(5)}{entry.value.slice(5)}
{/if} @@ -379,7 +381,7 @@ itemName="Variable" extraField="path" loadItems={async () => - (await VariableService.listVariable({ workspace: $workspaceStore ?? '' })).map((x) => ({ + (await VariableService.listVariable({ workspace: opWs ?? '' })).map((x) => ({ name: x.path, ...x }))} diff --git a/frontend/src/lib/components/flows/content/FlowInput.svelte b/frontend/src/lib/components/flows/content/FlowInput.svelte index 5c7002f147..e20f445e00 100644 --- a/frontend/src/lib/components/flows/content/FlowInput.svelte +++ b/frontend/src/lib/components/flows/content/FlowInput.svelte @@ -7,6 +7,7 @@ import JsonInputs from '$lib/components/JsonInputs.svelte' import { convert } from '@redocly/json-to-json-schema' import { sendUserToast } from '$lib/toast' + import { workspaceStore } from '$lib/stores' import EditableSchemaForm from '$lib/components/EditableSchemaForm.svelte' import AddPropertyV2 from '$lib/components/schema/AddPropertyV2.svelte' import FlowInputViewer from '$lib/components/FlowInputViewer.svelte' @@ -74,8 +75,11 @@ pathStore, initialPathStore, fakeInitialPath, - flowInputEditorState + flowInputEditorState, + opWorkspace } = getContext('FlowEditorContext') + // Acting workspace when the flow editor runs in an AI session; else the nav workspace. + let opWs = $derived(opWorkspace?.() ?? $workspaceStore) // Get diffManager from the graph const diffManager = $derived(flowModuleSchemaMap?.getDiffManager()) @@ -887,6 +891,7 @@ > { @@ -910,6 +915,7 @@
{ updatePreviewSchemaAndArgs(e.detail ?? undefined) }} @@ -929,6 +935,7 @@ title="Saved inputs" > { diff --git a/frontend/src/lib/components/flows/content/FlowInputsFlow.svelte b/frontend/src/lib/components/flows/content/FlowInputsFlow.svelte index 9f16d95bab..4938b470af 100644 --- a/frontend/src/lib/components/flows/content/FlowInputsFlow.svelte +++ b/frontend/src/lib/components/flows/content/FlowInputsFlow.svelte @@ -6,7 +6,8 @@ import { workspaceStore } from '$lib/stores' import { emptyString } from '$lib/utils' - import { createEventDispatcher, untrack } from 'svelte' + import { createEventDispatcher, getContext, untrack } from 'svelte' + import type { FlowEditorContext } from '../types' import { flip } from 'svelte/animate' import { fade } from 'svelte/transition' interface Props { @@ -15,6 +16,9 @@ let { children }: Props = $props() + const flowEditorContext = getContext('FlowEditorContext') + let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) + // export let failureModule: boolean const dispatch = createEventDispatcher() @@ -25,10 +29,10 @@ let ownerFilter: string | undefined = $state(undefined) async function loadFlows() { - items = await FlowService.listFlows({ workspace: $workspaceStore!, withoutDescription: true }) + items = await FlowService.listFlows({ workspace: opWs!, withoutDescription: true }) } $effect(() => { - $workspaceStore && untrack(() => loadFlows()) + opWs && untrack(() => loadFlows()) }) let prefilteredItems = $derived( ownerFilter ? items?.filter((x) => x.path.startsWith(ownerFilter!)) : items diff --git a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte index 1bf578a081..565eee9490 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte @@ -94,10 +94,12 @@ pathStore, saveDraft, customUi, - executionCount + executionCount, + opWorkspace } = getContext('FlowEditorContext') const selectedId = $derived(selectionManager.getSelectedId()) + let opWs = $derived(opWorkspace?.() ?? $workspaceStore) interface Props { flowModule: FlowModule @@ -398,7 +400,7 @@ let preparedSqlQueries = usePreparedAssetSqlQueries( () => flowGraphAssetsCtx?.val.sqlQueries[selectedId], - () => $workspaceStore + () => opWs ) // Debug mode state @@ -532,16 +534,12 @@ resetDAPClient() dapClient = getDAPClient(dapServerUrl) - const env = await fetchContextualVariables($workspaceStore ?? '') + const env = await fetchContextualVariables(opWs ?? '') const code = flowModule.value.content let signedPayload try { - signedPayload = await signDebugRequest( - $workspaceStore ?? '', - code ?? '', - rawScriptLang ?? 'python3' - ) + signedPayload = await signDebugRequest(opWs ?? '', code ?? '', rawScriptLang ?? 'python3') debugSessionJobId = signedPayload.job_id } catch (signError) { sendUserToast(getDebugErrorMessage(signError), true) @@ -759,14 +757,14 @@ on:toggleCache={() => selectAdvanced('cache')} on:toggleStopAfterIf={() => selectAdvanced('early-stop')} on:fork={async () => { - const [module, state] = await fork(flowModule) + const [module, state] = await fork(flowModule, opWs) flowModule = module flowStateStore.val[module.id] = state }} on:reload={async () => { if (flowModule.value.type == 'script') { if (flowModule.value.hash != undefined) { - flowModule.value.hash = await getLatestHashForScript(flowModule.value.path) + flowModule.value.hash = await getLatestHashForScript(flowModule.value.path, opWs) } forceReload++ await reload(flowModule) @@ -781,7 +779,8 @@ flowModule, selectedId, flowStateStore.val[flowModule.id]?.schema, - $pathStore + $pathStore, + opWs ) if (flowModule.value.type == 'rawscript') { module.value.input_transforms = flowModule.value.input_transforms @@ -797,6 +796,7 @@
{ @@ -904,7 +903,7 @@ }, {} )} - key={`flow-inline-${$workspaceStore}-${$pathStore}-${flowModule.id}`} + key={`flow-inline-${opWs}-${$pathStore}-${flowModule.id}`} moduleId={flowModule.id} preparedAssetsSqlQueries={preparedSqlQueries.current} customTag={flowModule.value.tag} @@ -916,7 +915,7 @@ client={dapClient} currentFrameId={currentDebugFrameId} onClose={() => (showDebugConsole = false)} - workspace={$workspaceStore} + workspace={opWs} jobId={debugSessionJobId ?? undefined} /> @@ -931,7 +930,6 @@ bind:this={editor} class="h-full relative" code={flowModule.value.content} - syncExternalCode scriptLang={flowModule?.value?.language} automaticLayout={true} cmdEnterAction={async () => { @@ -968,7 +966,7 @@ }, {} )} - key={`flow-inline-${$workspaceStore}-${$pathStore}-${flowModule.id}`} + key={`flow-inline-${opWs}-${$pathStore}-${flowModule.id}`} moduleId={flowModule.id} preparedAssetsSqlQueries={preparedSqlQueries.current} customTag={flowModule.value.tag} diff --git a/frontend/src/lib/components/flows/content/FlowModuleScript.svelte b/frontend/src/lib/components/flows/content/FlowModuleScript.svelte index cc3a75fde9..021437d20a 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleScript.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleScript.svelte @@ -20,7 +20,8 @@ import { getScriptByPath, scriptLangToEditorLang } from '$lib/scripts' import { workspaceStore } from '$lib/stores' import { Loader2 } from 'lucide-svelte' - import { untrack } from 'svelte' + import { getContext, untrack } from 'svelte' + import type { FlowEditorContext } from '../types' interface Props { path: string @@ -44,6 +45,9 @@ language = $bindable(undefined) }: Props = $props() + const flowEditorContext = getContext('FlowEditorContext') + let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) + let code: string | undefined = $state() let previousCode: string | undefined = $state() let lock: string | undefined = $state(undefined) @@ -51,7 +55,7 @@ let notFound = $state(false) function getCachedKey(path: string, hash: string | undefined) { - return `${$workspaceStore}-${path}-${hash ?? ''}` + return `${opWs}-${path}-${hash ?? ''}` } function getCachedValues(path: string, hash: string | undefined) { const key = getCachedKey(path, hash) @@ -64,12 +68,15 @@ notFound = cachedValues[key]?.notFound ?? false } - getCachedValues(untrack(() => path), untrack(() => hash)) + getCachedValues( + untrack(() => path), + untrack(() => hash) + ) async function loadPreviousCode(previousHash: string) { try { const previousScript = await ScriptService.getScriptByHash({ - workspace: $workspaceStore!, + workspace: opWs!, hash: previousHash }) previousCode = previousScript.content @@ -93,8 +100,8 @@ const script = path.startsWith('hub/') ? await getScriptByPath(path!) : hash - ? await ScriptService.getScriptByHash({ workspace: $workspaceStore!, hash }) - : await getScriptByPath(path!) + ? await ScriptService.getScriptByHash({ workspace: opWs!, hash }) + : await getScriptByPath(path!, opWs) code = script.content language = script.language @@ -133,7 +140,7 @@
tag: {tag}
{/if} {#if notFound} -
script not found at {path} in workspace {$workspaceStore}
+
script not found at {path} in workspace {opWs}
{:else if showAllCode} {#if showDiff} {#key (previousCode ?? '') + (code ?? '')} diff --git a/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte b/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte index d1d946c5cd..3a9e791d2d 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleSuspend.svelte @@ -18,7 +18,9 @@ import EditableSchemaDrawer from '$lib/components/schema/EditableSchemaDrawer.svelte' import AddProperty from '$lib/components/schema/AddProperty.svelte' - const { selectionManager, flowStateStore } = getContext('FlowEditorContext') + const { selectionManager, flowStateStore, opWorkspace } = + getContext('FlowEditorContext') + let opWs = $derived(opWorkspace?.() ?? $workspaceStore) const result = flowStateStore.val[selectionManager.getSelectedId()]?.previewResult ?? {} let editor: SimpleEditor | undefined = $state(undefined) @@ -37,7 +39,7 @@ let isSuspendEnabled = $derived(Boolean(flowModule.suspend)) async function loadGroups(): Promise { - allUserGroups = await GroupService.listGroupNames({ workspace: $workspaceStore! }) + allUserGroups = await GroupService.listGroupNames({ workspace: opWs! }) schema.properties['groups'] = { type: 'array', items: { diff --git a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte index 39aaaa66f2..62a3432995 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte @@ -108,9 +108,14 @@ {/if} -
- -

The timeout will be ignored when running "Test this step"

-
-
+ {#if flowModule.timeout && flowModule.timeout.type !== 'static'} +
+ +

+ A dynamic timeout expression is evaluated when running the full flow. It is ignored when + running "Test this step" — only a static timeout value applies there. +

+
+
+ {/if} diff --git a/frontend/src/lib/components/flows/content/FlowModuleWorkerTagSelect.svelte b/frontend/src/lib/components/flows/content/FlowModuleWorkerTagSelect.svelte index 139251d062..620f0136dd 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleWorkerTagSelect.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleWorkerTagSelect.svelte @@ -19,20 +19,39 @@ isPreprocessor: boolean } = $props() - const { flowStore, selectionManager } = getContext('FlowEditorContext') + const { flowStore, selectionManager, opWorkspace } = + getContext('FlowEditorContext') + + // A fork-scoped session deploys to opWorkspace, not $workspaceStore. Keep a local + // tag list in that case so the gate reflects the fork without clobbering the + // shared, navigation-scoped `workerTags` cache. + let effectiveWorkspace = $derived(opWorkspace?.() ?? $workspaceStore) + let usesLocal = $derived( + effectiveWorkspace != undefined && effectiveWorkspace !== $workspaceStore + ) + let localWorkerTags = $state(undefined) + let currentTags = $derived(usesLocal ? localWorkerTags : $workerTags) const dispatch = createEventDispatcher() loadWorkerGroups() async function loadWorkerGroups() { - if (!$workerTags) { - $workerTags = await WorkerService.getCustomTagsForWorkspace({ workspace: $workspaceStore! }) + if (usesLocal) { + if (!localWorkerTags) { + localWorkerTags = await WorkerService.getCustomTagsForWorkspace({ + workspace: effectiveWorkspace! + }) + } + } else if (!$workerTags) { + $workerTags = await WorkerService.getCustomTagsForWorkspace({ + workspace: effectiveWorkspace! + }) } } -{#if $workerTags} - {#if $workerTags?.length > 0} +{#if currentTags} + {#if currentTags?.length > 0}
{#if flowStore.val.tag == undefined || isPreprocessor || flowStore.val.value?.preserve_step_tags} dispatch('change', e.detail)} /> {:else} diff --git a/frontend/src/lib/components/flows/content/FlowPathViewer.svelte b/frontend/src/lib/components/flows/content/FlowPathViewer.svelte index e51535782f..649a4d9f6c 100644 --- a/frontend/src/lib/components/flows/content/FlowPathViewer.svelte +++ b/frontend/src/lib/components/flows/content/FlowPathViewer.svelte @@ -1,5 +1,5 @@
diff --git a/frontend/src/lib/components/flows/content/FlowResult.svelte b/frontend/src/lib/components/flows/content/FlowResult.svelte index a3241c7c27..70d783895d 100644 --- a/frontend/src/lib/components/flows/content/FlowResult.svelte +++ b/frontend/src/lib/components/flows/content/FlowResult.svelte @@ -2,6 +2,8 @@ import FlowExecutionStatus from '$lib/components/runs/FlowExecutionStatus.svelte' import type { Job } from '$lib/gen' import { workspaceStore } from '$lib/stores' + import { getContext } from 'svelte' + import type { FlowEditorContext } from '../types' import FlowCard from '../common/FlowCard.svelte' import Button from '$lib/components/common/button/Button.svelte' import type { StateStore } from '$lib/utils' @@ -18,6 +20,9 @@ } let { job, isOwner, suspendStatus, noEditor, onOpenDetails }: Props = $props() + + const flowEditorContext = getContext('FlowEditorContext') + let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) @@ -39,7 +44,7 @@ {#if isOwner !== undefined && suspendStatus} ('FlowEditorContext') const WM_DEPLOYERS_GROUP = 'wm_deployers' @@ -146,6 +147,7 @@ initialPath={$initialPathStore} namePlaceholder="flow" kind="flow" + workspaceOverride={opWorkspace?.()} /> {#if $initialPathStore && $pathStore && $pathStore !== $initialPathStore} - +
{#if flowStore.val.tag}
{#if flowStore.val.on_behalf_of_email && canPreserve} → { diff --git a/frontend/src/lib/components/flows/content/McpOAuthConnect.svelte b/frontend/src/lib/components/flows/content/McpOAuthConnect.svelte index f97036df20..b6ee442b43 100644 --- a/frontend/src/lib/components/flows/content/McpOAuthConnect.svelte +++ b/frontend/src/lib/components/flows/content/McpOAuthConnect.svelte @@ -12,7 +12,8 @@ import Path from '$lib/components/Path.svelte' import { sendUserToast } from '$lib/toast' import { sameTopDomainOrigin } from '$lib/cookies' - import { onDestroy } from 'svelte' + import { getContext, onDestroy } from 'svelte' + import type { FlowEditorContext } from '../types' interface Props { onConnected: (resourcePath: string, resourceName: string) => void @@ -21,6 +22,9 @@ let { onConnected, onCancel }: Props = $props() + const flowEditorContext = getContext('FlowEditorContext') + let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) + let serverUrl = $state('') let discoveryResult = $state(null) let selectedScopes = $state([]) @@ -112,7 +116,7 @@ let accountId: number | undefined if (data.expires_in && data.refresh_token) { const accountIdStr = await OauthService.createAccount({ - workspace: $workspaceStore!, + workspace: opWs!, requestBody: { refresh_token: data.refresh_token, expires_in: data.expires_in, @@ -124,7 +128,7 @@ } await VariableService.createVariable({ - workspace: $workspaceStore!, + workspace: opWs!, requestBody: { path: resourcePath, value: data.access_token, @@ -136,7 +140,7 @@ }) await ResourceService.createResource({ - workspace: $workspaceStore!, + workspace: opWs!, requestBody: { resource_type: 'mcp', path: resourcePath, @@ -228,6 +232,7 @@ initialPath="" namePlaceholder={resourceName} kind="resource" + workspaceOverride={opWs} /> diff --git a/frontend/src/lib/components/flows/conversations/FlowChatManager.svelte.ts b/frontend/src/lib/components/flows/conversations/FlowChatManager.svelte.ts index 2a57ad937e..7cf2c8d6f7 100644 --- a/frontend/src/lib/components/flows/conversations/FlowChatManager.svelte.ts +++ b/frontend/src/lib/components/flows/conversations/FlowChatManager.svelte.ts @@ -53,6 +53,15 @@ export class FlowChatManager { #useStreaming = $state(false) #path = $state(undefined) + // When the flow editor runs as an AI-session live editor, it acts on a workspace + // that can differ from the nav store. FlowChat.svelte wires this to + // FlowEditorContext.opWorkspace so workspace-scoped calls hit the acting workspace. + operatingWorkspace?: () => string | undefined + + #workspace(): string | undefined { + return this.operatingWorkspace?.() ?? get(workspaceStore) + } + initialize( onRunFlow: ( userMessage: string, @@ -112,7 +121,7 @@ export class FlowChatManager { // Create a new conversation object and add it to the top of the list const newConversation: ConversationWithDraft = { id: newConversationId, - workspace_id: get(workspaceStore)!, + workspace_id: this.#workspace()!, flow_path: this.#path!, title: 'New chat', created_at: new Date().toISOString(), @@ -160,7 +169,7 @@ export class FlowChatManager { try { this.deletingConversationId = conversationId await FlowConversationsService.deleteFlowConversation({ - workspace: get(workspaceStore)!, + workspace: this.#workspace()!, conversationId }) if (this.selectedConversationId === conversationId) { @@ -178,14 +187,14 @@ export class FlowChatManager { } async cancelCurrentJob() { - if (!get(workspaceStore)) { + if (!this.#workspace()) { return } try { if (this.currentJobId) { await JobService.cancelQueuedJob({ - workspace: get(workspaceStore)!, + workspace: this.#workspace()!, id: this.currentJobId, requestBody: {} }) @@ -206,11 +215,11 @@ export class FlowChatManager { // Only used by InfiniteList private async loadConversations(page: number, perPage: number) { - if (!get(workspaceStore) || !this.#path) return [] + if (!this.#workspace() || !this.#path) return [] try { const response = await FlowConversationsService.listFlowConversations({ - workspace: get(workspaceStore)!, + workspace: this.#workspace()!, flowPath: this.#path, page: page, perPage: perPage @@ -227,7 +236,7 @@ export class FlowChatManager { // Message loading private async loadMessages(reset: boolean, conversationId?: string) { let conversationIdToUse = conversationId ?? this.selectedConversationId - if (!get(workspaceStore) || !conversationIdToUse) return + if (!this.#workspace() || !conversationIdToUse) return if (reset) { if (this.#conversationsCache[conversationIdToUse]) { @@ -245,7 +254,7 @@ export class FlowChatManager { const previousScrollHeight = this.messagesContainer?.scrollHeight || 0 const response = await FlowConversationsService.listConversationMessages({ - workspace: get(workspaceStore)!, + workspace: this.#workspace()!, conversationId: conversationIdToUse, page: pageToFetch, perPage: this.#perPage @@ -318,7 +327,7 @@ export class FlowChatManager { // Polling private async pollJobResult(jobId: string) { try { - await waitJob(jobId) + await waitJob(jobId, this.#workspace()) } catch (error) { console.error('Error polling job result:', error) } finally { @@ -338,12 +347,12 @@ export class FlowChatManager { conversationId: string, options?: { isNewConversation?: boolean; removeTempMessages?: boolean } ) { - if (!get(workspaceStore)) return + if (!this.#workspace()) return try { const lastSeq = this.getLastPersistedMessageSeq() const response = await FlowConversationsService.listConversationMessages({ - workspace: get(workspaceStore)!, + workspace: this.#workspace()!, conversationId: conversationId, page: 1, perPage: 50, @@ -488,7 +497,7 @@ export class FlowChatManager { } // Build the EventSource URL - const streamUrl = `/api/w/${get(workspaceStore)}/jobs_u/getupdate_sse/${jobId}` + const streamUrl = `/api/w/${this.#workspace()}/jobs_u/getupdate_sse/${jobId}` const url = new URL(streamUrl, window.location.origin) url.searchParams.set('poll_delay_ms', '50') url.searchParams.set('fast', 'true') diff --git a/frontend/src/lib/components/flows/flowModuleNextId.test.ts b/frontend/src/lib/components/flows/flowModuleNextId.test.ts new file mode 100644 index 0000000000..dcf007e925 --- /dev/null +++ b/frontend/src/lib/components/flows/flowModuleNextId.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it } from 'vitest' + +import type { OpenFlow } from '$lib/gen' +import type { FlowState } from './flowState' +import { nextId } from './flowModuleNextId' + +function flowWith(ids: string[]): OpenFlow { + return { + summary: '', + value: { + modules: ids.map((id) => ({ id, value: { type: 'identity' } as any })) + } + } as OpenFlow +} + +function stateWith(keys: string[]): FlowState { + return Object.fromEntries(keys.map((k) => [k, {}])) as FlowState +} + +describe('nextId', () => { + it('produces a, b, c, ... for a fresh flow', () => { + expect(nextId(stateWith(['failure']), flowWith([]))).toBe('a') + expect(nextId(stateWith(['a', 'failure']), flowWith(['a']))).toBe('b') + expect(nextId(stateWith(['a', 'b', 'c', 'failure']), flowWith(['a', 'b', 'c']))).toBe('d') + }) + + it('ignores the reserved failure/preprocessor keys always present in flowState', () => { + expect(nextId(stateWith(['failure', 'preprocessor']), flowWith([]))).toBe('a') + }) + + // Regression: copy ids ("z2"), subflow result keys and other non-canonical keys land in + // flowState; charsToNumber on them used to leak into the max and made new steps jump to + // garbage ids like "bzw". + it('is not poisoned by copy ids', () => { + const ids = ['a', 'b', 'c'] + const state = stateWith([...ids, 'c2', 'a2', 'z2', 'c10', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('d') + }) + + it('is not poisoned by subflow result keys', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'subflow:abcd', 'Result', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) + + // A step renamed to a long lowercase word ("process") is a valid base-26 string and would + // otherwise inflate the max; the length cutoff keeps such renames out of the sequence. + it('is not poisoned by renames to long lowercase words or underscored ids', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'process', 'my_step', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) +}) diff --git a/frontend/src/lib/components/flows/flowModuleNextId.ts b/frontend/src/lib/components/flows/flowModuleNextId.ts index e10c900982..48b2eb5ac2 100644 --- a/frontend/src/lib/components/flows/flowModuleNextId.ts +++ b/frontend/src/lib/components/flows/flowModuleNextId.ts @@ -1,19 +1,35 @@ import type { OpenFlow } from '$lib/gen' import { dfs } from './dfs' import type { FlowState } from './flowState' -import { charsToNumber, numberToChars } from './idUtils' +import { charsToNumber, forbiddenIds, numberToChars } from './idUtils' + +const reservedIds = new Set(forbiddenIds) + +// Returns the base-26 value of a key only if it is a short, auto-generated step id +// (a, b, ..., z, aa, ...). flowState/module-id keys also include copy ids ("a2"), subflow +// result keys ("subflow:..."), reserved keys and user-renamed ids; feeding those through +// charsToNumber yields meaningless (often huge) numbers that would poison id generation and +// make new steps jump to ids like "bzw". Short non-canonical keys are rejected via a +// round-trip check; longer keys are skipped entirely, which also leaves user renames to long +// lowercase words (e.g. "process") out of the sequence. +function autoIdNumber(key: string): number | undefined { + if (key.length >= 4 || reservedIds.has(key)) { + return undefined + } + const num = charsToNumber(key) + if (num < 0 || numberToChars(num) !== key) { + return undefined + } + return num +} // Computes the next available id export function nextId(flowState: FlowState, fullFlow: OpenFlow): string { const allIds = dfs(fullFlow.value.modules, (fm) => fm.id) const max = allIds.concat(Object.keys(flowState)).reduce((acc, key) => { - if (key.length >= 4) { - return acc - } else { - const num = charsToNumber(key) - return Math.max(acc, num + 1) - } + const num = autoIdNumber(key) + return num === undefined ? acc : Math.max(acc, num + 1) }, 0) return numberToChars(max) } diff --git a/frontend/src/lib/components/flows/flowStateUtils.svelte.ts b/frontend/src/lib/components/flows/flowStateUtils.svelte.ts index 04db00b63d..cc784286ce 100644 --- a/frontend/src/lib/components/flows/flowStateUtils.svelte.ts +++ b/frontend/src/lib/components/flows/flowStateUtils.svelte.ts @@ -213,14 +213,17 @@ export async function createFlow(id: string): Promise<[FlowModule, FlowModuleSta } export async function fork( - flowModule: FlowModule + flowModule: FlowModule, + // The acting workspace when the flow editor runs in an AI session; else the nav workspace. + workspace?: string ): Promise<[FlowModule & { value: RawScript }, FlowModuleState]> { if (flowModule.value.type !== 'script') { throw new Error('Can only fork a script module') } const forkedFlowModule = await createInlineScriptModuleFromPath( flowModule.value.path ?? '', - flowModule.id + flowModule.id, + workspace ) const flowModuleState = await loadFlowModuleState(forkedFlowModule) return [forkedFlowModule, flowModuleState] @@ -228,9 +231,10 @@ export async function fork( async function createInlineScriptModuleFromPath( path: string, - id: string + id: string, + workspace?: string ): Promise { - const { content, language } = await getScriptByPath(path) + const { content, language } = await getScriptByPath(path, workspace) return { id, @@ -255,7 +259,10 @@ export async function createScriptFromInlineScript( flowModule: FlowModule, suffix: string, schema: Schema | undefined, - flowPath: string + flowPath: string, + // The session's acting workspace when the flow editor runs in an AI session; + // falls back to the navigation workspace outside a session. + workspace?: string ): Promise<[FlowModule & { value: PathScript }, FlowModuleState]> { const user = get(userStore) @@ -275,10 +282,10 @@ export async function createScriptFromInlineScript( const forkedDescription = wasForked ? `as a fork of ${originalScriptPath}` : '' const description = `This script was edited in place of flow ${flowPath} ${forkedDescription} by ${user?.username}.` - const availablePath = await findNextAvailablePath(path) + const availablePath = await findNextAvailablePath(path, workspace) const hash = await ScriptService.createScript({ - workspace: get(workspaceStore)!, + workspace: workspace ?? get(workspaceStore)!, requestBody: { path: availablePath, summary: flowModule.summary ?? '', diff --git a/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte b/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte index 2f1c016385..dbebc82f70 100644 --- a/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte +++ b/frontend/src/lib/components/flows/header/FlowPreviewButtons.svelte @@ -1,5 +1,6 @@
diff --git a/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte b/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte index ce1a7e63e0..3b93e0c05c 100644 --- a/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte +++ b/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte @@ -1,6 +1,7 @@ @@ -22,6 +27,7 @@ label={data.label} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data.eventHandlers.select(data.id)) }} diff --git a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte index 6333d32845..9902582ee0 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte @@ -6,6 +6,7 @@ import { X } from 'lucide-svelte' import type { BranchOneStartN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { computeBorderStatus } from '../utils' interface Props { data: BranchOneStartN['data'] id: string @@ -13,6 +14,12 @@ const { selectionManager } = getGraphContext() let { data, id }: Props = $props() + + // branchIndex is -1 for the default branch and 0-based for explicit branches; + // branchChosen is 0 for default and 1-based, hence the +1. + let borderStatus = $derived( + computeBorderStatus(data.branchIndex + 1, 'branchone', data.flowModuleState) + ) @@ -22,11 +29,12 @@ preLabel={data.preLabel} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data?.eventHandlers?.select(data.id)) }} /> - {#if data.insertable} + {#if data.insertable && data.branchIndex >= 0} + + {#if $open && active} +
+ {#if showDoc} + +
+
+
+ +
+
+
+

{active.label}

+ {#if active.badge} + + {active.badge.label} + + {/if} +
+

{active.tagline}

+
+
+ +

{active.description}

+ +
    + {#each active.bullets as bullet (bullet)} +
  • + + {bullet} +
  • + {/each} +
+ + +
+ {/if} + + +
+ {#snippet rowBody(option: Option, ac: (typeof accentClasses)[string])} +
+ +
+ + {option.label} + + {#if option.badge} + + {option.badge.label} + + {/if} + {/snippet} + {#each allOptions as option (option.key)} + {@const ac = accentClasses[option.accent]} + {@const rowClass = + 'w-full flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover'} + {#if option.variants} + + {#if $wacSubOpen} +
+ {#each option.variants ?? [] as variant (variant.label)} + {@const VariantIcon = variant.icon} + + {/each} +
+ {/if} + {:else} + + {/if} + {/each} + + +
+ + {#if $importSubOpen} +
+ {#each importActions as action (action.label)} + + {/each} +
+ {/if} + + {#if !showDoc} + + {/if} +
+
+ {/if} +
+ + + + importDrawer?.closeDrawer?.()}> + + + + {#snippet content()} +
+ {#key importType} + {#await import('$lib/components/SimpleEditor.svelte')} + + {:then Module} + + {/await} + {/key} +
+ {/snippet} +
+ {#snippet actions()} + + {/snippet} +
+
diff --git a/frontend/src/lib/components/home/ItemsList.svelte b/frontend/src/lib/components/home/ItemsList.svelte index 256b2277f5..49a0dc62ee 100644 --- a/frontend/src/lib/components/home/ItemsList.svelte +++ b/frontend/src/lib/components/home/ItemsList.svelte @@ -13,6 +13,7 @@ type ListableRawApp } from '$lib/gen' import { resource } from 'runed' + import { getDraftItems } from '$lib/workspaceDrafts.svelte' import { userStore, workspaceStore } from '$lib/stores' import type uFuzzy from '@leeoniya/ufuzzy' import { @@ -73,23 +74,44 @@ type TableApp = TableItem type TableRawApp = TableItem - // Folders with ≥1 pipeline script (auto_kind='pipeline'). Used by - // TreeView to surface a "Pipeline" entry inside those folders. Cheap - // thanks to the partial index on script.auto_kind. + // Folders that are data pipelines, surfaced as their own "Pipeline" entry + // (the member scripts are folded into it, not listed individually). Two + // sources: deployed pipelines (folders with ≥1 `auto_kind='pipeline'` script, + // cheap via the partial index) AND bundle-phase pipelines that only exist as a + // `data_pipeline` draft so far — so a pipeline shows up the moment its first + // node is drafted, before anything is deployed. let pipelineFoldersRes = resource( () => $workspaceStore, async (ws) => { if (!ws) return new Set() + const folders = new Set() try { - const rows = await AssetService.listPipelineFolders({ workspace: ws }) - return new Set(rows.map((r) => r.folder)) + for (const r of await AssetService.listPipelineFolders({ workspace: ws })) + folders.add(r.folder) } catch { - // Decorative tree entry — degrade to "no pipelines" on failure. - return new Set() + // Decorative entry — degrade gracefully on failure. } + try { + for (const d of await getDraftItems(ws)) { + if (d.kind !== 'data_pipeline') continue + const m = d.path.match(/^f\/([^/]+)\/data_pipeline$/) + if (m) folders.add(m[1]) + } + } catch { + // Drafts unavailable — show deployed pipelines only. + } + return folders } ) - let pipelineFolders = $derived(pipelineFoldersRes.current ?? new Set()) + // Folders of pipeline-member scripts present in the current listing (captured + // in loadScripts before they're filtered out). Unioned in so a folder whose + // only pipeline node is a never-deployed `// pipeline` script draft — not in + // listPipelineFolders (deployed-only) nor a `data_pipeline` bundle — still gets + // a pipeline entry instead of vanishing. + let pipelineMemberFolders = $state(new Set()) + let pipelineFolders = $derived( + new Set([...(pipelineFoldersRes.current ?? []), ...pipelineMemberFolders]) + ) let scripts: TableScript[] | undefined = $state() let flows: TableFlow[] | undefined = $state() @@ -115,12 +137,26 @@ withoutDescription: true }) - scripts = loadedScripts.map((script: Script) => { - return { - canWrite: canWrite(script.path, script.extra_perms, $userStore) && !$userStore?.operator, - ...script - } - }) + // Pipeline-member scripts (`auto_kind='pipeline'`) are represented by their + // pipeline's entry, not listed individually — but capture their folders so + // the pipeline entry still surfaces (incl. a members-only / draft-only folder). + const memberFolders = new Set() + scripts = loadedScripts + .filter((script: Script) => { + if (script.auto_kind === 'pipeline') { + const m = script.path.match(/^f\/([^/]+)\//) + if (m) memberFolders.add(m[1]) + return false + } + return true + }) + .map((script: Script) => { + return { + canWrite: canWrite(script.path, script.extra_perms, $userStore) && !$userStore?.operator, + ...script + } + }) + pipelineMemberFolders = memberFolders loading = false } @@ -239,13 +275,32 @@ const INCLUDE_WITHOUT_MAIN_SETTING_NAME = 'includeWithoutMain' let treeView = $state(getLocalSetting(TREE_VIEW_SETTING_NAME) == 'true') let filterUserFoldersType: 'only f/*' | 'u/username and f/*' | undefined = $derived( - $userStore?.is_super_admin && $userStore.username.includes('@') + $userStore?.non_member ? 'only f/*' : $userStore?.is_admin || $userStore?.is_super_admin ? 'u/username and f/*' : undefined ) let filterUserFolders = $state(getLocalSetting(FILTER_USER_FOLDER_SETTING_NAME) == 'true') + + // Pipeline entries are rendered independently of the item list, so apply the + // same gates the items get — otherwise a pipeline would still show under the + // Flows/Apps tabs, in the archived view, under a label filter, or outside a + // selected owner. Pipelines are script-based units always at `f/`, so + // kind=script and the user-folder toggle always include them; kind=flow/app, + // archived, a label filter (pipelines carry no labels), and a non-matching + // owner exclude them. + let visiblePipelineFolders = $derived.by(() => { + if (archived) return new Set() + if (itemKind !== 'all' && itemKind !== 'script') return new Set() + if (labelFilter != undefined) return new Set() + if (ownerFilter == undefined) return pipelineFolders + return new Set( + [...pipelineFolders].filter( + (f) => `f/${f}` === ownerFilter || `f/${f}`.startsWith(ownerFilter + '/') + ) + ) + }) let includeWithoutMain = $state( getLocalSetting(INCLUDE_WITHOUT_MAIN_SETTING_NAME) ? getLocalSetting(INCLUDE_WITHOUT_MAIN_SETTING_NAME) == 'true' @@ -506,7 +561,8 @@ if (menuItem) { if (e.key === 'ArrowUp' || e.key === 'ArrowDown') { const menu = menuItem.closest('[role="menu"]') - if (menu) { + // menus marked data-arrow-loop keep melt's cyclic wrap instead of exiting + if (menu && !menu.hasAttribute('data-arrow-loop')) { const items = Array.from(menu.querySelectorAll('[role="menuitem"]')) const idx = items.indexOf(menuItem) const isFirst = idx === 0 @@ -825,14 +881,17 @@ {#each new Array(6) as _} {/each} - {:else if filteredItems.length === 0} + {:else if filteredItems.length === 0 && (filter !== '' || visiblePipelineFolders.size === 0)} + {:else if treeView} loadScripts(includeWithoutMain)} on:flowChanged={loadFlows} @@ -849,7 +908,7 @@ {:else}
{#if filter === ''} - {#each [...pipelineFolders].sort() as folder (folder)} + {#each [...visiblePipelineFolders].sort() as folder (folder)} i && 'folderName' in i + // Hidden while searching: pipelines aren't part of the text filter (the list + // view hides their rows on a query too), so a folder matching the search + // shouldn't surface an unrelated Pipeline row. let hasPipeline = $derived( - depth === 0 && isFolderItem(item) && (pipelineFolders?.has(item.folderName) ?? false) + depth === 0 && + !isSearching && + isFolderItem(item) && + (pipelineFolders?.has(item.folderName) ?? false) ) const isFolder = isFolderItem diff --git a/frontend/src/lib/components/home/TreeViewRoot.svelte b/frontend/src/lib/components/home/TreeViewRoot.svelte index 59adf1aaa9..ef5cf25d58 100644 --- a/frontend/src/lib/components/home/TreeViewRoot.svelte +++ b/frontend/src/lib/components/home/TreeViewRoot.svelte @@ -24,7 +24,42 @@ let groupedItems: ReturnType | 'loading' = $state('loading') $effect(() => { items - untrack(() => (groupedItems = groupItems(items))) + pipelineFolders + isSearching + untrack(() => { + const grouped = groupItems(items) + // Ensure every pipeline folder is present at the top level so its + // "Pipeline" entry shows even when it has no listed items — a bundle-phase + // pipeline (only a draft so far) or a folder whose only scripts are + // pipeline members (folded into the pipeline, hidden from the list). + // Skip while searching: pipelines aren't part of the text filter (list view + // hides them on `filter !== ''`), so injecting them would surface unrelated + // folders in the results. + if (!isSearching) { + const present = new Set( + grouped + .filter((g) => 'folderName' in g) + .map((g) => (g as { folderName: string }).folderName) + ) + // Insert each missing pipeline folder among the existing folders in name + // order — `groupItems` already sorts user groups first then folders + // alphabetically, so inserting before the first greater-named folder + // keeps that ordering (rather than prepending out of order). + for (const folderName of [...(pipelineFolders ?? [])] + .filter((f) => !present.has(f)) + .sort()) { + const item = { folderName, items: [] } + const idx = grouped.findIndex( + (g) => + 'folderName' in g && + (g as { folderName: string }).folderName.localeCompare(folderName) > 0 + ) + if (idx < 0) grouped.push(item) + else grouped.splice(idx, 0, item) + } + } + groupedItems = grouped + }) }) diff --git a/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte b/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte new file mode 100644 index 0000000000..fc9c2843ec --- /dev/null +++ b/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte @@ -0,0 +1,20 @@ + + + + + diff --git a/frontend/src/lib/components/icons/fileIcon.ts b/frontend/src/lib/components/icons/fileIcon.ts new file mode 100644 index 0000000000..fef893e398 --- /dev/null +++ b/frontend/src/lib/components/icons/fileIcon.ts @@ -0,0 +1,74 @@ +/** + * Resolve a file name (or relative path) to an icon by extension. Shared by the + * raw-app file tree and the AI-chat file attachments so both stay consistent. + */ +import { File, ImageIcon } from 'lucide-svelte' +import TypeScript from '../common/languageIcons/TypeScript.svelte' +import JavaScriptIcon from './JavaScriptIcon.svelte' +import JsonIcon from './JsonIcon.svelte' +import ReactIcon from './ReactIcon.svelte' +import SvelteIcon from './SvelteIcon.svelte' +import VueIcon from './VueIcon.svelte' +import CssIcon from './CssIcon.svelte' +import SassIcon from './SassIcon.svelte' +import LessIcon from './LessIcon.svelte' +import HtmlIcon from './HtmlIcon.svelte' +import MarkdownIcon from './MarkdownIcon.svelte' +import YamlIcon from './YamlIcon.svelte' + +export interface ResolvedFileIcon { + icon: any + className?: string +} + +/** Lowercased extension of a file name or path (basename only); '' if none. */ +export function getFileExtension(filename: string): string { + const base = filename.split('/').pop() ?? filename + const parts = base.split('.') + return parts.length > 1 ? parts[parts.length - 1].toLowerCase() : '' +} + +/** Icon (and optional color class) for a file, by extension. */ +export function getFileIcon(filename: string): ResolvedFileIcon { + switch (getFileExtension(filename)) { + case 'json': + return { icon: JsonIcon } + case 'tsx': + case 'jsx': + return { icon: ReactIcon } + case 'ts': + return { icon: TypeScript } + case 'js': + return { icon: JavaScriptIcon } + case 'svelte': + return { icon: SvelteIcon } + case 'vue': + return { icon: VueIcon } + case 'css': + return { icon: CssIcon } + case 'scss': + case 'sass': + return { icon: SassIcon } + case 'less': + return { icon: LessIcon } + case 'png': + case 'jpg': + case 'jpeg': + case 'gif': + case 'svg': + case 'webp': + case 'ico': + return { icon: ImageIcon, className: 'text-purple-500' } + case 'html': + case 'htm': + return { icon: HtmlIcon } + case 'md': + case 'markdown': + return { icon: MarkdownIcon } + case 'yaml': + case 'yml': + return { icon: YamlIcon } + default: + return { icon: File, className: 'text-tertiary' } + } +} diff --git a/frontend/src/lib/components/icons/index.ts b/frontend/src/lib/components/icons/index.ts index 9c4a70c041..b39df4310e 100644 --- a/frontend/src/lib/components/icons/index.ts +++ b/frontend/src/lib/components/icons/index.ts @@ -185,6 +185,7 @@ import TwitchIcon from './TwitchIcon.svelte' import TwitterIcon from './TwitterIcon.svelte' import VercelIcon from './VercelIcon.svelte' import WebflowIcon from './WebflowIcon.svelte' +import WhatsappBusinessIcon from './WhatsappBusinessIcon.svelte' import WooCommerceIcon from './WooCommerceIcon.svelte' import WordpressIcon from './WordpressIcon.svelte' import XataIcon from './XataIcon.svelte' @@ -413,6 +414,7 @@ export const APP_TO_ICON_COMPONENT = { twitter: TwitterIcon, vercel: VercelIcon, webflow: WebflowIcon, + whatsapp_business: WhatsappBusinessIcon, woocommerce: WooCommerceIcon, wordpress: WordpressIcon, xata: XataIcon, diff --git a/frontend/src/lib/components/instanceSettings.ts b/frontend/src/lib/components/instanceSettings.ts index 72ccbeca2b..81a7b841b7 100644 --- a/frontend/src/lib/components/instanceSettings.ts +++ b/frontend/src/lib/components/instanceSettings.ts @@ -443,7 +443,7 @@ export const settings: Record = { { label: 'Store audit logs in object storage', description: - 'When enabled and instance object storage is configured, audit logs are also exported as newline-delimited JSON to the dedicated logs/audit/ folder (partitioned by day). Export is incremental and runs off the hot path. Pre-existing history is not backfilled: export starts from when the setting is enabled (transactions in flight at that moment may include a bounded set of just-prior rows). No audit log committed after enabling is ever skipped.', + 'When enabled and instance object storage is configured, audit logs are also exported as newline-delimited JSON to the dedicated logs/audit/ folder (partitioned by day). Export is incremental and runs off the hot path. Enabling (or re-enabling) anchors the export at ~now: while it stays enabled, every audit log committed from that point on is exported (transactions in flight at the moment of enabling may include a bounded set of just-prior rows). Pre-existing history, and any window during which export was disabled, are NOT exported by this cursor — use the opt-in backfill API to export a chosen historical range, back to when audit-log partitioning was introduced (older rows in the legacy audit table are not exported, and a window overlapping them is rejected): POST /settings/audit_logs_s3_backfill {from, to} (status at GET /settings/audit_logs_s3_backfill_status).', key: 'store_audit_logs_s3', fieldType: 'boolean', storage: 'setting', diff --git a/frontend/src/lib/components/instanceSettings/DbHealth.svelte b/frontend/src/lib/components/instanceSettings/DbHealth.svelte index b1a16bdad5..647c4390d8 100644 --- a/frontend/src/lib/components/instanceSettings/DbHealth.svelte +++ b/frontend/src/lib/components/instanceSettings/DbHealth.svelte @@ -289,6 +289,11 @@

Total connections: {data.connection_pool.pg_total_connections} / Max: {data.connection_pool.pg_max_connections} + {#if data.connection_pool.pg_superuser_reserved_connections > 0} + ({data.connection_pool.pg_superuser_reserved_connections} reserved for superuser) + {/if}

Active: {data.connection_pool.pg_active_connections} @@ -297,6 +302,67 @@

{data.connection_pool.message}

+ + {#if data.connection_pool.sizing} + {@const sizing = data.connection_pool.sizing} + {@const undersized = + sizing.recommended_max_connections > data.connection_pool.pg_max_connections} +
+

Connection sizing guidance

+
+ Live worker instances + {formatNumber(sizing.live_worker_instances)} + Live workers + {formatNumber(sizing.live_workers)} + {#if sizing.live_agent_workers > 0} + Agent workers (HTTP, excluded) + {formatNumber(sizing.live_agent_workers)} + {/if} + Est. peak worker connections + {formatNumber(sizing.estimated_worker_connections)} + Per-server pool {sizing.database_connections_override != null + ? '(DATABASE_CONNECTIONS)' + : '(default)'} + {formatNumber(sizing.server_pool_size)} + Per-worker-instance pool {sizing.database_connections_override != null + ? '(DATABASE_CONNECTIONS)' + : '(default)'} + {formatNumber(sizing.worker_pool_size)} + Recommended max_connections + ≥ {formatNumber(sizing.recommended_max_connections)} +
+

{sizing.message}

+ {#if undersized} +

+ Current max_connections ({formatNumber( + data.connection_pool.pg_max_connections + )}) is below the recommended floor for a single server. Increase max_connections + or lower per-process pools via DATABASE_CONNECTIONS. +

+ {/if} +
+ {/if}
{/if} diff --git a/frontend/src/lib/components/instanceSettings/SettingCard.svelte b/frontend/src/lib/components/instanceSettings/SettingCard.svelte index 892a3c0d62..96a1244cce 100644 --- a/frontend/src/lib/components/instanceSettings/SettingCard.svelte +++ b/frontend/src/lib/components/instanceSettings/SettingCard.svelte @@ -19,6 +19,7 @@ } values?: Record children: import('svelte').Snippet + headerAction?: import('svelte').Snippet class?: string } @@ -31,6 +32,7 @@ actionButton, values, children, + headerAction, class: clazz }: Props = $props() @@ -54,7 +56,9 @@ {tooltip} {/if}
- {#if actionButton} + {#if headerAction} + {@render headerAction()} + {:else if actionButton} - + {/snippet} {#snippet content()} - -
-
Default Datatable & Schema
+
+
Default Datatable & Schema
-

- {description} -

+

+ {description} +

-
- Database - schema ?? '', (v) => onChange?.(datatable, v || undefined)} - placeholder="public" - size="sm" - /> -
+
+ Database + schema ?? '', (v) => onChange?.(datatable, v || undefined)} + placeholder="public" + size="sm" + /> +
+
{/snippet} diff --git a/frontend/src/lib/components/raw_apps/FileIcon.svelte b/frontend/src/lib/components/raw_apps/FileIcon.svelte new file mode 100644 index 0000000000..10ff3f4868 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/FileIcon.svelte @@ -0,0 +1,88 @@ + + + +{#if spec} + {@const Icon = spec.icon} + +{/if} diff --git a/frontend/src/lib/components/raw_apps/FileTreeNode.svelte b/frontend/src/lib/components/raw_apps/FileTreeNode.svelte index 9819ed8796..3d428b9aaf 100644 --- a/frontend/src/lib/components/raw_apps/FileTreeNode.svelte +++ b/frontend/src/lib/components/raw_apps/FileTreeNode.svelte @@ -1,32 +1,11 @@
diff --git a/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte b/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte index 73abdf6c34..6cdcd6a427 100644 --- a/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte @@ -15,6 +15,26 @@ jobsById?: Record editor: boolean workspace: string + /** + * Restrict waitJob/getJob/streamJob to job ids launched by this app + * instance (WIN-2006): a SANDBOXED bundle must not read arbitrary + * workspace jobs through the credentialed bridge. Off for unsandboxed + * renders (the default, and editor preview) — there the bundle holds + * the same credential as the bridge, so gating adds nothing and would + * only break unsandboxed apps that poll persisted or runnable-returned + * job ids. + */ + gateJobIds?: boolean + /** + * Additional trusted message source beyond the bundle iframe: the + * detached preview window opened from the editor ("open preview in a + * separate window"). Its app bundle posts runnable requests to + * `window.opener` (this window), so the bridge must accept its + * `event.source` and reply to it. A getter so it tracks the live handle + * without a reactive prop. Editor-only — the detached window runs the + * same unsandboxed bundle as the inline preview. + */ + extraSourceWindow?: () => Window | null | undefined } let { @@ -24,16 +44,31 @@ jobs = $bindable([]), jobsById = $bindable({}), editor, - workspace + workspace, + gateJobIds = true, + extraSourceWindow }: Props = $props() + // Job ids launched by this app instance — see `gateJobIds`. + const launchedJobs = new Set() + let listener = async (event) => { - if (!iframe || event.source !== iframe.contentWindow) return + // Only accept messages from the bundle iframe (opaque origin) or the + // detached preview window we opened, so other frames/extensions can't + // drive the runnable bridge (WIN-2006). Reject unconditionally until the + // iframe is bound — never process a message from an unknown source. + const detachedWindow = extraSourceWindow?.() + const sourceWindow = event.source as Window | null + if (!iframe || !sourceWindow) return + if (sourceWindow !== iframe.contentWindow && sourceWindow !== detachedWindow) return const data = event.data + // Reply to whichever window sent the request (inline iframe or the + // detached preview), not a hardcoded target — otherwise the detached + // window's calls would hang waiting for a response routed elsewhere. function respond(o: object) { - iframe?.contentWindow?.postMessage({ type: data.type + 'Res', ...o, reqId: data.reqId }, '*') + sourceWindow?.postMessage({ type: data.type + 'Res', ...o, reqId: data.reqId }, '*') } async function respondWithResult(uuid: string) { let error = false @@ -115,6 +150,7 @@ }, undefined ) + launchedJobs.add(uuid) let job: JobById = { component: runnable_id, created_at: Date.now(), job: uuid } if (event.data.type == 'backendAsync') { let result = uuid @@ -134,14 +170,29 @@ console.error('No runnable found for', runnable_id) } } else if (event.data.type == 'waitJob') { + if (gateJobIds && !launchedJobs.has(data.jobId)) { + respond({ result: { message: 'Unknown job' }, error: true }) + return + } await respondWithResult(data.jobId) } else if (event.data.type == 'getJob') { + if (gateJobIds && !launchedJobs.has(data.jobId)) { + respond({ result: { message: 'Unknown job' }, error: true }) + return + } const job = await JobService.getJob({ workspace, id: data.jobId }) respond({ result: job }) } else if (event.data.type == 'streamJob') { // Stream job results using SSE const jobId = data.jobId const reqId = data.reqId + if (gateJobIds && !launchedJobs.has(jobId)) { + sourceWindow?.postMessage( + { type: 'streamJobRes', reqId, error: true, result: { message: 'Unknown job' } }, + '*' + ) + return + } const params = new URLSearchParams() params.set('fast', 'true') params.set('only_result', 'true') @@ -163,7 +214,7 @@ if (type === 'error') { eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, @@ -177,7 +228,7 @@ if (type === 'not_found') { eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, @@ -191,7 +242,7 @@ // Send stream update if there's new stream data if (update.new_result_stream !== undefined) { - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobUpdate', reqId, @@ -205,7 +256,7 @@ // Check if job is completed if (update.completed) { eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, @@ -223,7 +274,7 @@ eventSource.onerror = (error) => { console.warn('SSE stream error:', error) eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, diff --git a/frontend/src/lib/components/raw_apps/RawAppDataTableDrawer.svelte b/frontend/src/lib/components/raw_apps/RawAppDataTableDrawer.svelte index aa53999453..6a6a2a646f 100644 --- a/frontend/src/lib/components/raw_apps/RawAppDataTableDrawer.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppDataTableDrawer.svelte @@ -12,6 +12,10 @@ import DBManagerContent from '../DBManagerContent.svelte' import type { DbInput } from '../dbTypes' import type { SelectedTable } from '../DBManager.svelte' + import { getRawAppOperatingWorkspace } from './rawAppWorkspace' + + const getOpWs = getRawAppOperatingWorkspace() + let opWs = $derived(getOpWs?.() ?? $workspaceStore) interface Props { onAdd?: (ref: DataTableRef) => void @@ -40,11 +44,9 @@ // Load available datatables from workspace const datatables = resource([], async () => { - if (!$workspaceStore) return [] + if (!opWs) return [] try { - return (await WorkspaceService.listDataTables({ workspace: $workspaceStore })).map( - (d) => d.name - ) + return (await WorkspaceService.listDataTables({ workspace: opWs })).map((d) => d.name) } catch (e) { console.error('Failed to load datatables:', e) return [] @@ -163,11 +165,12 @@ CloseIcon={hasReplResult ? ArrowLeft : undefined} noPadding > - {#if dbInput && $workspaceStore} + {#if dbInput && opWs} {#key selectedDatatable} void onRuntimeLogRequester?: (requester: RawAppRuntimeLogRequester | undefined) => void onRunsProvider?: (provider: RawAppRunsProvider | undefined) => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void + // Deploy created the app at a new path; the page navigates to it. Callback + // prop for the same reason as `onRestore` — `on:savedNewAppPath` forwarding + // through these runes-mode components is dropped. + onSavedNewAppPath?: (path: string) => void + // Condensed top bar: smaller (sm) buttons, a shorter bar, and the + // EditorHeader's path/breadcrumb row dropped (summary only). Used by the + // session preview to save vertical room. + condensedHeader?: boolean } let { @@ -125,6 +148,7 @@ summary = $bindable(''), path, newPath = undefined, + labels = undefined, savedApp = $bindable(undefined), diffDrawer = undefined, onNavigate, @@ -141,10 +165,21 @@ othersDraftsCount = 0, onOpenOthersDrafts, onRuntimeLogRequester = undefined, - onRunsProvider = undefined + onRunsProvider = undefined, + onRestore, + onSavedNewAppPath, + condensedHeader = false }: Props = $props() export const version: number | undefined = undefined + // Workspace this editor operates on: the session's acting workspace when + // embedded in a session preview (autosaveWorkspace), else the navigation + // workspace. Deploy/save/background-runner must target it, not $workspaceStore. + const opWorkspace = $derived(autosaveWorkspace ?? $workspaceStore) + // Expose it to the sidebar sub-components (inline scripts, datatable/shared-UI + // drawers, DB selector) so their lookups target the app's workspace too. + setRawAppOperatingWorkspace(() => opWorkspace) + // Convert to object format for child components let dataTableRefsObjects = $derived(data.tables.map(parseDataTableRef)) let dataTableWhitelist = $derived(buildDataTableWhitelist(dataTableRefsObjects)) @@ -240,6 +275,10 @@ let previewIframe: HTMLIFrameElement | undefined = $state(undefined) let previewIframeLoaded = $state(false) let lastBuild: { css: string; js: string } | undefined = undefined + // Detached preview tab/window rendering the same app-preview bundle as the + // inline pane. Kept live-synced: every build is replayed into it until the + // user closes it. Not reactive — it's a window handle, not UI state. + let externalPreviewWindow: Window | null = null let inspectorEnabled = $state(false) let bundlerType: 'esbuild' | 'rolldown' = $state('esbuild') @@ -250,6 +289,8 @@ // Latest UI Builder error; cleared on next successful build. let buildError = $state(undefined) + // Latest uncaught runtime error thrown by the rendered app; cleared on next build. + let runtimeError = $state(undefined) let logsCollapsed = $state(false) let logsDiv: HTMLDivElement | undefined = $state(undefined) $effect(() => { @@ -493,6 +534,24 @@ 'boolean' ) + // Auto-compact when the editor opens in a narrow container (e.g. the session + // preview pane): drop to the merged single-pane view and retract the file + // sidebar. Applied once, on the first measured layout — later resizes are the + // user's call. The sidebar is set without persisting so a transient narrow + // open never overrides the user's saved expand/collapse preference. + let rootWidth = $state(0) + const NARROW_PX = 900 + let appliedNarrowDefault = false + $effect(() => { + const w = rootWidth + if (appliedNarrowDefault || w <= 0) return + appliedNarrowDefault = true + if (w < NARROW_PX) { + splitWithPreview = false + sidebarCollapsed.setWithoutPersist(true) + } + }) + function handleYamlApply(update: RawAppYamlUpdate) { if (update.summary !== undefined) { summary = update.summary @@ -575,10 +634,10 @@ let sharedUiLoaded = $state(false) async function loadSharedUi() { - if (!$workspaceStore) return + if (!opWorkspace) return try { const res = (await WorkspaceService.getSharedUi({ - workspace: $workspaceStore + workspace: opWorkspace })) as { files?: Record; version?: number } sharedUiFiles = res.files ?? {} sharedUiVersion = res.version ?? 0 @@ -862,12 +921,12 @@ handleHistorySelect(id) }, listDatatableTables: async (): Promise => { - if (!$workspaceStore) { + if (!opWorkspace) { return [] } const tables = await WorkspaceService.listDataTableTables({ - workspace: $workspaceStore + workspace: opWorkspace }) return filterDatatableTables(tables) }, @@ -876,7 +935,7 @@ schemaName: string, tableName: string ): Promise> => { - if (!$workspaceStore) { + if (!opWorkspace) { return {} } @@ -890,7 +949,7 @@ } const schema = await WorkspaceService.getDataTableTableSchema({ - workspace: $workspaceStore, + workspace: opWorkspace, datatableName, schemaName, tableName @@ -906,13 +965,13 @@ sql: string, newTable?: { schema: string; name: string } ): Promise<{ success: boolean; result?: Record[]; error?: string }> => { - if (!$workspaceStore) { + if (!opWorkspace) { return { success: false, error: 'Workspace not available' } } try { const result = await runScriptAndPollResult({ - workspace: $workspaceStore, + workspace: opWorkspace, requestBody: { language: 'postgresql', content: sql, @@ -933,6 +992,7 @@ // Clear the cached schema so it gets refreshed with the new table const resourcePath = `datatable://${datatableName}` delete $dbSchemas[resourcePath] + delete $dbSchemas[`${opWorkspace}:${resourcePath}`] } } @@ -982,6 +1042,22 @@ } function listener(e: MessageEvent) { + // The detached preview window asks for the build every time it (re)loads, + // including a manual browser refresh — its app-preview.html shell starts + // blank and the one-shot `load` feed can't survive the tab reloading + // itself. Re-feed it here so it repaints. Gated to our own window handle + // AND a same-origin sender: the preview runs user app code that can + // navigate the window away, and a cross-origin doc must not be able to + // trigger a bundle replay (the build can carry app source/secrets). + if ( + e.data?.type === 'appPreviewReady' && + e.source === externalPreviewWindow && + e.origin === window.location.origin + ) { + feedExternalPreview() + return + } + // Two children speak to us now: the UI Builder iframe (source editor) // and the preview iframe (rendered user app). Gate by source so they // can't be confused or spoofed. @@ -993,10 +1069,8 @@ // the preview iframe so it renders the new app. if (fromUiBuilder && e.data.type === 'preview') { lastBuild = { css: e.data.css, js: e.data.js } - previewIframe?.contentWindow?.postMessage( - { type: 'preview', css: e.data.css, js: e.data.js }, - '*' - ) + feedPreviewIframe(lastBuild) + syncExternalPreview() return } @@ -1025,6 +1099,16 @@ return } + // Uncaught error/rejection from the rendered app — surfaced in the preview + // overlay so a runtime crash isn't a silent blank error. + if (fromPreview && e.data.type === 'runtimeError') { + runtimeError = + typeof e.data.message === 'string' && e.data.message + ? e.data.message + : 'Unknown runtime error' + return + } + // Inspector events come exclusively from the preview iframe. if (fromPreview && e.data.type === 'inspectorSelect') { inspectorElement = e.data.element as InspectorElementInfo @@ -1097,6 +1181,78 @@ } } + function postToExternalPreview(msg: Record) { + if (!externalPreviewWindow || externalPreviewWindow.closed) { + externalPreviewWindow = null + return + } + // Restrict to our own origin: the detached window loads same-origin + // app-preview.html, but user app code can navigate it elsewhere — don't + // post the build (potential app source/secrets) to a cross-origin doc. + externalPreviewWindow.postMessage(msg, window.location.origin) + } + + function syncExternalPreview() { + if (lastBuild) { + postToExternalPreview({ type: 'preview', css: lastBuild.css, js: lastBuild.js }) + } + } + + // Feed a build into the inline preview iframe. Clears any prior runtime-error + // overlay first: a fresh render supersedes the old crash, and if the new + // render throws again app-preview.html re-posts `runtimeError`. + function feedPreviewIframe(build: { css: string; js: string }) { + runtimeError = undefined + previewIframe?.contentWindow?.postMessage( + { type: 'preview', css: build.css, js: build.js }, + '*' + ) + } + + // Full (re)feed of the detached window: theme first, then the build. Used + // when (re)attaching to a window — open, focus-reuse, load, handshake — so + // it always matches the editor's current state. Plain rebuilds use + // `syncExternalPreview` alone (the theme hasn't changed). + function feedExternalPreview() { + postToExternalPreview({ type: 'setDarkMode', dark: darkMode }) + syncExternalPreview() + } + + onDestroy(() => { + // Don't leave a detached preview behind when the editor unmounts: it + // would stop receiving builds and, once refreshed, has no opener to + // re-feed it — a permanently blank orphan. + if (externalPreviewWindow && !externalPreviewWindow.closed) externalPreviewWindow.close() + externalPreviewWindow = null + }) + + function openExternalPreview() { + // Reuse an already-open window instead of spawning duplicates. + if (externalPreviewWindow && !externalPreviewWindow.closed) { + externalPreviewWindow.focus() + feedExternalPreview() + return + } + // Scope the window name per app path so two open editors don't fight over + // (or take over / close) one shared OS-level preview window. + const win = window.open( + '/ui_builder/app-preview.html', + `windmillRawAppPreview:${encodeURIComponent(path)}` + ) + if (!win) { + sendUserToast('Could not open the preview window (popup blocked?)', true) + return + } + externalPreviewWindow = win + // Initial feed: fires once when the freshly opened tab loads. This is the + // only feed path against an app-preview.html that predates the + // `appPreviewReady` handshake, so the window isn't blank on first open + // regardless of the pinned UI Builder artifact. A manual refresh is + // covered separately by the handshake in `listener` (this listener is + // bound to the now-stale document and won't fire again). + win.addEventListener('load', () => feedExternalPreview()) + } + let getBundleResolve: (({ css, js }: { css: string; js: string }) => void) | undefined = undefined async function getBundle(): Promise<{ css: string; js: string }> { @@ -1197,10 +1353,7 @@ // Replay the last build so the preview repopulates without // waiting for the user to trigger another bundle. if (lastBuild) { - previewIframe?.contentWindow?.postMessage( - { type: 'preview', css: lastBuild.css, js: lastBuild.js }, - '*' - ) + feedPreviewIframe(lastBuild) } // Escape inside the preview exits inspect mode — the keydown fires in // the iframe's document, so the parent window listener can't see it. @@ -1226,6 +1379,7 @@ if (previewIframe && previewIframeLoaded) { previewIframe.contentWindow?.postMessage({ type: 'setDarkMode', dark: darkMode }, '*') } + postToExternalPreview({ type: 'setDarkMode', dark: darkMode }) }) $effect(() => { // Match VS Code's editor font size to Windmill's text-xs. @@ -1419,9 +1573,9 @@ // Force an immediate flush. No toast — the AutosaveIndicator narrates the // result, and `flush` never rejects (postSave routes errors to the failures map). function flushDraft() { - if (!$workspaceStore || !liveEditorDraftStoragePath) return + if (!opWorkspace || !liveEditorDraftStoragePath) return void UserDraftDbSyncer.flush({ - workspace: $workspaceStore, + workspace: opWorkspace, itemKind: 'raw_app', path: liveEditorDraftStoragePath }) @@ -1489,27 +1643,30 @@ externalPreviewWindow} /> -
+
yamlEditorDrawer?.openDrawer()} sidebarCollapsed={sidebarCollapsed.val} onToggleSidebar={() => (sidebarCollapsed.val = !sidebarCollapsed.val)} + {condensedHeader} /> { if (lastBuild) { - previewIframe?.contentWindow?.postMessage( - { - type: 'preview', - css: lastBuild.css, - js: lastBuild.js - }, - '*' - ) + feedPreviewIframe(lastBuild) } }} > +
+ {:else if runtimeError} + {/if} {#if logs}
files: Record | undefined @@ -126,6 +137,10 @@ onOpenYamlEditor?: () => void sidebarCollapsed?: boolean onToggleSidebar?: () => void + /** Condensed top bar: smaller (sm) buttons, a shorter bar, and the + * EditorHeader's path/breadcrumb row dropped (summary only). Used by the + * session preview to save vertical room. */ + condensedHeader?: boolean onNavigate?: (item: import('$lib/components/workspacePicker').WorkspaceItem) => void liveEditorDraftStoragePath?: string /** Indicator-only overrides for the sessions preview: the AutosaveIndicator @@ -150,6 +165,14 @@ loadedFromDraft?: boolean othersDraftsCount?: number onOpenOthersDrafts?: () => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void + // Deploy created the app at a new path; the page navigates to it. Callback + // prop for the same reason as `onRestore` — `on:savedNewAppPath` forwarding + // through these runes-mode components is dropped. + onSavedNewAppPath?: (path: string) => void } let { @@ -160,6 +183,7 @@ version = $bindable(undefined), newApp, newPath = '', + labels: initialLabels = undefined, appPath, runnables, data, @@ -174,6 +198,7 @@ onOpenYamlEditor = undefined, sidebarCollapsed = false, onToggleSidebar = undefined, + condensedHeader = false, onNavigate = undefined, liveEditorDraftStoragePath = undefined, autosaveWorkspace = undefined, @@ -183,12 +208,19 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore, + onSavedNewAppPath }: Props = $props() + // Set by the on-behalf-of selector when the publisher picks a user other than + // themselves. Forwarded as `preserve_on_behalf_of` so the backend keeps the + // policy's on_behalf_of instead of resetting it to the deploying user. + let preserveOnBehalfOf = $state(false) + // The AutosaveIndicator watches these; in the sessions preview they're the // session's (workspace, path), else the full-page editor's own values. - const indicatorWorkspace = $derived(autosaveWorkspace ?? $workspaceStore) + const opWorkspace = $derived(autosaveWorkspace ?? $workspaceStore) const indicatorPath = $derived(autosavePath ?? liveEditorDraftStoragePath) $effect(() => { @@ -215,8 +247,8 @@ ) $effect(() => { - if (liveEditorDraftStoragePath === undefined || !$workspaceStore) return - const workspace = $workspaceStore + if (liveEditorDraftStoragePath === undefined || !opWorkspace) return + const workspace = opWorkspace UserDraft.setLiveEditorDraft({ workspace, itemKind: 'raw_app', @@ -255,6 +287,10 @@ let topbarWidth = $state(0) const compactTopbar = $derived(topbarWidth > 0 && topbarWidth < 720) + // Top-bar button size + bar height. Condensed (session preview) uses the + // smallest well-supported unified size (`sm`) so the bar is thinner. + const headerBtnSize = $derived(condensedHeader ? 'sm' : 'md') + async function publishToHub() { if (!app) return publishingToHub = true @@ -306,7 +342,7 @@ try { const { js, css } = await getBundle() await AppService.createAppRaw({ - workspace: $workspaceStore!, + workspace: opWorkspace!, formData: { app: { value: app, @@ -314,7 +350,9 @@ summary: summary, policy, deployment_message: deploymentMsg, - custom_path: customPath + custom_path: customPath, + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels }, js, css @@ -322,13 +360,14 @@ }) // New path now exists server-side — drop the autocomplete cache so // it shows up immediately instead of after the 60s TTL. - invalidateWorkspacePaths($workspaceStore!) + invalidateWorkspacePaths(opWorkspace!) savedApp = { summary: summary, value: structuredClone(stateSnapshot(app)), path: path, policy: policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } closeSaveDrawer() sendUserToast('App deployed successfully') @@ -345,7 +384,7 @@ path: appPath }) } - dispatch('savedNewAppPath', path) + onSavedNewAppPath?.(path) onDeploy?.({ path }) } catch (e) { sendUserToast(`Error creating app: ${e.body ?? e.message}`, true) @@ -369,15 +408,7 @@ savedApp && app && orderedJsonStringify(deployedValue) === - orderedJsonStringify( - replaceFalseWithUndefined({ - summary: summary, - value: app, - path: newEditedPath || savedApp.path, - policy, - custom_path: customPath - }) - ) + orderedJsonStringify(replaceFalseWithUndefined(currentDiffValue)) ) { await updateApp(npath) } else { @@ -393,22 +424,16 @@ async function syncWithDeployed() { const deployedApp = await AppService.getAppByPath({ - workspace: $workspaceStore!, + workspace: opWorkspace!, path: appPath!, withStarredInfo: true }) deployedBy = deployedApp.created_by - // Strip off extra information - deployedValue = replaceFalseWithUndefined({ - ...deployedApp, - id: undefined, - created_at: undefined, - created_by: undefined, - versions: undefined, - extra_perms: undefined - }) + // Normalize away post-deploy noise (see stripRawAppDiffNoise) so the + // diff/comparison only reflects what the editor actually changed. + deployedValue = replaceFalseWithUndefined(stripRawAppDiffNoise(deployedApp)) } async function openDiffDrawer() { @@ -422,14 +447,8 @@ diffDrawer?.openDrawer() diffDrawer?.setDiff({ mode: 'normal', - deployed: deployedValue ?? savedApp, - current: { - summary: summary, - value: app, - path: newEditedPath || savedApp.path, - policy, - custom_path: customPath - } + deployed: deployedValue ?? stripRawAppDiffNoise(savedApp), + current: currentDiffValue }) } @@ -444,7 +463,7 @@ policy.execution_mode = 'publisher' } await AppService.updateAppRaw({ - workspace: $workspaceStore!, + workspace: opWorkspace!, path: appPath!, formData: { app: { @@ -453,25 +472,28 @@ policy, path: npath, deployment_message: deploymentMsg, + preserve_on_behalf_of: preserveOnBehalfOf || undefined, // custom_path requires admin so to accept update without it, we need to send as undefined when non-admin (when undefined, it will be ignored) // it also means that customPath needs to be set to '' instead of undefined to unset it (when admin) custom_path: - $userStore?.is_admin || $userStore?.is_super_admin ? (customPath ?? '') : undefined + $userStore?.is_admin || $userStore?.is_super_admin ? (customPath ?? '') : undefined, + labels }, js, css } }) - invalidateWorkspacePaths($workspaceStore!) + invalidateWorkspacePaths(opWorkspace!) savedApp = { summary: summary, value: structuredClone(stateSnapshot(app)), path: npath, policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } const appHistory = await AppService.getAppHistoryByPath({ - workspace: $workspaceStore!, + workspace: opWorkspace!, path: npath }) version = appHistory[0]?.version @@ -487,19 +509,21 @@ }) } if (appPath !== npath) { - dispatch('savedNewAppPath', npath) + onSavedNewAppPath?.(npath) } onDeploy?.({ path: npath }) } - async function setPublishState() { + async function setPublishState(message?: string) { await computeTriggerables() await AppService.updateApp({ - workspace: $workspaceStore!, + workspace: opWorkspace!, path: appPath, requestBody: { policy } }) - if (policy.execution_mode == 'anonymous') { + if (message) { + sendUserToast(message) + } else if (policy.execution_mode == 'anonymous') { sendUserToast('App require no login to be accessed') } else { sendUserToast('App require login and read-access') @@ -518,7 +542,7 @@ } try { const appVersion = await AppService.getAppLatestVersion({ - workspace: $workspaceStore!, + workspace: opWorkspace!, path: appPath }) onLatest = appVersion?.version === undefined || version === appVersion?.version @@ -585,15 +609,27 @@ } ]) - const dispatch = createEventDispatcher() - let customPath = $state(savedApp?.custom_path) let customPathError = $state('') + let labels = $state(untrack(() => initialLabels)) let jobsDrawerOpen = $state(false) let app = $derived(files ? { runnables: runnables, files, data } : undefined) + // Editor-side value for diffing/comparison against the deployed app, with the + // same noise stripped as the deployed side (see stripRawAppDiffNoise). + let currentDiffValue = $derived( + stripRawAppDiffNoise({ + summary: summary, + value: app, + path: newEditedPath || savedApp?.path, + policy, + custom_path: customPath, + labels + }) + ) + $effect(() => { saveDrawerOpen && compareVersions() }) @@ -605,13 +641,7 @@ bind:open {diffDrawer} bind:deployedValue - currentValue={{ - summary: summary, - value: app, - path: newEditedPath || savedApp?.path, - policy, - custom_path: customPath - }} + currentValue={currentDiffValue} /> @@ -632,14 +662,8 @@ diffDrawer?.openDrawer() diffDrawer?.setDiff({ mode: 'normal', - deployed: deployedValue ?? savedApp, - current: { - summary: summary, - value: app, - path: newEditedPath || savedApp.path, - policy, - custom_path: customPath - }, + deployed: deployedValue ?? stripRawAppDiffNoise(savedApp), + current: currentDiffValue, button: { text: 'Looks good, deploy', onClick: () => { @@ -685,19 +709,22 @@ {onLatest} {savedApp} rawApp + operatingWorkspace={opWorkspace} bind:summary bind:customPath bind:deploymentMsg bind:customPathError bind:pathError bind:newEditedPath + bind:preserveOnBehalfOf + bind:labels /> (historyBrowserDrawerOpen = false)}> - + onRestore?.(e.detail)} {appPath} /> @@ -753,9 +780,15 @@
-
+ +
{#if onToggleSidebar}
- {#if !inSessionPane} - - {/if} + + opWorkspace && indicatorPath !== undefined + ? UserDraftDbSyncer.flush({ + workspace: opWorkspace, + itemKind: 'raw_app', + path: indicatorPath + }) + : undefined + } + : undefined} + > + {#snippet fallback()} + + {/snippet} + +
+ {/if} + {#if guarded} +
+ + Large file — {lineCount.toLocaleString()} lines. + + +
+ {:else} + {#await import('$lib/components/DiffEditor.svelte')} +
+ {:then Module} +
+ +
+ {/await} + {/if} +
diff --git a/frontend/src/lib/components/raw_apps/RawAppInlineScriptEditor.svelte b/frontend/src/lib/components/raw_apps/RawAppInlineScriptEditor.svelte index 7abaa40b52..42c048878b 100644 --- a/frontend/src/lib/components/raw_apps/RawAppInlineScriptEditor.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppInlineScriptEditor.svelte @@ -24,6 +24,7 @@ import { usePreparedAssetSqlQueries } from '$lib/infer.svelte' import AssetsDropdownButton from '../assets/AssetsDropdownButton.svelte' import { workspaceStore } from '$lib/stores' + import { getRawAppOperatingWorkspace } from './rawAppWorkspace' import { SvelteSet } from 'svelte/reactivity' import { Pane, Splitpanes } from 'svelte-splitpanes' import { editor as meditor } from 'monaco-editor' @@ -76,6 +77,10 @@ onSelectionChange, delete_after_secs = $bindable() }: Props = $props() + + const getOpWs = getRawAppOperatingWorkspace() + let opWs = $derived(getOpWs?.() ?? $workspaceStore) + let diffEditor = $state() as DiffEditor | undefined let validCode = $state(true) @@ -145,7 +150,7 @@ ) let preparedSqlQueries = usePreparedAssetSqlQueries( () => inferAssetsRes.current?.sql_queries, - () => $workspaceStore + () => opWs ) $effect(() => { if (!inlineScript || !inferAssetsRes.current || inferAssetsRes.current.status === 'error') @@ -303,13 +308,13 @@ resetDAPClient() dapClient = getDAPClient(dapServerUrl) - const env = await fetchContextualVariables($workspaceStore ?? '') + const env = await fetchContextualVariables(opWs ?? '') const code = inlineScript.content let signedPayload try { signedPayload = await signDebugRequest( - $workspaceStore ?? '', + opWs ?? '', code ?? '', inlineScript.language ?? 'python3' ) @@ -636,6 +641,7 @@
(showDebugConsole = false)} - workspace={$workspaceStore} + workspace={opWs} jobId={debugSessionJobId ?? undefined} /> diff --git a/frontend/src/lib/components/raw_apps/RawAppInlineScriptRunnable.svelte b/frontend/src/lib/components/raw_apps/RawAppInlineScriptRunnable.svelte index c4af7309b4..ca48eb74dc 100644 --- a/frontend/src/lib/components/raw_apps/RawAppInlineScriptRunnable.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppInlineScriptRunnable.svelte @@ -29,6 +29,10 @@ import { userStore, workspaceStore } from '$lib/stores' import { isHubFlowPath } from '$lib/utils' import { sendUserToast } from '$lib/toast' + import { getRawAppOperatingWorkspace } from './rawAppWorkspace' + + const getOpWs = getRawAppOperatingWorkspace() + let opWs = $derived(getOpWs?.() ?? $workspaceStore) type RunnableWithInlineScript = RunnableWithFields & { inlineScript?: InlineScript & { language: ScriptLang } @@ -131,7 +135,7 @@ case 'groups': return $userStore?.groups ?? [] case 'workspace': - return $workspaceStore ?? '' + return opWs ?? '' case 'author': return $userStore?.email ?? '' // In editor, author is the current user default: @@ -188,6 +192,7 @@ v.type == 'static') .map(([k]) => k)} diff --git a/frontend/src/lib/components/raw_apps/RawAppInlineScriptsPanel.svelte b/frontend/src/lib/components/raw_apps/RawAppInlineScriptsPanel.svelte index 17b981c2a9..6d0854ee54 100644 --- a/frontend/src/lib/components/raw_apps/RawAppInlineScriptsPanel.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppInlineScriptsPanel.svelte @@ -2,6 +2,10 @@ import { workspaceStore } from '$lib/stores' import RawAppInlineScripRunnable, { type Runnable } from './RawAppInlineScriptRunnable.svelte' import { createScriptFromInlineScript } from '../apps/editor/inlineScriptsPanel/utils' + import { getRawAppOperatingWorkspace } from './rawAppWorkspace' + + const getOpWs = getRawAppOperatingWorkspace() + let opWs = $derived(getOpWs?.() ?? $workspaceStore) interface Props { runnables: Record @@ -36,12 +40,7 @@ { - createScriptFromInlineScript( - selectedRunnable ?? '', - e.detail, - $workspaceStore ?? '', - appPath - ) + createScriptFromInlineScript(selectedRunnable ?? '', e.detail, opWs ?? '', appPath) }} on:delete={() => { if (selectedRunnable) { diff --git a/frontend/src/lib/components/raw_apps/RawAppInputsSpecEditor.svelte b/frontend/src/lib/components/raw_apps/RawAppInputsSpecEditor.svelte index be2025baf2..7b3639f41d 100644 --- a/frontend/src/lib/components/raw_apps/RawAppInputsSpecEditor.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppInputsSpecEditor.svelte @@ -14,6 +14,10 @@ import type { InputType } from '../apps/inputType' import Select from '$lib/components/select/Select.svelte' import { userStore, workspaceStore } from '$lib/stores' + import { getRawAppOperatingWorkspace } from './rawAppWorkspace' + + const getOpWs = getRawAppOperatingWorkspace() + let opWs = $derived(getOpWs?.() ?? $workspaceStore) // Build ctx properties with current user's actual values let ctxProperties = $derived([ @@ -31,7 +35,7 @@ { value: 'workspace', label: 'Workspace', - subtitle: `string — "${$workspaceStore ?? 'unknown'}"` + subtitle: `string — "${opWs ?? 'unknown'}"` }, { value: 'author', label: 'Author', subtitle: `string — "${$userStore?.email ?? 'unknown'}"` } ]) diff --git a/frontend/src/lib/components/raw_apps/RawAppPreview.svelte b/frontend/src/lib/components/raw_apps/RawAppPreview.svelte index 296590a0e4..9a70b162d7 100644 --- a/frontend/src/lib/components/raw_apps/RawAppPreview.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppPreview.svelte @@ -2,8 +2,8 @@ import { type UserExt } from '$lib/stores' import RawAppBackgroundRunner from './RawAppBackgroundRunner.svelte' import type { Runnable } from './rawAppPolicy' - import { htmlContent } from './utils' - import { onMount, untrack } from 'svelte' + import { getContext, onMount, untrack } from 'svelte' + import { unsandboxedRawAppHtml } from './utils' interface Props { workspace: string @@ -17,43 +17,190 @@ let iframe = $state() as HTMLIFrameElement | undefined - // Get initial hash from parent URL to pass to iframe - let initialHash = $state('') + // Get initial hash from parent URL to pass to the iframe + let initialHash = '' - onMount(() => { - initialHash = window.location.hash || '' + // WIN-2006: unless the publisher opted into sandbox isolation, run the bundle + // same-origin with full access (the default); otherwise the opaque-origin sandbox. + const unsandboxedCtx = getContext<{ value: boolean }>('IS_APP_UNSANDBOXED') + let unsandboxed = $derived(unsandboxedCtx?.value ?? false) + // Unsandboxed (the default) must match the pre-isolation viewer exactly: NO + // sandbox attribute (a same-origin blob with full session — an attribute would + // only break leftover features like unsandboxed popups for OAuth flows, while + // adding no isolation). The sandboxed path keeps the restrictive attribute; the + // wrapper document's `CSP: sandbox` response header enforces the opaque origin + // regardless. + let sandboxAttr = $derived( + unsandboxed + ? undefined + : 'allow-scripts allow-forms allow-popups allow-popups-to-escape-sandbox allow-downloads allow-modals allow-top-navigation' + ) + + // WIN-2006: source of the bundle iframe. + // - DEFAULT (isolated): a real API URL serving a sandboxed, opaque-origin + // document (`CSP: sandbox` response header + the iframe sandbox attribute), + // so a malicious bundle can never reach the authenticated Windmill origin + // (no cookie, no window.parent, no token). Root-relative so it resolves + // against the real host even when this component itself runs inside an opaque + // viewer (where `location.origin` is "null"). Context is handed over via + // postMessage — never baked into the document, never a credential. + // - UNSANDBOXED (the default — publisher did not opt into isolation): a + // client-built blob: wrapper (same-origin with the SPA) loaded with `allow-same-origin`, + // so relative `fetch('/api/...')` and the session cookie work. The backend + // `.html` is ALWAYS sandboxed, so we must build the same-origin wrapper here + // rather than relax a real-origin endpoint a victim could be linked to. + let iframeSrc = $derived.by(() => { + if (!secret || typeof window === 'undefined') return undefined + if (unsandboxed) { + // untrack(user) so userStore refreshes don't regenerate the blob URL and + // reload the iframe (losing state); ctx is only needed for initial render. + // Always pass the wrapper object — pre-sandbox bundles rely on + // `window.ctx.workspace` even for anonymous viewers (ctx.ctx undefined). + const u = untrack(() => user) + const html = unsandboxedRawAppHtml( + workspace, + secret, + { ctx: u, workspace }, + window.location.origin, + window.location.hash || '' + ) + return URL.createObjectURL(new Blob([html], { type: 'text/html' })) + } + // `wm_coep` (embed-in-cross-origin-isolated-page opt-in) must be propagated + // to the wrapper document: under a COEP `require-corp` embedder, a nested + // document is only allowed to load if it asserts COEP itself, so the + // backend adds the header when the flag is present. + const coep = new URLSearchParams(window.location.search).has('wm_coep') ? '?wm_coep=1' : '' + return `/api/w/${workspace}/apps_u/get_data/v/${secret}.html${coep}` }) - // Use blob URL instead of srcDoc to give the iframe a proper origin. - // srcDoc iframes have "null" origin which breaks URL constructor in routers. - // untrack(user) so that userStore refreshes don't regenerate the blob URL - // and cause the iframe to fully reload (losing all state). - // The user context is only needed for initial render. - let blobUrl = $derived.by(() => { - if (!secret) return undefined - const u = untrack(() => user) - const baseUrl = typeof window !== 'undefined' ? window.location.origin : '' - const html = htmlContent(workspace, secret, { ctx: u, workspace }, baseUrl, initialHash) - const blob = new Blob([html], { type: 'text/html' }) - return URL.createObjectURL(blob) - }) - - // Cleanup blob URL when it changes or component unmounts + // Revoke blob: URLs (unsandboxed path) when they change or on unmount. $effect(() => { - const url = blobUrl + const url = iframeSrc return () => { - if (url) URL.revokeObjectURL(url) + if (url && url.startsWith('blob:')) URL.revokeObjectURL(url) + } + }) + + // Persistence for the bundle's (opaque-origin) localStorage, backed by a store + // scoped PER APP (keyed by workspace + app path) so one sandboxed app can't read + // or clobber another's (even two apps at the same path in different workspaces). On a real origin (workspace viewer, public page — even when + // that page sits inside someone else's iframe) it reads/writes real localStorage + // directly. Only inside an opaque frame (the Windmill embed viewer), where Web + // Storage throws, does it relay per-key ops up to the embedder, the persistence + // authority. `framed` therefore probes storage rather than just `window.parent`: + // an externally-embedded public page is framed too, but its parent is not the + // Windmill embedder and would never answer the relay (leaving the bundle without + // ctx). The snapshot is handed to the bundle before it evaluates so its + // localStorage is hydrated synchronously. + const SHARED_LS_KEY = `wm_apps_localstorage:${workspace}:${path}` + function storageAccessible(): boolean { + try { + localStorage.getItem(SHARED_LS_KEY) + return true + } catch (_) { + return false + } + } + const framed = typeof window !== 'undefined' && window.parent !== window && !storageAccessible() + let bundleStorage: Record | undefined = undefined + let pendingReady = false + + function readDirect(): Record { + try { + return JSON.parse(localStorage.getItem(SHARED_LS_KEY) || '{}') + } catch (_) { + return {} + } + } + + function applyDirectOp(d: any) { + try { + const s = readDirect() + if (d.op === 'set') s[d.key] = String(d.value) + else if (d.op === 'remove') delete s[d.key] + else if (d.op === 'clear') for (const k in s) delete s[k] + localStorage.setItem(SHARED_LS_KEY, JSON.stringify(s)) + } catch (_) {} + } + + function respondCtx() { + iframe?.contentWindow?.postMessage( + { + type: 'windmill:ctx', + // Same shape as the unsandboxed wrapper: always the object, so + // `window.ctx.workspace` works for anonymous viewers too. + ctx: { ctx: user, workspace }, + initialHash, + storage: { local: bundleStorage ?? {}, session: {} } + }, + '*' + ) + } + + onMount(() => { + initialHash = window.location.hash || '' + if (framed) { + // Pre-fetch the shared store from the embedder. + try { + window.parent.postMessage({ type: 'wm_ls_req' }, '*') + } catch (_) {} + // If the parent never answers (it isn't the Windmill embedder, e.g. an + // opaque context created by a third party), don't hold the bundle's ctx + // hostage: proceed with empty storage. Must beat the backend wrapper's + // own 1.5s no-ctx fallback. + const fallback = setTimeout(() => { + if (bundleStorage === undefined) { + bundleStorage = {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + } + }, 750) + return () => clearTimeout(fallback) } }) - // Listen for hash changes from iframe and update parent URL $effect(() => { function handleMessage(event: MessageEvent) { - console.log('[Parent] Received message:', event.data) - if (event.data?.type === 'windmill:hashchange') { - const newHash = event.data.hash || '' - console.log('[Parent] Updating hash to:', newHash) - // Update parent URL without triggering navigation + const data = event.data + // Shared-store hydration from the embedder (public mode only). + if (framed && event.source === window.parent && data?.type === 'wm_ls_hydrate') { + bundleStorage = data.data || {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + return + } + // Everything else must come from the bundle iframe. + if (event.source !== iframe?.contentWindow) return + if (data?.type === 'windmill:ready') { + // Hand the bundle its context + shared storage before it evaluates. + if (!framed) { + bundleStorage = readDirect() + respondCtx() + } else if (bundleStorage !== undefined) { + respondCtx() + } else { + pendingReady = true + } + } else if (data?.type === 'wm_ls_op') { + // The bundle mutated localStorage — apply it to the shared store. + if (!framed) { + applyDirectOp(data) + } else { + try { + window.parent.postMessage( + { type: 'wm_ls_op', op: data.op, key: data.key, value: data.value }, + '*' + ) + } catch (_) {} + } + } else if (data?.type === 'windmill:hashchange') { + // Keep the parent URL hash in sync for shareable URLs. + const newHash = data.hash || '' if (window.location.hash !== newHash) { history.replaceState(null, '', newHash || window.location.pathname) } @@ -65,13 +212,29 @@ }) - + -{#if blobUrl} +{#if iframeSrc} + + {/if} diff --git a/frontend/src/lib/components/raw_apps/RawAppSharedUiDrawer.svelte b/frontend/src/lib/components/raw_apps/RawAppSharedUiDrawer.svelte index eedf10239f..b84a60ca0f 100644 --- a/frontend/src/lib/components/raw_apps/RawAppSharedUiDrawer.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppSharedUiDrawer.svelte @@ -6,6 +6,10 @@ import DrawerContent from '../common/drawer/DrawerContent.svelte' import Editor from '$lib/components/Editor.svelte' import { Pane, Splitpanes } from 'svelte-splitpanes' + import { getRawAppOperatingWorkspace } from './rawAppWorkspace' + + const getOpWs = getRawAppOperatingWorkspace() + let opWs = $derived(getOpWs?.() ?? $workspaceStore) let open = $state(false) let files: Record = $state({}) @@ -20,10 +24,10 @@ } async function load() { - if (!$workspaceStore) return + if (!opWs) return loading = true try { - const res = (await WorkspaceService.getSharedUi({ workspace: $workspaceStore })) as any + const res = (await WorkspaceService.getSharedUi({ workspace: opWs })) as any files = res.files ?? {} version = res.version ?? 0 editedBy = res.edited_by ?? '' diff --git a/frontend/src/lib/components/raw_apps/RawAppTemplatePicker.svelte b/frontend/src/lib/components/raw_apps/RawAppTemplatePicker.svelte index 308c554891..e37892516b 100644 --- a/frontend/src/lib/components/raw_apps/RawAppTemplatePicker.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppTemplatePicker.svelte @@ -3,6 +3,7 @@ import type { Policy } from '$lib/gen' import { userStore, workspaceStore } from '$lib/stores' import { sendUserToast } from '$lib/toast' + import { getRawAppOperatingWorkspace } from './rawAppWorkspace' import Modal from '$lib/components/common/modal/Modal.svelte' import Button from '$lib/components/common/button/Button.svelte' import TextInput from '$lib/components/text_input/TextInput.svelte' @@ -60,8 +61,14 @@ let preWhitelistedTables = $state([]) let dataTableDrawer: RawAppDataTableDrawer | undefined = $state() - const datatables = createDatatablesResource(() => $workspaceStore) - const schemas = createSchemasResource(() => selectedDatatable) + const getOpWs = getRawAppOperatingWorkspace() + let opWs = $derived(getOpWs?.() ?? $workspaceStore) + + const datatables = createDatatablesResource(() => opWs) + const schemas = createSchemasResource( + () => selectedDatatable, + () => opWs + ) const availableDatatables = $derived(datatables.current) const availableSchemas = $derived(schemas.current) @@ -115,11 +122,11 @@ async function start(withPrompt: boolean) { const template = templates[selectedTemplateIndex] - if (schemaMode === 'new' && newSchemaName && selectedDatatable && $workspaceStore) { + if (schemaMode === 'new' && newSchemaName && selectedDatatable && opWs) { try { const { dbSchemaOpsWithPreviewScripts } = await import('$lib/components/dbOps') const dbOps = dbSchemaOpsWithPreviewScripts({ - workspace: $workspaceStore, + workspace: opWs, input: { type: 'database', resourceType: 'postgresql', diff --git a/frontend/src/lib/components/raw_apps/datatableUtils.svelte.ts b/frontend/src/lib/components/raw_apps/datatableUtils.svelte.ts index bdb93355c6..5e25b73eca 100644 --- a/frontend/src/lib/components/raw_apps/datatableUtils.svelte.ts +++ b/frontend/src/lib/components/raw_apps/datatableUtils.svelte.ts @@ -25,22 +25,29 @@ export function createDatatablesResource(getWorkspace: () => string | undefined) * Creates a resource that loads schemas for a given datatable. * The getDatatable getter is used as a reactive dependency - when it changes, schemas are refetched. */ -export function createSchemasResource(getDatatable: () => string | undefined) { - return resource([() => getDatatable() ?? ''], async () => { +export function createSchemasResource( + getDatatable: () => string | undefined, + getWorkspace: () => string | undefined = () => get(workspaceStore) +) { + return resource([() => getDatatable() ?? '', () => getWorkspace() ?? ''], async () => { const datatable = getDatatable() - const workspace = get(workspaceStore) + const workspace = getWorkspace() if (!datatable || !workspace) return [] const resourcePath = `datatable://${datatable}` + // Key the schema cache by workspace too: a datatable of the same name can + // exist in both the nav and the acting workspace, so `datatable://` + // alone would let one workspace's schema be reused for the other. + const cacheKey = `${workspace}:${resourcePath}` const schemas = get(dbSchemas) - let dbSchema = schemas[resourcePath] + let dbSchema = schemas[cacheKey] if (!dbSchema) { try { - schemas[resourcePath] = await getDbSchemas('postgresql', resourcePath, workspace, (msg) => + schemas[cacheKey] = await getDbSchemas('postgresql', resourcePath, workspace, (msg) => console.error('Schema error:', msg) ) - dbSchema = get(dbSchemas)[resourcePath] + dbSchema = get(dbSchemas)[cacheKey] } catch (e) { console.error(`Failed to load schema for ${datatable}:`, e) return [] diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts new file mode 100644 index 0000000000..300b5dde1e --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts @@ -0,0 +1,299 @@ +import { describe, expect, it } from 'vitest' +import { + parseRawAppDiff, + rawAppDiffToItems, + RAW_APP_METADATA_PATH, + type RawAppDiffEntry +} from './rawAppDiffUtils' + +function byPath(entries: RawAppDiffEntry[], path: string): RawAppDiffEntry | undefined { + return entries.find((e) => e.path === path) +} + +describe('parseRawAppDiff — files', () => { + it('detects added, removed and modified files, omits unchanged', () => { + const original = { + files: { 'index.html': '

hi

', 'styles.css': 'body{}', 'gone.js': 'x' } + } + const current = { + files: { 'index.html': '

hello

', 'styles.css': 'body{}', 'new.ts': 'y' } + } + const entries = parseRawAppDiff(original, current) + const paths = entries.map((e) => e.path).sort() + // styles.css unchanged → omitted + expect(paths).toEqual(['gone.js', 'index.html', 'new.ts']) + + expect(byPath(entries, 'index.html')?.status).toBe('modified') + expect(byPath(entries, 'index.html')?.lang).toBe('html') + expect(byPath(entries, 'gone.js')?.status).toBe('removed') + expect(byPath(entries, 'new.ts')?.status).toBe('added') + expect(byPath(entries, 'new.ts')?.lang).toBe('typescript') + }) + + it('carries original/current content on the right sides', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': 'old', 'b.txt': 'keep' } }, + { files: { 'a.txt': 'new', 'b.txt': 'keep' } } + ) + const a = byPath(entries, 'a.txt')! + expect(a.original).toBe('old') + expect(a.current).toBe('new') + }) +}) + +describe('parseRawAppDiff — runnables', () => { + it('emits per-runnable leaves for add/remove/modify', () => { + const original = { + runnables: { a: { path: 'u/x/a' }, b: { path: 'u/x/b' }, same: { path: 'u/x/same' } } + } + const current = { + runnables: { a: { path: 'u/x/a2' }, c: { path: 'u/x/c' }, same: { path: 'u/x/same' } } + } + const entries = parseRawAppDiff(original, current) + expect(byPath(entries, 'runnables/a')?.status).toBe('modified') + expect(byPath(entries, 'runnables/a')?.lang).toBe('yaml') + expect(byPath(entries, 'runnables/b')?.status).toBe('removed') + expect(byPath(entries, 'runnables/c')?.status).toBe('added') + // identical runnable omitted + expect(byPath(entries, 'runnables/same')).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — metadata', () => { + it('collapses summary/data/policy/custom_path into one app.yaml leaf', () => { + const entries = parseRawAppDiff( + { summary: 'old summary', custom_path: 'foo' }, + { summary: 'new summary', custom_path: 'foo' } + ) + const meta = byPath(entries, RAW_APP_METADATA_PATH) + expect(meta?.status).toBe('modified') + expect(meta?.lang).toBe('yaml') + expect(meta?.original).toContain('old summary') + expect(meta?.current).toContain('new summary') + }) + + it('omits app.yaml when no metadata field changed', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': '1' }, summary: 's' }, + { files: { 'a.txt': '2' }, summary: 's' } + ) + expect(byPath(entries, RAW_APP_METADATA_PATH)).toBeUndefined() + expect(entries).toHaveLength(1) + }) +}) + +describe('parseRawAppDiff — whole app added / removed', () => { + it('marks everything added when the original side is absent', () => { + const current = { + files: { 'index.html': '

hi

' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'brand new' + } + const entries = parseRawAppDiff(undefined, current) + expect(entries.every((e) => e.status === 'added')).toBe(true) + expect(byPath(entries, 'index.html')?.original).toBeUndefined() + expect(byPath(entries, RAW_APP_METADATA_PATH)?.status).toBe('added') + }) + + it('marks everything removed when the current side is absent', () => { + const original = { + files: { 'index.html': '

hi

' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'going away' + } + const entries = parseRawAppDiff(original, undefined) + expect(entries.every((e) => e.status === 'removed')).toBe(true) + expect(byPath(entries, 'index.html')?.current).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — collisions', () => { + it('disambiguates synthesized paths against real files', () => { + const original = { files: { 'app.yaml': 'real-old' }, summary: 'sa' } + const current = { files: { 'app.yaml': 'real-new' }, summary: 'sb' } + const entries = parseRawAppDiff(original, current) + // The real file keeps the natural path. + const realFile = byPath(entries, 'app.yaml')! + expect(realFile.original).toBe('real-old') + // The metadata leaf is pushed to a non-colliding path. + const meta = byPath(entries, 'app.yaml~2')! + expect(meta.original).toContain('sa') + expect(meta.current).toContain('sb') + // No two entries share a path. + const paths = entries.map((e) => e.path) + expect(new Set(paths).size).toBe(paths.length) + }) +}) + +describe('parseRawAppDiff — input shapes', () => { + // getItemValue returns the deployed app row: files/runnables/data live under + // `value`; summary/policy/custom_path at the top level. + it('reads files/runnables/data from the `value` wrapper (app-row shape)', () => { + const original = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

a

' }, runnables: {}, data: {} } + } + const current = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

b

' }, runnables: {}, data: {} } + } + const entries = parseRawAppDiff(original, current) + const file = byPath(entries, 'index.html') + expect(file?.status).toBe('modified') + expect(file?.original).toBe('

a

') + expect(file?.current).toBe('

b

') + }) + + it('still handles the flat draft shape (files at top level)', () => { + const entries = parseRawAppDiff({ files: { 'a.ts': 'x' } }, { files: { 'a.ts': 'y' } }) + expect(byPath(entries, 'a.ts')?.status).toBe('modified') + }) + + it('prefers `value` over a stray top-level files key', () => { + const entries = parseRawAppDiff( + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'a' } } }, + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'b' } } } + ) + expect(byPath(entries, 'real.ts')?.status).toBe('modified') + expect(byPath(entries, 'top.ts')).toBeUndefined() + }) +}) + +describe('rawAppDiffToItems', () => { + const appPath = 'u/admin/classy_app' + + it('produces composite-pathed items with status-derived exists flags', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'a', '/gone.ts': 'x' }, runnables: { r: { p: 1 } } } }, + { value: { files: { '/App.tsx': 'b', '/new.ts': 'y' }, runnables: {} } } + ) + const byPath = (p: string) => items.find((i) => i.path === p) + + const app = byPath('u/admin/classy_app/App.tsx')! + expect(app.kind).toBe('raw_app_file') + expect(app.status).toBe('modified') + expect(app.exists_in_source).toBe(true) + expect(app.exists_in_fork).toBe(true) + expect(app.appPath).toBe(appPath) + expect((app as any).lang).toBe('typescript') + + const gone = byPath('u/admin/classy_app/gone.ts')! + expect(gone.status).toBe('removed') + expect(gone.exists_in_source).toBe(true) + expect(gone.exists_in_fork).toBe(false) + + const added = byPath('u/admin/classy_app/new.ts')! + expect(added.status).toBe('added') + expect(added.exists_in_source).toBe(false) + expect(added.exists_in_fork).toBe(true) + + // Runnables render as script/flow rows, not file leaves. + const runnable = byPath('u/admin/classy_app/runnables/r')! + expect(runnable.kind).toBe('script') + expect(runnable.status).toBe('removed') + }) + + it('renders an inline-script runnable as a script row with code hoisted', () => { + const mk = (code: string) => ({ + value: { + files: {}, + runnables: { + a: { name: 'a', type: 'inline', inlineScript: { content: code, language: 'bun' } } + } + } + }) + const items = rawAppDiffToItems(appPath, mk('old code'), mk('new code')) + const r = items.find((i) => i.path === `${appPath}/runnables/a`) + expect(r?.kind).toBe('script') + expect(r?.status).toBe('modified') + // content + language hoisted to the top level for the script-style viewer + const cur = (r as any).currentRaw + expect(cur.content).toBe('new code') + expect(cur.language).toBe('bun') + expect(cur.inlineScript.content).toBeUndefined() + }) + + it('picks the flow kind for a runType:flow runnable', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: {}, runnables: {} } }, + { value: { files: {}, runnables: { f: { runType: 'flow', path: 'u/x/f' } } } } + ) + const r = items.find((i) => i.path === `${appPath}/runnables/f`) + expect(r?.kind).toBe('flow') + expect(r?.status).toBe('added') + }) + + it('flags the metadata item and attaches whole-app YAML for the expand view', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: {} } }, + { summary: 'new', value: { files: {} } } + ) + const meta = items.find((i) => i.path === `${appPath}/${RAW_APP_METADATA_PATH}`) as any + expect(meta.kind).toBe('raw_app_file') + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlOriginal).toContain('old') + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named app.yaml', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { 'app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { 'app.yaml': 'real-new' } } } + ) + // Real file keeps the natural path and is NOT the metadata item. + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + expect(realFile.isMetadata).toBe(false) + expect(realFile.fullYamlCurrent).toBeUndefined() + // Synthesized metadata moved to app.yaml~2 but still carries the flag + YAML. + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named /app.yaml (leading slash)', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { '/app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { '/app.yaml': 'real-new' } } } + ) + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(realFile.isMetadata).toBe(false) + expect(meta.isMetadata).toBe(true) + // distinct composite paths → distinct row keys + expect(realFile.path).not.toBe(meta.path) + }) + + it('treats a file keyed /App.tsx on one side and App.tsx on the other as one item', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'old' } } }, + { value: { files: { 'App.tsx': 'new' } } } + ) + const files = items.filter((i) => i.path === `${appPath}/App.tsx`) + expect(files).toHaveLength(1) + expect(files[0].status).toBe('modified') + expect((files[0] as any).original).toBe('old') + expect((files[0] as any).current).toBe('new') + }) + + it('dedups a runnable leaf against a real file named runnables/', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/runnables/foo': 'old' }, runnables: { foo: { p: 1 } } } }, + { value: { files: { '/runnables/foo': 'new' }, runnables: { foo: { p: 2 } } } } + ) + const realFile = items.find((i) => i.kind === 'raw_app_file')! + const runnable = items.find((i) => i.kind === 'script')! + expect(realFile.path).toBe(`${appPath}/runnables/foo`) + // Reserved away from the real file's composite path (slash-normalized). + expect(runnable.path).toBe(`${appPath}/runnables/foo~2`) + expect(runnable.path).not.toBe(realFile.path) + }) +}) diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts new file mode 100644 index 0000000000..444e03a0a3 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts @@ -0,0 +1,383 @@ +import { extToLang } from '$lib/editorLangUtils' +import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils' + +// A raw app rendered as a *folder of files* for diffing. Each entry is one +// virtual file: real `files` keep their natural path, runnables become +// `runnables/`, and the remaining app metadata collapses into a single +// `app.yaml` leaf. Only changed entries are emitted (unchanged are omitted). +export type RawAppDiffStatus = 'added' | 'removed' | 'modified' + +export interface RawAppDiffEntry { + /** Tree path. Real file path, `runnables/`, or `app.yaml`. */ + path: string + status: RawAppDiffStatus + /** Original (parent) side content. Undefined when status is `added`. */ + original?: string + /** Current (fork) side content. Undefined when status is `removed`. */ + current?: string + /** Monaco language id for the per-file diff editor. */ + lang: string + /** True for the synthesized metadata leaf. Tracked as a flag (not by matching + * `path === 'app.yaml'`) so it survives `reserveUnique` moving the leaf to + * `app.yaml~2` when a real file is literally named `app.yaml`. */ + isMetadata?: boolean +} + +// Loose shape — diff inputs come from `getItemValue` / drafts and aren't +// strictly typed, and arrive in TWO shapes: +// - the deployed app row (getAppByPath): `{ summary, policy, custom_path?, +// value: { files, runnables, data } }` — files/runnables/data nested under +// `value`. +// - the flat draft/runtime shape (RawAppDraft / RuntimeRawApp): everything at +// the top level. +// `normalizeRawApp` coalesces both. Anything not present is empty/absent. +export interface RawAppish { + value?: Record + files?: Record + runnables?: Record + summary?: unknown + data?: unknown + policy?: unknown + custom_path?: unknown + [k: string]: unknown +} + +interface NormalizedRawApp { + files: Record + runnables: Record + summary: unknown + data: unknown + policy: unknown + custom_path: unknown +} + +export const RAW_APP_METADATA_PATH = 'app.yaml' +const RUNNABLES_PREFIX = 'runnables/' +const METADATA_FIELDS = ['summary', 'data', 'policy', 'custom_path'] as const + +// Real file keys may carry a leading slash (`/App.tsx`) which `joinAppPath` +// strips. Collision reservation must compare in the stripped space, else a real +// `/app.yaml` and the synthetic `app.yaml` leaf both become `/app.yaml`. +const stripLeadingSlash = (p: string) => p.replace(/^\/+/, '') + +function isObject(v: unknown): v is Record { + return !!v && typeof v === 'object' +} + +function asFileMap(f: unknown): Record { + if (!isObject(f)) return {} + const out: Record = {} + for (const [k, v] of Object.entries(f)) { + // Canonicalize the key (strip the leading slash `joinAppPath` would strip + // anyway) so the same file keyed `/App.tsx` on one side and `App.tsx` on the + // other is treated as one file — not two leaves colliding at the same + // composite path. Coerce non-string content so the diff editor gets a string. + out[stripLeadingSlash(k)] = typeof v === 'string' ? v : String(v ?? '') + } + return out +} + +// Coalesce the two raw-app shapes (app row with a `value` wrapper vs flat draft) +// into a canonical view. Returns undefined when the whole app is absent. +// +// Per-field precedence is deliberately asymmetric and mirrors the app-row shape: +// the editor payload (`files`/`runnables`/`data`) lives under `value`, so those +// prefer `value` first; the row-level metadata (`summary`/`policy`/`custom_path`) +// lives at the top level, so those prefer `raw` first. Each falls back to the +// other side so a flat draft (everything top-level) still normalizes correctly. +function normalizeRawApp(raw: RawAppish | undefined): NormalizedRawApp | undefined { + if (!isObject(raw)) return undefined + const value = isObject(raw.value) ? raw.value : undefined + return { + files: asFileMap(value?.files ?? raw.files), + runnables: isObject(value?.runnables ?? raw.runnables) + ? ((value?.runnables ?? raw.runnables) as Record) + : {}, + summary: raw.summary ?? value?.summary, + data: value?.data ?? raw.data, + policy: raw.policy ?? value?.policy, + custom_path: raw.custom_path ?? value?.custom_path + } +} + +// The serialized metadata blob for one side, or undefined when the whole app +// is absent (so the `app.yaml` leaf reads as added/removed). +function metadataYaml(app: NormalizedRawApp | undefined): string | undefined { + if (!app) return undefined + const meta: Record = {} + for (const field of METADATA_FIELDS) { + if (app[field] !== undefined) meta[field] = app[field] + } + return orderedYamlStringify(meta) +} + +function extOf(path: string): string { + const base = path.split('/').pop() ?? path + const dot = base.lastIndexOf('.') + return dot >= 0 ? base.slice(dot + 1).toLowerCase() : '' +} + +function diffStatus(o: string | undefined, c: string | undefined): RawAppDiffStatus | undefined { + if (o === undefined && c !== undefined) return 'added' + if (o !== undefined && c === undefined) return 'removed' + if (o !== c) return 'modified' + return undefined +} + +// Reserve a synthesized path, disambiguating against already-taken paths so a +// real file literally named `app.yaml` or under `runnables/` never collides +// with a synthesized leaf. +function reserveUnique(path: string, taken: Set): string { + if (!taken.has(path)) { + taken.add(path) + return path + } + let i = 2 + while (taken.has(`${path}~${i}`)) i++ + const p = `${path}~${i}` + taken.add(p) + return p +} + +/** + * Diff two raw-app objects into a flat list of changed virtual files. Either + * side may be undefined (whole app added or removed). Consumers build a tree + * from the returned paths with `buildFileTree`. + */ +export function parseRawAppDiff( + original: RawAppish | undefined, + current: RawAppish | undefined, + opts: { includeRunnables?: boolean } = {} +): RawAppDiffEntry[] { + const { includeRunnables = true } = opts + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oFiles = oApp?.files ?? {} + const cFiles = cApp?.files ?? {} + const oRunnables = oApp?.runnables ?? {} + const cRunnables = cApp?.runnables ?? {} + + // Reserve every real file path (changed or not) up front so synthesized + // runnable/metadata paths can dodge collisions — slash-normalized so a real + // `/app.yaml` or `/runnables/x` is seen as colliding with the synthetic leaf. + const taken = new Set( + [...Object.keys(oFiles), ...Object.keys(cFiles)].map(stripLeadingSlash) + ) + + const entries: RawAppDiffEntry[] = [] + + // Real files. + for (const path of [...new Set([...Object.keys(oFiles), ...Object.keys(cFiles)])].sort()) { + const o = Object.prototype.hasOwnProperty.call(oFiles, path) ? oFiles[path] : undefined + const c = Object.prototype.hasOwnProperty.call(cFiles, path) ? cFiles[path] : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ path, status, original: o, current: c, lang: extToLang(extOf(path)) }) + } + + // Runnables → one YAML leaf each under `runnables/`. Consumers that render + // runnables as script/flow rows (rawAppDiffToItems) skip these and diff the + // runnable objects themselves. + const runnableNames = includeRunnables + ? [...new Set([...Object.keys(oRunnables), ...Object.keys(cRunnables)])].sort() + : [] + for (const name of runnableNames) { + const inO = Object.prototype.hasOwnProperty.call(oRunnables, name) + const inC = Object.prototype.hasOwnProperty.call(cRunnables, name) + const o = inO ? orderedYamlStringify(oRunnables[name]) : undefined + const c = inC ? orderedYamlStringify(cRunnables[name]) : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ + path: reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken), + status, + original: o, + current: c, + lang: 'yaml' + }) + } + + // Remaining metadata → single `app.yaml` leaf. + const oMeta = metadataYaml(oApp) + const cMeta = metadataYaml(cApp) + const metaStatus = diffStatus(oMeta, cMeta) + if (metaStatus) { + entries.push({ + path: reserveUnique(RAW_APP_METADATA_PATH, taken), + status: metaStatus, + original: oMeta, + current: cMeta, + lang: 'yaml', + isMetadata: true + }) + } + + return entries +} + +// A raw-app file rendered as a standalone diff item, shaped like a +// WorkspaceItemDiff (so it flows through the existing list / tree / search / +// count machinery) plus the embedded diff payload. `kind: 'raw_app_file'` +// keeps it distinct from the backend kinds. The composite `path` +// (`/`) nests it under the app's folder in the tree. +export interface RawAppFileItem { + kind: 'raw_app_file' + path: string + /** Friendly composite path (`/`) for tree display only; + * `path` stays storage-keyed so a never-deployed draft still loads/edits via + * its `…/draft_` path. Defaults to `path` when no friendly path differs. */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + /** Workspace path of the owning raw app (for the edit link). */ + appPath: string + status: RawAppDiffStatus + original?: string + current?: string + lang: string + /** True for the synthesized `app.yaml` metadata item. */ + isMetadata: boolean + /** Whole serialized app (both sides) — only on the metadata item, for its + * optional "expand to full YAML" view. */ + fullYamlOriginal?: string + fullYamlCurrent?: string +} + +// A raw-app runnable rendered as a script/flow item (what it actually is). It +// carries the reshaped runnable object per side so the normal script-style +// diff (Content + Metadata) renders it — inline code shows with proper syntax +// highlighting instead of a YAML blob. `kind` drives the row icon/label +// (script vs flow); the body is always rendered script-style. +export interface RawAppRunnableItem { + kind: 'script' | 'flow' + path: string + /** Friendly composite path for tree display only (see RawAppFileItem). */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + appPath: string + status: RawAppDiffStatus + /** Reshaped runnable (content/language hoisted) for the diff viewer. */ + originalRaw?: unknown + currentRaw?: unknown +} + +export type RawAppSyntheticItem = RawAppFileItem | RawAppRunnableItem + +function joinAppPath(appPath: string, filePath: string): string { + // File keys may carry a leading slash (e.g. `/App.tsx`); strip it so the + // composite path has single separators and splits cleanly in the tree. + return `${appPath}/${filePath.replace(/^\/+/, '')}` +} + +// The whole serialized app for one side, matching the previous YAML escape +// hatch (cleaned + ordered). Undefined when the app is absent on that side. +function wholeAppYaml(raw: RawAppish | undefined): string | undefined { + if (!isObject(raw)) return undefined + return orderedYamlStringify(cleanValueProperties(replaceFalseWithUndefined(raw))) +} + +// Hoist an inline runnable's code + language to the top level so the +// script-style diff viewer (which reads top-level `content`/`language`) shows +// the code in the Content tab and the rest as Metadata. Path-referencing +// runnables (no inline script) just fall through to a YAML metadata diff. +function reshapeRunnable(runnable: unknown): unknown { + if (!isObject(runnable)) return runnable + const inline = isObject(runnable.inlineScript) ? runnable.inlineScript : undefined + if (!inline) return runnable + return { + ...runnable, + content: inline.content, + language: inline.language, + // Drop the now-hoisted code so it isn't duplicated in the Metadata tab. + inlineScript: { ...inline, content: undefined } + } +} + +// Runnables become script/flow rows; `runType: 'flow'` picks the flow icon. +function runnableKind(...sides: unknown[]): 'script' | 'flow' { + return sides.some((s) => isObject(s) && s.runType === 'flow') ? 'flow' : 'script' +} + +/** + * Expand a raw-app diff into standalone items for `appPath`: + * - files + the `app.yaml` metadata leaf → `RawAppFileItem`s (the metadata item + * also carries the whole-app YAML for its expand view); + * - runnables → `RawAppRunnableItem`s (script/flow rows). + */ +export function rawAppDiffToItems( + appPath: string, + original: RawAppish | undefined, + current: RawAppish | undefined, + displayAppPath: string = appPath +): RawAppSyntheticItem[] { + // Files + metadata (runnables handled separately as script/flow rows). + const fileItems = parseRawAppDiff(original, current, { includeRunnables: false }).map( + (e): RawAppFileItem => ({ + kind: 'raw_app_file', + path: joinAppPath(appPath, e.path), + displayPath: joinAppPath(displayAppPath, e.path), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: e.status !== 'added', + exists_in_fork: e.status !== 'removed', + appPath, + status: e.status, + original: e.original, + current: e.current, + lang: e.lang, + isMetadata: e.isMetadata ?? false + }) + ) + const metaItem = fileItems.find((i) => i.isMetadata) + if (metaItem) { + metaItem.fullYamlOriginal = wholeAppYaml(original) + metaItem.fullYamlCurrent = wholeAppYaml(current) + } + + // Runnables → script/flow rows, diffed on their object value. + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oRun = oApp?.runnables ?? {} + const cRun = cApp?.runnables ?? {} + // Reserve each runnable's composite leaf against the real file paths, mirroring + // parseRawAppDiff, so a real file literally named `runnables/` can't yield + // a second leaf at the same composite path. Normalize the leading slash (which + // joinAppPath strips) so `/runnables/x` and `runnables/x` are seen as equal. + const taken = new Set( + [...Object.keys(oApp?.files ?? {}), ...Object.keys(cApp?.files ?? {})].map(stripLeadingSlash) + ) + const runnableItems: RawAppRunnableItem[] = [] + for (const name of [...new Set([...Object.keys(oRun), ...Object.keys(cRun)])].sort()) { + const inO = Object.prototype.hasOwnProperty.call(oRun, name) + const inC = Object.prototype.hasOwnProperty.call(cRun, name) + const oStr = inO ? orderedYamlStringify(oRun[name]) : undefined + const cStr = inC ? orderedYamlStringify(cRun[name]) : undefined + const status = diffStatus(oStr, cStr) + if (!status) continue + const rel = reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken) + runnableItems.push({ + kind: runnableKind(oRun[name], cRun[name]), + path: joinAppPath(appPath, rel), + displayPath: joinAppPath(displayAppPath, rel), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: status !== 'added', + exists_in_fork: status !== 'removed', + appPath, + status, + originalRaw: inO ? reshapeRunnable(oRun[name]) : undefined, + currentRaw: inC ? reshapeRunnable(cRun[name]) : undefined + }) + } + + return [...fileItems, ...runnableItems] +} diff --git a/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts b/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts index b99704436b..35ff0805d3 100644 --- a/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts +++ b/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts @@ -47,6 +47,12 @@ export function appSourceToDraftValue(app: any, fallback?: any): AppDraftValue { runnables: { ...(value.runnables ?? {}) }, data: normalizeRawAppData(value), policy: app.policy ?? fallback?.policy, - custom_path: app.custom_path ?? fallback?.custom_path + custom_path: app.custom_path ?? fallback?.custom_path, + // Pin the fork base: a deployed app exposes `versions` (head = last); an + // existing draft already carries `parent_version` — preserve it. + parent_version: + app.parent_version ?? + (Array.isArray(app.versions) ? app.versions[app.versions.length - 1] : undefined) ?? + fallback?.parent_version } } diff --git a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts index f33c6e7eb6..16f588c993 100644 --- a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts +++ b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts @@ -19,10 +19,11 @@ export async function updateRawAppPolicy( ) ).filter((entry): entry is [string, TriggerableV2] => entry != null) const triggerables_v2 = Object.fromEntries(entries) - return { + const next: Policy = { ...currentPolicy, triggerables_v2 } + return next } type RunnableWithInlineScript = RunnableWithFields & { diff --git a/frontend/src/lib/components/raw_apps/rawAppWorkspace.ts b/frontend/src/lib/components/raw_apps/rawAppWorkspace.ts new file mode 100644 index 0000000000..71702fd826 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppWorkspace.ts @@ -0,0 +1,22 @@ +import { getContext, setContext } from 'svelte' + +// The workspace a raw-app editor operates on. In a session preview this is the +// session's acting workspace, which differs from the navigation `$workspaceStore` +// (a session deliberately leaves the nav store on the workspace the top nav +// points at). RawAppEditor provides it once; the sidebar sub-components (inline +// scripts, datatable/shared-UI drawers, DB selector, …) read it so their lookups +// target the workspace the app actually lives in rather than the nav workspace. +// +// A getter (not a value) so the live `$derived` opWorkspace is read reactively at +// each call site. Consumers fall back to `$workspaceStore` when unset — e.g. the +// full-page app editor, where the nav workspace IS the operating workspace. + +const KEY = 'RawAppOperatingWorkspace' + +export function setRawAppOperatingWorkspace(get: () => string | undefined): void { + setContext(KEY, get) +} + +export function getRawAppOperatingWorkspace(): (() => string | undefined) | undefined { + return getContext(KEY) +} diff --git a/frontend/src/lib/components/raw_apps/utils.test.ts b/frontend/src/lib/components/raw_apps/utils.test.ts index b8ca6d207d..7d087a61ec 100644 --- a/frontend/src/lib/components/raw_apps/utils.test.ts +++ b/frontend/src/lib/components/raw_apps/utils.test.ts @@ -1,9 +1,11 @@ import { describe, expect, it } from 'vitest' import { + canonicalRawAppDiffValue, formatRuntimeLogsForChat, genWmillTs, normalizeRawAppRuntimeLogs, + stripRawAppDiffNoise, type Runnable } from './utils' @@ -61,3 +63,100 @@ describe('normalizeRawAppRuntimeLogs', () => { expect(formatRuntimeLogsForChat(entries)).toBe('[06:13:20.000] LOG: ready') }) }) + +// A deployed raw-app row as returned by getAppByPath: nested `value`, plus the +// server-managed columns and a recomputed inline-script lock. +function deployedRow() { + return { + id: 42, + raw_app: true, + is_draft: false, + created_at: '2024-01-01', + created_by: 'admin', + versions: [1, 2], + extra_perms: { 'u/admin': true }, + summary: 'app', + path: 'u/admin/app', + policy: { execution_mode: 'publisher' }, + value: { + files: { '/App.tsx': 'export default 1' }, + runnables: { + a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun', lock: 'deps\n' } } + } + } + } +} + +describe('stripRawAppDiffNoise', () => { + it('drops server-managed columns, nulls inline locks and canonicalizes data', () => { + const cleaned = stripRawAppDiffNoise(deployedRow()) + + for (const key of [ + 'raw_app', + 'id', + 'created_at', + 'created_by', + 'versions', + 'extra_perms', + 'is_draft' + ]) { + expect(cleaned).not.toHaveProperty(key) + } + expect(cleaned.value.runnables.a.inlineScript.lock).toBeUndefined() + // absent `data` is canonicalized to the default empty shape + expect(cleaned.value.data).toEqual({ tables: [], datatable: undefined, schema: undefined }) + }) + + it('does not mutate the input (live editor state)', () => { + const input = deployedRow() + stripRawAppDiffNoise(input) + expect(input.raw_app).toBe(true) + expect(input.value.runnables.a.inlineScript.lock).toBe('deps\n') + }) + + it('handles the flat editor/draft shape (files/runnables top-level)', () => { + const flat = { + summary: 'app', + files: { '/App.tsx': 'x' }, + runnables: { + a: { type: 'inline', inlineScript: { content: 'm()', language: 'bun', lock: 'l' } } + } + } + const cleaned = stripRawAppDiffNoise(flat) + expect(cleaned.runnables.a.inlineScript.lock).toBeUndefined() + expect(cleaned.data).toEqual({ tables: [], datatable: undefined, schema: undefined }) + }) +}) + +describe('canonicalRawAppDiffValue', () => { + it('collapses a nested deployed row and a flat draft to an identical value when content matches', () => { + const deployed = deployedRow() + // The flat draft shape a raw app autosaves: top-level files/runnables/data, + // no server columns, lock cleared on edit. + const draft = { + summary: 'app', + files: { '/App.tsx': 'export default 1' }, + runnables: { a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun' } } }, + data: { tables: [] }, + policy: { execution_mode: 'publisher' } + } + + expect(canonicalRawAppDiffValue(deployed)).toEqual(canonicalRawAppDiffValue(draft)) + }) + + it('still surfaces a real change (summary edit)', () => { + const deployed = deployedRow() + const draft = { + summary: 'app EDITED', + files: { '/App.tsx': 'export default 1' }, + runnables: { a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun' } } }, + data: { tables: [] } + } + + const a = canonicalRawAppDiffValue(deployed) + const b = canonicalRawAppDiffValue(draft) + expect(a).not.toEqual(b) + expect(a.summary).toBe('app') + expect(b.summary).toBe('app EDITED') + }) +}) diff --git a/frontend/src/lib/components/raw_apps/utils.ts b/frontend/src/lib/components/raw_apps/utils.ts index 53d84fd97c..c0fba48e18 100644 --- a/frontend/src/lib/components/raw_apps/utils.ts +++ b/frontend/src/lib/components/raw_apps/utils.ts @@ -3,6 +3,8 @@ import type { Schema } from '../../common' import { schemaToTsType } from '../../schema' import { isRunnableByName, isRunnableByPath, type RunnableWithFields } from '../apps/inputType' import type { InlineScript } from '../apps/sharedTypes' +import { stateSnapshot } from '$lib/svelte5Utils.svelte' +import { appSourceToDraftValue, normalizeRawAppData } from './rawAppDraftValue' // export type RunnableWithFields = any @@ -15,6 +17,66 @@ export type RawApp = { files: string[] } +// Server-managed columns the deployed app row (getAppByPath) carries but the +// editor's current value never does — leaving them in renders as spurious diff. +const RAW_APP_DEPLOYED_METADATA_KEYS = [ + 'raw_app', + 'id', + 'created_at', + 'created_by', + 'versions', + 'extra_perms', + 'is_draft' +] as const + +/** + * Normalize a raw-app value before a deployed-vs-current diff (or unsaved-change + * comparison). Three sources of spurious post-deploy diff: + * - the deployed row carries server-managed columns (`raw_app`, timestamps, …) + * that the editor's current value lacks; + * - inline-script `lock`s are recomputed server-side at every deploy and the + * editor clears them on edit, so the editor value and the freshly deployed + * one always diverge on `lock` even though the user changed nothing there; + * - the deployed value omits an empty `data` while the editor always carries + * the default `{ tables: [] }`, so even an untouched app reads as changed. + * All three must be neutralized symmetrically on both sides. Returns a deep + * clone; never mutates the input (the current side is live editor state). + */ +export function stripRawAppDiffNoise>(value: T): T { + const cloned = structuredClone(stateSnapshot(value)) as Record + for (const key of RAW_APP_DEPLOYED_METADATA_KEYS) { + delete cloned[key] + } + // Runnables/data live under `.value` on a deployed row and on the editor's + // diff value alike, but a flat draft shape carries them top-level. + const source = cloned.value ?? cloned + const runnables = source.runnables + if (runnables && typeof runnables === 'object') { + for (const k of Object.keys(runnables)) { + const inlineScript = runnables[k]?.inlineScript + if (inlineScript && inlineScript.lock != undefined) { + inlineScript.lock = undefined + } + } + } + // Canonicalize `data` so an absent and a default-empty `data` compare equal. + source.data = normalizeRawAppData(source) + return cloned as T +} + +/** + * Canonical raw-app value for diffing a *draft* against a *deployed* row. On top + * of the noise stripped by stripRawAppDiffNoise, the two also differ in shape: a + * deployed row nests its source under `value`, whereas a draft carries + * `files`/`runnables`/`data` at the top level. `appSourceToDraftValue` collapses + * both onto the same flat field set first. Use this for the session/compare + * draft diff so it matches the editor's Diff button (which shares + * stripRawAppDiffNoise). + */ +export function canonicalRawAppDiffValue(source: Record) { + return stripRawAppDiffNoise(appSourceToDraftValue(source)) +} + export type RawAppRuntimeLogLevel = 'log' | 'info' | 'warn' | 'error' | 'debug' export type RawAppRuntimeLogEntry = { level: RawAppRuntimeLogLevel @@ -79,65 +141,50 @@ export function formatAppRunsForChat(runs: RawAppRunSummary[]): string { return JSON.stringify(runs, null, 2) } -export function htmlContent( +// The sandboxed (isolated) raw-app wrapper is generated server-side and served as +// a sandboxed, opaque-origin document (see `get_raw_app_data` in the backend +// `apps.rs`, WIN-2006) — a blob: URL cannot carry the `CSP: sandbox` response +// header that enforces isolation, so the wrapper must come from the backend. +// +// The function below is used ONLY for the unsandboxed path (the default — the +// publisher did not opt into sandbox isolation). It is loaded as a blob: URL — +// same-origin with the SPA — so, with `allow-same-origin`, the bundle runs with +// the viewer's full session. Crucially this is an in-memory blob, not a +// real-origin endpoint, so it is not a URL an attacker can navigate a logged-in +// victim to in order to gain isolation-bypassing access — the backend `.html` +// document stays sandboxed whenever the publisher did opt in. +export function unsandboxedRawAppHtml( workspace: string, - secret: string | undefined, + secret: string, ctx: any, - baseUrl: string = '', - initialHash: string = '' + baseUrl: string, + initialHash: string ) { return ` - App Preview + App diff --git a/frontend/src/lib/components/runs/RunsQueue.svelte b/frontend/src/lib/components/runs/RunsQueue.svelte index b0c5ed5adb..2cd2914878 100644 --- a/frontend/src/lib/components/runs/RunsQueue.svelte +++ b/frontend/src/lib/components/runs/RunsQueue.svelte @@ -1,7 +1,7 @@ + +{#if self} +
+
{HANDLER_LABEL[self.kind]}
+
+ {#if self.handledJob} + + {/if} + {#if self.schedulePath} + + for schedule {self.schedulePath} + + {/if} +
+
+{:else} + {#if retries.length > 1} +
+
Retries ({retries.length - 1})
+
+ {#each retries as attempt, i (attempt.id)} + + {/each} +
+
+ {/if} + {#if handlers.length > 0} +
+
Handlers
+
+ {#each handlers as handler (handler.id)} + + {/each} +
+
+ {/if} +{/if} diff --git a/frontend/src/lib/components/runs/UpstreamSnapshotsPanel.svelte b/frontend/src/lib/components/runs/UpstreamSnapshotsPanel.svelte new file mode 100644 index 0000000000..ee265d5874 --- /dev/null +++ b/frontend/src/lib/components/runs/UpstreamSnapshotsPanel.svelte @@ -0,0 +1,82 @@ + + +{#if snapshots.length > 0} +
+

+ Upstream snapshots + + Version of each upstream asset when this run was dispatched. Recorded for debugging only — + the run reads the latest data, not these versions. To inspect what this run saw, query the + asset with the copied AT (VERSION => n) clause. For partitioned + assets, the partition shown is the slice whose write produced that snapshot — the snapshot itself + covers the whole table. + +

+
+
+ + + + + + + + + {#each snapshots as s (s.asset)} + + + + + + {/each} + +
AssetSnapshotTime travel
+ {s.asset} + + @ {s.snapshot_id} + {#if s.partition} + + · partition {s.partition} + {/if} + + +
+ + +{/if} diff --git a/frontend/src/lib/components/script_builder.ts b/frontend/src/lib/components/script_builder.ts index fb9d667da0..0fbd24d904 100644 --- a/frontend/src/lib/components/script_builder.ts +++ b/frontend/src/lib/components/script_builder.ts @@ -92,4 +92,8 @@ export interface ScriptBuilderProps { othersDraftsCount?: number // Wired by the route to flip the OtherUsersDraftsModal open. onOpenOthersDrafts?: () => void + // Condensed top bar: smaller (sm) buttons, a shorter bar, and the + // EditorHeader's path/breadcrumb row dropped (summary only). Used by the + // session preview to save vertical room. + condensedHeader?: boolean } diff --git a/frontend/src/lib/components/search/GlobalSearchModal.svelte b/frontend/src/lib/components/search/GlobalSearchModal.svelte index 19e71530e6..3e631a55e9 100644 --- a/frontend/src/lib/components/search/GlobalSearchModal.svelte +++ b/frontend/src/lib/components/search/GlobalSearchModal.svelte @@ -505,12 +505,16 @@ time: new Date(x.edited_at).getTime(), search_id: x.path })), - ...scripts.map((x) => ({ - ...x, - type: 'script' as 'script', - time: new Date(x.created_at).getTime(), - search_id: x.path - })), + // Pipeline-member scripts (`auto_kind='pipeline'`) are reached through + // their pipeline, not searched individually. + ...scripts + .filter((x) => x.auto_kind !== 'pipeline') + .map((x) => ({ + ...x, + type: 'script' as 'script', + time: new Date(x.created_at).getTime(), + search_id: x.path + })), ...apps.map((x) => ({ ...x, type: 'app' as 'app', diff --git a/frontend/src/lib/components/secretArgUtils.ts b/frontend/src/lib/components/secretArgUtils.ts index 82aee1f53d..91101d4816 100644 --- a/frontend/src/lib/components/secretArgUtils.ts +++ b/frontend/src/lib/components/secretArgUtils.ts @@ -11,11 +11,14 @@ import { generateRandomString } from '$lib/utils' */ export async function processSecretArgs( args: Record, - schema: Schema | undefined + schema: Schema | undefined, + // Workspace the ephemeral secret variable is created in — must match the + // workspace the preview job runs in, else $jsonvar: resolves to a missing var. + forceWorkspace?: string ): Promise> { if (!schema?.properties) return args - const workspace = get(workspaceStore) + const workspace = forceWorkspace ?? get(workspaceStore) const user = get(userStore) if (!workspace || !user) return args diff --git a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte b/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte deleted file mode 100644 index 0716398131..0000000000 --- a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte +++ /dev/null @@ -1,118 +0,0 @@ - - - buildEditUrl(d as unknown as WorkspaceItemDiff, workspaceId)} -> - {#snippet titleExtra()} -
- - {ws?.name ?? workspaceId} -
- {/snippet} -
diff --git a/frontend/src/lib/components/sessions/FlowEditorView.svelte b/frontend/src/lib/components/sessions/FlowEditorView.svelte index fb98e05496..845c6e2bdc 100644 --- a/frontend/src/lib/components/sessions/FlowEditorView.svelte +++ b/frontend/src/lib/components/sessions/FlowEditorView.svelte @@ -15,7 +15,8 @@ path, workspaceId, onNavigate, - isActiveSession = true + isActiveSession = true, + active = true }: { runtime: SessionRuntime path: string @@ -24,8 +25,12 @@ /** Forwarded to SessionEditorTarget — only the visible session claims the * workspace's single live-editor slot. */ isActiveSession?: boolean + /** Whether this is the visible preview tab (forwarded as isActiveTab). */ + active?: boolean } = $props() + // This tab's own flow cell; each open flow editor binds its own store. + const cell = $derived(runtime.flowCell(path)) let selectedId = $state('settings-metadata') let diffDrawer: DiffDrawer | undefined = $state() @@ -35,7 +40,7 @@ // baseline — useUserDraftSync's inbound effect then syncs the editor preview. // Mirrors ScriptEditorView. async function restoreDeployed() { - const saved = runtime.savedFlow.val + const saved = cell.saved.val if (!saved) { sendUserToast('Could not restore to deployed', true) return @@ -61,7 +66,7 @@ } -{#if runtime.savedFlow.val} +{#if cell.saved.val} {/if} runtime.flowStore.val?.path ?? path} + isActiveTab={active} + effectivePath={() => cell.store.val?.path ?? path} > {#snippet editor()} + + { // FlowBuilder has no deploy toast and the session stays put, so toast diff --git a/frontend/src/lib/components/sessions/ForkDiffDrawer.svelte b/frontend/src/lib/components/sessions/ForkDiffDrawer.svelte deleted file mode 100644 index e03fc5a49c..0000000000 --- a/frontend/src/lib/components/sessions/ForkDiffDrawer.svelte +++ /dev/null @@ -1,125 +0,0 @@ - - - buildEditUrl(d as unknown as WorkspaceItemDiff, forkWorkspaceId)} -> - {#snippet titleExtra()} -
- - {forkWs?.name ?? forkWorkspaceId} - - {parentWs?.name ?? parentWorkspaceId} - {#if comparison} - - {comparison.summary.total_diffs} item{comparison.summary.total_diffs !== 1 ? 's' : ''} - - {#if comparison.summary.conflicts > 0} - - - {comparison.summary.conflicts} conflict{comparison.summary.conflicts !== 1 ? 's' : ''} - - {/if} - {/if} -
- {/snippet} -
diff --git a/frontend/src/lib/components/sessions/OpenInSessionButton.svelte b/frontend/src/lib/components/sessions/OpenInSessionButton.svelte new file mode 100644 index 0000000000..6470e556bd --- /dev/null +++ b/frontend/src/lib/components/sessions/OpenInSessionButton.svelte @@ -0,0 +1,61 @@ + + + + +{#if show} + +{:else if !inSessionPanel} + {@render fallback?.()} +{/if} diff --git a/frontend/src/lib/components/sessions/PipelineEditorView.svelte b/frontend/src/lib/components/sessions/PipelineEditorView.svelte new file mode 100644 index 0000000000..0e30d0959d --- /dev/null +++ b/frontend/src/lib/components/sessions/PipelineEditorView.svelte @@ -0,0 +1,404 @@ + + +
+
+ + f/{path} + · data pipeline +
+
+ + {#if graphRes.loading && !graphRes.current && pe.drafts.size === 0} +
+ + Loading pipeline… +
+ {:else if graphRes.error && pe.drafts.size === 0} +
+ Failed to load pipeline: {graphRes.error.message} +
+ {:else} + + runNode(path, args)} + canRunByPath + onTestStateChange={(running) => { + const openPath = pe.openScriptPath + if (running && openPath) { + activeRunnable = { kind: 'script', path: openPath } + activeRunnables.arm(`script:${openPath}`) + activeRunnableJobId = undefined + } else if (!running && activeRunnable?.path === openPath) { + // Only clear the hint for the script the pane just finished — a + // canvas per-node run of a different script keeps its own hint. + activeRunnable = undefined + activeRunnableJobId = undefined + } + }} + onRunCompleted={() => { + activeRunnable = undefined + activeRunnableJobId = undefined + }} + onSelect={handleCanvasSelect} + onDraftSaved={afterSaved} + onPersistedSaved={afterSaved} + onScriptRemoved={async (removedPath) => { + pe.forgetPath(removedPath) + await graphRes.refetch() + }} + onScriptRenamed={async (oldPath, newPath) => { + // Repoint the selection so the canvas follows the renamed node instead + // of staying on the now-gone old path until an unrelated refetch. + if (pe.selection?.kind === 'runnable' && pe.selection.path === oldPath) { + pe.selection = { ...pe.selection, path: newPath } + } + await graphRes.refetch() + }} + onDiscard={() => { + if (pe.activeDraftPath) pe.discardDraft(pe.activeDraftPath) + }} + onClose={() => { + pe.selection = undefined + pe.activeDraftPath = undefined + pe.clearLiveOverlays() + }} + /> + {/if} +
+
+ + + graphRes.refetch()} +/> diff --git a/frontend/src/lib/components/sessions/PreviewRouterPicker.svelte b/frontend/src/lib/components/sessions/PreviewRouterPicker.svelte new file mode 100644 index 0000000000..80d09b6004 --- /dev/null +++ b/frontend/src/lib/components/sessions/PreviewRouterPicker.svelte @@ -0,0 +1,192 @@ + + + + + +{#snippet leafIcon(leaf: DrillLeaf)} + {#if leaf.data.type === 'item'} + + {:else if leaf.icon} + {@const Icon = leaf.icon} + + {/if} +{/snippet} + +{#snippet branchIcon(branch: DrillBranch)} + {#if branch.key === 'kind:flow' || branch.key === 'kind:script' || branch.key === 'kind:app'} + {@const k = branch.key.slice(5) as Kind} + + {:else if branch.icon} + {@const Icon = branch.icon} + + {/if} +{/snippet} + + onPick(leaf.data)} + initialScope={computedInitialScope} + {initialHighlight} + {externalFilter} + {autoFocus} + {flush} + {leafIcon} + {branchIcon} + leafSecondary={(leaf, scope) => + leaf.data.type === 'item' ? relativizeWorkspacePath(leaf.data.item.path, scope) : undefined} + onScopeChange={(scope) => { + if (scope.length > 0) loader.ensureForScopeSegment(scope[0]) + }} + onFilterChange={loader.onFilterChange} +/> diff --git a/frontend/src/lib/components/sessions/PreviewTabHost.svelte b/frontend/src/lib/components/sessions/PreviewTabHost.svelte new file mode 100644 index 0000000000..2a25cff3a5 --- /dev/null +++ b/frontend/src/lib/components/sessions/PreviewTabHost.svelte @@ -0,0 +1,124 @@ + + +{#if slot.kind === 'editor' && mounted && runtime} +
+ {#if slot.editorKind === 'flow'} + + {:else if slot.editorKind === 'script'} + + {:else if slot.editorKind === 'pipeline'} + + {:else} + + {/if} +
+{:else if mounted} + +{/if} diff --git a/frontend/src/lib/components/sessions/RawAppEditorView.svelte b/frontend/src/lib/components/sessions/RawAppEditorView.svelte index 3e600e5114..e5f9dfe30e 100644 --- a/frontend/src/lib/components/sessions/RawAppEditorView.svelte +++ b/frontend/src/lib/components/sessions/RawAppEditorView.svelte @@ -17,7 +17,8 @@ path, workspaceId, onNavigate, - isActiveSession = true + isActiveSession = true, + active = true }: { runtime: SessionRuntime path: string @@ -26,13 +27,17 @@ /** Forwarded to SessionEditorTarget — only the visible session claims the * workspace's single live-editor slot. */ isActiveSession?: boolean + /** Whether this is the visible preview tab (forwarded as isActiveTab). */ + active?: boolean } = $props() + // This tab's own raw-app cell; each open app editor binds its own store. + const cell = $derived(runtime.rawAppCell(path)) let diffDrawer: DiffDrawer | undefined = $state() // Path typed in the editor header, surfaced when it differs from the stored // path. Mirror it into the runtime draft as `draft_path` so the rename - // mutates runtime.rawApp.val → the autosave sig changes → the draft is saved + // mutates this cell's store → the autosave sig changes → the draft is saved // (and the home/review/Drafts lists show the friendly name). Mirrors the // full-page /apps_raw/edit route. let pendingDraftPath = $state(undefined) @@ -45,7 +50,7 @@ $effect(() => { const dp = pendingDraftPath untrack(() => { - const val = runtime.rawApp.val + const val = cell.store.val if (!val) return if (dp !== undefined) { surfacedDraftPath = true @@ -80,7 +85,7 @@ } -{#if runtime.savedRawApp.val} +{#if cell.saved.val} {/if} runtime.rawApp.val?.path ?? path} + isActiveTab={active} + effectivePath={() => cell.store.val?.path ?? path} > {#snippet editor()} - {#if runtime.rawApp.val} + {#if cell.store.val} + + { // Sync the preview to deployed (raw apps deploy only from this editor). diff --git a/frontend/src/lib/components/sessions/ScriptEditorView.svelte b/frontend/src/lib/components/sessions/ScriptEditorView.svelte index 563ddf9e55..e040fec744 100644 --- a/frontend/src/lib/components/sessions/ScriptEditorView.svelte +++ b/frontend/src/lib/components/sessions/ScriptEditorView.svelte @@ -16,7 +16,8 @@ workspaceId, onNavigate, initialTestPanelCollapsed = false, - isActiveSession = true + isActiveSession = true, + active = true }: { runtime: SessionRuntime path: string @@ -26,8 +27,12 @@ /** Forwarded to SessionEditorTarget — only the visible session claims the * workspace's single live-editor slot. */ isActiveSession?: boolean + /** Whether this is the visible preview tab (forwarded as isActiveTab). */ + active?: boolean } = $props() + // This tab's own script cell; each open script editor binds its own store. + const cell = $derived(runtime.scriptCell(path)) let diffDrawer: DiffDrawer | undefined = $state() // Restore actions for the diff drawer. The previous shared @@ -36,7 +41,7 @@ // reset the live UserDraft handle to the target baseline — the inbound // effect then syncs the editor preview. Mirrors /scripts/edit's restore. async function restoreDeployed() { - const saved = runtime.savedScript.val + const saved = cell.saved.val if (!saved) { sendUserToast('Could not restore to deployed', true) return @@ -71,7 +76,7 @@ } -{#if runtime.savedScript.val} +{#if cell.saved.val} {/if} runtime.scriptStore.val?.path ?? path} + isActiveTab={active} + effectivePath={() => cell.store.val?.path ?? path} > {#snippet editor()} - {#if runtime.scriptStore.val} + {#if cell.store.val} + import { + Archive, + ExternalLink, + GitPullRequestClosed, + MoveRight, + Pencil, + Trash2 + } from 'lucide-svelte' + import { Button } from '$lib/components/common' + import Badge from '$lib/components/common/badge/Badge.svelte' + import WorkspaceFamilyPicker from './WorkspaceFamilyPicker.svelte' + import { isPremiumStore, userStore, userWorkspaces, workspaceStore } from '$lib/stores' + import { canCreateFork } from '$lib/utils/editInFork' + import { isCloudHosted } from '$lib/cloud' + import { sessionState, type Session } from './sessionState.svelte' + import { getRuntime } from './sessionRuntime.svelte' + import SessionDiffDrawer from './SessionDiffDrawer.svelte' + import { useWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' + import { badgeCounts, buildDeployItems } from './sessionDeployModel' + import { useExistingMaskKeys } from './sessionDeployModel.svelte' + import JobsSegment from '$lib/components/copilot/chat/JobsSegment.svelte' + + // Unified session bar: surfaces what the CURRENT chat changed — pending + // drafts and deployed items — as one count badge per status that opens the + // session diff drawer. + let { + session, + onMove, + onCreateForkAndMove, + onArchive, + onDelete + }: { + session: Session + onMove?: (workspaceId: string) => void + onCreateForkAndMove?: (fork: { + parent_workspace_id: string + id: string + name: string + }) => void | Promise + onArchive?: () => void + onDelete?: () => void + } = $props() + + // Only meaningful once the session committed to a workspace (post first send). + const committedId = $derived(session.workspace_id) + const sessionWorkspace = $derived( + committedId ? $userWorkspaces.find((w) => w.id === committedId) : undefined + ) + const parentWorkspaceId = $derived(sessionWorkspace?.parent_workspace_id ?? undefined) + const parentWorkspace = $derived( + parentWorkspaceId ? $userWorkspaces.find((w) => w.id === parentWorkspaceId) : undefined + ) + const isFork = $derived(!!parentWorkspaceId) + + // Same gate as the sidebar WorkspaceMenu / SessionWorkspaceBar. On cloud, + // forking is a premium-only feature (backend caps it per paid seat). + const forksAllowed = $derived( + (!isCloudHosted() || $isPremiumStore) && + canCreateFork($userStore) && + $workspaceStore !== 'admins' + ) + + const runtime = $derived(getRuntime(session.id)) + + // The chat's modified-items mask (`${UserDraftItemKind}:${storagePath}`). + // undefined = legacy/untracked chat → fall back to showing every draft. + // A Set (even empty) → filter to just this chat's items. + const mask = $derived(runtime?.manager.modifiedItems) + + // Workspace Drafts (fetches on mount and on every Server-Draft invalidation); + // scoped to the chat's mask inside the deploy model below. + const drafts = useWorkspaceDrafts(() => committedId) + + // The committed workspace vanished from the user's list (deleted, archived, + // or access revoked) — the session can't operate in it anymore. + const isUnavailable = $derived(!!committedId && !sessionWorkspace) + + // The committed workspace is gone, so we can't read its parent to know if + // it was a fork — fall back to the fork-id convention for the wording. + const committedIsFork = $derived(committedId?.startsWith('wm-fork-') ?? false) + + // Full dock refresh: re-run existence checks + draft list. Both are + // stale-while-revalidate — a hard reset would blank dockItems for the + // round-trip, hiding the bar (and unmounting the drawer) mid-deploy. + function refreshDock() { + existing.refresh() + drafts.refresh() + } + + // Refresh the draft list when the AI finishes a turn (loading true → false): + // tool calls may have created/edited/deleted items. Deploys happen + // server-side, so the frontend only has this coarse signal. + let wasLoading = $state(false) + $effect(() => { + const isLoading = runtime?.manager.loading ?? false + if (wasLoading && !isLoading) refreshDock() + wasLoading = isLoading + }) + + // Refresh when the user comes back to this view — covers edits made + // elsewhere while we were away. Two signals: visibilitychange (another tab + // in the same window) and window focus (a second browser window, where this + // tab never goes hidden so visibilitychange never fires — e.g. deploying + // from the full-page editor side by side). + $effect(() => { + if (!committedId) return + function refreshIfCurrent() { + if (document.visibilityState !== 'visible') return + if (sessionState.currentSessionId !== session.id) return + refreshDock() + } + document.addEventListener('visibilitychange', refreshIfCurrent) + window.addEventListener('focus', refreshIfCurrent) + return () => { + document.removeEventListener('visibilitychange', refreshIfCurrent) + window.removeEventListener('focus', refreshIfCurrent) + } + }) + + let diffDrawer: SessionDiffDrawer | undefined = $state(undefined) + + // Opening the drawer re-fetches its own data, so it can show fresher state + // than the badge just clicked (e.g. "1 draft" that was deployed elsewhere in + // the meantime). Re-sync the dock alongside so the two never disagree. + function openDrawer() { + refreshDock() + diffDrawer?.open() + } + + // The dock counts are computed from the same pure model over this chat's + // drafts; the readout mirrors the drawer's item states. + // Existence check for mask-only (deployed) items — without it they'd be + // dropped from the dock counts while the drawer (which runs the same check) + // still shows them as Deployed. + const existing = useExistingMaskKeys(() => ({ + draftItems: drafts.items, + mask: committedId ? mask : undefined, + workspaceId: committedId ?? '' + })) + const dockItems = $derived( + committedId + ? buildDeployItems({ draftItems: drafts.items, mask, existingKeys: existing.keys }) + : [] + ) + const dockCounts = $derived(badgeCounts(dockItems)) + + // Deletion-only fork chat: the mask has entries but every one failed the + // (resolved) existence check — the chat's edits were deletions, which have no + // dock row by design. The pending fork→parent removal is still reviewable on + // the compare page, so the bar must keep that doorway instead of vanishing. + // Gated on a resolved check (undefined = still loading) so the bar doesn't + // flash this state while deployed rows are being confirmed, and on isFork — + // a non-fork deletion is immediate and final, with nothing left to review. + const deletionOnly = $derived( + isFork && (mask?.size ?? 0) > 0 && existing.keys !== undefined && dockItems.length === 0 + ) + const compareHref = $derived( + committedId + ? `/forks/compare?workspace_id=${encodeURIComponent(committedId)}&mode=fork` + + (session.chatId ? `&from_session=${encodeURIComponent(session.chatId)}` : '') + : undefined + ) + + // One "Edits" bar for both fork and non-fork sessions, shown when this chat + // edited anything. The fork's identity lives in the modal (SessionDiffDrawer's + // title), not on the bar. Fork sessions still need the forking gate + a + // resolvable fork/parent pair. + const showBar = $derived( + !!committedId && + (dockItems.length > 0 || deletionOnly) && + (!isFork || (forksAllowed && !!sessionWorkspace && !!parentWorkspace && !!parentWorkspaceId)) + ) + + // Background jobs the chat started (rendered as the Jobs segment). The session + // bar shows if there are edits OR jobs; each segment hides when its side is empty. + const hasJobs = $derived((runtime?.manager.backgroundJobs.length ?? 0) > 0) + + +{#snippet dock()} + +
+ {#if dockCounts.draft > 0} + + {dockCounts.draft} draft{dockCounts.draft === 1 ? '' : 's'} + + {/if} + {#if dockCounts.deployed > 0} + + {dockCounts.deployed} deployed + + {/if} +
+{/snippet} + +{#if committedId && isUnavailable} + +
+
+ +
+ The {committedIsFork ? 'fork' : 'workspace'} has been archived or deleted + + Move this session to another workspace, or discard it. + {committedId} + +
+
+
+ onMove?.(workspaceId)} + onCreateFork={async (fork) => { + await onCreateForkAndMove?.(fork) + }} + createForkCaption="Created immediately and the session moved into it." + > + {#snippet trigger()} + + {/snippet} + + +
+
+{:else if committedId && (showBar || hasJobs)} + +
+ {#if showBar} + {#if deletionOnly && compareHref} + + + + Edits + + Review deletions + + + {:else} + + + {/if} + {:else} + +
+ {/if} + {#if hasJobs} + + + {/if} +
+{/if} + + +{#if committedId && !isUnavailable} + + void runtime?.manager.renameModifiedItem(item.draftKind, item.path, item.displayPath)} + onItemDiscarded={(item) => void runtime?.manager.removeModifiedItem(item.draftKind, item.path)} + /> +{/if} diff --git a/frontend/src/lib/components/sessions/SessionDiffDrawer.svelte b/frontend/src/lib/components/sessions/SessionDiffDrawer.svelte new file mode 100644 index 0000000000..ff50e8d438 --- /dev/null +++ b/frontend/src/lib/components/sessions/SessionDiffDrawer.svelte @@ -0,0 +1,112 @@ + + + + {#snippet titleExtra()} +
+ {#if isFork} + + + {ws?.name ?? workspaceId} + + {#if ws?.is_dev_workspace} + {devBadgeText(ws.dev_workspace_label)} + {/if} + + + {parentWs?.name ?? parentWorkspaceId} + + {:else} + + + {ws?.name ?? workspaceId} + + {/if} +
+ {/snippet} +
diff --git a/frontend/src/lib/components/sessions/SessionDraftBar.svelte b/frontend/src/lib/components/sessions/SessionDraftBar.svelte deleted file mode 100644 index dcc5640f2c..0000000000 --- a/frontend/src/lib/components/sessions/SessionDraftBar.svelte +++ /dev/null @@ -1,70 +0,0 @@ - - -{#if committedId && count > 0} -
-
- - {count} draft{count === 1 ? '' : 's'} -
-
- drawer?.open()} /> - -
-
- - -{/if} diff --git a/frontend/src/lib/components/sessions/SessionEditorTarget.svelte b/frontend/src/lib/components/sessions/SessionEditorTarget.svelte index f4c29188f9..ce1fd38221 100644 --- a/frontend/src/lib/components/sessions/SessionEditorTarget.svelte +++ b/frontend/src/lib/components/sessions/SessionEditorTarget.svelte @@ -1,5 +1,5 @@ + + + +{#if $subOpen} +
+ +
+
+ + {#if archivedCount > 0} + + {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + + {/if} +
+
+
+{/if} diff --git a/frontend/src/lib/components/sessions/SessionForkBar.svelte b/frontend/src/lib/components/sessions/SessionForkBar.svelte deleted file mode 100644 index b1e74b43eb..0000000000 --- a/frontend/src/lib/components/sessions/SessionForkBar.svelte +++ /dev/null @@ -1,201 +0,0 @@ - - -{#if committedId && isUnavailable} - -
-
- -
- The fork has been archived or deleted - - Move this session to another workspace, or discard it. - {committedId} - -
-
-
- onMove?.(workspaceId)} - onCreateFork={async (fork) => { - await onCreateForkAndMove?.(fork) - }} - createForkCaption="Created immediately and the session moved into it." - > - {#snippet trigger()} - - {/snippet} - - -
-
-{:else if forksAllowed && isFork && sessionWorkspace && parentWorkspace && parentWorkspaceId && committedId} - {@const StatusIcon = - forkStatus === 'ahead' - ? GitPullRequestArrow - : forkStatus === 'diverged' - ? GitCompareArrows - : GitFork} - {@const statusColor = - forkStatus === 'ahead' - ? 'text-blue-500' - : forkStatus === 'diverged' - ? 'text-amber-500' - : 'text-secondary'} - {@const statusTitle = - forkStatus === 'ahead' - ? 'Ahead of parent' - : forkStatus === 'diverged' - ? 'Diverged from parent' - : forkStatus === 'in_sync' - ? 'In sync with parent' - : 'Fork'} -
-
- - - - - {sessionWorkspace.name} - - - - {parentWorkspace.name} - -
-
- diffDrawer?.open()} - /> - -
-
- - -{/if} diff --git a/frontend/src/lib/components/sessions/SessionItemNotFound.svelte b/frontend/src/lib/components/sessions/SessionItemNotFound.svelte index dc1a088c53..77c5173813 100644 --- a/frontend/src/lib/components/sessions/SessionItemNotFound.svelte +++ b/frontend/src/lib/components/sessions/SessionItemNotFound.svelte @@ -3,7 +3,11 @@ import type { WorkspaceItem, WorkspaceItemKind } from '$lib/components/workspacePicker' import type { SessionTarget } from './sessionState.svelte' - const KIND_NOT_FOUND_LABEL: Record = { + // `pipeline` targets never hit this component (they aren't slot-loaded, so they + // can't 404 through SessionEditorTarget) — exclude it from the kinds here. + type NotFoundKind = Exclude + + const KIND_NOT_FOUND_LABEL: Record = { flow: 'Flow', script: 'Script', raw_app: 'Raw app' @@ -14,7 +18,7 @@ path, onNavigate }: { - kind: SessionTarget['kind'] + kind: NotFoundKind path: string onNavigate?: (item: WorkspaceItem) => void } = $props() diff --git a/frontend/src/lib/components/sessions/SessionModeSwitch.svelte b/frontend/src/lib/components/sessions/SessionModeSwitch.svelte new file mode 100644 index 0000000000..5b297b4bc8 --- /dev/null +++ b/frontend/src/lib/components/sessions/SessionModeSwitch.svelte @@ -0,0 +1,55 @@ + + + +*]:w-full' : 'w-full [&>*]:flex-1'} +> + {#snippet children({ item })} + + + {/snippet} + diff --git a/frontend/src/lib/components/sessions/SessionPicker.svelte b/frontend/src/lib/components/sessions/SessionPicker.svelte index 55683b4d96..e3d1c5d21e 100644 --- a/frontend/src/lib/components/sessions/SessionPicker.svelte +++ b/frontend/src/lib/components/sessions/SessionPicker.svelte @@ -1,13 +1,14 @@
- Run in + Acting on {#snippet trigger()} - - {#if pendingFork || (currentWs && currentWs.id !== root?.id)} - - {:else} - - {/if} - - {pendingFork?.name ?? currentWs?.name ?? effectiveId ?? 'Pick workspace'} - - {#if pendingFork} - (new) - {/if} - - + {/snippet}
diff --git a/frontend/src/lib/components/sessions/SessionWrapper.svelte b/frontend/src/lib/components/sessions/SessionWrapper.svelte index c74bf294e6..f5fa09c6d0 100644 --- a/frontend/src/lib/components/sessions/SessionWrapper.svelte +++ b/frontend/src/lib/components/sessions/SessionWrapper.svelte @@ -3,11 +3,12 @@ import { Pane, Splitpanes } from 'svelte-splitpanes' import AIChat from '$lib/components/copilot/chat/AIChat.svelte' import EditableInput from '$lib/components/common/EditableInput.svelte' - import { Button } from '$lib/components/common' + import { Button, NameIdTooltip } from '$lib/components/common' import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { AIChatManager } from '$lib/components/copilot/chat/AIChatManager.svelte' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' + import { workspaceIsFork } from '$lib/utils/workspaceHierarchy' import { WorkspaceService } from '$lib/gen' import { sendUserToast } from '$lib/toast' import Toggle from '$lib/components/Toggle.svelte' @@ -15,45 +16,46 @@ import { Archive, ArchiveRestore, + ArrowUpRight, EllipsisVertical, - PanelRightClose, - PanelRightOpen, + ExternalLink, Pencil, + Settings, Trash2 } from 'lucide-svelte' - import type { WorkspaceItem } from '$lib/components/workspacePicker' - import Popover from '$lib/components/meltComponents/Popover.svelte' - import WorkspaceItemDrillPicker from '$lib/components/WorkspaceItemDrillPicker.svelte' - import FlowEditorView from './FlowEditorView.svelte' - import ScriptEditorView from './ScriptEditorView.svelte' - import RawAppEditorView from './RawAppEditorView.svelte' + import { type Item } from '$lib/utils' + import WorkspaceScopeTrigger from '$lib/components/WorkspaceScopeTrigger.svelte' import SessionWorkspaceBar from './SessionWorkspaceBar.svelte' - import SessionForkBar from './SessionForkBar.svelte' - import SessionDraftBar from './SessionDraftBar.svelte' + import SessionChangesBar from './SessionChangesBar.svelte' import { createSession, + deleteSessionsForWorkspace, getEffectiveWorkspaceId, moveSessionToNewFork, moveSessionToWorkspace, + peekTransientDraftPrompt, + queueTransientDraftPrompt, + reconcileAfterWorkspaceChange, renameSession, selectSession, sessionState, setSessionArchived, - setSessionTarget, - syncWorkspaceTo, - type SessionTarget + syncWorkspaceTo } from './sessionState.svelte' - import { editorWarmIds, getOrCreateRuntime, removeSession } from './sessionRuntime.svelte' + import { getOrCreateRuntime, removeSession } from './sessionRuntime.svelte' import { goto } from '$lib/navigation' - import { slide } from 'svelte/transition' + import { base } from '$app/paths' + import { splitterPointerCapture } from '$lib/utils/splitterPointerCapture' - let { sessionId }: { sessionId: string } = $props() - - // LRU-warm sessions get their editor pane mounted; others render - // chat-only. Reading from the reactive Set keeps SessionWrapper in - // sync with promoteEditorWarm without an explicit prop round-trip - // through the page route. - const mountEditor = $derived(editorWarmIds.has(sessionId)) + // headerInset: extra left padding on the chat header so it clears a floating + // control (the collapsed-rail launcher) sitting at the screen's top-left. + let { + sessionId, + headerInset = false + }: { + sessionId: string + headerInset?: boolean + } = $props() // Parent keys by sessionId; this wrapper only mounts when the session exists. // Captured at script-init so we can synchronously bind context. @@ -67,9 +69,51 @@ // Reactive session reference (mutations to summary/target propagate via the $state proxy) const session = $derived(sessionState.sessions.find((s) => s.id === sessionId)) + // Seed the composer with the unsent prompt a reload preserved in the + // transient draft slot (script-init: AIChatInput reads it once at mount). + const restoredDraftPrompt = peekTransientDraftPrompt(sessionId) + + // The workspace the session acts on, shown in the header "Acting on" strip via the shared + // WorkspaceScopeTrigger chip. `targetId` is also the workspace the chip's ellipsis menu targets. + const acting = $derived.by(() => { + const wsId = session ? getEffectiveWorkspaceId(session) : undefined + if (!wsId) return undefined + const name = $userWorkspaces.find((w) => w.id === wsId)?.name ?? wsId + return { targetId: wsId, name } + }) + + // Ellipsis menu on the "Acting on" chip. Both entries are real links (so + // modifier/middle clicks open a new tab); the `workspace` query param + // points the navigation at the acting workspace — the layout applies it on + // both full loads and client-side query changes. The trailing external-link + // glyphs make the leave-the-session navigation explicit. + const actingMenu = $derived([ + { + displayName: 'Workspace settings', + icon: Settings, + href: `${base}/workspace_settings?workspace=${acting?.targetId ?? ''}`, + extra: externalLinkHint + }, + { + displayName: 'Go to this workspace', + icon: ArrowUpRight, + href: `${base}/?workspace=${acting?.targetId ?? ''}`, + extra: externalLinkHint + } + ]) + + // Load copilot config (models, providers) for the workspace the session acts + // on, not the navigation workspace — a session deliberately leaves + // $workspaceStore on the nav workspace, so keying off it would pick the model + // and provider from the wrong workspace's AI config for a fork-scoped session. + // Only the active session may write this: copilotInfo/copilotSessionModel are + // global, and warm background wrappers (each in their own workspace) would + // otherwise race to clobber the active chat's model config. $effect(() => { - if ($workspaceStore) { - loadCopilot($workspaceStore) + if (sessionState.currentSessionId !== sessionId) return + const ws = acting?.targetId ?? $workspaceStore + if (ws) { + loadCopilot(ws) } }) @@ -81,6 +125,9 @@ async function resetToNewSession() { const fresh = createSession() selectSession(fresh.id) + // The page derives the visible session from the `session_name` query, not + // currentSessionId — navigate so the URL leaves the deleted/archived session + // (else it renders the not-found state or stays on the archived one). await goto(`/sessions?session_name=${encodeURIComponent(fresh.name)}`) } @@ -89,10 +136,13 @@ // the fork lingers as an orphan whose only purpose was this session. const sessionForkId = $derived.by(() => { const wsId = session?.workspace_id - if (!wsId || !wsId.startsWith('wm-fork-')) return undefined + if (!wsId) return undefined const ws = $userWorkspaces.find((w) => w.id === wsId) - // Don't offer the option if the fork is gone or not user-accessible. - if (!ws || !ws.parent_workspace_id) return undefined + // Don't offer the option if the fork is gone/not user-accessible or isn't a fork (prefix OR + // parent, so an orphaned wm-fork- fork still qualifies). + if (!ws || !workspaceIsFork(wsId, $userWorkspaces)) return undefined + // A persistent dev workspace is not an ephemeral session fork — never offer to delete it. + if (ws.is_dev_workspace) return undefined return wsId }) @@ -101,16 +151,6 @@ let archiveConfirmOpen = $state(false) let archiveAlsoFork = $state(false) - async function refreshWorkspaceList() { - // Match the SidebarContent.deleteFork pattern: replace the in-memory - // list rather than nulling it. See B1 fix. - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces', e) - } - } - async function handleConfirmedDelete() { deleteConfirmOpen = false if (!session) return @@ -125,8 +165,9 @@ if (forkToDelete) { try { await WorkspaceService.deleteWorkspace({ workspace: forkToDelete }) + await deleteSessionsForWorkspace(forkToDelete) sendUserToast(`Deleted forked workspace ${forkToDelete}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to delete fork ${forkToDelete}: ${e?.body ?? e}`, true) } @@ -150,7 +191,7 @@ try { await WorkspaceService.archiveWorkspace({ workspace: forkToArchive }) sendUserToast(`Archived forked workspace ${forkToArchive}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to archive fork ${forkToArchive}: ${e?.body ?? e}`, true) } @@ -184,43 +225,6 @@ runtime?.manager.displayMessages.some((m) => m.role === 'user') ?? false ) - // Effective workspace for routing editor views — committed if set, - // otherwise the pending pick, otherwise the current active workspace. - const effectiveWorkspaceId = $derived( - session ? (getEffectiveWorkspaceId(session) ?? $workspaceStore ?? '') : '' - ) - - // Core mutation: assign a target via the canonical setter, then re-open - // the editor pane. Shared by every code path that swaps the session's - // editor target (drill picker, fork-bar dropdown, …). - function applyEditorTarget(target: SessionTarget, summary?: string) { - if (!session) return - setSessionTarget(session.id, target, summary) - // Picking a target also re-opens the editor pane (the user just chose - // what to view). - editorVisible = true - } - - function pickEditorTarget(item: WorkspaceItem) { - // Legacy drag-and-drop apps aren't hosted in the session preview pane — - // open them in the standalone app editor instead. Only code-based raw - // apps (item.raw_app) are previewable here. - if (item.kind === 'app' && !item.raw_app) { - goto(`/apps/edit/${item.path}?workspace=${effectiveWorkspaceId}`) - return - } - // WorkspaceItem.kind is 'flow'|'script'|'app'; any 'app' reaching here is - // a raw app. The diff-API uses 'raw_app' as its kind so we align - // SessionTarget on the same canonical string. - const kind: SessionTarget['kind'] = item.kind === 'app' ? 'raw_app' : item.kind - applyEditorTarget({ kind, path: item.path }, item.summary) - } - - // Editor pane visibility. Toggling this just hides/shows the pane via CSS - // — the editor stays mounted, so re-opening doesn't pay a remount cost - // and xy-flow / Monaco keep their viewport state. - let editorVisible = $state(true) - // Focus the chat input whenever this session is the active one. // The textarea is disabled until copilotInfo loads (otherwise focus is // a silent no-op), so we wait for that too. Triggers on initial mount, @@ -237,7 +241,7 @@ // True when the session committed to a workspace that's no longer in // the user's list (deleted / archived / access revoked). The chat is - // disabled and SessionForkBar shows a move/discard banner. + // disabled and SessionChangesBar shows a move/discard banner. const isUnavailable = $derived( !!session?.workspace_id && !$userWorkspaces.find((w) => w.id === session!.workspace_id) ) @@ -262,32 +266,52 @@ } +{#snippet externalLinkHint()} + +{/snippet} + {#if !session || !runtime}
Session not found
{:else} - {@const hasTarget = - session.target?.kind === 'flow' || - session.target?.kind === 'script' || - session.target?.kind === 'raw_app'} - {@const hasEditor = mountEditor && hasTarget && editorVisible} - {#snippet inputPreface()} {#if !hasFirstUserMessage} {/if} - +
- +
+
+ + This session is archived +
+ +
+ {/if} + moveAndActivate(workspaceId)} onCreateForkAndMove={(fork) => createForkAndMove(fork)} onArchive={() => archiveAndReset()} onDelete={() => (deleteConfirmOpen = true)} /> -
{/snippet} @@ -295,159 +319,116 @@ sessions have their own empty-state affordances above. --> {#snippet sessionEmptyHint()}{/snippet} - - - -
- renameSession(session.id, v)} - class="text-sm font-semibold" - inputClass="!text-sm !font-semibold" - /> - summaryInput?.edit() - }, - session.archived - ? { - displayName: 'Unarchive', - icon: ArchiveRestore, - action: () => setSessionArchived(session.id, false) - } - : { - displayName: 'Archive', - icon: Archive, - action: () => archiveAndReset() - }, - { - displayName: 'Delete', - icon: Trash2, - type: 'delete', - action: () => (deleteConfirmOpen = true) - } - ]} + +
+ + +
- {#snippet buttonReplacement()} - - - - {/snippet} - - {#if !session.target && hasFirstUserMessage} - -
- - {#snippet trigger()} - - {/snippet} - {#snippet content()} - pickEditorTarget(item)} + renameSession(session.id, v)} + class="text-sm font-semibold" + inputClass="!text-sm !font-semibold" + /> + summaryInput?.edit() + }, + ...(session.archived + ? // No Unarchive when the workspace is gone — it can't persist + // (putSession guard) and reconcile would re-archive it. + isUnavailable + ? [] + : [ + { + displayName: 'Unarchive', + icon: ArchiveRestore, + action: () => setSessionArchived(session.id, false) + } + ] + : [ + { + displayName: 'Archive', + icon: Archive, + action: () => archiveAndReset() + } + ]), + { + displayName: 'Delete', + icon: Trash2, + type: 'delete', + action: () => (deleteConfirmOpen = true) + } + ]} + > + {#snippet buttonReplacement()} + + + + {/snippet} + + {#if acting && hasFirstUserMessage} + +
+ Acting on + + + - {/snippet} - -
- {:else if hasTarget && mountEditor && !editorVisible} -
- -
- {:else if hasEditor} -
- -
- {/if} -
-
- -
-
- {#if hasEditor && session.target} - -
- {#if session.target.kind === 'flow'} - - {:else if session.target.kind === 'script'} - - {:else if session.target.kind === 'raw_app'} - + +
{/if} +
+
+ queueTransientDraftPrompt(sessionId, text)} + forceDisabled={isUnavailable || !!session.archived} + forceDisabledMessage={isUnavailable + ? 'This session is linked to a workspace that no longer exists. Move it or discard it from the banner above to keep working.' + : session.archived + ? 'This session is archived. Unarchive it from the banner above to keep working.' + : ''} + emptyHint={sessionEmptyHint} + {inputPreface} + />
- {/if} -
+ + - :global(.splitter-hidden .splitpanes__splitter) { + /* Invisible-but-draggable splitter: a real (layout-occupying) gutter, wide + enough to grab. No overlap tricks — the zone can't cover the left pane's + scrollbar or the right pane's edge. */ + :global(.splitpanes--vertical.splitter-hidden) > :global(.splitpanes__splitter) { background-color: transparent !important; border: none !important; opacity: 0 !important; + width: 10px !important; } diff --git a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte index 472ea1515b..360c08a9f3 100644 --- a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte +++ b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte @@ -1,301 +1,384 @@ - - searchableText(d)} + items={displayEntries} + bind:filteredItems={searchedEntries} + f={(e: DisplayEntry) => searchableText(e)} /> -{#snippet renderTreeNode(node: TreeNode, depth: number)} + +{#snippet rowBadge(item: DeployItem)} + {#if badgeOf(item) === 'draft'} + {#if model.staleOf(item.key)} + + + {#snippet trigger()} + + {/snippet} + {#snippet content()} +
+ Started from an older deployed version. A newer version was deployed after this draft + began. Review the latest deploy before deploying. +
+ {/snippet} +
+ {/if} + + {item.draftOnly ? 'Draft only' : 'Draft'} + + {:else} + + + + + + + {/if} +{/snippet} + + +{#snippet deployFailed(item: DeployItem)} + {@const s = model.statusOf(item.key)} + {#if s?.status === 'failed'} + + Failed + + + {/if} +{/snippet} + +{#snippet renderTreeNode(node: TreeNode, depth: number)} {#if node.type === 'folder'} {@const isUserScope = node.isScope && node.name.startsWith('u/')} - {@const fkey = folderKey(node)} + {@const fkey = node.key} {@const open = isFolderOpen(fkey)} {@const isHl = fkey === highlightedKey}
(folderOpen[fkey] = (e.currentTarget as HTMLDetailsElement).open)} + ontoggle={(e) => { + // Record real user toggles only; skip the echo fired when the `open` + // attribute is driven by state (search force-open, expandApp). + const domOpen = (e.currentTarget as HTMLDetailsElement).open + if (domOpen !== isFolderOpen(fkey)) folderOpen[fkey] = domOpen + }} class="select-none" > - setHoverHighlight(fkey)} - class="flex items-center gap-1.5 px-3 py-1 cursor-pointer text-xs font-medium font-mono text-emphasis hover:bg-surface-hover list-none [&::-webkit-details-marker]:hidden tree-summary {isHl - ? 'bg-surface-hover' - : ''}" - style="padding-left: {depth * 12 + 8}px" - > - - - {#if isUserScope} - - {:else} - - {/if} - {node.name} - -
+ {#if node.app} + {@const appItem = segmentItems.find((it) => it.key === node.app?.summaryKey)} + + setHoverHighlight(fkey)} + onclick={(e) => { + e.preventDefault() + if (appItem) revealDiff(appItem, fkey) + }} + title={node.fullPath} + class="flex items-center gap-2 pl-3 pr-1 py-2 rounded-md cursor-pointer hover:bg-surface-hover list-none [&::-webkit-details-marker]:hidden tree-summary {isHl + ? 'bg-surface-hover' + : ''}" + style="padding-left: {depth * 12 + 8}px" + > + + + {node.app.summary ?? node.name} + + {#if appItem} + {@render rowBadge(staged[appItem.key] ?? appItem)} + {/if} + + + + {:else} + setHoverHighlight(fkey)} + class="flex items-center gap-2 pl-3 pr-1 py-2 rounded-md cursor-pointer text-xs font-normal font-mono text-secondary hover:bg-surface-hover list-none [&::-webkit-details-marker]:hidden tree-summary {isHl + ? 'bg-surface-hover' + : ''}" + style="padding-left: {depth * 12 + 8}px" + > + {#if isUserScope} + + {:else} + + {/if} + {node.name} + + + + + + {/if} +
+ {#each node.children as child} {@render renderTreeNode(child, depth + 1)} {/each}
{:else} - {@const status = node.diff.status} - {@const key = itemKey(node.diff)} - { - highlightedKey = key - scrollToDiff(node.diff) - }} - onmouseenter={() => setHoverHighlight(key)} - > - {#snippet extras()} - - {/snippet} - + {@const d = node.data} + {@const key = node.key} + +
+ {#if isSynthetic(d)} + void revealSynthetic(d, key)} + onmouseenter={() => setHoverHighlight(key)} + > + {#snippet extras()} + + {/snippet} + + {:else} + revealDiff(d, key)} + onmouseenter={() => setHoverHighlight(key)} + > + {#snippet extras()} + {@render rowBadge(staged[d.key] ?? d)} + {#if d.deployKind === 'raw_app'} + + { + e.stopPropagation() + expandApp(d) + }} + > + {#if loadedDiffs[d.key]?.state === 'loading'} + + {:else} + + {/if} + + {:else} + + {/if} + {/snippet} + + {/if} +
{/if} {/snippet} - + +{#snippet diffBlock(item: DeployItem)} + {@const loaded = loadedDiffs[item.key]} + {#if !mountedRows[item.key]} +
+ + Diff loads on scroll… +
+ {:else if !loaded || loaded.state === 'loading'} +
+ + Loading diff… +
+ {:else if loaded.state === 'error'} +
{loaded.error}
+ {:else if item.deployKind === 'raw_app'} +
+ {#each rawAppItems(item, loaded) as sub (displayKey(sub))} + +
+
+ {sub.path} +
+ {#if sub.kind === 'raw_app_file'} + + {:else} + {@const runnable = sub as RawAppRunnableItem} + + {/if} +
+ {/each} +
+ {:else} + + {/if} +{/snippet} + + {#snippet actions()} - + + { + // Ignore the programmatic flip to 'inline' when `narrow` forces it — + // only a real click (when not narrow) should change the preference, + // so widening again restores the user's choice. + if (!narrow) diffStyle = v + }} + noWFull + > {#snippet children({ item })} @@ -503,147 +911,220 @@ /> {/snippet} - {/snippet} -
- {#if diffs.length > 0} - - {/if} -
-
- {#if loading && diffs.length === 0} -
- - Loading comparison... +
+
+ {#if model.items.length > 0} + + {/if} +
+
+ {#if model.loading && model.items.length === 0} +
+ + Loading changes... +
+ {:else if model.error} +
{model.error}
+ {:else if model.items.length === 0} +
No changes.
+ {:else if orderedItems.length === 0} +
No files match.
+ {:else} +
+ {#each orderedItems as d (d.key)} + + {@const view = staged[d.key] ?? d} + {@const action = actionFor(view)} + {@const editUrl = editUrlFor?.(d)} + {@const status = model.statusOf(d.key)} +
- - -
- {#if editUrl} - + + +
+ {#if editUrl} + + {d.displayPath} + + {:else} +
+ {d.displayPath} +
+ {/if} +
+
+ {@render rowBadge(view)} + {#if status?.status === 'failed'} + {@render deployFailed(d)} + {:else if action.op !== 'none'} + {#if action.secondary?.length} + + {/if} +
+ + {#if staged[d.key]} + +
+ +
+ {/if} +
+ {/if} +
+
+
+ + {#if view.done} +
- {dpath} - + Deployed — no pending changes. +
{:else} -
- {dpath} +
+ {@render diffBlock(view)}
{/if}
-
- {#if d.ahead && d.ahead > 0} - {d.ahead} ahead - {/if} - {#if d.behind && d.behind > 0} - {d.behind} behind - {/if} - - - {status} - -
- -
- {#if !loaded || loaded.state === 'loading'} -
- - Loading diff… -
- {:else if loaded.state === 'error'} -
{loaded.error}
- {:else if loaded.state === 'ready'} - - {/if}
- - {/each} -
- {/if} -
+ {/each} +
+ + + {/if} +
+
+
+ + {#if model.items.length > 0 && compareSessionHref} + + {/if} + +
diff --git a/frontend/src/lib/components/sidebar/UserMenu.svelte b/frontend/src/lib/components/sidebar/UserMenu.svelte index b6e6d6a9cc..1ec649cf12 100644 --- a/frontend/src/lib/components/sidebar/UserMenu.svelte +++ b/frontend/src/lib/components/sidebar/UserMenu.svelte @@ -42,6 +42,7 @@ label={`User (${$userStore?.username ?? $userStore?.email})`} {isCollapsed} {lightMode} + showChevron {trigger} /> {/snippet} @@ -52,12 +53,22 @@ {$userStore?.email}

- {#if $userStore?.is_admin} + {#if $userStore?.non_member} + Superadmin, not a member of this workspace + {:else if $userStore?.is_admin} Admin of this workspace {:else if $userStore?.operator} Operator in this workspace {/if} + {#if $userStore?.non_member} + + You are not a member, but as a superadmin you can access this workspace. You act here + under the username + {$userStore?.username} + with admin permissions. + + {/if}
diff --git a/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte b/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte index a834d00317..e84f62daef 100644 --- a/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte +++ b/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte @@ -9,13 +9,17 @@ workspaceUsageStore, workspaceColor, clearWorkspaceFromStorage, - globalForkModal + globalForkModal, + type UserWorkspace } from '$lib/stores' - import { isRuleActive } from '$lib/workspaceProtectionRules.svelte' - import { Building, Plus, Settings, GitFork } from 'lucide-svelte' + import { Building, Check, ChevronDown, ChevronRight, Plus, Settings } from 'lucide-svelte' + import { forkAccentStyle } from '$lib/utils/forkColor' + import { SvelteSet } from 'svelte/reactivity' + import { Badge, CopyButton, NameIdTooltip } from '$lib/components/common' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' import { Menu, MenuItem } from '$lib/components/meltComponents' import WorkspaceIcon from '$lib/components/workspace/WorkspaceIcon.svelte' + import { fixupUrlAfterWorkspaceSwitch } from './workspaceSwitchUrl' import { goto } from '$lib/navigation' import { base } from '$lib/base' import { page } from '$app/state' @@ -27,7 +31,10 @@ import { twMerge } from 'tailwind-merge' import type { MenubarBuilders } from '@melt-ui/svelte' import { buildWorkspaceHierarchy } from '$lib/utils/workspaceHierarchy' + import { canCreateFork } from '$lib/utils/editInFork' import { getContrastTextColor } from '$lib/utils' + import { workspaceRootId } from '$lib/components/sessions/sessionScope.svelte' + import { devBadgeText } from '$lib/utils/devWorkspaceLabel' interface Props { isCollapsed?: boolean @@ -36,13 +43,6 @@ strictWorkspaceSelect?: boolean } - function removePrefix(str: string, prefix: string): string { - if (str.startsWith(prefix)) { - return str.substring(prefix.length) - } - return str - } - let { isCollapsed = false, createMenu, strictWorkspaceSelect = false }: Props = $props() async function toggleSwitchWorkspace(id: string) { @@ -50,38 +50,24 @@ return } workspaceAIClients.init(id) - const editPages = [ - '/scripts/edit/', - '/flows/edit/', - '/apps/edit/', - '/scripts/get/', - '/flows/get/', - '/apps/get/' - ] - const isOnEditPage = editPages.some((editPage) => page.route.id?.includes(editPage) ?? false) - // An AI session is scoped to its (forked) workspace, so it makes no sense - // to keep showing it after the user switches workspace — go home instead. - const isOnSessionPage = page.route.id?.includes('/sessions') ?? false - switchWorkspace(id) - if (isOnEditPage || isOnSessionPage) { - await goto('/') - } else if (page.url.searchParams.get('workspace')) { - page.url.searchParams.set('workspace', id) - } + // The sessions page needs no navigation here: the item's link navigation + // (workspaceHref) keeps the route, and the page's family reconcile swaps + // out a chat that doesn't belong to the new workspace's family. + await fixupUrlAfterWorkspaceSwitch(id) } - // An AI session is scoped to its (forked) workspace, so switching workspace - // should leave for home (the link's navigation wins over onClick's - // preventDefault; onClick still performs the switch). Pure logic + - // new-tab/workspace-param handling lives in workspaceMenuHref (unit-tested). + // Href for the item's navigation (including modifier/middle clicks opening a + // new tab): same page, `workspace` param swapped to the clicked id, the open + // session kept only within its family. Pure logic lives in workspaceMenuHref + // (unit-tested). function workspaceHref(id: string): string { + const all = $userWorkspaces ?? [] return workspaceMenuHref({ - routeId: page.route.id, - base, pathname: page.url.pathname, searchParams: page.url.searchParams, - id + id, + sameFamily: workspaceRootId(id, all) === workspaceRootId($workspaceStore ?? undefined, all) }) } @@ -98,121 +84,284 @@ toggleSwitchWorkspace(workspace.id) } - function getForkedWorkspace(workspaceId: string) { - if (!$userWorkspaces) return undefined - return $userWorkspaces.find((w) => w.id === workspaceId && w.parent_workspace_id != null) - } - - function getParentWorkspace(parentId: string) { - if (!$userWorkspaces) return undefined - return $userWorkspaces.find((w) => w.id === parentId) - } - - // Group workspaces into parent-child hierarchy using Svelte 5 derived and the new utility - const groupedWorkspaces = $derived.by(() => { - if (!$userWorkspaces) return [] - return buildWorkspaceHierarchy($userWorkspaces) + // Family-first picker: list the workspace families (roots) with their forks + // collapsed behind a per-family chevron, so direct fork navigation stays + // available without flattening fork-heavy instances into a long menu. The + // scope header's WorkspaceFamilyPicker remains the primary fork surface. + // + // strictWorkspaceSelect is used on standalone pages (e.g. svix webhook + // creation) that render this menu with no scope header, so there forks must + // stay directly selectable — list the full hierarchy unconditionally. + const hierarchy = $derived($userWorkspaces ? buildWorkspaceHierarchy($userWorkspaces) : []) + const expandedFamilies = new SvelteSet() + // Root ids with at least one fork — only they get the expand chevron. + // hierarchy is a DFS (parent before child), so a depth>0 row belongs to the + // last depth-0 row seen. + const familiesWithForks = $derived.by(() => { + const withForks = new Set() + let rootId: string | undefined + for (const h of hierarchy) { + if (h.depth === 0) rootId = h.workspace.id + else if (rootId) withForks.add(rootId) + } + return withForks + }) + // Gate for the "Workspace fork" entry pinned below the list (the global fork + // modal carries its own base-workspace picker). Hidden on non-premium cloud, + // in the admins workspace, or when forking is disabled. + const canForkHere = $derived( + (!isCloudHosted() || $isPremiumStore) && + $workspaceStore !== 'admins' && + canCreateFork($userStore) + ) + const familyWorkspaces = $derived.by(() => { + if (strictWorkspaceSelect) return hierarchy + let rootId: string | undefined + return hierarchy.filter((h) => { + if (h.depth === 0) { + rootId = h.workspace.id + return true + } + return !!rootId && expandedFamilies.has(rootId) + }) }) + let menuOpen = $state(false) + + // ArrowRight/ArrowLeft expand/collapse the keyboard-highlighted family (melt + // stamps data-highlighted on the item; the row wrapper carries the workspace + // id). Capture phase, so melt's menubar left/right (adjacent-menu switching) + // doesn't fire when the keypress means expansion here. + function onExpandKeydown(e: KeyboardEvent) { + if (!menuOpen || strictWorkspaceSelect) return + if (e.key !== 'ArrowRight' && e.key !== 'ArrowLeft') return + // Scoped inside a row wrapper: the menubar trigger also carries + // data-highlighted while its menu is open. + const id = document + .querySelector('[data-workspace-id] [data-highlighted]') + ?.closest('[data-workspace-id]') + ?.getAttribute('data-workspace-id') + if (!id || !familiesWithForks.has(id)) return + if (e.key === 'ArrowRight' && !expandedFamilies.has(id)) expandedFamilies.add(id) + else if (e.key === 'ArrowLeft' && expandedFamilies.has(id)) expandedFamilies.delete(id) + else return + e.preventDefault() + e.stopPropagation() + } + + function findRoot(id: string | undefined): UserWorkspace | undefined { + if (!id || !$userWorkspaces) return undefined + let current = $userWorkspaces.find((w) => w.id === id) + while (current?.parent_workspace_id) { + const parent = $userWorkspaces.find((w) => w.id === current!.parent_workspace_id) + if (!parent) break + current = parent + } + return current + } + + // The active workspace's family root — shown in the trigger so a forked + // active workspace still surfaces its family name here (the fork itself is + // shown in the breadcrumb). + const currentFamily = $derived(findRoot($workspaceStore ?? undefined)) + + // The active workspace itself (fork included) — names the settings entry. + const activeWorkspace = $derived($userWorkspaces?.find((w) => w.id === $workspaceStore)) + const canManageWorkspace = $derived($userStore?.is_admin || $superadmin) + + // font-normal is explicit: href-less MenuItems render as + {:else} + + + {/if} +
{/each} - {#if (isCloudHosted() || $superadmin) && !strictWorkspaceSelect} + {#if (isCloudHosted() || $superadmin || canForkHere) && !strictWorkspaceSelect}
- - - Workspace - + {#if isCloudHosted() || $superadmin} + + + Workspace + + {/if} + {#if canForkHere} + (globalForkModal.val = { opened: true })} + {item} + > + + Workspace fork + + {/if}
{/if} - {#if !strictWorkspaceSelect && !isCloudHosted() && !isRuleActive('DisableWorkspaceForking') && $workspaceStore !== 'admins'} + {#if canManageWorkspace && !strictWorkspaceSelect}
(globalForkModal.val = { opened: true })} > - - Fork current workspace + + {activeWorkspace?.name ?? $workspaceStore} settings
{/if} @@ -228,14 +377,6 @@ {/if} - {#if ($userStore?.is_admin || $superadmin) && !strictWorkspaceSelect} -
- - - Workspace settings - -
- {/if} {#if isCloudHosted() && !$isPremiumStore && !strictWorkspaceSelect}
diff --git a/frontend/src/lib/components/sidebar/WorkspaceScopeHeader.svelte b/frontend/src/lib/components/sidebar/WorkspaceScopeHeader.svelte new file mode 100644 index 0000000000..ba6ae625d9 --- /dev/null +++ b/frontend/src/lib/components/sidebar/WorkspaceScopeHeader.svelte @@ -0,0 +1,69 @@ + + +
+ + {#snippet trigger()} + + {/snippet} + +
diff --git a/frontend/src/lib/components/sidebar/changelogs.ts b/frontend/src/lib/components/sidebar/changelogs.ts index 80851e64ec..e57eb4bf5b 100644 --- a/frontend/src/lib/components/sidebar/changelogs.ts +++ b/frontend/src/lib/components/sidebar/changelogs.ts @@ -384,4 +384,22 @@ const changelogs: Changelog[] = [ } ] -export { changelogs } \ No newline at end of file +export { changelogs } +// Single owner of the "which changelogs are new" localStorage key — every +// menu surfacing changelogs must read/stamp through these, or two surfaces +// with independent state would fight over the same key. +const LAST_OPENED_KEY = 'changelogsLastOpened' + +export function readRecentChangelogs(): { recent: Changelog[]; hasNew: boolean } { + const lastOpened = localStorage.getItem(LAST_OPENED_KEY) + const recent = lastOpened + ? changelogs.filter((changelog) => changelog.date > lastOpened) + : changelogs.slice(0, 3) + const hasNew = + lastOpened != null && recent.length > 0 && lastOpened !== new Date().toISOString().split('T')[0] + return { recent, hasNew } +} + +export function markChangelogsOpened(): void { + localStorage.setItem(LAST_OPENED_KEY, new Date().toISOString().split('T')[0]) +} diff --git a/frontend/src/lib/components/sidebar/leaveWorkspace.ts b/frontend/src/lib/components/sidebar/leaveWorkspace.ts new file mode 100644 index 0000000000..043748b0c4 --- /dev/null +++ b/frontend/src/lib/components/sidebar/leaveWorkspace.ts @@ -0,0 +1,15 @@ +import { get } from 'svelte/store' +import { goto } from '$lib/navigation' +import { WorkspaceService } from '$lib/gen' +import { sendUserToast } from '$lib/toast' +import { clearStores } from '$lib/storeUtils' +import { workspaceStore } from '$lib/stores' + +// Leave the active workspace and land on the workspace list. Shared by the +// settings dropdown and the sidebar menu. +export async function leaveCurrentWorkspace(): Promise { + await WorkspaceService.leaveWorkspace({ workspace: get(workspaceStore) ?? '' }) + sendUserToast('You left the workspace') + clearStores() + await goto('/user/workspaces') +} diff --git a/frontend/src/lib/components/sidebar/sidebarChrome.ts b/frontend/src/lib/components/sidebar/sidebarChrome.ts new file mode 100644 index 0000000000..110435c058 --- /dev/null +++ b/frontend/src/lib/components/sidebar/sidebarChrome.ts @@ -0,0 +1,5 @@ +// Sidebar rail background. Deliberately raw hex — no theme token exists for +// the rail chrome — kept here as the single source for every host that renders +// or mimics the sidebar (app layout, kitchen-sink harness). +export const SIDEBAR_BG = '#F3F3F7' +export const SIDEBAR_BG_DARK = '#1e232e' diff --git a/frontend/src/lib/components/sidebar/workspaceMenuHref.test.ts b/frontend/src/lib/components/sidebar/workspaceMenuHref.test.ts index 2291d2d125..7c165b6e80 100644 --- a/frontend/src/lib/components/sidebar/workspaceMenuHref.test.ts +++ b/frontend/src/lib/components/sidebar/workspaceMenuHref.test.ts @@ -2,38 +2,35 @@ import { describe, it, expect } from 'vitest' import { workspaceMenuHref } from './workspaceMenuHref' describe('workspaceMenuHref', () => { - it('on a session route, keeps the workspace id (so new-tab lands in the right workspace)', () => { + it('keeps the open session for a same-family target (new-tab stays on the chat)', () => { expect( workspaceMenuHref({ - routeId: '/(root)/(logged)/sessions', - base: '', pathname: '/sessions', searchParams: new URLSearchParams('session_name=foo'), - id: 'wm-fork-bar' + id: 'wm-fork-bar', + sameFamily: true }) - ).toBe('/?workspace=wm-fork-bar') + ).toBe('/sessions?session_name=foo&workspace=wm-fork-bar') }) - it('respects the base prefix on a session route', () => { + it('drops the open session for a cross-family target', () => { expect( workspaceMenuHref({ - routeId: '/(root)/(logged)/sessions', - base: '/wm', - pathname: '/wm/sessions', - searchParams: new URLSearchParams(), - id: 'ws2' + pathname: '/sessions', + searchParams: new URLSearchParams('session_name=foo'), + id: 'other-root', + sameFamily: false }) - ).toBe('/wm/?workspace=ws2') + ).toBe('/sessions?workspace=other-root') }) - it('off a session route, swaps the workspace param on the current path', () => { + it('swaps the workspace param on the current path', () => { expect( workspaceMenuHref({ - routeId: '/(root)/(logged)/scripts/edit/[...path]', - base: '', pathname: '/scripts/edit/u/me/x', searchParams: new URLSearchParams('workspace=old&foo=1'), - id: 'new_ws' + id: 'new_ws', + sameFamily: false }) ).toBe('/scripts/edit/u/me/x?workspace=new_ws&foo=1') }) @@ -41,11 +38,10 @@ describe('workspaceMenuHref', () => { it('adds the workspace param when none was present', () => { expect( workspaceMenuHref({ - routeId: '/(root)/(logged)/runs', - base: '', pathname: '/runs', searchParams: new URLSearchParams(), - id: 'w' + id: 'w', + sameFamily: true }) ).toBe('/runs?workspace=w') }) diff --git a/frontend/src/lib/components/sidebar/workspaceMenuHref.ts b/frontend/src/lib/components/sidebar/workspaceMenuHref.ts index 5e5ec11325..08f93d6384 100644 --- a/frontend/src/lib/components/sidebar/workspaceMenuHref.ts +++ b/frontend/src/lib/components/sidebar/workspaceMenuHref.ts @@ -1,21 +1,21 @@ -// Href for a workspace-switch link in the sidebar WorkspaceMenu. -// -// On an AI-session route, switching workspace leaves for home — but we keep the -// `?workspace=` param so a modifier/middle click (open in new tab, which -// bypasses the onClick fast-path) still lands in the *clicked* workspace's home -// rather than the default one. Everywhere else, stay on the current path and -// just swap the `workspace` query param. +// Href for a workspace-switch link in the sidebar WorkspaceMenu: stay on the +// current path and just swap the `workspace` query param, so a modifier/middle +// click (open in new tab, which bypasses the onClick fast-path) lands on the +// same page — session mode included — in the *clicked* workspace. A session +// named in the URL is kept only for same-family targets: a cross-family tab +// must not open a foreign family's chat, so it lands on the sessions page with +// nothing selected instead. export function workspaceMenuHref(args: { - routeId: string | null | undefined - base: string pathname: string searchParams: URLSearchParams id: string + // Whether `id` belongs to the same workspace family as the active workspace. + sameFamily?: boolean }): string { - if (args.routeId?.includes('/sessions')) { - return `${args.base}/?workspace=${args.id}` - } const params = new URLSearchParams(args.searchParams) params.set('workspace', args.id) + if (!args.sameFamily) { + params.delete('session_name') + } return `${args.pathname}?${params.toString()}` } diff --git a/frontend/src/lib/components/sidebar/workspaceSwitchUrl.ts b/frontend/src/lib/components/sidebar/workspaceSwitchUrl.ts new file mode 100644 index 0000000000..08bd5cb5d5 --- /dev/null +++ b/frontend/src/lib/components/sidebar/workspaceSwitchUrl.ts @@ -0,0 +1,30 @@ +import { page } from '$app/state' +import { replaceState } from '$app/navigation' +import { goto } from '$lib/navigation' + +// Post-workspace-switch URL fixup shared by the sidebar workspace pickers. +// Item-scoped pages would show a wrong-workspace (or missing) item after a +// switch — go home instead. Otherwise, really rewrite a ?workspace= param in +// the address bar (mutating page.url is a no-op there): left stale, it gets +// re-applied on reload or when exiting session mode restores the route, +// silently switching the workspace back. +const EDIT_PAGES = [ + '/scripts/edit/', + '/flows/edit/', + '/apps/edit/', + '/apps_raw/edit/', + '/scripts/get/', + '/flows/get/', + '/apps/get/', + '/apps_raw/get/' +] + +export async function fixupUrlAfterWorkspaceSwitch(id: string): Promise { + if (EDIT_PAGES.some((p) => page.route.id?.includes(p) ?? false)) { + await goto('/') + } else if (page.url.searchParams.get('workspace')) { + const url = new URL(window.location.href) + url.searchParams.set('workspace', id) + replaceState(url, page.state) + } +} diff --git a/frontend/src/lib/components/sqlDdl.ts b/frontend/src/lib/components/sqlDdl.ts new file mode 100644 index 0000000000..5897c08859 --- /dev/null +++ b/frontend/src/lib/components/sqlDdl.ts @@ -0,0 +1,142 @@ +// Heuristics for splitting a SQL script into individual statements and +// detecting Data Definition Language (DDL) statements. Shared by the datatable +// SQL REPL and the postgres editor to steer schema changes into migrations. + +// A dollar-quote tag: `$$` or `$name$` (the optional tag follows unquoted +// identifier rules, so it starts with a letter/underscore — `$1` is a +// parameter placeholder, not a dollar quote). +const DOLLAR_QUOTE_START = /^\$([A-Za-z_][A-Za-z0-9_]*)?\$/ + +// code may be composed of many sql statements separated by ';' +// this splits them while taking into account that ';' may appear inside a +// string, quoted identifier, dollar-quoted body ($$ ... $$), or a comment +// (-- ... or /* ... */) and is then not the end of a statement. +export function splitSqlStatements(code: string): string[] { + const statements: string[] = [] + let currentStatement = '' + let inSingleQuote = false + let inDoubleQuote = false + let inBacktick = false + let inLineComment = false + let inBlockComment = false + let dollarTag: string | null = null + + for (let i = 0; i < code.length; i++) { + const char = code[i] + const prevChar = i > 0 ? code[i - 1] : null + const nextChar = i + 1 < code.length ? code[i + 1] : null + + // Inside a protected region: append verbatim (';' is not a separator) + // and only watch for the region's end. + if (inLineComment) { + currentStatement += char + if (char === '\n') inLineComment = false + continue + } + if (inBlockComment) { + // Look ahead for the close so the '*' of the opening '/*' can't be + // reused (e.g. `/*/` stays open). + if (char === '*' && nextChar === '/') { + currentStatement += '*/' + i += 1 + inBlockComment = false + } else { + currentStatement += char + } + continue + } + if (dollarTag !== null) { + if (char === '$' && code.startsWith(dollarTag, i)) { + currentStatement += dollarTag + i += dollarTag.length - 1 + dollarTag = null + } else { + currentStatement += char + } + continue + } + if (inSingleQuote) { + currentStatement += char + if (char === "'" && prevChar !== '\\') inSingleQuote = false + continue + } + if (inDoubleQuote) { + currentStatement += char + if (char === '"' && prevChar !== '\\') inDoubleQuote = false + continue + } + if (inBacktick) { + currentStatement += char + if (char === '`' && prevChar !== '\\') inBacktick = false + continue + } + + // Not in any protected region: detect the start of one. + if (char === '-' && nextChar === '-') { + inLineComment = true + currentStatement += char + continue + } + if (char === '/' && nextChar === '*') { + // Consume both chars so the '*' of '/*' can't double as a '*/' close. + inBlockComment = true + currentStatement += '/*' + i += 1 + continue + } + if (char === '$') { + const m = DOLLAR_QUOTE_START.exec(code.slice(i)) + if (m) { + dollarTag = m[0] + currentStatement += dollarTag + i += dollarTag.length - 1 + continue + } + } + if (char === "'") { + inSingleQuote = true + currentStatement += char + continue + } + if (char === '"') { + inDoubleQuote = true + currentStatement += char + continue + } + if (char === '`') { + inBacktick = true + currentStatement += char + continue + } + + if (char === ';') { + statements.push(currentStatement.trim()) + currentStatement = '' + } else { + currentStatement += char + } + } + + if (currentStatement.trim()) { + statements.push(currentStatement.trim()) + } + + return statements.filter((s) => s.length > 0) +} + +export function pruneComments(code: string): string { + return code + .replace(/--.*?(\r?\n|$)/g, '') + .replace(/\/\*[\s\S]*?\*\//g, '') + .trim() +} + +// Schema-changing keywords. GRANT/REVOKE are included since permission changes +// also belong in migrations rather than ad-hoc execution. +const DDL_KEYWORDS = ['CREATE', 'ALTER', 'DROP', 'TRUNCATE', 'RENAME', 'COMMENT', 'GRANT', 'REVOKE'] + +/** Heuristic: a statement is DDL if its first keyword is schema-changing. */ +export function isDdlStatement(statement: string): boolean { + const firstWord = pruneComments(statement).trim().split(/\s+/)[0]?.toUpperCase() + return !!firstWord && DDL_KEYWORDS.includes(firstWord) +} diff --git a/frontend/src/lib/components/triggers/CaptureTable.svelte b/frontend/src/lib/components/triggers/CaptureTable.svelte index f6bc1d209f..18ab7d7e91 100644 --- a/frontend/src/lib/components/triggers/CaptureTable.svelte +++ b/frontend/src/lib/components/triggers/CaptureTable.svelte @@ -33,6 +33,10 @@ limitPayloadSize?: boolean noBorder?: boolean captureActiveIndicator?: boolean | undefined + // Workspace to scope capture list/get/delete calls to. Defaults to the nav + // `$workspaceStore`; an AI-session live editor passes the session's acting + // workspace (a fork) so captures hit the right workspace. + workspace?: string } let { @@ -47,9 +51,12 @@ fullHeight = true, limitPayloadSize = false, noBorder = false, - captureActiveIndicator = undefined + captureActiveIndicator = undefined, + workspace = undefined }: Props = $props() + let ws = $derived(workspace ?? $workspaceStore) + let selected: number | undefined = $state(undefined) let testKind: 'preprocessor' | 'main' = $state('main') let isEmpty: boolean = $state(true) @@ -94,7 +101,7 @@ function initLoadCaptures(kind: 'preprocessor' | 'main' = testKind) { const loadInputsPageFn = async (page: number, perPage: number) => { const captures = await CaptureService.listCaptures({ - workspace: $workspaceStore!, + workspace: ws!, runnableKind: isFlow ? 'flow' : 'script', path: path ?? '', triggerKind: captureType, @@ -114,7 +121,7 @@ payloadData: 'Too big to display here, select to view', getFullCapture: () => CaptureService.getCapture({ - workspace: $workspaceStore!, + workspace: ws!, id: capture.id }) } @@ -148,7 +155,7 @@ const deleteInputFn = async (id: any) => { await CaptureService.deleteCapture({ - workspace: $workspaceStore!, + workspace: ws!, id }) } diff --git a/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte index e628c7cc99..44fc57d4ef 100644 --- a/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte @@ -350,6 +350,7 @@ {#if useDrawer} draftSync.deployed} + reserveSpace={draftSync.hasBaseline} getCurrent={() => draftSync.current} onDiscard={() => draftSync.resetToDeployed(initialPath)} disabled={!can_write} diff --git a/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte index 34cf4e92c3..1af45e9f2f 100644 --- a/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte @@ -368,7 +368,7 @@ {#if mode === 'suspended'} {/if} -
+
+
-
+
-
+
- +
@@ -1380,6 +1380,7 @@ {#if useDrawer} draftSync.deployed} + reserveSpace={draftSync.hasBaseline} getCurrent={() => draftSync.current} onDiscard={() => draftSync.resetToDeployed(initialPath)} disabled={!can_write} diff --git a/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte index 6ca2d13ddd..01e4a28e29 100644 --- a/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte @@ -370,6 +370,7 @@ {#if useDrawer} draftSync.deployed} + reserveSpace={draftSync.hasBaseline} getCurrent={() => draftSync.current} onDiscard={() => draftSync.resetToDeployed(initialPath)} disabled={!can_write} diff --git a/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts b/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts index aba27de99f..ba09167d6d 100644 --- a/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts +++ b/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts @@ -48,6 +48,11 @@ export interface TriggerDraftSync { * loading and for brand-new triggers (no deployed baseline yet). */ readonly hasDraft: boolean + /** + * Whether a deployed baseline exists (banner can appear). Reactive, unlike + * reading `deployed` directly (a plain `let`), so callers can reserve its slot. + */ + readonly hasBaseline: boolean /** The deployed baseline the dirty check compares against. */ readonly deployed: Cfg | undefined /** The current form config (the live local draft). */ @@ -104,6 +109,10 @@ export function useTriggerDraftSync(opts: TriggerDraftSyncOptions): TriggerDraft cfgDiffers(opts.getCfg() as Cfg, opts.deployed() as Cfg) ) + // Reactive "banner is possible" — depends on `drawerLoading()` so it + // re-evaluates (and re-reads the plain-`let` baseline) once the load settles. + const hasBaseline = $derived(!opts.drawerLoading() && opts.deployed() != null) + /** `auto: true` marks a discard from the reactive persist-effect (not an * explicit user action), so it respects the "Enable auto-save" toggle — else * with autosave off the editor would delete server drafts while writing none. */ @@ -189,6 +198,9 @@ export function useTriggerDraftSync(opts: TriggerDraftSyncOptions): TriggerDraft get hasDraft() { return hasDraft }, + get hasBaseline() { + return hasBaseline + }, get deployed() { return opts.deployed() }, diff --git a/frontend/src/lib/components/triggers/utils.ts b/frontend/src/lib/components/triggers/utils.ts index 231de5cea4..3d17491791 100644 --- a/frontend/src/lib/components/triggers/utils.ts +++ b/frontend/src/lib/components/triggers/utils.ts @@ -1,4 +1,14 @@ -import { Webhook, Mail, Calendar, Route, Unplug, Database, Terminal } from 'lucide-svelte' +import { + Webhook, + Mail, + Calendar, + Route, + Unplug, + Database, + Terminal, + Timer, + Zap +} from 'lucide-svelte' import KafkaIcon from '$lib/components/icons/KafkaIcon.svelte' import NatsIcon from '$lib/components/icons/NatsIcon.svelte' import MqttIcon from '$lib/components/icons/MqttIcon.svelte' @@ -82,7 +92,8 @@ export const jobTriggerKinds: JobTriggerKind[] = [ 'azure', 'google', 'github', - 'asset' + 'asset', + 'freshness' ] export type Trigger = { @@ -118,7 +129,12 @@ export const triggerIconMap = { cli: Terminal, nextcloud: NextcloudIcon, google: GoogleIcon, - github: GithubIcon + github: GithubIcon, + // Job-attribution-only kinds (no trigger CRUD page): the pipeline asset + // cascade and the freshness watchdog. Needed so the Runs filter and job + // detail render these trigger kinds instead of a blank label / no icon. + asset: Zap, + freshness: Timer } export const triggerDisplayNamesMap = { @@ -139,8 +155,12 @@ export const triggerDisplayNamesMap = { cli: 'CLI', nextcloud: 'Nextcloud', google: 'Google', - github: 'GitHub' -} as const satisfies Record + github: 'GitHub', + asset: 'Asset cascade', + freshness: 'Freshness' + // `asset` / `freshness` are job-attribution-only (JobTriggerKind, not + // TriggerType) — hence the union in the satisfies below. +} as const satisfies Record /** * Converts a TriggerType to a CaptureTriggerKind when a mapping exists diff --git a/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte index 77b5149933..361d5b610b 100644 --- a/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte @@ -457,6 +457,7 @@ {#if useDrawer} draftSync.deployed} + reserveSpace={draftSync.hasBaseline} getCurrent={() => draftSync.current} onDiscard={() => draftSync.resetToDeployed(initialPath)} disabled={!can_write} diff --git a/frontend/src/lib/components/workspace/WorkspaceCard.svelte b/frontend/src/lib/components/workspace/WorkspaceCard.svelte index c4ac967b50..0942311805 100644 --- a/frontend/src/lib/components/workspace/WorkspaceCard.svelte +++ b/frontend/src/lib/components/workspace/WorkspaceCard.svelte @@ -6,10 +6,13 @@ import type { UserWorkspace } from '$lib/stores' import { superadmin } from '$lib/stores' import { WorkspaceService } from '$lib/gen' + import { reconcileAfterWorkspaceChange } from '$lib/components/sessions/sessionState.svelte' import { pluralize } from '$lib/utils' + import { forkAccentStyle } from '$lib/utils/forkColor' import WorkspaceIcon from './WorkspaceIcon.svelte' import WorkspaceCard from './WorkspaceCard.svelte' import { twMerge } from 'tailwind-merge' + import { devBadgeText } from '$lib/utils/devWorkspaceLabel' interface ExtendedWorkspace extends UserWorkspace { _children?: ExtendedWorkspace[] @@ -50,6 +53,17 @@ const paddingLeft = untrack(() => depth) * 24 const isSelected = $derived(selectedWorkspaceId === workspace.id) + // Colored forks render icon + name in the derived fork accent (the fork + // picker convention); the icon side is handled inside WorkspaceIcon. + const forkAccent = $derived(isForked ? forkAccentStyle(workspace.color) : undefined) + + // The canonical dev workspace sorts before throwaway forks. + const sortedChildren = $derived( + [...children].sort((a, b) => { + if (!!a.is_dev_workspace !== !!b.is_dev_workspace) return a.is_dev_workspace ? -1 : 1 + return a.name.localeCompare(b.name) + }) + ) // Helper functions function isWorkspaceArchived(workspace: UserWorkspace): boolean { @@ -64,6 +78,8 @@ if (onUnarchive) { await WorkspaceService.unarchiveWorkspace({ workspace: workspace.id }) await onUnarchive(workspace.id) + // Restore sessions auto-archived when this workspace was archived. + await reconcileAfterWorkspaceChange() } } @@ -108,6 +124,8 @@ @@ -115,13 +133,26 @@
- + {#if workspace.marked} {@html workspace.marked} {:else} {workspace.name} {/if} + {#if workspace.is_dev_workspace} + {devBadgeText(workspace.dev_workspace_label)} + {/if} - {#if workspace.id === 'admins'} {workspace.id} @@ -205,7 +236,7 @@ {#if children.length > 0 && isExpanded}
- {#each children as child (child.id)} + {#each sortedChildren as child (child.id)} -
+
{#if isForked} {#snippet text()} {#if isForked && parentName} - Fork of {parentName} + {isDevWorkspace ? `${devLabelWord(devWorkspaceLabel)} workspace of` : 'Fork of'} + {parentName} {/if} {/snippet} - + + {:else} - + {/if} -
\ No newline at end of file +
diff --git a/frontend/src/lib/components/workspace/WorkspaceTreeView.svelte b/frontend/src/lib/components/workspace/WorkspaceTreeView.svelte index 87e94fed14..c3c3afa049 100644 --- a/frontend/src/lib/components/workspace/WorkspaceTreeView.svelte +++ b/frontend/src/lib/components/workspace/WorkspaceTreeView.svelte @@ -41,8 +41,15 @@ // Computed expansion states that include auto-expansion for search results let expansionStates = $derived.by(() => { + // Prod nodes that have a canonical dev start expanded so the dev is visible without a click; + // manual toggles still win, so the user can collapse them. + const devExpanded: Record = {} + workspaces?.forEach((w) => { + if (w.is_dev_workspace && w.parent_workspace_id) devExpanded[w.parent_workspace_id] = true + }) + if (!searchFilter || !filteredWorkspaces || !workspaces) { - return manualExpansionStates + return { ...devExpanded, ...manualExpansionStates } } const matchedWorkspaceIds = new Set(filteredWorkspaces.map((w) => w.id)) @@ -75,8 +82,8 @@ } }) - // Combine manual and auto-expanded states - return { ...manualExpansionStates, ...autoExpanded } + // Combine dev-default, manual, and auto-expanded states + return { ...devExpanded, ...manualExpansionStates, ...autoExpanded } }) // Build nested hierarchy correctly - always use full workspace list for hierarchy diff --git a/frontend/src/lib/components/workspaceItemsLoader.svelte.ts b/frontend/src/lib/components/workspaceItemsLoader.svelte.ts index 45d64bd01b..4ac65f805b 100644 --- a/frontend/src/lib/components/workspaceItemsLoader.svelte.ts +++ b/frontend/src/lib/components/workspaceItemsLoader.svelte.ts @@ -21,13 +21,23 @@ import { * Both getters are read inside the returned closures so changing * workspace or kinds after mount Just Works. */ +function itemsEqual(a: WorkspaceItem[] | undefined, b: WorkspaceItem[]): boolean { + if (!a || a.length !== b.length) return false + return a.every( + (item, i) => + item.path === b[i].path && + item.summary === b[i].summary && + item.kind === b[i].kind && + item.raw_app === b[i].raw_app + ) +} + export function useWorkspaceItemsLoader( workspace: () => string | undefined, kinds: () => readonly WorkspaceItemKind[] ) { // Seed from the module-level cache so kinds already fetched in this - // session render on the first frame. Re-fetching `ensureLoaded` later - // quietly swaps in fresh data (stale-while-revalidate). + // session render on the first frame. let loaded = $state>>( (() => { const ws = untrack(workspace) @@ -41,18 +51,38 @@ export function useWorkspaceItemsLoader( })() ) let loadingKind = $state>>({}) + // Workspace+kind pairs this loader instance has refreshed — at most one + // background re-fetch per kind per mount. Marked on success only, so a + // failed fetch is retried by the next call instead of stranding stale data. + const revalidated = new Set() async function ensureLoaded(kind: WorkspaceItemKind) { const ws = workspace() if (!ws) return + const revalidateKey = `${ws}:${kind}` // `loaded[kind]` read inside `untrack` so callers wiring this into // a reactive context (DrillPicker's onFilterChange effect) don't // subscribe to a signal `ensureLoaded` itself writes — that would // re-fire the effect on every assignment and busy-loop. - if (!untrack(() => loaded[kind])) loadingKind[kind] = true + const hasCached = !!untrack(() => loaded[kind]) + if (hasCached && revalidated.has(revalidateKey)) return + if (!hasCached) loadingKind[kind] = true try { - const items = await loadKind(ws, kind) - loaded[kind] = items + const items = await loadKind(ws, kind, { revalidate: true }) + revalidated.add(revalidateKey) + // Keep the reference stable when nothing changed so an open picker's + // derived tree isn't rebuilt under the user on every revalidation. + if ( + !itemsEqual( + untrack(() => loaded[kind]), + items + ) + ) + loaded[kind] = items + } catch (e) { + // Callers fire-and-forget; surface the failure without an + // unhandled rejection. Cached items (if any) keep rendering. + console.error(`Failed to load workspace ${kind}s`, e) } finally { loadingKind[kind] = false } diff --git a/frontend/src/lib/components/workspacePicker.ts b/frontend/src/lib/components/workspacePicker.ts index ba2985ecbb..92ea48af20 100644 --- a/frontend/src/lib/components/workspacePicker.ts +++ b/frontend/src/lib/components/workspacePicker.ts @@ -44,9 +44,10 @@ type WorkspaceCache = { } /** Module-level session cache. Persists across picker mounts within a single - * page session. NOT invalidated automatically — call `invalidate()` after - * creating/deleting an item if the picker may be opened again before a full - * reload. */ + * page session so cached kinds render on the first frame. Pickers revalidate + * once per mount via `loadKind(..., { revalidate: true })`, so stale entries + * self-heal on the next open; call `invalidate()` after creating/deleting an + * item when even a brief flash of the stale list must be avoided. */ const cache = new Map() const inflight = new Map>() /** Bumped by `invalidate()`. Each in-flight `loadKind` captures the version @@ -87,11 +88,14 @@ export function invalidate(workspace: string, kind?: WorkspaceItemKind) { export async function loadKind( workspace: string, - kind: WorkspaceItemKind + kind: WorkspaceItemKind, + opts?: { revalidate?: boolean } ): Promise { const existing = cache.get(workspace)?.[kind] - if (existing) return existing + if (existing && !opts?.revalidate) return existing const key = cacheKey(workspace, kind) + // An in-flight fetch is already hitting the network, so it satisfies a + // revalidate request too. const flying = inflight.get(key) if (flying) return flying diff --git a/frontend/src/lib/components/workspaceSettings/AISettings.svelte b/frontend/src/lib/components/workspaceSettings/AISettings.svelte index 2080ecd332..f44059e395 100644 --- a/frontend/src/lib/components/workspaceSettings/AISettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/AISettings.svelte @@ -13,6 +13,8 @@ import { supportsAutocomplete } from '../copilot/utils' import TestAiKey from '../copilot/TestAIKey.svelte' import Label from '../Label.svelte' + import AiSkillsSettings from './AiSkillsSettings.svelte' + import { isGlobalAiEnabled } from '../copilot/chat/global/gate' import SettingsPageHeader from '../settings/SettingsPageHeader.svelte' import ResourcePicker from '../ResourcePicker.svelte' import Toggle from '../Toggle.svelte' @@ -587,6 +589,10 @@
{/if} + + {#if promptScope === 'workspace' && isGlobalAiEnabled()} + + {/if}
+ import { onMount } from 'svelte' + import { createDropdownMenu, melt } from '@melt-ui/svelte' + import Button from '../common/button/Button.svelte' + import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte' + import Modal2 from '../common/modal/Modal2.svelte' + import Toggle from '../Toggle.svelte' + import DropdownV2 from '../DropdownV2.svelte' + import Checkbox from '../common/checkbox/Checkbox.svelte' + import Markdown from 'svelte-exmarkdown' + import { gfmPlugin } from 'svelte-exmarkdown/gfm' + import ToggleButton from '../common/toggleButton-v2/ToggleButton.svelte' + import ToggleButtonGroup from '../common/toggleButton-v2/ToggleButtonGroup.svelte' + import SettingCard from '../instanceSettings/SettingCard.svelte' + import autosize from '$lib/autosize' + import { conditionalMelt } from '$lib/utils' + import { workspaceStore } from '$lib/stores' + import { sendUserToast } from '$lib/toast' + import { WorkspaceService } from '$lib/gen' + import { buildSkillMd, parseAndValidateSkill, parseSkillMd, type SkillUpload } from './aiSkills' + import { + ChevronDown, + ClipboardPaste, + Eye, + FolderUp, + ListChecks, + Pencil, + Plus, + Trash2 + } from 'lucide-svelte' + + type SkillListItem = { name: string; description: string } + + // `//SKILL.md` is 3 path segments; SKILL.md files nested deeper + // are likely vendored/incidental and are skipped so importing a parent dir + // doesn't sweep in unrelated skills. + const MAX_SKILL_DEPTH = 3 + const MAX_SKILLS_PER_IMPORT = 50 + const MAX_SKILLS_PER_WORKSPACE = 100 + const SAMPLE_SKILL_PLACEHOLDER = + '---\nname: my-skill\ndescription: what this skill helps with\n---\n\n# My skill\n\nInstructions for the assistant…' + const menuItemClass = + 'w-full flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover' + + let skills: SkillListItem[] = $state([]) + let uploading: boolean = $state(false) + let pasteContent: string = $state('') + // The content the modal opened with, so Save can be gated on unsaved changes. + let originalContent: string = $state('') + let pasteModalOpen: boolean = $state(false) + // Set while the paste modal is editing an existing skill; holds the skill's + // name before edits so a rename can delete the old entry after save. + let editingOriginalName: string | undefined = $state(undefined) + let dirInput: HTMLInputElement | undefined = $state(undefined) + let toDelete: string | undefined = $state(undefined) + let pendingImport: SkillUpload[] | undefined = $state(undefined) + let pendingSkipped: string[] = $state([]) + // Per-conflict overwrite choice for a folder import, keyed by skill name. + let overwriteChoices: Record = $state({}) + // The skill detail modal opens in read mode with rendered markdown; a header + // toggle flips it to raw SKILL.md editing. + let detailMode: 'view' | 'edit' = $state('view') + // Multi-select "manage" mode: rows gain a checkbox for batch deletion. + let manageMode: boolean = $state(false) + let selected: Record = $state({}) + let confirmBatchDelete: boolean = $state(false) + let listRequestId = 0 + + let existingNames = $derived(new Set(skills.map((s) => s.name))) + let selectedCount = $derived(skills.filter((s) => selected[s.name]).length) + let allSelected = $derived(skills.length > 0 && selectedCount === skills.length) + + // Leave manage mode automatically once a batch delete empties it below the + // two-skill threshold that surfaces the "Manage skills" button. + $effect(() => { + if (manageMode && skills.length <= 1) exitManage() + }) + let pendingConflicts = $derived( + (pendingImport ?? ([] as SkillUpload[])).filter((s) => existingNames.has(s.name)) + ) + let pendingNew = $derived( + (pendingImport ?? ([] as SkillUpload[])).filter((s) => !existingNames.has(s.name)) + ) + // Parsed view of the modal's raw content, for rendering the skill in read mode. + let viewParsed = $derived(parseSkillMd(pasteContent)) + let isDirty = $derived(pasteContent !== originalContent) + // Validate through the shared schema; surfaced inline so Save can be gated + // without a toast. + let pasteResult = $derived(parseAndValidateSkill(pasteContent)) + let pasteError = $derived('error' in pasteResult ? pasteResult.error : undefined) + + // Reset edit mode whenever the paste modal closes so a later "Paste a skill" + // opens a blank creation form. + $effect(() => { + if (!pasteModalOpen) editingOriginalName = undefined + }) + + // melt dropdown for the "+ Add skills" button: arrow-key nav, outside/escape + // close and focus management come for free. + const { + elements: { trigger: addMenuTrigger, menu: addMenu, item: addMenuItem }, + states: { open: addMenuOpen } + } = createDropdownMenu({ + positioning: { placement: 'bottom-end', gutter: 4, fitViewport: true }, + loop: true, + forceVisible: true + }) + + // attach the menu trigger to the design-system +
+ {/if} +{/snippet} + + + {#snippet headerAction()} +
+ {#if manageMode} + + + {:else} + {#if skills.length > 1} + + {/if} + + {/if} +
+ {#if $addMenuOpen} +
+ + +
+ {/if} + {/snippet} + +
+ {#if skills.length === 0} +
+ No custom skills yet +
+ {:else} +
+ {#if manageMode} +
+ 0 && !allSelected} + onChange={toggleSelectAll} + /> + + {selectedCount ? `${selectedCount} selected` : 'Select all'} + +
+ {/if} + {#each skills as skill (skill.name)} +
+ {#if manageMode} + + {:else} +
+
{skill.name}
+
{skill.description}
+ +
+ openSkill(skill.name, 'edit') + }, + { + displayName: 'Delete', + icon: Trash2, + type: 'delete', + action: () => (toDelete = skill.name) + } + ]} + /> + {/if} +
+ {/each} +
+ {/if} +
+
+ + + + + + {#snippet headerRight()} + {#if editingOriginalName} + + {#snippet children({ item })} + + + {/snippet} + + {/if} + {/snippet} +
+ {#if detailMode === 'view'} +
+ {#if viewParsed.description} +

{viewParsed.description}

+ {/if} +
+ +
+
+ {:else} + {@render pasteZone()} + {/if} +
+
+ + { + const toImport = [...pendingNew, ...pendingConflicts.filter((s) => overwriteChoices[s.name])] + const skipped = pendingSkipped + pendingImport = undefined + pendingSkipped = [] + overwriteChoices = {} + if (toImport.length) await uploadSkills(toImport, skipped) + else sendUserToast('No skills imported.') + }} + onCanceled={() => { + pendingImport = undefined + pendingSkipped = [] + overwriteChoices = {} + }} +> +
+ {#if pendingNew.length} +
+ Add {pendingNew.length} new skill(s): + {pendingNew.map((s) => s.name).join(', ')} +
+ {/if} + {#if pendingConflicts.length} +
+ + {pendingConflicts.length} skill(s) already exist — choose which to overwrite: + +
+ {#each pendingConflicts as conflict (conflict.name)} +
+ {conflict.name} + +
+ {/each} +
+
+ {/if} + {#if pendingSkipped.length} + {pendingSkipped.length} file(s) will be skipped. + {/if} +
+
+ + { + const name = toDelete + toDelete = undefined + if (name) await deleteSkill(name) + }} + onCanceled={() => (toDelete = undefined)} +> + + Delete the skill {toDelete}? The AI chat will no longer be able to use it. + + + + { + confirmBatchDelete = false + await deleteSelected() + }} + onCanceled={() => (confirmBatchDelete = false)} +> + + Delete {selectedCount} selected skill(s)? The AI chat will no longer be able to use them. + + diff --git a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte index 024d94e8e6..5f74da4b80 100644 --- a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte +++ b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte @@ -14,14 +14,22 @@ import { validateUsername } from '$lib/utils' import { logoutWithRedirect } from '$lib/logoutKit' import { page } from '$app/state' - import { usersWorkspaceStore, workspaceStore } from '$lib/stores' - import { Button } from '$lib/components/common' + import { usersWorkspaceStore, userWorkspaces, workspaceStore } from '$lib/stores' + import { + workspaceIsFork, + findWorkspaceRoot, + findWorkspaceDescendants + } from '$lib/utils/workspaceHierarchy' + import { resource } from 'runed' + import { Badge, Button } from '$lib/components/common' + import { devBadgeText } from '$lib/utils/devWorkspaceLabel' import Toggle from '$lib/components/Toggle.svelte' import Tooltip from '$lib/components/Tooltip.svelte' import { onMount } from 'svelte' import { sendUserToast } from '$lib/toast' import TestAIKey from '$lib/components/copilot/TestAIKey.svelte' import { switchWorkspace } from '$lib/storeUtils' + import { deleteSessionsForWorkspace } from '$lib/components/sessions/sessionState.svelte' import { isCloudHosted } from '$lib/cloud' import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte' import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte' @@ -35,13 +43,95 @@ import { base } from '$lib/base' import Label from '../Label.svelte' import ForkDatatableSection from './ForkDatatableSection.svelte' + import Select from '../select/Select.svelte' + import WorkspaceScopeTrigger from '../WorkspaceScopeTrigger.svelte' + import DarkModeToggle from '../sidebar/DarkModeToggle.svelte' + import ForkDucklakeSection from './ForkDucklakeSection.svelte' interface Props { isFork?: boolean + // Rendered inside the global fork modal rather than the standalone + // /user/create_workspace page: content-driven height and no + // "Back to workspaces" navigation. + inModal?: boolean onFinish?: () => void } - let { isFork = false, onFinish }: Props = $props() + let { isFork = false, inModal = false, onFinish }: Props = $props() + + // Dev-workspace mode: create the fork as a persistent, prefix-less dev workspace and (optionally) + // lock the parent ("prod") against direct edits. + let createAsDevWorkspace = $state(false) + let lockProdDeploy = $state(true) + let lockProdForking = $state(true) + // Bring the parent's members into the fork (a shared env). Defaults on for a + // dev workspace, off for a throwaway fork; flipping the dev toggle resets it. + let copyMembers = $state(false) + $effect(() => { + copyMembers = createAsDevWorkspace + }) + + // A dev workspace can only be created off a root base (backend rejects a dev of a fork). Clear a + // stale toggle when the base no longer qualifies so we never submit is_dev_workspace against a fork. + $effect(() => { + if (!canDesignateDevWorkspace && createAsDevWorkspace) { + createAsDevWorkspace = false + } + }) + + // The base workspace to fork from. A fork's git branch is based on its parent's branch, so picking + // a fork here (rather than the root) yields a fork of a fork. + let baseWorkspaceId = $state(undefined) + + // Base candidates are the current workspace's family: its root first, then every fork/dev under it. + let familyRoot = $derived(findWorkspaceRoot($workspaceStore, $userWorkspaces)) + let baseCandidates = $derived( + familyRoot ? [familyRoot, ...findWorkspaceDescendants(familyRoot.id, $userWorkspaces)] : [] + ) + let baseItems = $derived( + baseCandidates.map((w) => ({ + value: w.id, + label: w.id === familyRoot?.id ? `${w.name} (root)` : w.name, + subtitle: w.is_dev_workspace ? 'dev workspace' : w.id === familyRoot?.id ? undefined : 'fork' + })) + ) + let defaultBaseWorkspaceId = $derived(familyRoot?.id) + // Seed the base once the family is known; keep an explicit user choice as long as it stays valid. + $effect(() => { + if (!isFork) return + if (baseWorkspaceId && baseCandidates.some((w) => w.id === baseWorkspaceId)) return + baseWorkspaceId = defaultBaseWorkspaceId + }) + + // Cosmetic display label for the new dev workspace: 'dev' | 'staging'. Purely visual (badge text + + // wording); reset when the dev toggle is turned off. + let devWorkspaceLabel = $state<'dev' | 'staging'>('dev') + $effect(() => { + if (!createAsDevWorkspace) devWorkspaceLabel = 'dev' + }) + + // The dev-workspace option is only offered when forking a root workspace that doesn't already + // have one: a workspace gets at most one dev, and dev workspaces don't nest (a dev of a dev). + let baseWorkspaceEntry = $derived($userWorkspaces.find((w) => w.id === baseWorkspaceId)) + // Require the base workspace to be loaded before treating it as a root: a missing entry must + // not read as root (it would offer invalid dev creation while the workspace list is still loading). + // `workspaceIsFork` (prefix OR parent) also excludes an orphaned `wm-fork-` workspace, whose parent + // FK was set null — it has no parent but is still a fork, so it can't host a dev workspace. + let currentIsRoot = $derived( + !!baseWorkspaceEntry && !workspaceIsFork(baseWorkspaceId, $userWorkspaces) + ) + // Ask the server whether a dev already exists: the caller may not be a member of this prod's dev, + // so the client workspace list can't see it and would offer an invalid "create dev" action. + const devWorkspaceResource = resource( + () => (currentIsRoot ? baseWorkspaceId : undefined), + async (ws) => (ws ? await WorkspaceService.getDevWorkspace({ workspace: ws }) : undefined) + ) + // Offer dev designation only once the server confirms there's no dev yet (returns null); stay + // conservative (no offer) while the check is loading (current is undefined). + let canDesignateDevWorkspace = $derived(currentIsRoot && devWorkspaceResource.current === null) + let currentWorkspaceName = $derived( + baseWorkspaceEntry?.name ?? baseWorkspaceId ?? 'the root workspace' + ) let id = $state('') let name = $state('') @@ -54,6 +144,7 @@ let checking = $state(false) let forkDatatableSection: ReturnType | undefined = $state(undefined) + let forkDucklakeSection: ReturnType | undefined = $state(undefined) let workspaceColor: string | undefined = $state(undefined) let colorEnabled = $state(false) @@ -72,16 +163,20 @@ // For forks the actual workspace id is prefixed: checking the bare id // would report the name as free even when `wm-fork-` is taken // (e.g. by an archived fork, which keeps its id reserved). - const effectiveId = isFork ? `${WM_FORK_PREFIX}${id}` : id + const effectiveId = isFork && !createAsDevWorkspace ? `${WM_FORK_PREFIX}${id}` : id let exists = id != '' && (await WorkspaceService.existsWorkspace({ requestBody: { id: effectiveId } })) - forkIdTaken = isFork && exists + // The "delete existing fork to reclaim the id" affordance is only for prefixed forks. + forkIdTaken = isFork && !createAsDevWorkspace && exists if (exists) { - errorId = isFork + errorId = forkIdTaken ? `A workspace with id '${effectiveId}' already exists. It may be an archived fork: archiving keeps the id reserved.` : 'ID already exists' } else if (id != '' && !/^\w+(-\w+)*$/.test(id)) { errorId = 'ID can only contain letters, numbers and dashes and must not finish by a dash' + } else if (effectiveId.length > 50) { + // `wm-fork-` prefix included: matches the backend's 50-char (git-branch / DB) limit. + errorId = `ID '${effectiveId}' is too long (${effectiveId.length} chars). Maximum is 50.` } else { errorId = '' } @@ -90,6 +185,9 @@ const WM_FORK_PREFIX = 'wm-fork-' + // A dev workspace keeps its bare id; an ordinary fork is prefixed with `wm-fork-`. + const effectiveForkId = $derived(createAsDevWorkspace ? id : `${WM_FORK_PREFIX}${id}`) + let forkIdTaken = $state(false) let deleteExistingForkOpen = $state(false) let deletingExistingFork = $state(false) @@ -100,6 +198,14 @@ deletingExistingFork = true try { await WorkspaceService.deleteWorkspace({ workspace: prefixedId }) + // Drop local sessions bound to this id so they don't resurface (or + // auto-unarchive) against a new fork recreated under the same id. + // Fire-and-forget: neither a slow nor a failing IndexedDB op should + // block the delete/reuse flow (cleanup completes long before the UI + // could create a session in a recreated fork). + void deleteSessionsForWorkspace(prefixedId).catch((e) => + console.error(`Session cleanup for reused fork id ${prefixedId} failed`, e) + ) sendUserToast(`Permanently deleted workspace ${prefixedId}`) deleteExistingForkOpen = false await validateName(id) @@ -120,7 +226,7 @@ for (const job of jobs) { let j = await JobService.getCompletedJob({ id: job, - workspace: $workspaceStore! + workspace: baseWorkspaceId! }) ret.push(j) } @@ -148,16 +254,15 @@ } async function createOrForkWorkspace() { - const prefixed_id = `${WM_FORK_PREFIX}${id}` if (isFork) { - await forkWorkspace(prefixed_id) + await forkWorkspace(effectiveForkId) } else { await createWorkspace() } } async function forkWorkspace(prefixed_id: string): Promise { - if ($workspaceStore) { + if (baseWorkspaceId) { forkCreationLoading = true errorMsgs = [] failedSyncJobs = [] @@ -179,20 +284,38 @@ } async function completeFork(prefixed_id: string): Promise { - let gitSyncJobIds = await WorkspaceService.createWorkspaceForkGitBranch({ - workspace: $workspaceStore!, - requestBody: { - id: prefixed_id, - name, - color: colorEnabled && workspaceColor ? workspaceColor : undefined - } - }) + let gitSyncJobIds: string[] + try { + gitSyncJobIds = await WorkspaceService.createWorkspaceForkGitBranch({ + workspace: baseWorkspaceId!, + requestBody: { + id: prefixed_id, + name, + color: colorEnabled && workspaceColor ? workspaceColor : undefined, + is_dev_workspace: createAsDevWorkspace, + dev_workspace_label: createAsDevWorkspace ? devWorkspaceLabel : undefined, + // Send the lock intent in this first phase too so the backend can reject a non-admin's + // locked-dev request before any branch is created (avoids dangling branches). + lock_prod_deploy: createAsDevWorkspace && lockProdDeploy, + lock_prod_forking: createAsDevWorkspace && lockProdForking, + copy_members: copyMembers + } + }) + } catch (e) { + // The backend can reject here (fork cap, depth limit, premium, non-admin lock). Reset the + // loading state and surface the error rather than leaving the button spinning. + forkCreationError = `Failed to create fork '${prefixed_id}'` + errorMsgs.push(e?.body ?? e ?? 'Unknown error') + forkCreationLoading = false + sendUserToast(`Could not create fork '${prefixed_id}' ${e?.body ?? e}`, true) + return + } try { await Promise.all( gitSyncJobIds.map((jobId) => jobManager.runWithProgress(() => Promise.resolve(jobId), { - workspace: $workspaceStore!, + workspace: baseWorkspaceId!, timeout: 60000, timeoutMessage: `Deploy fork job timed out after 60s`, onProgress: (status) => { @@ -207,7 +330,7 @@ } catch (error) { forkCreationLoading = false sendUserToast( - `Could not fork workspace ${$workspaceStore} because branch creation failed: ${errorMsgs} - ${error}`, + `Could not fork workspace ${baseWorkspaceId} because branch creation failed: ${errorMsgs} - ${error}`, true ) return @@ -216,7 +339,7 @@ forkCreationError = 'Failed to create a branch for this fork on the git sync repo(s)' forkCreationLoading = false sendUserToast( - `Could not fork workspace ${$workspaceStore} because branch creation failed: ${errorMsgs}`, + `Could not fork workspace ${baseWorkspaceId} because branch creation failed: ${errorMsgs}`, true ) return @@ -232,12 +355,18 @@ try { await WorkspaceService.createWorkspaceFork({ - workspace: $workspaceStore!, + workspace: baseWorkspaceId!, requestBody: { id: prefixed_id, name, color: colorEnabled && workspaceColor ? workspaceColor : undefined, - forked_datatables: forkedDatatables + forked_datatables: forkedDatatables, + shared_ducklakes: forkDucklakeSection?.getSharedDucklakes() ?? [], + is_dev_workspace: createAsDevWorkspace, + dev_workspace_label: createAsDevWorkspace ? devWorkspaceLabel : undefined, + lock_prod_deploy: createAsDevWorkspace && lockProdDeploy, + lock_prod_forking: createAsDevWorkspace && lockProdForking, + copy_members: copyMembers } }) } catch (e) { @@ -249,7 +378,11 @@ } forkCreationLoading = false - sendUserToast(`Successfully forked workspace ${$workspaceStore} as: wm-fork-${id}`) + sendUserToast( + createAsDevWorkspace + ? `Created ${devWorkspaceLabel === 'staging' ? 'staging' : 'dev'} workspace ${effectiveForkId} for ${baseWorkspaceId}` + : `Successfully forked workspace ${baseWorkspaceId} as: wm-fork-${id}` + ) usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) switchWorkspace(prefixed_id) @@ -388,7 +521,27 @@ let autoAdd = $state(true) let selected: Exclude = $state('openai') run(() => { - id = name.toLowerCase().replace(/\s/gi, '-') + if (isFork) { + // Forks have no separate display name — the unique id is the name. + name = id + } else { + id = name.toLowerCase().replace(/\s/gi, '-') + } + }) + // When creating a dev workspace, prefill the fork name with `-dev` / `-stg` (the effect + // above slugifies it into the id). Only fill an empty field or one still holding a prior suggestion, + // so a user-typed name is never overwritten; flipping Dev<->Staging updates the suffix, and turning + // the dev toggle back off clears the suggestion. + let lastAutoDevName = $state(undefined) + $effect(() => { + const target = + createAsDevWorkspace && $workspaceStore + ? `${$workspaceStore}-${devWorkspaceLabel === 'staging' ? 'stg' : 'dev'}` + : '' + if (name === '' || name === lastAutoDevName) { + name = target + lastAutoDevName = target === '' ? undefined : target + } }) run(() => { validateName(id) @@ -402,9 +555,16 @@ let domain = $derived($usersWorkspaceStore?.email.split('@')[1]) -
-
-
+
+ +
+
{#if errorMsgs.length != 0}
    @@ -424,7 +584,7 @@ {job.id} @@ -449,7 +609,7 @@ {jobId} @@ -460,34 +620,32 @@ {/if} {/if} - + {/if} -
{/if}
+ {#if isFork && colorEnabled} +
+
+ Fork picker preview + + +
+
+ +
+
+ {/if} + {#if isFork && canDesignateDevWorkspace} + + {/if} + {#if isFork && createAsDevWorkspace} + + {/if} + {#if isFork} +
+ {/each} +
+ {#if anyShared} +
+ + + A shared lake is NOT isolated: pipeline runs in the fork read and write this workspace's + tables directly. + +
+ {/if} + +{/if} diff --git a/frontend/src/lib/components/workspaceSettings/NewDataTableMigrationModal.svelte b/frontend/src/lib/components/workspaceSettings/NewDataTableMigrationModal.svelte new file mode 100644 index 0000000000..d4dad04ac1 --- /dev/null +++ b/frontend/src/lib/components/workspaceSettings/NewDataTableMigrationModal.svelte @@ -0,0 +1,230 @@ + + + + +
+ + + + + {#snippet content()} + + + + +
+ + {#if enableDown} +
+ +
+ {/if} +
+
+ {/snippet} +
+
+ +
+
+
+ + + + diff --git a/frontend/src/lib/components/workspaceSettings/StorageSettings.svelte b/frontend/src/lib/components/workspaceSettings/StorageSettings.svelte index ef46189444..1e6c990819 100644 --- a/frontend/src/lib/components/workspaceSettings/StorageSettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/StorageSettings.svelte @@ -1,7 +1,7 @@ @@ -114,10 +188,9 @@ link="https://www.windmill.dev/docs/core_concepts/object_storage_in_windmill#workspace-object-storage" /> {#if !$enterpriseLicense} - - Windmill S3 bucket browser will not work for buckets containing more than 20 files and uploads - are limited to files {'<'} 50MB. Consider upgrading to Windmill EE to use this feature with large - buckets. + + Total workspace storage is capped at {quotaDisplay} in the Community Edition: writes that would exceed + the quota are rejected. Consider upgrading to Windmill EE for unlimited workspace storage. {:else} @@ -129,6 +202,67 @@ >, set by the superadmins in the instance settings UI. {/if} +{#if primaryStorageSaved} +
+
+ Storage usage +
+ {#if storageUsage} + {#if storageUsage.quota_bytes !== undefined && usedFraction !== undefined} +
+
+
+ + {displaySize(storageUsage.total_bytes)} of {quotaDisplay} used + {#if storageUsage.storages.length > 1} + ({storageUsage.storages + .map( + (s) => + `${s.storage === '_default_' ? 'primary' : s.storage}: ${displaySize(s.bytes)}` + ) + .join(', ')}) + {/if} + + {#if overQuota} + + Writes to workspace storage are rejected until usage drops below {quotaDisplay}. Delete + files from workspace storage or upgrade to Windmill EE for unlimited storage. + + {/if} + {:else} + + {displaySize(storageUsage.total_bytes)} used + {#if storageUsage.storages.length > 1} + ({storageUsage.storages + .map( + (s) => + `${s.storage === '_default_' ? 'primary' : s.storage}: ${displaySize(s.bytes)}` + ) + .join(', ')}) + {/if} + + {/if} + {:else if storageUsageLoading} + Computing storage usage... + {/if} +
+{/if} {#if s3ResourceSettings} @@ -144,7 +278,7 @@ - {#each tableRows as tableRow, idx} + {#each tableRows as tableRow} {#if tableRow[0] === null} @@ -161,27 +295,30 @@
{#if tableRow[1].resourceType === 'filesystem'} - - {/if} + + + {/if}
{#if tableRow[1].resourceType === 'filesystem'} @@ -195,6 +332,7 @@ class="flex-1" bind:value={tableRow[1].resourcePath} resourceType={tableRow[1].resourceType} + error={emptyString(tableRow[1].resourcePath)} /> {/if}
@@ -223,19 +361,15 @@ class="cursor-not-allowed" > {#snippet trigger()} - - - - {/snippet} + + {/snippet} {#snippet content()} - - {#if emptyString(tableRow[1].resourcePath)} - Please select a storage resource - {:else if isDirty(tableRow[0])} - Please save your changes - {/if} - - {/snippet} + {#if emptyString(tableRow[1].resourcePath)} + Please select a storage resource + {:else if isDirty(tableRow[0])} + Please save your changes + {/if} + {/snippet} {:else} { if (s3ResourceSettings.secondaryStorage) { - s3ResourceSettings.secondaryStorage.splice(idx - 1, 1) - s3ResourceSettings.secondaryStorage = [...s3ResourceSettings.secondaryStorage] + const realIdx = s3ResourceSettings.secondaryStorage.findIndex( + (s) => s === tableRow + ) + if (realIdx !== -1) { + s3ResourceSettings.secondaryStorage.splice(realIdx, 1) + s3ResourceSettings.secondaryStorage = [...s3ResourceSettings.secondaryStorage] + } } }} /> + {:else if (s3ResourceSettings.secondaryStorage?.length ?? 0) === 0} + {/if} @@ -295,7 +436,16 @@ {/snippet}
- {#if !s3ResourceSettings.resourcePath} + {#if !showPrimaryRow} + + {:else if !s3ResourceSettings.resourcePath} onDiscard?.()} saveLabel="Save storage settings" diff --git a/frontend/src/lib/components/workspaceSettings/aiSkills.test.ts b/frontend/src/lib/components/workspaceSettings/aiSkills.test.ts new file mode 100644 index 0000000000..87f84d865a --- /dev/null +++ b/frontend/src/lib/components/workspaceSettings/aiSkills.test.ts @@ -0,0 +1,197 @@ +import { describe, expect, it } from 'vitest' + +import { + MAX_SKILL_DESCRIPTION_LENGTH, + MAX_SKILL_INSTRUCTIONS_LENGTH, + MAX_SKILL_NAME_LENGTH, + buildSkillMd, + parseAndValidateSkill, + parseSkillMd, + validateSkill +} from './aiSkills' + +describe('parseSkillMd', () => { + it('splits frontmatter name/description from the body', () => { + const md = '---\nname: my-skill\ndescription: does a thing\n---\n\n# Title\n\nBody text.' + expect(parseSkillMd(md)).toEqual({ + name: 'my-skill', + description: 'does a thing', + instructions: '# Title\n\nBody text.' + }) + }) + + it('trims frontmatter values and the body', () => { + const md = '---\nname: spaced \ndescription: padded \n---\n\n body ' + const parsed = parseSkillMd(md) + expect(parsed.name).toBe('spaced') + expect(parsed.description).toBe('padded') + expect(parsed.instructions).toBe('body') + }) + + it('returns undefined name/description when there is no frontmatter', () => { + expect(parseSkillMd('just a body')).toEqual({ + name: undefined, + description: undefined, + instructions: 'just a body' + }) + }) + + it('strips a leading UTF-8 BOM before matching frontmatter', () => { + const md = '---\nname: bom-skill\ndescription: d\n---\n\nbody' + const parsed = parseSkillMd(md) + expect(parsed.name).toBe('bom-skill') + expect(parsed.description).toBe('d') + expect(parsed.instructions).toBe('body') + }) + + it('handles CRLF line endings in the frontmatter fence', () => { + const md = '---\r\nname: crlf\r\ndescription: d\r\n---\r\n\r\nbody' + const parsed = parseSkillMd(md) + expect(parsed.name).toBe('crlf') + expect(parsed.description).toBe('d') + }) + + it('leaves name/description undefined for missing frontmatter keys', () => { + const parsed = parseSkillMd('---\nname: only-name\n---\n\nbody') + expect(parsed.name).toBe('only-name') + expect(parsed.description).toBeUndefined() + expect(parsed.instructions).toBe('body') + }) + + it('ignores non-string frontmatter values', () => { + const parsed = parseSkillMd('---\nname: 123\ndescription: [a, b]\n---\n\nbody') + expect(parsed.name).toBeUndefined() + expect(parsed.description).toBeUndefined() + }) + + it('does not throw on malformed YAML frontmatter', () => { + const parsed = parseSkillMd('---\nname: "unterminated\n---\n\nbody') + expect(parsed.instructions).toBe('body') + expect(parsed.name).toBeUndefined() + }) +}) + +describe('validateSkill', () => { + const valid = { name: 'a-skill', description: 'a description', instructions: 'body' } + + it('returns undefined for a valid skill', () => { + expect(validateSkill(valid)).toBeUndefined() + }) + + it('flags a missing name', () => { + expect(validateSkill({ ...valid, name: '' })).toBe('name is required') + }) + + it('flags a missing description', () => { + expect(validateSkill({ ...valid, description: '' })).toBe('description is required') + }) + + it('flags a missing body', () => { + expect(validateSkill({ ...valid, instructions: '' })).toBe('body is required') + }) + + it('rejects names with disallowed characters and echoes the value', () => { + expect(validateSkill({ ...valid, name: 'Bad Name!' })).toBe( + `name "Bad Name!" must only contain lowercase letters, digits or '-'` + ) + }) + + it('accepts names of lowercase letters, digits and hyphens', () => { + expect(validateSkill({ ...valid, name: 'skill-123' })).toBeUndefined() + }) + + it('accepts a name exactly at the length limit but rejects one over', () => { + expect(validateSkill({ ...valid, name: 'a'.repeat(MAX_SKILL_NAME_LENGTH) })).toBeUndefined() + expect(validateSkill({ ...valid, name: 'a'.repeat(MAX_SKILL_NAME_LENGTH + 1) })).toBe( + `name is longer than ${MAX_SKILL_NAME_LENGTH} characters` + ) + }) + + it('counts the description limit in code points, not UTF-16 units', () => { + // Astral emoji are 2 UTF-16 units but 1 code point each. + const desc = '😀'.repeat(MAX_SKILL_DESCRIPTION_LENGTH) + expect(validateSkill({ ...valid, description: desc })).toBeUndefined() + expect(validateSkill({ ...valid, description: desc + '😀' })).toBe( + `description is longer than ${MAX_SKILL_DESCRIPTION_LENGTH} characters` + ) + }) + + it('counts the body limit in bytes', () => { + // A 4-byte emoji fills the byte budget four times faster than its char count. + const bodyAtLimit = 'a'.repeat(MAX_SKILL_INSTRUCTIONS_LENGTH) + expect(validateSkill({ ...valid, instructions: bodyAtLimit })).toBeUndefined() + expect(validateSkill({ ...valid, instructions: bodyAtLimit + 'a' })).toBe( + `body is longer than ${MAX_SKILL_INSTRUCTIONS_LENGTH} bytes` + ) + const multibyte = '😀'.repeat(MAX_SKILL_INSTRUCTIONS_LENGTH / 4 + 1) + expect(validateSkill({ ...valid, instructions: multibyte })).toBe( + `body is longer than ${MAX_SKILL_INSTRUCTIONS_LENGTH} bytes` + ) + }) + + it('reports the name issue first when several fields are invalid', () => { + expect(validateSkill({ name: '', description: '', instructions: '' })).toBe('name is required') + }) +}) + +describe('parseAndValidateSkill', () => { + it('parses and validates a well-formed SKILL.md', () => { + const md = '---\nname: good-skill\ndescription: a good one\n---\n\nbody' + expect(parseAndValidateSkill(md)).toEqual({ + skill: { name: 'good-skill', description: 'a good one', instructions: 'body' } + }) + }) + + it('uses nameOverride instead of the frontmatter name', () => { + const md = '---\nname: frontmatter-name\ndescription: d\n---\n\nbody' + const result = parseAndValidateSkill(md, 'folder-name') + expect(result).toEqual({ + skill: { name: 'folder-name', description: 'd', instructions: 'body' } + }) + }) + + it('validates the nameOverride, not the frontmatter name', () => { + const md = '---\nname: valid-frontmatter\ndescription: d\n---\n\nbody' + expect(parseAndValidateSkill(md, 'Bad Folder!')).toEqual({ + error: `name "Bad Folder!" must only contain lowercase letters, digits or '-'` + }) + }) + + it('does not fall back to the frontmatter name for an empty override', () => { + const md = '---\nname: has-name\ndescription: d\n---\n\nbody' + expect(parseAndValidateSkill(md, '')).toEqual({ error: 'name is required' }) + }) + + it('returns an error for a missing description', () => { + expect(parseAndValidateSkill('---\nname: s\n---\n\nbody')).toEqual({ + error: 'description is required' + }) + }) + + it('returns an error for an empty body', () => { + expect(parseAndValidateSkill('---\nname: s\ndescription: d\n---\n\n')).toEqual({ + error: 'body is required' + }) + }) +}) + +describe('buildSkillMd', () => { + it('round-trips a skill through parseSkillMd', () => { + const skill = { name: 'round-trip', description: 'desc', instructions: '# Body\n\ntext' } + const parsed = parseSkillMd(buildSkillMd(skill)) + expect(parsed.name).toBe(skill.name) + expect(parsed.description).toBe(skill.description) + expect(parsed.instructions).toBe(skill.instructions) + }) + + it('quotes descriptions with YAML-special characters so they round-trip', () => { + const skill = { + name: 'colon-desc', + description: 'value: with colon, #hash and : more', + instructions: 'body' + } + const parsed = parseSkillMd(buildSkillMd(skill)) + expect(parsed.description).toBe(skill.description) + expect(validateSkill(skill)).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/workspaceSettings/aiSkills.ts b/frontend/src/lib/components/workspaceSettings/aiSkills.ts new file mode 100644 index 0000000000..aeb148f7c1 --- /dev/null +++ b/frontend/src/lib/components/workspaceSettings/aiSkills.ts @@ -0,0 +1,101 @@ +import YAML from 'yaml' +import { z } from 'zod' + +export type SkillUpload = { name: string; description: string; instructions: string } + +// `name` + `description` mirror the Claude SKILL.md spec (counted in characters); +// the body is a byte-bounded payload. Keep these in sync with backend `validate_skill`. +export const MAX_SKILL_NAME_LENGTH = 64 +export const MAX_SKILL_DESCRIPTION_LENGTH = 1_024 +export const MAX_SKILL_INSTRUCTIONS_LENGTH = 64 * 1024 + +const textEncoder = new TextEncoder() + +// Single source of truth for skill field validation, shared by the paste/edit +// modal and the folder importer. Lengths are code-point / byte bounded to match +// the backend, so `.refine` (not `.max`, which counts UTF-16 units) is used. +export const skillSchema = z.object({ + name: z + .string() + .min(1, 'name is required') + .refine( + (v) => [...v].length <= MAX_SKILL_NAME_LENGTH, + `name is longer than ${MAX_SKILL_NAME_LENGTH} characters` + ) + .refine((v) => /^[a-z0-9-]+$/.test(v), { + error: (iss) => + `name ${JSON.stringify(iss.input)} must only contain lowercase letters, digits or '-'` + }), + description: z + .string() + .min(1, 'description is required') + .refine( + (v) => [...v].length <= MAX_SKILL_DESCRIPTION_LENGTH, + `description is longer than ${MAX_SKILL_DESCRIPTION_LENGTH} characters` + ), + instructions: z + .string() + .min(1, 'body is required') + .refine( + (v) => textEncoder.encode(v).byteLength <= MAX_SKILL_INSTRUCTIONS_LENGTH, + `body is longer than ${MAX_SKILL_INSTRUCTIONS_LENGTH} bytes` + ) +}) + +/** First validation error for a skill, or `undefined` if it is valid. */ +export function validateSkill(skill: SkillUpload): string | undefined { + const result = skillSchema.safeParse(skill) + return result.success ? undefined : result.error.issues[0]?.message +} + +/** Split a SKILL.md into its frontmatter `name`/`description` and the markdown body. */ +export function parseSkillMd(raw: string): { + name: string | undefined + description: string | undefined + instructions: string +} { + const text = raw.replace(/^/, '') + const fm = /^---\s*\r?\n([\s\S]*?)\r?\n---\s*\r?\n?/.exec(text) + if (!fm) { + return { name: undefined, description: undefined, instructions: text.trim() } + } + let name: string | undefined + let description: string | undefined + try { + const data = YAML.parse(fm[1]) ?? {} + if (typeof data?.name === 'string') name = data.name.trim() + if (typeof data?.description === 'string') description = data.description.trim() + } catch { + // Malformed frontmatter — fall through so the skill is reported as invalid + // rather than silently dropped. + } + return { name, description, instructions: text.slice(fm[0].length).trim() } +} + +/** + * Parse a SKILL.md and validate it in one step. `nameOverride` lets the folder + * importer supply the skill name from its containing folder instead of the + * frontmatter. Returns the validated skill or the first error message. + */ +export function parseAndValidateSkill( + raw: string, + nameOverride?: string +): { skill: SkillUpload } | { error: string } { + const parsed = parseSkillMd(raw) + const candidate: SkillUpload = { + name: nameOverride ?? parsed.name ?? '', + description: parsed.description ?? '', + instructions: parsed.instructions + } + const error = validateSkill(candidate) + return error ? { error } : { skill: candidate } +} + +/** Reconstruct a SKILL.md from its stored parts for editing/rendering. */ +export function buildSkillMd(skill: SkillUpload): string { + const frontmatter = YAML.stringify({ + name: skill.name, + description: skill.description + }).trimEnd() + return `---\n${frontmatter}\n---\n\n${skill.instructions}\n` +} diff --git a/frontend/src/lib/components/workspaceSettings/datatableMigrationUtils.ts b/frontend/src/lib/components/workspaceSettings/datatableMigrationUtils.ts new file mode 100644 index 0000000000..ed046418f2 --- /dev/null +++ b/frontend/src/lib/components/workspaceSettings/datatableMigrationUtils.ts @@ -0,0 +1,30 @@ +import { WorkspaceService, type DatatableMigrationWithStatus } from '$lib/gen' + +/** + * Migrations that are defined but not yet applied. A newly-created migration + * always gets the highest timestamp, so every pending migration is "earlier": + * running the new one on its own would apply it ahead of them (out of order). + */ +export function pendingMigrations( + migrations: DatatableMigrationWithStatus[] +): DatatableMigrationWithStatus[] { + return migrations.filter((m) => m.status !== 'ran') +} + +/** Fetch the data table's migration status and return the pending ones. */ +export async function fetchPendingMigrations( + workspace: string, + datatableName: string +): Promise { + const { migrations } = await WorkspaceService.getDatatableMigrationsStatus({ + workspace, + datatableName + }) + return pendingMigrations(migrations) +} + +/** Confirmation copy shown before running a just-created migration ahead of + * `count` still-pending earlier ones (mirrors the row-level Run warning). */ +export function outOfOrderRunMessage(count: number): string { + return `${count} earlier migration(s) have not been run yet. This migration might depend on them. Run it anyway?` +} diff --git a/frontend/src/lib/components/workspaceSettings/utils.svelte.ts b/frontend/src/lib/components/workspaceSettings/utils.svelte.ts index 3d5d70afec..6925be9c04 100644 --- a/frontend/src/lib/components/workspaceSettings/utils.svelte.ts +++ b/frontend/src/lib/components/workspaceSettings/utils.svelte.ts @@ -7,3 +7,31 @@ export let isCustomInstanceDbEnabled = derived( [superadmin], ([superadmin_]) => superadmin_ && !isCloudHosted() ) + +// Postgres caps identifiers at 63 bytes; the backend rejects longer db names. +const MAX_INSTANCE_DB_NAME_LEN = 63 + +// Builds a default instance database name scoped to the workspace (e.g. `dt_myworkspace`), +// appending `_1`, `_2`... until an unused name is found. Workspace ids may contain hyphens, +// which are not valid in unquoted postgres identifiers, so they are replaced with underscores. +// The result is truncated to keep it within the postgres identifier length limit. +export function getUnusedInstanceDbName( + prefix: string, + workspaceId: string, + usedNames: Iterable +): string { + const used = new Set(usedNames) + const base = `${prefix}_${workspaceId.toLowerCase().replace(/-/g, '_')}`.slice( + 0, + MAX_INSTANCE_DB_NAME_LEN + ) + if (!used.has(base)) return base + let i = 1 + let candidate: string + do { + const suffix = `_${i}` + candidate = base.slice(0, MAX_INSTANCE_DB_NAME_LEN - suffix.length) + suffix + i++ + } while (used.has(candidate)) + return candidate +} diff --git a/frontend/src/lib/components/workspaceSettings/utils.test.ts b/frontend/src/lib/components/workspaceSettings/utils.test.ts new file mode 100644 index 0000000000..68115e6a62 --- /dev/null +++ b/frontend/src/lib/components/workspaceSettings/utils.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'vitest' + +import { getUnusedInstanceDbName } from './utils.svelte' + +describe('getUnusedInstanceDbName', () => { + it('scopes the name to the workspace with the given prefix', () => { + expect(getUnusedInstanceDbName('dt', 'myworkspace', [])).toBe('dt_myworkspace') + expect(getUnusedInstanceDbName('dl', 'myworkspace', [])).toBe('dl_myworkspace') + }) + + it('lowercases and replaces hyphens with underscores', () => { + expect(getUnusedInstanceDbName('dt', 'My-Team', [])).toBe('dt_my_team') + }) + + it('appends an incrementing suffix when the name is already used', () => { + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc'])).toBe('dt_abc_1') + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc', 'dt_abc_1'])).toBe('dt_abc_2') + }) + + it('skips over already-used suffixed names', () => { + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc', 'dt_abc_2'])).toBe('dt_abc_1') + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc', 'dt_abc_1', 'dt_abc_2'])).toBe( + 'dt_abc_3' + ) + }) + + it('accepts any iterable of used names', () => { + expect(getUnusedInstanceDbName('dt', 'abc', new Set(['dt_abc']))).toBe('dt_abc_1') + }) + + it('truncates the base name to the postgres 63-char identifier limit', () => { + const longId = 'w'.repeat(100) + const name = getUnusedInstanceDbName('dt', longId, []) + expect(name.length).toBe(63) + expect(name.startsWith('dt_')).toBe(true) + }) + + it('keeps the result within 63 chars even when appending a suffix', () => { + const longId = 'w'.repeat(100) + const base = getUnusedInstanceDbName('dt', longId, []) // length 63 + const name = getUnusedInstanceDbName('dt', longId, [base]) + expect(name.length).toBeLessThanOrEqual(63) + expect(name.endsWith('_1')).toBe(true) + }) +}) diff --git a/frontend/src/lib/editorLangUtils.ts b/frontend/src/lib/editorLangUtils.ts index 3152eff946..5706ad8bcb 100644 --- a/frontend/src/lib/editorLangUtils.ts +++ b/frontend/src/lib/editorLangUtils.ts @@ -85,6 +85,22 @@ export function extToLang(ext: string) { return 'graphql' case 'css': return 'css' + case 'scss': + return 'scss' + case 'less': + return 'less' + case 'html': + case 'htm': + return 'html' + case 'md': + case 'markdown': + return 'markdown' + case 'xml': + return 'xml' + case 'svg': + return 'xml' + case 'txt': + return 'plaintext' case 'yml': return 'ansible' case 'cs': diff --git a/frontend/src/lib/infer.ts b/frontend/src/lib/infer.ts index 7c8b83a64f..a70dd7c745 100644 --- a/frontend/src/lib/infer.ts +++ b/frontend/src/lib/infer.ts @@ -68,6 +68,7 @@ import wasmUrlWac from 'windmill-parser-wasm-wac/windmill_parser_wasm_bg.wasm?ur import { workspaceStore } from './stores.js' import { argSigToJsonSchemaType } from 'windmill-utils-internal' import { type AssetWithAccessType } from './components/assets/lib.js' +import { type ColumnLineage } from './components/assets/AssetGraph/parsePipelineAnnotations' const loadSchemaLastRun = writable< | [ @@ -169,6 +170,10 @@ type InferAssetsResult = assets: AssetWithAccessType[] sql_queries?: InferAssetsSqlQueryDetails[] columns?: Record + // Body-inferred column lineage (DuckDB SQL AST). Present once the + // `windmill-parser-wasm-asset` package is rebuilt with the inference; + // the spread below already forwards it from the parser output. + column_lineage?: ColumnLineage[] } | { status: 'error' diff --git a/frontend/src/lib/mcpEndpointTools.ts b/frontend/src/lib/mcpEndpointTools.ts index c5bc08d606..8a46163009 100644 --- a/frontend/src/lib/mcpEndpointTools.ts +++ b/frontend/src/lib/mcpEndpointTools.ts @@ -17,25 +17,53 @@ export interface EndpointTool { export const mcpEndpointTools: EndpointTool[] = [ { - name: "queryDocumentation", - description: "query Windmill AI documentation assistant (EE only)", + name: "searchDocs", + description: "Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more.", instructions: "", - path: "/inkeep", - method: "POST", + path: "/docs/search", + method: "GET", pathParamsSchema: undefined, - queryParamsSchema: undefined, - bodySchema: { + queryParamsSchema: { "type": "object", "properties": { "query": { "type": "string", - "description": "The documentation query to send to the AI assistant" + "description": "Keywords to search for in the documentation body, e.g. \"chromium worker tag\" or \"retry exponential backoff\". Fewer, more distinctive words match better." } }, "required": [ "query" ] }, + bodySchema: undefined, + pathFieldRenames: undefined, + queryFieldRenames: undefined, + bodyFieldRenames: undefined + }, + { + name: "readDocsPage", + description: "Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section.", + instructions: "", + path: "/docs/page", + method: "GET", + pathParamsSchema: undefined, + queryParamsSchema: { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted." + }, + "section": { + "type": "string", + "description": "Optional. A heading title from the page outline to read just that section instead of the full page." + } + }, + "required": [ + "url" + ] +}, + bodySchema: undefined, pathFieldRenames: undefined, queryFieldRenames: undefined, bodyFieldRenames: undefined @@ -94,6 +122,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -179,6 +210,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the variable (body parameter)" @@ -220,6 +254,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "include_encrypted": { "type": "boolean", "description": "ask to include the encrypted value if secret and decrypt secret is not true (default: false)\n" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -270,6 +308,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft variables whose path has no\ndeployed variable. Synthesized rows carry `draft_only: true`\nso the home page can render a \"Draft\" badge.\n" } }, "required": [] @@ -319,6 +361,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -393,6 +438,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the resource (body parameter)" @@ -424,7 +472,16 @@ export const mcpEndpointTools: EndpointTool[] = [ "path" ] }, - queryParamsSchema: undefined, + queryParamsSchema: { + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +}, bodySchema: undefined, pathFieldRenames: undefined, queryFieldRenames: undefined, @@ -479,6 +536,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft resources whose path has\nno deployed resource. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] @@ -728,6 +789,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -856,6 +921,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -1193,6 +1262,14 @@ export const mcpEndpointTools: EndpointTool[] = [ "language" ] } + }, + "temp_script_refs": { + "type": "object", + "nullable": true, + "description": "Map of relative-import script path -> temp storage hash so the preview job resolves those imports from not-yet-deployed local content instead of the deployed script", + "additionalProperties": { + "type": "string" + } } }, "required": [ @@ -1465,6 +1542,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "boolean", "description": "filter on successful jobs" }, + "status": { + "type": "string", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.. Possible values: success, failure, canceled, skipped" + }, "all_workspaces": { "type": "boolean", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)" @@ -1473,6 +1554,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "boolean", "description": "is not a scheduled job" }, + "excludes_entrypoint_override": { + "type": "boolean", + "description": "exclude jobs that were started with a `_ENTRYPOINT_OVERRIDE` arg (e.g. dynamic-select helper runs and preprocessor previews)" + }, "broad_filter": { "type": "string", "description": "broad search across multiple fields (case-insensitive substring match on path, tag, schedule path, trigger kind, label)" @@ -1511,6 +1596,10 @@ export const mcpEndpointTools: EndpointTool[] = [ }, "no_code": { "type": "boolean" + }, + "approval_token": { + "type": "string", + "description": "Approval token granting read access to the job when not logged in. The token must be the one issued for this job's flow (i.e. the flow id used when generating the approval URL)." } }, "required": [] @@ -1565,7 +1654,7 @@ export const mcpEndpointTools: EndpointTool[] = [ "properties": { "path": { "type": "string", - "description": "The unique path identifier for this schedule" + "description": "The unique Windmill path for this schedule. Must be of the form `u//` or `f//`." }, "schedule": { "type": "string", @@ -2005,7 +2094,16 @@ export const mcpEndpointTools: EndpointTool[] = [ "path" ] }, - queryParamsSchema: undefined, + queryParamsSchema: { + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +}, bodySchema: undefined, pathFieldRenames: undefined, queryFieldRenames: undefined, @@ -2064,6 +2162,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft schedules whose path has\nno deployed schedule. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] diff --git a/frontend/src/lib/navigation.test.ts b/frontend/src/lib/navigation.test.ts new file mode 100644 index 0000000000..b6233d44f8 --- /dev/null +++ b/frontend/src/lib/navigation.test.ts @@ -0,0 +1,52 @@ +import { describe, it, expect } from 'vitest' +import { buildFilterUrl } from './navigation' + +// Parse the (un-based) app path the builder returns against a dummy origin so we can +// assert on pathname / searchParams / hash without caring about ordering. +function parse(appPath: string): URL { + return new URL(appPath, 'http://x') +} + +describe('buildFilterUrl', () => { + it('drops keys not in validKeys, and nullish/empty values', () => { + const u = parse( + buildFilterUrl( + '/runs', + { + status: 'failure', + path: 'f/foo/bar', + bogus: 'x', + user: '', + tag: null, + worker: undefined + }, + { validKeys: ['status', 'path', 'user', 'tag', 'worker'] } + ) + ) + expect(u.pathname).toBe('/runs') + expect(u.searchParams.get('status')).toBe('failure') + expect(u.searchParams.get('path')).toBe('f/foo/bar') + expect(u.searchParams.has('bogus')).toBe(false) // not in validKeys + expect(u.searchParams.has('user')).toBe(false) // empty string dropped + expect(u.searchParams.has('tag')).toBe(false) // null dropped + expect(u.searchParams.has('worker')).toBe(false) // undefined dropped + }) + + it('keeps every key when no validKeys are given', () => { + const u = parse(buildFilterUrl('/runs', { anything: 'yes' })) + expect(u.searchParams.get('anything')).toBe('yes') + }) + + it('appends a hash and no query when there are no params', () => { + const u = parse(buildFilterUrl('/schedules', {}, { hash: 'f/a/b' })) + expect(u.pathname).toBe('/schedules') + expect(u.search).toBe('') + expect(u.hash).toBe('#f/a/b') + }) + + it('combines params and hash', () => { + const u = parse(buildFilterUrl('/schedules', { path: 'f/x' }, { hash: 'f/a/b' })) + expect(u.searchParams.get('path')).toBe('f/x') + expect(u.hash).toBe('#f/a/b') + }) +}) diff --git a/frontend/src/lib/navigation.ts b/frontend/src/lib/navigation.ts index 96545718b2..fd3a35800e 100644 --- a/frontend/src/lib/navigation.ts +++ b/frontend/src/lib/navigation.ts @@ -1,5 +1,6 @@ import { goto as svelteGoto } from '$app/navigation' import { base as svelteBase } from '$app/paths' +import { serializeParam } from '$lib/svelte5UtilsKit.svelte' export function goto(path: string, options = {}) { if (svelteBase == '' || path.startsWith('?')) { @@ -10,6 +11,31 @@ export function goto(path: string, options = {}) { } } +/** + * Build an in-app deep-link to `pathname` with query-param filters, encoded exactly + * as the pages write them (via `serializeParam`) so `useUrlSyncedFilterInstance` + * round-trips them back into filter state. Nullish/empty values are dropped; when + * `validKeys` is provided, unknown keys are dropped too (structured output guarantees + * shape, not truth). Returns an un-prefixed app path — pass it to `goto`, which adds + * the SvelteKit base. + */ +export function buildFilterUrl( + pathname: string, + values: Record, + opts?: { validKeys?: Iterable; hash?: string } +): string { + const allow = opts?.validKeys ? new Set(opts.validKeys) : undefined + const sp = new URLSearchParams() + for (const [key, value] of Object.entries(values)) { + if (value === undefined || value === null || value === '') continue + if (allow && !allow.has(key)) continue + sp.set(key, serializeParam(value)) + } + const qs = sp.toString() + const hash = opts?.hash ? `#${opts.hash}` : '' + return qs ? `${pathname}?${qs}${hash}` : `${pathname}${hash}` +} + export async function setQuery( url: URL, key: string, diff --git a/frontend/src/lib/path.ts b/frontend/src/lib/path.ts index 98840803fe..8b07a10a13 100644 --- a/frontend/src/lib/path.ts +++ b/frontend/src/lib/path.ts @@ -2,10 +2,10 @@ import { get } from 'svelte/store' import { ScriptService } from './gen' import { workspaceStore } from './stores' -export async function findNextAvailablePath(path: string): Promise { +export async function findNextAvailablePath(path: string, workspace?: string): Promise { try { await ScriptService.getScriptByPath({ - workspace: get(workspaceStore)!, + workspace: workspace ?? get(workspaceStore)!, path }) @@ -17,7 +17,7 @@ export async function findNextAvailablePath(path: string): Promise { path = `${path}_${Number(version) + 1}` - return findNextAvailablePath(path) + return findNextAvailablePath(path, workspace) } catch (e) { // Catching an error means the path is available return path diff --git a/frontend/src/lib/rawAppDeploy.ts b/frontend/src/lib/rawAppDeploy.ts index 3439cb6b6d..b13951c5f8 100644 --- a/frontend/src/lib/rawAppDeploy.ts +++ b/frontend/src/lib/rawAppDeploy.ts @@ -75,7 +75,11 @@ export async function deployRawAppDraft( summary, policy, deployment_message: deploymentMessage, - custom_path: isAdmin ? (value.custom_path ?? '') : undefined + custom_path: isAdmin ? (value.custom_path ?? '') : undefined, + // Preserve the policy's on_behalf_of: this draft-deploy path has no + // on-behalf-of selector, so without the flag the backend resets it to + // the deploying user (gated server-side by can_preserve_on_behalf_of). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined }, js: bundle.js, css: bundle.css @@ -91,7 +95,9 @@ export async function deployRawAppDraft( summary, policy, deployment_message: deploymentMessage, - custom_path: value.custom_path + custom_path: value.custom_path, + // Preserve the policy's on_behalf_of (see update branch above). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined }, js: bundle.js, css: bundle.css diff --git a/frontend/src/lib/scripts.ts b/frontend/src/lib/scripts.ts index f1518f4368..bf88dafb9b 100644 --- a/frontend/src/lib/scripts.ts +++ b/frontend/src/lib/scripts.ts @@ -189,7 +189,12 @@ export function processLangs(selected: string | undefined, langs: string[]): str export const defaultScriptLanguages = Object.fromEntries(scriptLanguagesArray) -export async function getScriptByPath(path: string): Promise<{ +export async function getScriptByPath( + path: string, + // The acting workspace when called from a session live editor; defaults to + // the navigation workspace for full-page callers. + workspace?: string +): Promise<{ content: string language: SupportedLanguage schema: any @@ -216,7 +221,7 @@ export async function getScriptByPath(path: string): Promise<{ } } else { const script = await ScriptService.getScriptByPath({ - workspace: get(workspaceStore)!, + workspace: workspace ?? get(workspaceStore)!, path: path ?? '' }) return { @@ -234,9 +239,9 @@ export async function getScriptByPath(path: string): Promise<{ } } -export async function getLatestHashForScript(path: string): Promise { +export async function getLatestHashForScript(path: string, workspace?: string): Promise { const script = await ScriptService.getScriptByPath({ - workspace: get(workspaceStore)!, + workspace: workspace ?? get(workspaceStore)!, path: path ?? '' }) return script.hash diff --git a/frontend/src/lib/stores.ts b/frontend/src/lib/stores.ts index 92c8f18946..0449796d21 100644 --- a/frontend/src/lib/stores.ts +++ b/frontend/src/lib/stores.ts @@ -27,9 +27,12 @@ export interface UserExt { groups: string[] pgroups: string[] folders: string[] + folders_read: string[] folders_owners: string[] is_service_account?: boolean impersonating_email?: string + // true when the user is a superadmin viewing a workspace they are not a member of + non_member?: boolean } export interface UserWorkspace { @@ -39,6 +42,8 @@ export interface UserWorkspace { color?: string operator_settings?: OperatorSettings parent_workspace_id?: string | null + is_dev_workspace?: boolean + dev_workspace_label?: string | null disabled: boolean } @@ -185,6 +190,10 @@ export interface SQLSchema { schema: SQLBaseSchema publicOnly: boolean | undefined stringified: string + /** MySQL only: the connection's default database (`DATABASE()`), surfaced by the + * introspection script. Lets the table picker render the default db's tables + * unprefixed even when the connection can also see other (non-system) schemas. */ + defaultDb?: string } export interface GraphqlSchema { diff --git a/frontend/src/lib/svelte5UtilsKit.svelte.ts b/frontend/src/lib/svelte5UtilsKit.svelte.ts index 921d41614e..139d6985df 100644 --- a/frontend/src/lib/svelte5UtilsKit.svelte.ts +++ b/frontend/src/lib/svelte5UtilsKit.svelte.ts @@ -9,7 +9,7 @@ export type SearchParamsResult = : z.infer & Record /** Serialize a value to a URL search param string. Primitives are written as-is; anything else is JSON. */ -function serializeParam(value: unknown): string { +export function serializeParam(value: unknown): string { if (typeof value === 'string') return value if (typeof value === 'number') return String(value) if (typeof value === 'boolean') return String(value) diff --git a/frontend/src/lib/userDraft.svelte.ts b/frontend/src/lib/userDraft.svelte.ts index 8f7942b9e4..8437a1b87c 100644 --- a/frontend/src/lib/userDraft.svelte.ts +++ b/frontend/src/lib/userDraft.svelte.ts @@ -204,6 +204,7 @@ const DRAFT_COMPARE_IGNORED_FIELDS = [ 'edited_by', 'workspace_id', 'version_id', + 'parent_version', 'is_draft' ] as const diff --git a/frontend/src/lib/userDraftDbMigration.ts b/frontend/src/lib/userDraftDbMigration.ts index 824cd1c952..4c3079ea94 100644 --- a/frontend/src/lib/userDraftDbMigration.ts +++ b/frontend/src/lib/userDraftDbMigration.ts @@ -1,11 +1,12 @@ /** * One-off migration from the localStorage UserDraft autosave to the - * DB-backed `draft` table. Runs after `migrateLegacyUserDrafts` (which - * produces the `userdraft/w/{workspace}/{kind}/{path}` keys this reads), - * POSTing each to `/drafts/update` and clearing the source key only on - * success — so it's idempotent without a sentinel; failed entries retry next - * mount. Not workspace-gated: keys embed their own workspace and the token - * covers all of them, so gating would orphan other-workspace entries. + * DB-backed `draft` table. Reads the workspace-scoped + * `userdraft/w/{workspace}/{kind}/{path}` keys (written by the editor during + * the interim LS-backed phase, so the embedded workspace is correct), POSTing + * each to `/drafts/update` and clearing the source key only on success — so + * it's idempotent without a sentinel; failed entries retry next mount. Not + * workspace-gated: keys embed their own workspace and the token covers all of + * them, so gating would orphan other-workspace entries. * * Before uploading, each draft is compared against its deployed version * (script / flow / app); a draft that's deep-equal to what's deployed carries diff --git a/frontend/src/lib/userDraftLegacyMigration.test.ts b/frontend/src/lib/userDraftLegacyMigration.test.ts index 7069929897..f7963e55a0 100644 --- a/frontend/src/lib/userDraftLegacyMigration.test.ts +++ b/frontend/src/lib/userDraftLegacyMigration.test.ts @@ -1,6 +1,6 @@ import { describe, it, expect, beforeEach } from 'vitest' import { - migrateLegacyUserDrafts, + purgeLegacyUserDrafts, __resetUserDraftLegacyMigrationForTesting } from './userDraftLegacyMigration' @@ -8,18 +8,12 @@ function encodeLegacy(value: unknown): string { return btoa(encodeURIComponent(JSON.stringify(value))) } -function wrapped(value: V): string { - return JSON.stringify({ value }) -} - -// Read a migrated entry, strip the GC `lastWrittenAt` stamp so assertions -// can match the `{ value }` shape regardless of when the migration ran. -function storedShape(key: string): string | null { - const raw = localStorage.getItem(key) - if (raw == null) return null - const parsed = JSON.parse(raw) - delete parsed.lastWrittenAt - return JSON.stringify(parsed) +const legacyApp = { + grid: [], + fullscreen: false, + theme: undefined, + unusedInlineScripts: [], + hiddenInlineScripts: [] } beforeEach(() => { @@ -27,197 +21,110 @@ beforeEach(() => { __resetUserDraftLegacyMigrationForTesting() }) -describe('migrateLegacyUserDrafts', () => { - it('migrates a legacy app draft to the workspace-scoped key with a { value } wrapper', () => { - // Shape mirrors what the legacy AppEditor wrote: `encodeState($appStore)`, - // i.e. the inner App value, not the wrapping AppWithLastVersion. - const legacyApp = { - grid: [], - fullscreen: false, - theme: undefined, - unusedInlineScripts: [], - hiddenInlineScripts: [] - } +describe('purgeLegacyUserDrafts', () => { + it('drops a recognised legacy app draft without re-creating it under any key', () => { localStorage.setItem('app-u/me/dashboard', encodeLegacy(legacyApp)) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app-u/me/dashboard')).toBeNull() - expect(storedShape('userdraft/w/main/app/u/me/dashboard')).toBe(wrapped(legacyApp)) + // The workspace-blind key is gone, NOT promoted to a guessed workspace. + expect(localStorage.getItem('userdraft/w/main/app/u/me/dashboard')).toBeNull() }) - it('migrates a legacy empty-path app draft (the `app` literal key)', () => { - const legacyApp = { - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - } + it('drops the empty-path legacy keys (`app` / `flow` / `rawapp` literals)', () => { localStorage.setItem('app', encodeLegacy(legacyApp)) + localStorage.setItem('flow', encodeLegacy({ flow: { summary: '', value: { modules: [] } } })) + localStorage.setItem('rawapp', encodeLegacy({ files: {}, runnables: {}, data: {} })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app')).toBeNull() - expect(storedShape('userdraft/w/main/app/')).toBe(wrapped(legacyApp)) + expect(localStorage.getItem('flow')).toBeNull() + expect(localStorage.getItem('rawapp')).toBeNull() }) - it('migrates a legacy flow draft and strips the view-state envelope', () => { - const flow = { summary: 'f', value: { modules: [] }, path: 'u/me/myflow' } - const legacyBundle = { - flow, - path: 'u/me/myflow', - selectedId: 'settings', - draft_triggers: [{ id: 't1' }], - selected_trigger: null, - loadedFromHistory: undefined - } - localStorage.setItem('flow-u/me/myflow', encodeLegacy(legacyBundle)) + it('drops recognised legacy flow and raw-app drafts', () => { + localStorage.setItem( + 'flow-u/me/myflow', + encodeLegacy({ flow: { summary: 'f', value: { modules: [] } }, selectedId: 'settings' }) + ) + localStorage.setItem( + 'rawapp-u/me/site', + encodeLegacy({ files: { 'index.tsx': 'x' }, runnables: {}, data: {} }) + ) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('flow-u/me/myflow')).toBeNull() - // Only the inner Flow survives; the view-state envelope is dropped. - expect(storedShape('userdraft/w/main/flow/u/me/myflow')).toBe(wrapped(flow)) - }) - - it('migrates a legacy raw-app draft, defaulting the new `summary` field', () => { - const legacy = { - files: { 'index.tsx': 'export default () => null' }, - runnables: {}, - data: { tables: [] } - } - localStorage.setItem('rawapp-u/me/site', encodeLegacy(legacy)) - - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('rawapp-u/me/site')).toBeNull() - expect(storedShape('userdraft/w/main/raw_app/u/me/site')).toBe( - wrapped({ ...legacy, summary: '' }) - ) }) - it('preserves an existing new-format entry instead of overwriting it', () => { - // Old and new both exist for the same item — the new one is presumed - // fresher. - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - const existingNew = wrapped({ value: 'new' }) - localStorage.setItem('userdraft/w/main/app/u/me/dash', existingNew) + it('leaves the workspace-scoped interim keys untouched (migrateUserDraftsToDb owns those)', () => { + const interim = JSON.stringify({ value: { modules: [] } }) + localStorage.setItem('userdraft/w/main/flow/u/me/keep', interim) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('app-u/me/dash')).toBeNull() - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBe(existingNew) - }) - - it('is idempotent — the second invocation is a no-op', () => { - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).not.toBeNull() - - // Drop the migrated entry to detect any re-migration attempt. - localStorage.removeItem('userdraft/w/main/app/u/me/dash') - // Drop the source too, so re-running couldn't even find a source. - // (The sentinel alone should be enough; this just clarifies the intent.) - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBeNull() - }) - - it('skips entirely when no workspace is available', () => { - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - migrateLegacyUserDrafts('') - - expect(localStorage.getItem('app-u/me/dash')).not.toBeNull() - }) - - it('handles malformed legacy payloads without throwing', () => { - localStorage.setItem('app-u/me/garbled', 'not-base64!!!') - expect(() => migrateLegacyUserDrafts('main')).not.toThrow() - // Migration didn't migrate, didn't crash — leaves the entry alone. - expect(localStorage.getItem('app-u/me/garbled')).toBe('not-base64!!!') + expect(localStorage.getItem('userdraft/w/main/flow/u/me/keep')).toBe(interim) }) it('leaves keys whose path does not match the legacy `u|f/owner/name` shape alone', () => { - // A future feature or neighbouring code might pick a key like - // `app-recent` for its own purposes. The path doesn't look like a - // Windmill item path, so the migration must skip it. + // `app-recent` / `app-some_other_app` look like the legacy prefix but the + // suffix isn't a Windmill item path — a future feature might own them. localStorage.setItem('app-recent', 'whatever') localStorage.setItem('app-some_other_app', 'whatever') - // `flow-u/me/foo` matches the shape and would be migrated, but the - // payload also needs to look like a Windmill draft (asserted below). - localStorage.setItem('flow-u/me/foo', encodeLegacy({ flow: { value: { modules: [] } } })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app-recent')).toBe('whatever') expect(localStorage.getItem('app-some_other_app')).toBe('whatever') - expect(localStorage.getItem('userdraft/w/main/flow/u/me/foo')).not.toBeNull() }) - it('skips legacy-shaped keys whose payload does not look like a Windmill draft', () => { - // `app-u/me/dash` matches LEGACY_PATH_SHAPE and decodes to valid JSON, - // but none of the App-shape fields (grid/fullscreen/theme/ - // unusedInlineScripts/hiddenInlineScripts) are present. Treat it as - // unrelated and leave it untouched. - const unrelated = encodeLegacy({ random: 'data', count: 7 }) - localStorage.setItem('app-u/me/dash', unrelated) + it('leaves legacy-shaped keys whose payload does not look like a Windmill draft', () => { + // Matches LEGACY_PATH_SHAPE and decodes to valid JSON, but carries none of + // the App/flow draft fields — treat as unrelated, do not delete. + const unrelatedApp = encodeLegacy({ random: 'data', count: 7 }) + localStorage.setItem('app-u/me/dash', unrelatedApp) const unrelatedFlow = encodeLegacy({ stepsState: {} }) localStorage.setItem('flow-u/me/bar', unrelatedFlow) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('app-u/me/dash')).toBe(unrelated) - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBeNull() + expect(localStorage.getItem('app-u/me/dash')).toBe(unrelatedApp) expect(localStorage.getItem('flow-u/me/bar')).toBe(unrelatedFlow) - expect(localStorage.getItem('userdraft/w/main/flow/u/me/bar')).toBeNull() }) - it('migrates multiple legacy entries in a single invocation', () => { - localStorage.setItem( - 'app-u/me/a', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) + it('leaves a malformed (non-base64) legacy payload alone and does not throw', () => { + localStorage.setItem('app-u/me/garbled', 'not-base64!!!') + + expect(() => purgeLegacyUserDrafts()).not.toThrow() + expect(localStorage.getItem('app-u/me/garbled')).toBe('not-base64!!!') + }) + + it('is idempotent — once the sentinel is set, a later legacy key survives', () => { + localStorage.setItem('app-u/me/a', encodeLegacy(legacyApp)) + purgeLegacyUserDrafts() + expect(localStorage.getItem('app-u/me/a')).toBeNull() + + // A key written after the first run is NOT swept (the sentinel short-circuits). + localStorage.setItem('app-u/me/b', encodeLegacy(legacyApp)) + purgeLegacyUserDrafts() + expect(localStorage.getItem('app-u/me/b')).not.toBeNull() + }) + + it('purges multiple legacy entries in a single invocation', () => { + localStorage.setItem('app-u/me/a', encodeLegacy(legacyApp)) localStorage.setItem( 'flow-u/me/b', - encodeLegacy({ flow: { summary: '', value: { modules: [] }, path: 'u/me/b' } }) - ) - localStorage.setItem( - 'rawapp-u/me/c', - encodeLegacy({ files: {}, runnables: {}, data: { tables: [] } }) + encodeLegacy({ flow: { summary: '', value: { modules: [] } } }) ) + localStorage.setItem('rawapp-u/me/c', encodeLegacy({ files: {}, runnables: {}, data: {} })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('userdraft/w/main/app/u/me/a')).not.toBeNull() - expect(localStorage.getItem('userdraft/w/main/flow/u/me/b')).not.toBeNull() - expect(localStorage.getItem('userdraft/w/main/raw_app/u/me/c')).not.toBeNull() + expect(localStorage.getItem('app-u/me/a')).toBeNull() + expect(localStorage.getItem('flow-u/me/b')).toBeNull() + expect(localStorage.getItem('rawapp-u/me/c')).toBeNull() }) }) diff --git a/frontend/src/lib/userDraftLegacyMigration.ts b/frontend/src/lib/userDraftLegacyMigration.ts index d4ee04ee61..4239bc0e8a 100644 --- a/frontend/src/lib/userDraftLegacyMigration.ts +++ b/frontend/src/lib/userDraftLegacyMigration.ts @@ -1,22 +1,29 @@ /** - * One-off migration from the pre-UserDraft localStorage autosave entries to - * the workspace-scoped `userdraft/w/{ws}/{kind}/{path}` format. + * One-shot purge of the pre-UserDraft browser-local autosave keys. * - * Legacy keys (global, not workspace-scoped — assumed to belong to the user's - * current workspace at migration time): + * The original autosave (pre-#9121) wrote workspace-BLIND keys: * - * `flow` / `flow-{path}` base64 of `encodeState({ flow, path, selectedId, draft_triggers, ... })` - * `app` / `app-{path}` base64 of `encodeState(App)` - * `rawapp` / `rawapp-{path}` base64 of `encodeState({ files, runnables, data })` + * `flow` / `flow-{path}` base64 of `encodeState({ flow, path, selectedId, draft_triggers, ... })` + * `app` / `app-{path}` base64 of `encodeState(App)` + * `rawapp` / `rawapp-{path}` base64 of `encodeState({ files, runnables, data })` * - * Target keys: `userdraft/w/{workspace}/{flow|app|raw_app}/{path}` storing - * `JSON.stringify({ value: })`. + * Neither the key nor the decoded value records a workspace (the value carries + * only workspace-agnostic item paths like `u/me/x`), so these drafts cannot be + * attributed to the workspace they were edited in. The current editors are + * DB-backed and never read these keys, so they are dead data with one dangerous + * property: promoting them to the DB would force a GUESS of the workspace, which + * mis-files drafts into whatever workspace happened to be active when the + * migration first ran (a single global sentinel gates it). We therefore drop + * them instead of migrating them. + * + * Only keys that BOTH match the legacy path shape AND decode to a plausible + * legacy draft are removed; unrelated look-alikes (`app-recent`, garbage, + * non-Windmill payloads) are left untouched. The workspace-scoped interim keys + * (`userdraft/w/{ws}/...`, written by the editor with the correct workspace) + * are NOT touched here — `migrateUserDraftsToDb` still pushes those to the DB. * * Idempotent: writes a sentinel under `MIGRATION_FLAG` after the first run so - * subsequent invocations are no-ops. Existing new-format entries are never - * overwritten — when both an old and a new entry exist for the same item, the - * old one is simply dropped on the assumption that the new entry is the more - * recent edit. + * subsequent invocations are no-ops. * * This file is intentionally standalone — it does not import from * `userDraft.svelte.ts` so the new code stays uncluttered by the legacy @@ -71,10 +78,9 @@ function decodeLegacyState(raw: string): unknown { * Per-kind shape gate. The legacy keys (`app-foo`, `flow-foo`, ...) are * unusual enough that nothing else in the codebase has used them, but * matching `LEGACY_PATH_SHAPE` doesn't prove the payload is actually a - * Windmill draft (any base64-of-JSON could pass). Promoting a stray payload - * would silently surface as a phantom "Restored from local storage" toast - * on the next edit, so we reject anything that doesn't carry the fields the - * legacy writers actually produced. + * Windmill draft (any base64-of-JSON could pass). We only delete keys we can + * positively recognise as legacy drafts, so a stray look-alike that happens to + * use this key shape is left untouched rather than silently dropped. */ function isPlausibleLegacyValue(kind: LegacyKind, decoded: unknown): boolean { if (decoded == null || typeof decoded !== 'object') return false @@ -103,32 +109,6 @@ function isPlausibleLegacyValue(kind: LegacyKind, decoded: unknown): boolean { } } -function transformLegacyValue(kind: LegacyKind, decoded: unknown): unknown { - const obj = decoded as Record - switch (kind) { - case 'flow': - // The legacy bundle wrapped the Flow alongside view-state fields - // (selectedId, draft_triggers, ...). The new entry stores only the - // Flow — the view-state lives elsewhere or is re-derived. - return obj.flow - case 'app': - // Legacy stored the App directly. - return obj - case 'raw_app': - // Legacy bundle missed the `summary` field that the new editor adds. - return { - files: obj.files ?? {}, - runnables: obj.runnables ?? {}, - data: obj.data ?? {}, - summary: typeof obj.summary === 'string' ? obj.summary : '' - } - } -} - -function newKey(workspace: string, kind: LegacyKind, path: string): string { - return `userdraft/w/${workspace}/${kind}/${path}` -} - function listLocalStorageKeys(): string[] { const out: string[] = [] for (let i = 0; i < localStorage.length; i++) { @@ -139,16 +119,11 @@ function listLocalStorageKeys(): string[] { } /** - * Run the legacy → new-format migration. Idempotent: returns immediately if a - * previous run completed (signalled by `MIGRATION_FLAG`). - * - * The migration is workspace-scoped because the legacy keys had no notion of - * workspace — we treat the caller's current workspace as the owner of any - * surviving legacy entries. + * Remove the workspace-blind legacy autosave keys (see file header). Idempotent: + * returns immediately if a previous run completed (signalled by `MIGRATION_FLAG`). */ -export function migrateLegacyUserDrafts(workspace: string): void { +export function purgeLegacyUserDrafts(): void { if (typeof localStorage === 'undefined') return - if (!workspace) return if (localStorage.getItem(MIGRATION_FLAG) !== null) return try { @@ -157,32 +132,18 @@ export function migrateLegacyUserDrafts(workspace: string): void { if (!match) continue const raw = localStorage.getItem(key) if (raw == null) continue - - try { - const decoded = decodeLegacyState(raw) - if (!isPlausibleLegacyValue(match.newKind, decoded)) continue - const value = transformLegacyValue(match.newKind, decoded) - const target = newKey(workspace, match.newKind, match.path) - if (value !== undefined && localStorage.getItem(target) == null) { - // `lastWrittenAt` makes the migrated entry visible to - // `gcUserDrafts`. We stamp it as "now" so a freshly-migrated - // autosave gets the full retention window — sweeping it - // immediately on the first GC pass would lose work the - // legacy migration just rescued. - localStorage.setItem(target, JSON.stringify({ value, lastWrittenAt: Date.now() })) - } - localStorage.removeItem(key) - } catch (e) { - console.error('UserDraft legacy migration: failed to migrate', key, e) - } + // Only drop keys we can positively recognise as legacy Windmill + // drafts; leave unrelated or unparseable look-alikes in place. + if (!isPlausibleLegacyValue(match.newKind, decodeLegacyState(raw))) continue + localStorage.removeItem(key) } localStorage.setItem(MIGRATION_FLAG, new Date().toISOString()) } catch (e) { - console.error('UserDraft legacy migration: aborted', e) + console.error('UserDraft legacy purge: aborted', e) } } -/** Test-only: clear the sentinel so the migration can re-run. */ +/** Test-only: clear the sentinel so the purge can re-run. */ export function __resetUserDraftLegacyMigrationForTesting(): void { try { localStorage.removeItem(MIGRATION_FLAG) diff --git a/frontend/src/lib/utils.test.ts b/frontend/src/lib/utils.test.ts index 1b2b72a306..accdf73b15 100644 --- a/frontend/src/lib/utils.test.ts +++ b/frontend/src/lib/utils.test.ts @@ -1,5 +1,42 @@ import { describe, it, expect } from 'vitest' -import { cleanValueProperties, getQueryStmtCountHeuristic } from './utils' +import { + cleanValueProperties, + computeSharableHash, + extractTagFromSharableHash, + isDynamicTag, + getQueryStmtCountHeuristic, + parseDbInputFromAssetSyntax +} from './utils' + +describe('parseDbInputFromAssetSyntax', () => { + it('parses a table path', () => { + expect(parseDbInputFromAssetSyntax('ducklake://main/orders')).toEqual({ + type: 'ducklake', + ducklake: 'main', + specificTable: 'orders', + specificSchema: undefined + }) + }) + + it('parses a schema-qualified table path', () => { + expect(parseDbInputFromAssetSyntax('ducklake://main/analytics.orders')).toEqual({ + type: 'ducklake', + ducklake: 'main', + specificTable: 'orders', + specificSchema: 'analytics' + }) + }) + + it('handles a catalog-only path without throwing (no table segment)', () => { + // e.g. `// materialize ducklake` → `ducklake://main` — must not throw. + expect(parseDbInputFromAssetSyntax('ducklake://main')).toEqual({ + type: 'ducklake', + ducklake: 'main', + specificTable: undefined, + specificSchema: undefined + }) + }) +}) describe('getQueryStmtCountHeuristic', () => { describe('basic statements', () => { @@ -356,3 +393,61 @@ describe('cleanValueProperties', () => { expect(input).toHaveProperty('created_at') }) }) + +describe('computeSharableHash / extractTagFromSharableHash', () => { + function roundTrip(hash: string) { + const params = new URLSearchParams(hash) + const tag = extractTagFromSharableHash(params) + const args = Object.fromEntries([...params.entries()].map(([k, v]) => [k, JSON.parse(v)])) + return { tag, args } + } + + it('carries the tag under the reserved __tag key alongside JSON-encoded args', () => { + const hash = computeSharableHash({ name: 'world' }, 'my-custom-tag') + expect(roundTrip(hash)).toEqual({ tag: 'my-custom-tag', args: { name: 'world' } }) + }) + + it('omits __tag when no tag is given', () => { + const hash = computeSharableHash({ name: 'world' }) + expect(roundTrip(hash)).toEqual({ tag: undefined, args: { name: 'world' } }) + }) + + it('preserves an arg named __tag instead of misreading it as a tag', () => { + const hash = computeSharableHash({ __tag: 'value', name: 'world' }) + expect(roundTrip(hash)).toEqual({ tag: undefined, args: { __tag: 'value', name: 'world' } }) + }) + + it('carries a tag alongside an arg named __tag, preserving both', () => { + const hash = computeSharableHash({ __tag: 'value', name: 'world' }, 'my-custom-tag') + expect(roundTrip(hash)).toEqual({ + tag: 'my-custom-tag', + args: { __tag: 'value', name: 'world' } + }) + }) + + it('carries JSON-parseable tags like 123 or true without corrupting args', () => { + expect(roundTrip(computeSharableHash({ name: 'world' }, '123'))).toEqual({ + tag: '123', + args: { name: 'world' } + }) + expect(roundTrip(computeSharableHash({}, 'true'))).toEqual({ tag: 'true', args: {} }) + }) + + it('carries a tag that itself starts with the value prefix', () => { + expect(roundTrip(computeSharableHash({}, 't:odd'))).toEqual({ tag: 't:odd', args: {} }) + }) +}) + +describe('isDynamicTag', () => { + it('detects args interpolation placeholders', () => { + expect(isDynamicTag('worker-$args[env]')).toBe(true) + }) + + it('is false for plain tags, $workspace-only tags, and undefined', () => { + expect(isDynamicTag('gpu-heavy')).toBe(false) + // $workspace resolves identically on a re-run, so pinning the resolved value is fine + expect(isDynamicTag('$workspace-gpu')).toBe(false) + expect(isDynamicTag('')).toBe(false) + expect(isDynamicTag(undefined)).toBe(false) + }) +}) diff --git a/frontend/src/lib/utils.ts b/frontend/src/lib/utils.ts index 9e14e7ab9a..b2f8e0d346 100644 --- a/frontend/src/lib/utils.ts +++ b/frontend/src/lib/utils.ts @@ -1137,8 +1137,19 @@ export function extractCustomProperties(styleStr: string): string { return customStyleStr } -export function computeSharableHash(args: any) { - let nargs = {} +// Value prefix marking the reserved `__tag` key as a carried tag: no JSON-encoded +// arg value can start with `t:` (JSON strings start with `"`, numbers with a digit +// or `-`, etc.), so it cannot be confused with a genuine arg named `__tag` +const SHARABLE_HASH_TAG_PREFIX = 't:' + +// `tag` is carried as the reserved key `__tag` with a SHARABLE_HASH_TAG_PREFIX value; +// entry pairs allow a duplicate `__tag` key so an arg with that name can coexist with +// the carried tag (they are told apart by the value prefix) +export function computeSharableHash(args: any, tag?: string) { + let entries: [string, string][] = [] + if (tag) { + entries.push(['__tag', SHARABLE_HASH_TAG_PREFIX + tag]) + } for (let k in args) { let v = args[k] if (v !== undefined) { @@ -1148,11 +1159,11 @@ export function computeSharableHash(args: any) { console.error(`Value at key ${k} too big (${size}) to be shared`) return '' } - nargs[k] = JSON.stringify(v) + entries.push([k, JSON.stringify(v)]) } } try { - let r = new URLSearchParams(nargs).toString() + let r = new URLSearchParams(entries).toString() return r.length > 1000000 ? '' : r } catch (e) { console.error('Error computing sharable hash', e) @@ -1160,6 +1171,32 @@ export function computeSharableHash(args: any) { } } +// `$args[...]` tags are resolved by the backend at push time from the run's args; a +// job's stored tag is the resolved value, so re-running with it would pin a value +// that no longer matches edited args. `$workspace` is also interpolated but resolves +// identically on a re-run (same workspace), so it does not make a tag dynamic here. +export function isDynamicTag(tag: string | undefined): boolean { + return !!tag && tag.includes('$args[') +} + +// Counterpart of computeSharableHash's `tag`: extracts and removes the carried tag. +// Only SHARABLE_HASH_TAG_PREFIX-prefixed `__tag` values are carried tags; any other +// `__tag` value is a genuine arg with that name and is left in `params` for arg parsing. +export function extractTagFromSharableHash(params: URLSearchParams): string | undefined { + const values = params.getAll('__tag') + const carried = values.find((v) => v.startsWith(SHARABLE_HASH_TAG_PREFIX)) + if (carried == undefined) { + return undefined + } + params.delete('__tag') + for (const v of values) { + if (!v.startsWith(SHARABLE_HASH_TAG_PREFIX)) { + params.append('__tag', v) + } + } + return carried.slice(SHARABLE_HASH_TAG_PREFIX.length) +} + export function toCamel(s: string) { return s.replace(/([-_][a-z])/gi, ($1) => { return $1.toUpperCase().replace('-', '').replace('_', '') @@ -1190,8 +1227,10 @@ export function isCodeInjection(expr: string | undefined): boolean { // app logic via the `query` context. Only params we actually own are listed // here — the `wm_` prefix is a naming convention, not a reserved namespace, so // we don't strip it wholesale (that would break apps reading their own `wm_*` -// params). `wm_coep` is a transport flag for cross-origin isolation headers. -export const WINDMILL_RESERVED_QUERY_PARAMS = new Set(['wm_coep']) +// params). `wm_coep` is a transport flag for cross-origin isolation headers; +// `wm_embed`/`wm_embedder_origin` are the opaque app viewer transport params +// (see PublicAppFrame). +export const WINDMILL_RESERVED_QUERY_PARAMS = new Set(['wm_coep', 'wm_embed', 'wm_embedder_origin']) export function urlParamsToObject( params: URLSearchParams, @@ -2118,22 +2157,27 @@ export function pick(obj: T, keys: readonly export function parseDbInputFromAssetSyntax(path: string): DbInput | null { const [p1, _p2] = path.split('://') - const [p2, _p3] = _p2.split('/') - const [p3, p4] = _p3.split('.') + const [p2, _p3] = (_p2 ?? '').split('/') + // `_p3` is undefined for a catalog-only path (e.g. `ducklake://main`, no + // table segment) — guard the split so the helper returns a table-less input + // instead of throwing. + const [p3, p4] = (_p3 ?? '').split('.') + const specificTable = p4 || p3 || undefined + const specificSchema = p4 ? p3 : undefined return p1 === 'ducklake' ? { type: 'ducklake', ducklake: p2 || 'main', - specificTable: p4 ?? p3, - specificSchema: p4 ? p3 : undefined + specificTable, + specificSchema } : p1 === 'datatable' ? { type: 'database', resourcePath: `datatable://${p2 || 'main'}`, resourceType: 'postgresql', - specificTable: p4 ?? p3, - specificSchema: p4 ? p3 : undefined + specificTable, + specificSchema } : null } diff --git a/frontend/src/lib/utils/devWorkspaceLabel.ts b/frontend/src/lib/utils/devWorkspaceLabel.ts new file mode 100644 index 0000000000..026ec9a746 --- /dev/null +++ b/frontend/src/lib/utils/devWorkspaceLabel.ts @@ -0,0 +1,25 @@ +// Cosmetic display label for a dev workspace. The paired-fork machinery is unchanged; this only +// swaps the badge text and identity wording so a team can present the environment as "staging" +// instead of "dev". A null/unknown stored value renders as "dev" (the default). + +export type DevWorkspaceLabelKey = 'dev' | 'staging' + +/** Resolve the stored `dev_workspace_label` to a known key; anything unset/unknown is 'dev'. */ +export function devLabelKey(label: string | null | undefined): DevWorkspaceLabelKey { + return label === 'staging' ? 'staging' : 'dev' +} + +/** Short badge text: 'dev' or 'stg'. */ +export function devBadgeText(label: string | null | undefined): string { + return devLabelKey(label) === 'staging' ? 'stg' : 'dev' +} + +/** Capitalized word for identity wording, e.g. `${devLabelWord(l)} workspace of X`. */ +export function devLabelWord(label: string | null | undefined): string { + return devLabelKey(label) === 'staging' ? 'Staging' : 'Dev' +} + +/** Lowercase noun phrase for prose, e.g. "made in its ${devLabelNoun(l)}". */ +export function devLabelNoun(label: string | null | undefined): string { + return devLabelKey(label) === 'staging' ? 'staging workspace' : 'dev workspace' +} diff --git a/frontend/src/lib/utils/editInFork.ts b/frontend/src/lib/utils/editInFork.ts index aa6d653961..c446732b7d 100644 --- a/frontend/src/lib/utils/editInFork.ts +++ b/frontend/src/lib/utils/editInFork.ts @@ -1,22 +1,89 @@ import { base } from '$lib/base' +import { get } from 'svelte/store' +import { + userStore, + userWorkspaces, + workspaceStore, + type UserWorkspace, + type UserExt +} from '$lib/stores' +import { findCanonicalDevWorkspace } from '$lib/utils/workspaceHierarchy' +import { isRuleActive, canUserBypassRuleKind } from '$lib/workspaceProtectionRules.svelte' type ItemType = 'script' | 'flow' | 'app' | 'raw_app' -export function buildForkEditUrl(itemType: ItemType, itemPath: string): string { - let editPath: string +/** + * Whether to show the "edit in fork / dev workspace" affordance. Allowed when forking isn't disabled, + * when the user can bypass the forking rule (workspace admins, mirroring `canCreateFork`), OR when the + * current workspace has a canonical dev to route to — routing into an existing dev workspace creates + * no fork, so it survives a locked prod that has `DisableWorkspaceForking` set. User identity is read + * non-reactively (it's stable within a session); reactivity comes from the workspace args. + */ +export function editInForkAllowed( + currentWorkspaceId: string | undefined, + allWorkspaces: UserWorkspace[] +): boolean { + return ( + !isRuleActive('DisableWorkspaceForking') || + canUserBypassRuleKind('DisableWorkspaceForking', get(userStore)) || + !!findCanonicalDevWorkspace(currentWorkspaceId, allWorkspaces) + ) +} + +/** Label for the affordance: "Edit in " when routed to a canonical dev, else "Edit in fork". */ +export function editInForkLabel( + currentWorkspaceId: string | undefined, + allWorkspaces: UserWorkspace[] +): string { + const dev = findCanonicalDevWorkspace(currentWorkspaceId, allWorkspaces) + return dev ? `Edit in ${dev.name}` : 'Edit in fork' +} + +/** + * Whether the user may CREATE a new fork of the current workspace: forking not disabled, or the user + * can bypass the rule (workspace admins). Keeps the "Fork workspace" entry available to admins as the + * last-resort escape hatch on a locked prod. + */ +export function canCreateFork(user: UserExt | undefined): boolean { + return ( + !isRuleActive('DisableWorkspaceForking') || + canUserBypassRuleKind('DisableWorkspaceForking', user) + ) +} + +function editPathFor(itemType: ItemType, itemPath: string): string { switch (itemType) { case 'script': - editPath = `${base}/scripts/edit/${itemPath}` - break + return `${base}/scripts/edit/${itemPath}` case 'flow': - editPath = `${base}/flows/edit/${itemPath}` - break + return `${base}/flows/edit/${itemPath}` case 'app': - editPath = `${base}/apps/edit/${itemPath}` - break + return `${base}/apps/edit/${itemPath}` case 'raw_app': - editPath = `${base}/apps_raw/edit/${itemPath}` - break + return `${base}/apps_raw/edit/${itemPath}` } - return `${base}/user/fork_workspace?rd=${encodeURIComponent(editPath)}` +} + +function viewPathFor(itemType: ItemType, itemPath: string): string { + switch (itemType) { + case 'script': + return `${base}/scripts/get/${itemPath}` + case 'flow': + return `${base}/flows/get/${itemPath}` + case 'app': + return `${base}/apps/get/${itemPath}` + case 'raw_app': + return `${base}/apps_raw/get/${itemPath}` + } +} + +export function buildForkEditUrl(itemType: ItemType, itemPath: string): string { + // When the current ("prod") workspace has a canonical dev workspace, edits are funneled there: + // land on the item's page in the dev workspace (not straight in the editor) so the workspace + // switch is legible and the user opens the editor deliberately from there. + const dev = findCanonicalDevWorkspace(get(workspaceStore), get(userWorkspaces)) + if (dev) { + return `${viewPathFor(itemType, itemPath)}?workspace=${encodeURIComponent(dev.id)}` + } + return `${base}/user/fork_workspace?rd=${encodeURIComponent(editPathFor(itemType, itemPath))}` } diff --git a/frontend/src/lib/utils/forkColor.ts b/frontend/src/lib/utils/forkColor.ts new file mode 100644 index 0000000000..232acbbe90 --- /dev/null +++ b/frontend/src/lib/utils/forkColor.ts @@ -0,0 +1,50 @@ +// Derives the fork-chip accent palette from a workspace color: the default +// blue's light/dark bg+text profile, re-hued. Lightness is fixed per role so +// any picked hue stays readable; saturation follows the user color within a +// safe band so grayish picks yield grayish chips. + +function hexToHsl(hex: string): { h: number; s: number; l: number } | undefined { + const m = /^#?([0-9a-f]{3}|[0-9a-f]{6})$/i.exec(hex.trim()) + if (!m) return undefined + let c = m[1] + if (c.length === 3) c = [...c].map((x) => x + x).join('') + const r = parseInt(c.slice(0, 2), 16) / 255 + const g = parseInt(c.slice(2, 4), 16) / 255 + const b = parseInt(c.slice(4, 6), 16) / 255 + const max = Math.max(r, g, b) + const min = Math.min(r, g, b) + const l = (max + min) / 2 + const d = max - min + if (d === 0) return { h: 0, s: 0, l } + const s = d / (1 - Math.abs(2 * l - 1)) + let h: number + if (max === r) h = ((g - b) / d + (g < b ? 6 : 0)) * 60 + else if (max === g) h = ((b - r) / d + 2) * 60 + else h = ((r - g) / d + 4) * 60 + return { h, s, l } +} + +function clamp(x: number, lo: number, hi: number): number { + return Math.min(hi, Math.max(lo, x)) +} + +function hsl(h: number, s: number, l: number): string { + return `hsl(${Math.round(h)}, ${Math.round(s * 100)}%, ${Math.round(l * 100)}%)` +} + +/** + * Inline-style string setting the four `--fork-accent-*` custom properties a + * colored fork chip consumes (see WorkspaceScopeTrigger). Returns undefined + * for an unparsable color, letting the chip fall back to the default accent. + */ +export function forkAccentStyle(color: string | undefined): string | undefined { + if (!color) return undefined + const parsed = hexToHsl(color) + if (!parsed) return undefined + const { h, s } = parsed + const lightBg = hsl(h, clamp(s, 0.3, 1), 0.955) + const lightText = hsl(h, clamp(s, 0.3, 0.8), 0.42) + const darkBg = hsl(h, clamp(s * 0.35, 0.08, 0.25), 0.26) + const darkText = hsl(h, clamp(s, 0.3, 0.9), 0.86) + return `--fork-accent-bg: ${lightBg}; --fork-accent-text: ${lightText}; --fork-accent-bg-dark: ${darkBg}; --fork-accent-text-dark: ${darkText};` +} diff --git a/frontend/src/lib/utils/placementFly.ts b/frontend/src/lib/utils/placementFly.ts new file mode 100644 index 0000000000..397f9787fb --- /dev/null +++ b/frontend/src/lib/utils/placementFly.ts @@ -0,0 +1,26 @@ +import { fly, type TransitionConfig } from 'svelte/transition' + +/** + * Fly a melt-positioned floating element in from its anchor. Melt stamps the + * resolved side (after flip/fitViewport) on `data-side`, but only once its + * async computePosition settles — after the intro's config is read. So the + * caller passes its requested `placement` as the intro fallback, and + * `data-side` (read via a deferred config) takes over when present, i.e. on + * the outro and whenever positioning already settled. Missing both falls back + * to the common case (opens below → slides down from the trigger). + */ +export function placementFly( + node: Element, + { + duration = 100, + distance = 16, + placement + }: { duration?: number; distance?: number; placement?: string } = {} +): () => TransitionConfig { + return () => { + const side = node.getAttribute('data-side') ?? placement?.split('-')[0] ?? 'bottom' + const x = side === 'left' ? distance : side === 'right' ? -distance : 0 + const y = side === 'top' ? distance : side === 'left' || side === 'right' ? 0 : -distance + return fly(node, { duration, x, y }) + } +} diff --git a/frontend/src/lib/utils/splitterPointerCapture.ts b/frontend/src/lib/utils/splitterPointerCapture.ts new file mode 100644 index 0000000000..2737375445 --- /dev/null +++ b/frontend/src/lib/utils/splitterPointerCapture.ts @@ -0,0 +1,24 @@ +/** + * svelte-splitpanes tracks drags with document-level mousemove/mouseup only. + * A button release outside the browser window never reaches the document, so + * the drag state sticks: global col-resize cursor and pointer-events:none on + * every pane until the next click. Capturing the pointer on the splitter at + * pointerdown makes the browser deliver the release (and its compatibility + * mouseup, which bubbles to the library's document listener) even when it + * happens outside the window. Attach to any ancestor of the Splitpanes. + */ +export function splitterPointerCapture(node: HTMLElement) { + function onPointerDown(e: PointerEvent) { + const splitter = (e.target as Element | null)?.closest?.('.splitpanes__splitter') + if (splitter instanceof HTMLElement) { + try { + splitter.setPointerCapture(e.pointerId) + } catch { + // Non-capturable pointer (already released) — the plain document + // listeners still handle the in-window case. + } + } + } + node.addEventListener('pointerdown', onPointerDown, true) + return { destroy: () => node.removeEventListener('pointerdown', onPointerDown, true) } +} diff --git a/frontend/src/lib/utils/workspaceHierarchy.ts b/frontend/src/lib/utils/workspaceHierarchy.ts index fb1114362e..e1ff76f934 100644 --- a/frontend/src/lib/utils/workspaceHierarchy.ts +++ b/frontend/src/lib/utils/workspaceHierarchy.ts @@ -19,7 +19,7 @@ export function buildWorkspaceHierarchy(workspaces: UserWorkspace[]): WorkspaceH } // Create maps for quick lookups - const workspaceMap = new Map(workspaces.map(w => [w.id, w])) + const workspaceMap = new Map(workspaces.map((w) => [w.id, w])) const childrenMap = new Map() const hasChildrenSet = new Set() @@ -35,7 +35,7 @@ export function buildWorkspaceHierarchy(workspaces: UserWorkspace[]): WorkspaceH } // Find root workspaces (those without a parent or whose parent is not in the current list) - const rootWorkspaces = workspaces.filter(w => { + const rootWorkspaces = workspaces.filter((w) => { if (!w.parent_workspace_id) { return true // Definitely a root } @@ -46,7 +46,12 @@ export function buildWorkspaceHierarchy(workspaces: UserWorkspace[]): WorkspaceH const result: WorkspaceHierarchyItem[] = [] // Recursively build the hierarchy - function addWorkspaceAndChildren(workspace: UserWorkspace, depth: number, isForked: boolean, parentName?: string) { + function addWorkspaceAndChildren( + workspace: UserWorkspace, + depth: number, + isForked: boolean, + parentName?: string + ) { // Add the current workspace result.push({ workspace, @@ -56,11 +61,14 @@ export function buildWorkspaceHierarchy(workspaces: UserWorkspace[]): WorkspaceH hasChildren: hasChildrenSet.has(workspace.id) }) - // Add its children (sorted by name for consistency) + // Add its children: the canonical dev workspace first, then throwaway forks by name. const children = childrenMap.get(workspace.id) || [] children - .sort((a, b) => a.name.localeCompare(b.name)) - .forEach(child => { + .sort((a, b) => { + if (!!a.is_dev_workspace !== !!b.is_dev_workspace) return a.is_dev_workspace ? -1 : 1 + return a.name.localeCompare(b.name) + }) + .forEach((child) => { addWorkspaceAndChildren(child, depth + 1, true, workspace.name) }) } @@ -68,12 +76,13 @@ export function buildWorkspaceHierarchy(workspaces: UserWorkspace[]): WorkspaceH // Process root workspaces (sorted by name for consistency) rootWorkspaces .sort((a, b) => a.name.localeCompare(b.name)) - .forEach(workspace => { + .forEach((workspace) => { const isRootForked = workspace.parent_workspace_id != null - const parentName = isRootForked && workspace.parent_workspace_id - ? workspace.parent_workspace_id // Use parent ID as fallback if parent not in list - : undefined - + const parentName = + isRootForked && workspace.parent_workspace_id + ? workspace.parent_workspace_id // Use parent ID as fallback if parent not in list + : undefined + addWorkspaceAndChildren(workspace, 0, isRootForked, parentName) }) @@ -95,11 +104,60 @@ export function isRootWorkspace(workspace: UserWorkspace): boolean { return workspace.parent_workspace_id == null } +/** + * Walk up `parent_workspace_id` to the top of a workspace's family. Stops at the first ancestor not + * present in `allWorkspaces` (e.g. a parent the user can't see) and returns it, so the result is + * always the highest reachable ancestor. Returns undefined when the id itself isn't in the list. + */ +export function findWorkspaceRoot( + workspaceId: string | undefined, + allWorkspaces: UserWorkspace[] +): UserWorkspace | undefined { + if (!workspaceId) return undefined + let current = allWorkspaces.find((w) => w.id === workspaceId) + while (current?.parent_workspace_id) { + const parent = allWorkspaces.find((w) => w.id === current!.parent_workspace_id) + if (!parent) break + current = parent + } + return current +} + +/** + * Whether a workspace (by id) is a fork or dev workspace. Forks and dev workspaces both set + * `parent_workspace_id` (a dev workspace has no `wm-fork-` id prefix), but a `wm-fork-` workspace can + * outlive its parent (the parent FK is `ON DELETE SET NULL`), so treat the prefix as fork-ness too — + * otherwise an orphaned fork would lose its fork-only affordances (e.g. owner self-delete). + */ +export function workspaceIsFork( + workspaceId: string | undefined, + allWorkspaces: UserWorkspace[] +): boolean { + if (!workspaceId) return false + if (workspaceId.startsWith('wm-fork-')) return true + return allWorkspaces.find((w) => w.id === workspaceId)?.parent_workspace_id != null +} + +/** + * The canonical dev workspace of a prod workspace, if any (at most one per prod). Used to redirect + * edits from a locked prod workspace into its dev workspace. Disabled dev workspaces are excluded: + * redirecting edits to one the user can't select would be a dead end. + */ +export function findCanonicalDevWorkspace( + prodWorkspaceId: string | undefined, + allWorkspaces: UserWorkspace[] +): UserWorkspace | undefined { + if (!prodWorkspaceId) return undefined + return allWorkspaces.find( + (w) => w.parent_workspace_id === prodWorkspaceId && w.is_dev_workspace && !w.disabled + ) +} + /** * Helper function to find all descendants of a workspace */ export function findWorkspaceDescendants( - workspaceId: string, + workspaceId: string, allWorkspaces: UserWorkspace[] ): UserWorkspace[] { const descendants: UserWorkspace[] = [] @@ -126,4 +184,4 @@ export function findWorkspaceDescendants( collectDescendants(workspaceId) return descendants -} \ No newline at end of file +} diff --git a/frontend/src/lib/utils_deployable.ts b/frontend/src/lib/utils_deployable.ts index 5f49eb5c49..556ac0a3c7 100644 --- a/frontend/src/lib/utils_deployable.ts +++ b/frontend/src/lib/utils_deployable.ts @@ -40,6 +40,8 @@ export type Kind = | 'gcp_trigger' | 'azure_trigger' | 'email_trigger' + // Data table migration, diffed per `/_` path. + | 'datatable_migration' // Legacy generic kind used by the cross-workspace `DeployWorkspace` UI, // which carries the trigger sub-kind in `additionalInformation`. | 'trigger' diff --git a/frontend/src/lib/utils_draft_deploy.test.ts b/frontend/src/lib/utils_draft_deploy.test.ts new file mode 100644 index 0000000000..bae50691cc --- /dev/null +++ b/frontend/src/lib/utils_draft_deploy.test.ts @@ -0,0 +1,40 @@ +import { describe, it, expect } from 'vitest' +import { draftBaseIsStale } from './utils_draft_deploy' + +// draftBaseIsStale compares a draft's base pointer against the deployed head +// of the item it was fetched with (`get_draft=true`). Shared by CompareDrafts +// and the session Edits drawer — a regression here silently hides (or +// fabricates) the "started from an older deployed version" warning. + +describe('draftBaseIsStale', () => { + it('script: stale iff the draft parent_hash differs from the deployed hash', () => { + expect(draftBaseIsStale('script', { hash: 'v2', draft: { parent_hash: 'v1' } })).toBe(true) + expect(draftBaseIsStale('script', { hash: 'v2', draft: { parent_hash: 'v2' } })).toBe(false) + }) + + it('script: no base pointer or no head → not stale (nothing to compare)', () => { + expect(draftBaseIsStale('script', { hash: 'v2', draft: {} })).toBe(false) + expect(draftBaseIsStale('script', { draft: { parent_hash: 'v1' } })).toBe(false) + }) + + it('flow: compares the pinned version_id against the deployed head', () => { + expect(draftBaseIsStale('flow', { version_id: 7, draft: { version_id: 5 } })).toBe(true) + expect(draftBaseIsStale('flow', { version_id: 7, draft: { version_id: 7 } })).toBe(false) + expect(draftBaseIsStale('flow', { version_id: 7, draft: {} })).toBe(false) + }) + + it('app/raw_app: compares parent_version against the last of versions', () => { + expect(draftBaseIsStale('app', { versions: [1, 2, 3], draft: { parent_version: 2 } })).toBe( + true + ) + expect(draftBaseIsStale('raw_app', { versions: [1, 2, 3], draft: { parent_version: 3 } })).toBe( + false + ) + expect(draftBaseIsStale('app', { versions: [], draft: { parent_version: 2 } })).toBe(false) + }) + + it('no draft on the response → not stale', () => { + expect(draftBaseIsStale('script', { hash: 'v2' })).toBe(false) + expect(draftBaseIsStale('script', undefined)).toBe(false) + }) +}) diff --git a/frontend/src/lib/utils_draft_deploy.ts b/frontend/src/lib/utils_draft_deploy.ts index 6781818bb5..ee8538e4db 100644 --- a/frontend/src/lib/utils_draft_deploy.ts +++ b/frontend/src/lib/utils_draft_deploy.ts @@ -37,6 +37,7 @@ import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import type { DeployResult } from '$lib/utils_workspace_deploy' import { TRIGGER_RUNTIME_IGNORE } from '$lib/utils_deployable' import { deployRawAppDraft } from '$lib/rawAppDeploy' +import { canonicalRawAppDiffValue } from '$lib/components/raw_apps/utils' import { invalidateWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' import { setLocalDraftHint } from '$lib/localDraftHints.svelte' import { userStore } from '$lib/stores' @@ -206,9 +207,12 @@ export async function getDraftDiffValues( draft_saved_at: _c, no_deployed: _n, other_drafts_users: _o, + version_id: _v, ...deployed } = r - const draftValue = draft ?? deployed + // Strip the draft's pinned base `version_id` (which differs from the deployed + // head for a stale draft) so it never renders as a spurious diff line. + const { version_id: _dv, ...draftValue } = (draft ?? deployed) as any return { deployed: draftOnly ? EMPTY_DEPLOYED.flow!(draftValue) : deployed, draft: draftValue } } else if (kind === 'app' || kind === 'raw_app') { // A never-deployed raw app has no `app` row; the backend resolves the @@ -219,6 +223,16 @@ export async function getDraftDiffValues( getDraft: true, rawApp: kind === 'raw_app' })) as any + if (kind === 'raw_app' || r.raw_app === true) { + // Raw-app drafts are stored flat (files/runnables/data top-level) while the + // deployed row nests them under `value`, and deployed inline scripts carry + // server-recomputed locks. Canonicalize both onto the same shape with the + // post-deploy noise stripped — the same module the editor's Diff button uses. + return { + deployed: draftOnly ? canonicalRawAppDiffValue({}) : canonicalRawAppDiffValue(r), + draft: canonicalRawAppDiffValue(r.draft ?? r) + } + } const deployed = { summary: r.summary, value: r.value, @@ -226,7 +240,9 @@ export async function getDraftDiffValues( path: r.path, custom_path: r.custom_path } - const draftValue = r.draft ?? deployed + // Strip the draft's pinned fork-base `parent_version` (the deployed allowlist + // above already omits it) so it never renders as a spurious diff line. + const { parent_version: _pv, ...draftValue } = (r.draft ?? deployed) as any return { deployed: draftOnly ? EMPTY_DEPLOYED.app!(draftValue) : deployed, draft: draftValue } } else { // Variables / resources / schedules / triggers: one overlay GET yields @@ -246,6 +262,56 @@ export async function getDraftDiffValues( } } +/** + * Whether a draft's base is stale: the deployed version the draft forked from + * no longer matches the current deployed head — a newer version was deployed + * after the draft began, so deploying the draft would silently revert it. + * Scripts compare the draft's `parent_hash` vs the deployed `hash`; flows the + * pinned `version_id` vs the deployed head `version_id`; apps (incl. raw) the + * pinned `parent_version` vs the head of `versions`. `r` is the item fetched + * with `get_draft=true`; only script/flow/app kinds carry a base pointer. + */ +export function draftBaseIsStale(draftKind: UserDraftItemKind, r: any): boolean { + const draft = r?.draft + if (!draft) return false + if (draftKind === 'script') { + return !!r.hash && !!draft.parent_hash && draft.parent_hash !== r.hash + } + if (draftKind === 'flow') { + return r.version_id != null && draft.version_id != null && draft.version_id !== r.version_id + } + const head = Array.isArray(r.versions) ? r.versions[r.versions.length - 1] : undefined + return head != null && draft.parent_version != null && draft.parent_version !== head +} + +/** Fetch-and-test wrapper over `draftBaseIsStale` for one draft item. Returns + * false for kinds without a base pointer and on fetch errors (warn, not block). */ +export async function fetchDraftBaseStale( + draftKind: UserDraftItemKind, + path: string, + workspace: string +): Promise { + try { + if (draftKind === 'script') { + const r = await ScriptService.getScriptByPath({ workspace, path, getDraft: true }) + return draftBaseIsStale(draftKind, r) + } + if (draftKind === 'flow') { + const r = await FlowService.getFlowByPath({ workspace, path, getDraft: true }) + return draftBaseIsStale(draftKind, r) + } + if (draftKind === 'app' || draftKind === 'raw_app') { + // The apps endpoint auto-detects a raw app and overlays its draft. + const r = await AppService.getAppByPath({ workspace, path, getDraft: true }) + return draftBaseIsStale(draftKind, r) + } + return false + } catch (e) { + console.error(`Stale-draft check failed for ${draftKind}:${path}`, e) + return false + } +} + /** * Deploy a script/flow draft's trigger changes the same way the editors do. * Scripts and flows can carry `draft_triggers`; the create/update call below @@ -276,9 +342,9 @@ export async function deployDraft( kind: DraftKind, path: string, workspace: string, - draftOnly = false, - rawApp = false + opts: { draftOnly?: boolean; rawApp?: boolean; deploymentMessage?: string } = {} ): Promise { + const { draftOnly = false, rawApp = false, deploymentMessage } = opts try { if (kind === 'raw_app' || (kind === 'app' && rawApp)) { // Raw apps bundle their source files and deploy via the raw-app @@ -286,7 +352,7 @@ export async function deployDraft( // `kind === 'app'` + `rawApp` (editor). Must route here: the // visual-app branch would `updateApp` with no `value` (RawAppDraft // has none) and silently drop the draft's files. - await deployRawAppDraft(workspace, path) + await deployRawAppDraft(workspace, path, deploymentMessage) } else if (kind === 'script') { const r = (await ScriptService.getScriptByPath({ workspace, path, getDraft: true })) as any const d = r.draft ?? r @@ -297,7 +363,12 @@ export async function deployDraft( // the editor: createScript at the new path with parent_hash links lineage). await ScriptService.createScript({ workspace, - requestBody: { ...rest, path: scriptPath, parent_hash: r.hash } + requestBody: { + ...rest, + path: scriptPath, + parent_hash: r.hash, + deployment_message: deploymentMessage + } }) // Then deploy any draft trigger edits, so they aren't dropped with the draft. await deployDraftTriggers(draftTriggers, workspace, scriptPath, true) @@ -319,7 +390,8 @@ export async function deployDraft( ws_error_handler_muted: d.ws_error_handler_muted, visible_to_runner_only: d.visible_to_runner_only, on_behalf_of_email: d.on_behalf_of_email, - labels: d.labels + labels: d.labels, + deployment_message: deploymentMessage } // Draft-only flows have NO flow row (they live solely in the // draft table), so they deploy via createFlow; a draft on a @@ -353,14 +425,20 @@ export async function deployDraft( // undefined so the backend preserves the existing route. The draft has no // custom_path, so admins fall back to the deployed route (`''` when none). const isAdmin = !!(get(userStore)?.is_admin || get(userStore)?.is_super_admin) + const policy = r.policy ?? { execution_mode: 'publisher' } const requestBody = { value: appValue, summary: draftSummary ?? r.summary ?? '', - policy: r.policy ?? { execution_mode: 'publisher' }, + policy, // Honor the draft's intended path; `draft_path` holds the user-typed path // for a never-deployed app parked at a `u/{user}/draft_{uuid}` storage key. path: draftPath ?? r.path ?? path, - custom_path: isAdmin ? (r.custom_path ?? '') : undefined + custom_path: isAdmin ? (r.custom_path ?? '') : undefined, + deployment_message: deploymentMessage, + // The draft carries no on-behalf-of selector — the policy comes straight + // from the deployed app. Preserve its on_behalf_of (the backend resets it + // to the deploying user without this flag, gated by can_preserve_on_behalf_of). + preserve_on_behalf_of: policy?.on_behalf_of ? true : undefined } // Same as flows: draft-only apps have no app row → create; // drafts on a deployed app update it. diff --git a/frontend/src/lib/utils_workspace_deploy.ts b/frontend/src/lib/utils_workspace_deploy.ts index bde0003882..72400bb962 100644 --- a/frontend/src/lib/utils_workspace_deploy.ts +++ b/frontend/src/lib/utils_workspace_deploy.ts @@ -14,9 +14,15 @@ import { ScheduleService, ScriptService, SqsTriggerService, + UserService, VariableService, - WebsocketTriggerService + WebsocketTriggerService, + WorkspaceService } from '$lib/gen' +import { + fetchProtectionRulesForWorkspace, + canUserBypassRuleKindInRulesets +} from '$lib/workspaceProtectionRules.svelte' import { existsTrigger, getTriggersDeployData, @@ -229,7 +235,11 @@ function makeProvider(): DeployProvider { getSchedule: (p) => ScheduleService.getSchedule(p), createSchedule: (p) => ScheduleService.createSchedule(p), updateSchedule: (p) => ScheduleService.updateSchedule(p), - deleteSchedule: (p) => ScheduleService.deleteSchedule(p) + deleteSchedule: (p) => ScheduleService.deleteSchedule(p), + // Datatable migrations + listDatatableMigrations: (p) => WorkspaceService.listDatatableMigrations(p), + upsertDatatableMigration: (p) => WorkspaceService.upsertDatatableMigration(p), + deleteDatatableMigration: (p) => WorkspaceService.deleteDatatableMigration(p) } } @@ -382,3 +392,42 @@ export async function getOnBehalfOf( } return sharedGetOnBehalfOf(makeProvider(), kind as DeployKind, path, workspace) } + +export type DeployPermission = { ok: boolean; reason?: string } + +/** + * Whether the current user may deploy into `workspace`. Mirrors the server-side + * deploy authorization (`check_user_against_rule` in windmill-common) so the UI + * can disable the action with a reason instead of letting the click 403: + * - operators can never deploy; + * - when the `RestrictDeployToDeployers` protection rule is active, only + * admins, `wm_deployers` members (implicitly), and per-ruleset bypass + * users/groups may deploy. + * Fails open on any error — the server still enforces on the actual deploy. + * Shared by the session dock and the compare page so both gate identically. + */ +export async function checkDeployPermission(workspace: string): Promise { + try { + const me = await UserService.whoami({ workspace }) + if (me.operator) { + return { ok: false, reason: "You're an operator in this workspace — operators can't deploy" } + } + // Admins and wm_deployers members always satisfy RestrictDeployToDeployers + // (the backend allows wm_deployers implicitly, so check it before the + // per-ruleset bypass_users/bypass_groups fallback). + const isDeployer = me.is_admin || (me.groups ?? []).includes('wm_deployers') + if (!isDeployer) { + const rulesets = await fetchProtectionRulesForWorkspace(workspace) + const userInfo = { is_admin: !!me.is_admin, username: me.username, groups: me.groups ?? [] } + if (!canUserBypassRuleKindInRulesets(rulesets, 'RestrictDeployToDeployers', userInfo)) { + return { + ok: false, + reason: 'Only workspace admins and members of wm_deployers can deploy here' + } + } + } + return { ok: true } + } catch { + return { ok: true } + } +} diff --git a/frontend/src/lib/workspaceProtectionRules.svelte.ts b/frontend/src/lib/workspaceProtectionRules.svelte.ts index fa9627ba02..1774ed60e2 100644 --- a/frontend/src/lib/workspaceProtectionRules.svelte.ts +++ b/frontend/src/lib/workspaceProtectionRules.svelte.ts @@ -1,6 +1,11 @@ import { WorkspaceService, type ProtectionRuleset, type ProtectionRuleKind } from './gen' import type { UserExt } from './stores' +// The slice of the user identity the bypass checks read — structural, so +// callers can pass a whoami response (normalised groups) as well as the +// UserExt store value. +export type RuleBypassUser = Pick + /** * Internal reactive state using Svelte 5 $state rune */ @@ -94,7 +99,7 @@ export async function fetchProtectionRulesForWorkspace( * @param userInfo The user information * @returns true if user can bypass (is admin, in bypass_users, or has group in bypass_groups) */ -export function canUserBypassRule(ruleset: ProtectionRuleset, userInfo: UserExt): boolean { +export function canUserBypassRule(ruleset: ProtectionRuleset, userInfo: RuleBypassUser): boolean { // Admin always bypasses if (userInfo.is_admin) { return true @@ -134,7 +139,7 @@ export function isRuleActive(ruleKind: ProtectionRuleKind): boolean { */ export function canUserBypassRuleKind( ruleKind: ProtectionRuleKind, - userInfo: UserExt | undefined + userInfo: RuleBypassUser | undefined ): boolean { // If no user info, default to permissive if (!userInfo) { @@ -187,7 +192,7 @@ export function isRuleActiveInRulesets( export function canUserBypassRuleKindInRulesets( rulesets: ProtectionRuleset[], ruleKind: ProtectionRuleKind, - userInfo: UserExt | undefined + userInfo: RuleBypassUser | undefined ): boolean { // If no user info, default to not allowing bypass if (!userInfo) { diff --git a/frontend/src/routes/(root)/(logged)/+layout.svelte b/frontend/src/routes/(root)/(logged)/+layout.svelte index fa9bf2122d..8bd5119945 100644 --- a/frontend/src/routes/(root)/(logged)/+layout.svelte +++ b/frontend/src/routes/(root)/(logged)/+layout.svelte @@ -12,8 +12,12 @@ WorkspaceService } from '$lib/gen' import { capitalize, classNames, getModifierKey, sendUserToast } from '$lib/utils' + import { useLocalStorageValue } from '$lib/svelte5Utils.svelte' import WorkspaceMenu from '$lib/components/sidebar/WorkspaceMenu.svelte' import SidebarContent from '$lib/components/sidebar/SidebarContent.svelte' + import SettingsMenu from '$lib/components/sidebar/SettingsMenu.svelte' + import SidebarScrollArea from '$lib/components/sidebar/SidebarScrollArea.svelte' + import { SIDEBAR_BG, SIDEBAR_BG_DARK } from '$lib/components/sidebar/sidebarChrome' import CriticalAlertModal from '$lib/components/sidebar/CriticalAlertModal.svelte' import ForkConflictModal from '$lib/components/ForkConflictModal.svelte' import { @@ -38,6 +42,7 @@ import CenteredModal from '$lib/components/CenteredModal.svelte' import { afterNavigate, beforeNavigate } from '$app/navigation' import { goto } from '$lib/navigation' + import { registerToolDisplayActionHandler } from '$lib/components/copilot/chat/createdResourceActions.svelte' import UserSettings from '$lib/components/UserSettings.svelte' import SuperadminSettings from '$lib/components/SuperadminSettings.svelte' import WindmillIcon from '$lib/components/icons/WindmillIcon.svelte' @@ -50,23 +55,31 @@ import { SUPERADMIN_SETTINGS_HASH, USER_SETTINGS_HASH } from '$lib/components/sidebar/settings' import { isCloudHosted } from '$lib/cloud' import { syncTutorialsTodos } from '$lib/tutorialUtils' - import { ArrowLeft, Search, WandSparkles } from 'lucide-svelte' + import { PanelLeftClose, PanelLeftOpen, Home, Play, Search, WandSparkles } from 'lucide-svelte' import { getUserExt } from '$lib/user' import { deepEqual } from 'fast-equals' import { twMerge } from 'tailwind-merge' import OperatorMenu from '$lib/components/sidebar/OperatorMenu.svelte' import GlobalSearchModal from '$lib/components/search/GlobalSearchModal.svelte' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' + import MenuLink from '$lib/components/sidebar/MenuLink.svelte' import { loadProtectionRules } from '$lib/workspaceProtectionRules.svelte' - import { migrateLegacyUserDrafts } from '$lib/userDraftLegacyMigration' + import { purgeLegacyUserDrafts } from '$lib/userDraftLegacyMigration' import { migrateUserDraftsToDb } from '$lib/userDraftDbMigration' import DraftMigrationErrorModal from '$lib/components/DraftMigrationErrorModal.svelte' - import { setContext, untrack } from 'svelte' + import { onDestroy, setContext, untrack } from 'svelte' import { base } from '$app/paths' import { Menubar } from '$lib/components/meltComponents' import { aiChatManager } from '$lib/components/copilot/chat/AIChatManager.svelte' import AiChatLayout from '$lib/components/copilot/chat/AiChatLayout.svelte' import SessionPicker from '$lib/components/sessions/SessionPicker.svelte' + import SessionModeSwitch from '$lib/components/sessions/SessionModeSwitch.svelte' + import { isGlobalAiEnabled } from '$lib/components/copilot/chat/global/gate' + import { parsePreviewItemRoute } from '$lib/components/sessions/previewRouter' + import { rememberNavRoute } from '$lib/components/sessions/sessionSwitch.svelte' + import { sessionState } from '$lib/components/sessions/sessionState.svelte' + import { currentWorkspaceRootId } from '$lib/components/sessions/sessionScope.svelte' + import WorkspaceScopeHeader from '$lib/components/sidebar/WorkspaceScopeHeader.svelte' import { DEFAULT_HUB_BASE_URL } from '$lib/hub' import DBManagerDrawer from '$lib/components/DBManagerDrawer.svelte' import { useIsDarkMode } from '$lib/components/DarkModeObserver.svelte' @@ -86,16 +99,138 @@ let { children }: Props = $props() OpenAPI.WITH_CREDENTIALS = true let menuOpen = $state(false) + // Set by the workspace⇄session switch before it navigates, so the mobile menu + // drawer stays open across a mode toggle (unlike a normal link navigation, + // which dismisses it). Consumed once in beforeNavigate. + let preserveMenuOnNextNav = false let globalSearchModal: GlobalSearchModal | undefined = $state(undefined) - let isCollapsed = $state(false) + // Persisted nav-rail collapse preference. A deliberate collapse writes to it — + // the manual toggle and a drag past the collapse threshold. The contextual + // auto-collapse (app-mode routes, narrow widths) mutates the in-memory + // `isCollapsed` without persisting, so it stays transient and never gets + // "stuck" collapsed across reloads. + const collapsePref = useLocalStorageValue('nav_menu_collapsed', false, 'boolean') + let isCollapsed = $state(collapsePref.val) + + // Resizable desktop rail, sized in REM so it scales with the root font-size the + // same way the old `w-52`/`w-12` classes did — `:root` jumps to 18px past 1760px + // wide (app.css), which grows the rem-based button content; a fixed-px rail would + // not grow with it and the content would overflow. SIDEBAR_MIN_REM is the default + // expanded width (the old w-52); the handle only resizes when expanded and only + // widens from there — collapsing is the toggle button's job, not the drag's. + const SIDEBAR_MIN_REM = 13 + const SIDEBAR_COLLAPSED_REM = 3 + // Root font-size in px, used to convert the pointer's clientX (px) into rem. + function rootFontPx(): number { + if (!BROWSER) return 16 + const px = parseFloat(getComputedStyle(document.documentElement).fontSize) + return Number.isFinite(px) && px > 0 ? px : 16 + } + const widthPref = useLocalStorageValue('nav_menu_width_rem', SIDEBAR_MIN_REM, 'number') + // A non-finite stored width (e.g. NaN from a corrupt entry) must never reach the + // `style:width` binding: `Math.max(13, NaN)` is NaN, which renders as the invalid + // `width: NaNrem` and collapses the fixed rail to content width, mangling every + // menu button. Clamp defensively. + function clampSidebarWidth(v: number): number { + return Number.isFinite(v) ? Math.max(SIDEBAR_MIN_REM, v) : SIDEBAR_MIN_REM + } + const initialSidebarWidth = clampSidebarWidth(widthPref.val) + let sidebarWidth = $state(initialSidebarWidth) + // Heal a corrupt stored value so it stops breaking the rail on every reload. + if (widthPref.val !== initialSidebarWidth) widthPref.val = initialSidebarWidth + let resizingSidebar = $state(false) + // Width (in rem) the content offset must track: the icon strip when collapsed, + // the user-chosen width otherwise. + let railWidth = $derived(isCollapsed ? SIDEBAR_COLLAPSED_REM : sidebarWidth) + // Width transition shared by the rail and the content offset: none for the whole + // drag (the rail tracks the pointer 1:1 and hits the min as a hard wall, no + // friction), a plain ease only for the collapse/expand toggle. + let sidebarTransitionClass = $derived( + resizingSidebar ? '' : 'transition-all duration-200 ease-in-out' + ) + // Set while a drag is live so it can be torn down if the layout unmounts + // mid-drag (otherwise the window listeners would leak). + let stopSidebarResize: (() => void) | null = null + + function startSidebarResize(e: PointerEvent) { + // One drag at a time: ignore a second concurrent pointer (multi-touch on the + // handle) so its window listeners can't outlive the first pointer's stop(). + if (resizingSidebar) return + e.preventDefault() + const handle = e.currentTarget as HTMLElement + // Capture the pointer so events keep flowing to us even when the cursor + // crosses an iframe in the content area (app/session previews) — without + // this the drag silently stalls the moment it enters the iframe. + try { + handle.setPointerCapture(e.pointerId) + } catch {} + resizingSidebar = true + // The rail is fixed at left:0, so the pointer's clientX is the width — in px. + // Convert to rem (the unit the rail is sized in) via the root font-size. Pure + // resize: clamp at the min so the rail stops there like a wall (dragging left + // never collapses — that's the toggle button's job). + const onMove = (ev: PointerEvent) => { + sidebarWidth = Math.max(SIDEBAR_MIN_REM, ev.clientX / rootFontPx()) + } + // pointercancel (and unmount, via onDestroy) must clear the state too, or + // `resizingSidebar` sticks true — the overlay and handle highlight stay up + // and the window listeners leak. + const stop = () => { + if (!resizingSidebar) return + resizingSidebar = false + widthPref.val = sidebarWidth + window.removeEventListener('pointermove', onMove) + window.removeEventListener('pointerup', stop) + window.removeEventListener('pointercancel', stop) + try { + handle.releasePointerCapture(e.pointerId) + } catch {} + stopSidebarResize = null + } + stopSidebarResize = stop + window.addEventListener('pointermove', onMove) + window.addEventListener('pointerup', stop) + window.addEventListener('pointercancel', stop) + } + + onDestroy(() => stopSidebarResize?.()) + + // Let AI chat's 'navigate' link chips route through the app router without the + // tool layer importing $app/navigation. + $effect(() => { + const unregisterNavigate = registerToolDisplayActionHandler('navigate', (action) => { + if (action.type === 'navigate') goto(action.url) + }) + return unregisterNavigate + }) let userSettings: UserSettings | undefined = $state() let superadminSettings: SuperadminSettings | undefined = $state() let menuHidden = $state(false) let isDarkMode = useIsDarkMode() let darkMode = $derived(isDarkMode.val) - const SIDEBAR_BG = '#F3F3F7' - const SIDEBAR_BG_DARK = '#1e232e' + // Session mode is route-derived: the rail shows the sessions sidebar on the + // /sessions page and the workspace navigation everywhere else. The switch + // (SessionModeSwitch) just navigates in and out of that route. + let sessionMode = $derived(page.url.pathname.startsWith(base + '/sessions')) + // Session mode points the bottom settings entry at the open session's own + // workspace. An unsent draft hasn't committed one yet, so it falls back to + // the family root. + let sessionSettingsWorkspace = $derived.by(() => { + if (!sessionMode) return undefined + const current = sessionState.sessions.find((s) => s.id === sessionState.currentSessionId) + return current?.workspace_id ?? $currentWorkspaceRootId ?? $workspaceStore ?? undefined + }) + // Inside a preview iframe the rail still renders (navigation mode), but the + // switch must not — entering session mode from within the preview would + // nest the whole experience. Hide it when embedded. + const embedded = BROWSER && window.self !== window.top + + // AI sessions are still dev-gated (localStorage wm_dev_global_ai=1), same as + // the global chat. The Workspace ⇄ Sessions switch is the only entry point, so + // gate it on the flag too — otherwise it would ship the unfinished experience + // to prod. The /sessions page has its own gate for direct navigation. + const globalAiEnabled = isGlobalAiEnabled() if (page.status == 404) { goto('/user/login') @@ -120,18 +255,46 @@ $workspaceStore = queryWorkspace } - menuHidden = - page.url.searchParams.get('nomenubar') === 'true' || - page.url.pathname.startsWith('/oauth/callback/') + // When this window is an iframe (e.g. the sessions preview), keep the menu + // hidden once `nomenubar` has been requested: navigating inside the preview + // drops the query param (both client-side routing and full-document loads), + // and we don't want the global nav to pop back in. Stickiness is stored in + // sessionStorage so it survives full reloads within the iframe's browsing + // context. The top window is unaffected (embedded is false there), so the + // oauth-callback case and ordinary navigation still toggle normally. + const embedded = typeof window !== 'undefined' && window.self !== window.top + const requested = page.url.searchParams.get('nomenubar') === 'true' + if (embedded && requested) { + try { + sessionStorage.setItem('nomenubar_embedded', 'true') + } catch {} + } + let stickyEmbedded = false + if (embedded) { + try { + stickyEmbedded = sessionStorage.getItem('nomenubar_embedded') === 'true' + } catch {} + } + menuHidden = requested || page.url.pathname.startsWith('/oauth/callback/') || stickyEmbedded } async function updateUserStore(workspace: string | undefined) { if (workspace) { - try { - sessionStorage.setItem('workspace', String(workspace)) - localStorage.setItem('workspace', String(workspace)) - } catch (e) { - console.error('Could not persist workspace to local storage', e) + // A preview iframe shares BOTH localStorage and sessionStorage with the + // top-level app (same-origin nested browsing contexts share the top-level + // session storage). Persisting its session-scoped workspace to either would + // clobber the workspace the user is actually navigating. Keep it in-memory + // only ($workspaceStore is still set from the ?workspace= param for the + // iframe's own API calls); the fork survives iframe reloads because the + // preview always reloads a URL that carries ?workspace= (see + // PreviewTabHost.reload). Only the top window owns the persisted keys. + if (!embedded) { + try { + sessionStorage.setItem('workspace', String(workspace)) + localStorage.setItem('workspace', String(workspace)) + } catch (e) { + console.error('Could not persist workspace to local storage', e) + } } const user = await getUserExt(workspace) if (!deepEqual(user, $userStore)) { @@ -147,16 +310,102 @@ } catch (e) { console.error('Could not persist username to local storage', e) } - if (isCloudHosted() && user?.is_admin) { - isPremiumStore.set(await WorkspaceService.getIsPremium({ workspace })) + // Populate for all members (not just admins) so non-admin developers also get premium-gated + // affordances like the fork entry points on cloud. The `is_premium` endpoint is a boolean + // and no longer admin-gated. Best-effort: a failure here must not block user-store init. + if (isCloudHosted()) { + try { + isPremiumStore.set(await WorkspaceService.getIsPremium({ workspace })) + } catch (e) { + console.error('Could not fetch premium status', e) + } } } else { userStore.set(undefined) } } + // True when this window is a sessions-preview iframe (embedded + nomenubar, + // which the preview always sets and stickies — see the menu-hide block above). + function isSessionPreviewEmbed(): boolean { + if (!embedded) return false + try { + return sessionStorage.getItem('nomenubar_embedded') === 'true' + } catch { + return false + } + } + + // A job-detail navigation (/run/) inside a preview tab should open the job in + // a NEW tab rather than navigate the current tab away from its page (e.g. clicking + // a job in the Runs tab keeps Runs put and opens the run beside it). Returns the + // href to open (nomenubar dropped — the preview host re-adds it) and a short label. + function previewRunTarget(url: URL | undefined): { href: string; label: string } | undefined { + if (!url) return undefined + const m = url.pathname.match(/\/run\/([^/?#]+)/) + if (!m) return undefined + const u = new URL(url.href) + u.searchParams.delete('nomenubar') + const id = decodeURIComponent(m[1]) + return { href: u.pathname + u.search, label: `Run ${id.slice(0, 8)}` } + } + + // Map a navigation target to the session editor it should open as a component, + // or undefined for anything without a live-editor wrapper (regular apps, pages, + // /get viewers). Only /edit routes — the wrappers are editors. + function previewEditorTarget( + url: URL | undefined + ): { kind: 'script' | 'flow' | 'raw_app'; path: string } | undefined { + if (!url || !/\/(scripts|flows|apps_raw)\/edit\//.test(url.pathname)) return undefined + const route = parsePreviewItemRoute(url.pathname) + if (!route) return undefined + const kind = + route.kind === 'script' + ? 'script' + : route.kind === 'flow' + ? 'flow' + : route.raw_app + ? 'raw_app' + : undefined + return kind ? { kind, path: route.itemPath } : undefined + } + beforeNavigate((navigation) => { - menuOpen = false + if (preserveMenuOnNextNav) { + preserveMenuOnNextNav = false + } else { + menuOpen = false + } + + // Inside a sessions-preview iframe, hand an editor-route navigation up to the + // parent so it mounts the in-process editor (sharing the session runtime) + // instead of booting a second, disconnected editor in this frame. Cancel so + // the heavy editor never mounts here at all. Runs before the apps_raw reload + // below so a raw-app editor promotes rather than full-reloading the iframe. + if (isSessionPreviewEmbed()) { + const target = previewEditorTarget(navigation.to?.url) + if (target) { + navigation.cancel() + try { + window.parent.postMessage( + { type: 'wm.session.openEditor', kind: target.kind, path: target.path }, + window.location.origin + ) + } catch {} + return + } + const runTarget = previewRunTarget(navigation.to?.url) + if (runTarget) { + navigation.cancel() + try { + window.parent.postMessage( + { type: 'wm.session.openRun', href: runTarget.href, label: runTarget.label }, + window.location.origin + ) + } catch {} + return + } + } // Force page reload when navigating to /apps_raw/add or /apps_raw/edit // This ensures the cross-origin isolation headers are fetched from the server @@ -164,8 +413,9 @@ const toPath = navigation.to?.url.pathname if (toPath && (toPath.startsWith('/apps_raw/add') || toPath.startsWith('/apps_raw/edit'))) { const currentPath = navigation.from?.url.pathname - // Reload if we're not on an apps_raw path, or if we're on /apps/get_raw/ (viewing a raw app) - // The /apps/get_raw/ path doesn't have cross-origin isolation headers, so we need to reload + // Reload if we're not on an apps_raw path, or if we're on the raw app viewer + // (/apps_raw/get/): the viewer doesn't have cross-origin isolation headers, so + // we need a full reload to fetch them for the editor. if (!currentPath?.startsWith('/apps_raw/') || currentPath?.startsWith('/apps_raw/get/')) { navigation.cancel() window.location.href = navigation.to!.url.href @@ -319,19 +569,12 @@ $usedTriggerKinds = usedKinds } - function pathInAppMode(pathname: string | undefined): boolean { - if (!pathname) return false - return ( - pathname.startsWith(base + '/apps') || - pathname.startsWith(base + '/flows/add') || - pathname.startsWith(base + '/flows/edit') || - pathname.startsWith(base + '/scripts/add') || - pathname.startsWith(base + '/scripts/edit') - ) - } afterNavigate((n) => { - if (pathInAppMode(n.to?.url.pathname) && innerWidth >= 768) { - isCollapsed = true + // Remember the last navigation-mode route so exiting session mode returns + // the user where they were rather than to the home page. + const to = n.to?.url + if (to && !to.pathname.startsWith(base + '/sessions')) { + rememberNavRoute(to.pathname + to.search) } }) @@ -342,9 +585,6 @@ } let devOnly = $derived(page.url.pathname.startsWith(base + '/scripts/dev')) - // Sessions own their own chat pane; suppress the global Ask-AI chat on the /sessions route - // so it doesn't render a second chat overlay on top of the session. - let inSessionRoute = $derived(page.url.pathname.startsWith(base + '/sessions')) async function loadDefaultScripts(workspace: string, user: UserExt | undefined) { if (!user?.operator) { @@ -435,16 +675,17 @@ $effect(() => { $workspaceStore && untrack(() => onLoad()) }) - // One-shot UserDraft migration chain. `migrateLegacyUserDrafts` folds - // the legacy `flow` / `app-…` / `rawapp-…` LS keys into the - // `userdraft/w/{ws}/{kind}/{path}` format; `migrateUserDraftsToDb` - // then pushes those onto the server-side draft table and clears LS - // on success. The order matters — the second step only sees what - // the first one normalized. + // One-shot UserDraft migration. `purgeLegacyUserDrafts` drops the oldest + // workspace-blind `flow` / `app-…` / `rawapp-…` LS autosave keys (they + // can't be attributed to a workspace, so promoting them would mis-file + // drafts). `migrateUserDraftsToDb` then pushes the workspace-scoped + // `userdraft/w/{ws}/{kind}/{path}` keys — written by the editor with the + // correct workspace — onto the server-side draft table, clearing LS on + // success. $effect(() => { if ($workspaceStore && $userStore) { untrack(() => { - migrateLegacyUserDrafts($workspaceStore!) + purgeLegacyUserDrafts() void migrateUserDraftsToDb() }) } @@ -501,6 +742,61 @@ + +{#snippet quickLinks(collapsed: boolean)} + + { + setTimeout(() => { + window.dispatchEvent(new Event('popstate')) + }, 100) + }} + /> +{/snippet} + + +{#snippet brandMark(collapsed: boolean)} +
+ + {#if !collapsed} + {$whitelabelNameStore ? capitalize($whitelabelNameStore) : 'Windmill'} + {/if} +
+{/snippet} + + +{#snippet settingsMenu(collapsed: boolean)} +
+ +
+{/snippet} + {#if page.status == 404} @@ -514,6 +810,11 @@ {/if}
+ {#if resizingSidebar} + +
+ {/if} {#if !menuHidden} {#if !$userStore?.operator} {#if innerWidth < 768} @@ -576,58 +877,101 @@ class="h-full flex flex-col" style:background-color={darkMode ? SIDEBAR_BG_DARK : SIDEBAR_BG} > -
- - {#if $whitelabelNameStore} - {$whitelabelNameStore} - {:else} - Windmill - {/if} -
-
- + +
+ {#snippet children({ createMenu })} - {/snippet} - openSearchModal()} - isCollapsed={false} - icon={Search} - label="Search" - class="!text-xs" - shortcut={`${getModifierKey()}k`} - /> - aiChatManager.toggleOpen()} - isCollapsed={false} - icon={WandSparkles} - iconProps={{ - forceDarkMode: true - }} - label="Ask AI" - class="!text-xs" - iconClasses="!text-ai" - shortcut={`${getModifierKey()}L`} - />
- -
- -
+ {#if !embedded && globalAiEnabled} + +
+ (preserveMenuOnNextNav = true)} + /> +
+ {/if} - + {#if !sessionMode} + +
+ +
+ {/if} + + {#if sessionMode} + +
+ +
+ {@render settingsMenu(false)} + {:else} + + + +
+ {@render quickLinks(false)} + + {#snippet children({ createMenu })} + + {/snippet} + + openSearchModal()} + isCollapsed={false} + icon={Search} + label="Search" + class="!text-xs" + shortcut={`${getModifierKey()}k`} + /> + {#if !globalAiEnabled} + + aiChatManager.toggleOpen()} + isCollapsed={false} + icon={WandSparkles} + iconProps={{ forceDarkMode: true }} + label="Ask AI" + class="!text-xs" + iconClasses="!text-ai" + shortcut={`${getModifierKey()}L`} + /> + {/if} +
+ + + + + +
+ {@render settingsMenu(false)} +
+
+ {/if} + +
+ {@render brandMark(false)} +
@@ -636,93 +980,150 @@
+ {#if job} + + {/if} + {#if isNotFlow(job?.job_kind)} {#if ['python3', 'bun', 'deno'].includes(job?.language ?? '') && (job?.job_kind == 'script' || isScriptPreview(job?.job_kind))} @@ -912,6 +920,7 @@
{#if job.id && job.workspace_id} + {/if} {/if} diff --git a/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte b/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte index 24bf997c50..9f7c718975 100644 --- a/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte @@ -14,11 +14,19 @@ canWrite, truncateHash, copyToClipboard, - urlParamsToObject + urlParamsToObject, + extractTagFromSharableHash, + isDynamicTag } from '$lib/utils' import Tooltip from '$lib/components/Tooltip.svelte' import ShareModal from '$lib/components/ShareModal.svelte' - import { enterpriseLicense, hubBaseUrlStore, userStore, workspaceStore } from '$lib/stores' + import { + enterpriseLicense, + hubBaseUrlStore, + userStore, + userWorkspaces, + workspaceStore + } from '$lib/stores' import { isDeployable, ALL_DEPLOYABLE } from '$lib/utils_deployable' import AIFormAssistant from '$lib/components/copilot/AIFormAssistant.svelte' @@ -89,8 +97,7 @@ import TriggersEditor from '$lib/components/triggers/TriggersEditor.svelte' import { Triggers } from '$lib/components/triggers/triggers.svelte' import { page } from '$app/state' - import { isRuleActive } from '$lib/workspaceProtectionRules.svelte' - import { buildForkEditUrl } from '$lib/utils/editInFork' + import { buildForkEditUrl, editInForkAllowed, editInForkLabel } from '$lib/utils/editInFork' import { isCloudHosted } from '$lib/cloud' import { isWorkflowAsCode } from '$lib/components/graph/wacToFlow' import WacDiagram from '$lib/components/graph/WacDiagram.svelte' @@ -112,6 +119,9 @@ let scheduledForStr: string | undefined = $state(undefined) let invisible_to_owner: boolean | undefined = $state(undefined) let overrideTag: string | undefined = $state(undefined) + let overrideTagNote: string | undefined = $state(undefined) + // Tag carried over from 'Run again', pending the dynamic-tag check in loadScript + let carriedTag: string | undefined = undefined let inputSelected: 'saved' | 'history' | undefined = $state(undefined) let jsonView = $state(false) @@ -253,6 +263,15 @@ return } } + // A carried tag is the previous run's resolved value; when the script's tag is + // dynamic, drop it so the backend re-resolves from the (possibly edited) args + if (carriedTag && isDynamicTag(script.tag)) { + if (overrideTag === carriedTag) { + overrideTag = undefined + overrideTagNote = `tag ${script.tag} is resolved at run time, so the previous run's tag ${carriedTag} was not applied` + } + carriedTag = undefined + } can_write = script.workspace_id == $workspaceStore && canWrite(script.path, script.extra_perms!, $userStore) @@ -329,6 +348,8 @@ if (hash.length > 1) { try { let searchParams = new URLSearchParams(hash.slice(1)) + carriedTag = extractTagFromSharableHash(searchParams) + overrideTag = carriedTag let params = [...Object.entries(urlParamsToObject(searchParams))].map(([k, v]) => [ k, JSON.parse(v) @@ -369,10 +390,10 @@ script && !$userStore?.operator && !isCloudHosted() && - !isRuleActive('DisableWorkspaceForking') + editInForkAllowed($workspaceStore, $userWorkspaces) ) { buttons.push({ - label: 'Edit in fork', + label: editInForkLabel($workspaceStore, $userWorkspaces), buttonProps: { href: buildForkEditUrl('script', script.path), unifiedSize: 'md', @@ -872,6 +893,7 @@ bind:scheduledForStr bind:invisible_to_owner bind:overrideTag + {overrideTagNote} viewKeybinding loading={runLoading} autofocus diff --git a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte index a733c325d6..70e82b2792 100644 --- a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte @@ -1,80 +1,111 @@ -{#if !globalEnabled} -
- Sessions are gated on the global-AI dev flag. Enable with - localStorage.setItem('wm_dev_global_ai', '1') and reload. -
-{:else if !sessionName} -
No session selected — pick one in the sidebar.
-{:else if !sessionByName} - -
-
-

Session not found

-

- No session named {sessionName} exists. It may have been - deleted, or this link was created in a different browser. -

-
- -
-{:else} -
- {#each warmSessions as s (s.id)} - + + diff --git a/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte b/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte index e71f0bb71e..19fa4f2553 100644 --- a/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte @@ -3,11 +3,12 @@ import { page } from '$app/stores' import { isCloudHosted } from '$lib/cloud' import CenteredPage from '$lib/components/CenteredPage.svelte' - import { Alert, Button, Section, Skeleton, Tab, Tabs } from '$lib/components/common' + import { Alert, Button, CopyButton, Section, Skeleton, Tab, Tabs } from '$lib/components/common' import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte' import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte' import DeployToSetting from '$lib/components/DeployToSetting.svelte' + import DevWorkspaceSetting from '$lib/components/DevWorkspaceSetting.svelte' import ErrorOrRecoveryHandler from '$lib/components/ErrorOrRecoveryHandler.svelte' import PageHeader from '$lib/components/PageHeader.svelte' import ScriptPicker from '$lib/components/ScriptPicker.svelte' @@ -68,6 +69,13 @@ type DucklakeSettingsType } from '$lib/components/workspaceSettings/DucklakeSettings.svelte' import UnsavedConfirmationModal from '$lib/components/common/confirmationModal/UnsavedConfirmationModal.svelte' + import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte' + import { + archiveSessionsForWorkspace, + countSessionsForWorkspace, + deleteSessionsForWorkspace, + reconcileAfterWorkspaceChange + } from '$lib/components/sessions/sessionState.svelte' import TextInput from '$lib/components/text_input/TextInput.svelte' import CollapseLink from '$lib/components/CollapseLink.svelte' import { validateWebhookUrl, validateEncryptionKey } from '$lib/validators/workspaceSettings' @@ -87,6 +95,65 @@ let slack_team_name: string | undefined = $state() let teams_team_id: string | undefined = $state() let teams_team_name: string | undefined = $state() + + // Workspace archive/delete cascade to the workspace's client-side AI sessions + // (archive → archive, delete → delete) via reconcileSessionsLifecycle. The + // confirmation modals warn how many sessions are affected first. + let archiveConfirmOpen = $state(false) + let deleteConfirmOpen = $state(false) + let affectedSessionCount = $state(0) + + async function openArchiveConfirm() { + affectedSessionCount = await countSessionsForWorkspace($workspaceStore ?? '') + archiveConfirmOpen = true + } + async function openDeleteConfirm() { + affectedSessionCount = await countSessionsForWorkspace($workspaceStore ?? '') + deleteConfirmOpen = true + } + async function doArchiveWorkspace() { + const ws = $workspaceStore ?? '' + // Land on the parent workspace if this is a fork and the parent is still + // accessible — otherwise fall back to the workspace picker. + const parentId = $userWorkspaces.find((w) => w.id === ws)?.parent_workspace_id + const parentStillAccessible = !!(parentId && $userWorkspaces.find((w) => w.id === parentId)) + await WorkspaceService.archiveWorkspace({ workspace: ws }) + sendUserToast(`Archived workspace ${ws}`) + // Best-effort client cleanup: a local IndexedDB failure must not strand the + // user on the just-archived workspace. The reconcile also refreshes the + // workspace list (dropping the archived one) so the parent-accessible check + // below — captured before the archive — still routes correctly. + try { + await archiveSessionsForWorkspace(ws) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after workspace archive failed', e) + } + if (parentStillAccessible && parentId) { + switchWorkspace(parentId) + await goto('/') + } else { + workspaceStore.set(undefined) + usersWorkspaceStore.set(undefined) + await goto('/user/workspaces') + } + } + async function doDeleteWorkspace() { + const ws = $workspaceStore ?? '' + await WorkspaceService.deleteWorkspace({ workspace: ws }) + sendUserToast(`Deleted workspace ${ws}`) + // Best-effort client cleanup — must not block navigation off the deleted + // workspace if a local IndexedDB op throws. + try { + await deleteSessionsForWorkspace(ws) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after workspace delete failed', e) + } + workspaceStore.set(undefined) + usersWorkspaceStore.set(undefined) + await goto('/user/workspaces') + } let useCustomSlackApp: boolean = $state(false) let slackAppType: 'instance' | 'workspace' = $state('instance') @@ -282,6 +349,7 @@ | 'general' | 'webhook' | 'deploy_to' + | 'dev_workspace' | 'error_handler' | 'success_handler' | 'critical_alerts' @@ -1046,6 +1114,14 @@ } } + // The Dev workspace tab is only meaningful on a root workspace (to pair/manage a dev) or on a + // dev workspace itself (to see its prod / detach). Hide it for ordinary forks — pairing isn't + // available there and the backend would reject it. + const currentWsForDevTab = $derived($userWorkspaces.find((w) => w.id === $workspaceStore)) + const showDevWorkspaceTab = $derived( + !currentWsForDevTab?.parent_workspace_id || (currentWsForDevTab?.is_dev_workspace ?? false) + ) + // Navigation groups for sidebar const navigationGroups = $derived([ { @@ -1094,6 +1170,17 @@ aiDescription: 'Deployment UI workspace settings', isEE: true }, + ...(showDevWorkspaceTab + ? [ + { + id: 'dev_workspace', + label: 'Dev workspace', + aiId: 'workspace-settings-dev-workspace', + aiDescription: + 'Pair this workspace with a dev workspace (same code, different environment)' + } + ] + : []), { id: 'rulesets', label: 'Rulesets', @@ -1217,8 +1304,13 @@ {#if $userStore?.is_admin || $superadmin} - {#if $superadmin} + + {#snippet titleActions()} + {#if $workspaceStore} + + {/if} + {/snippet} + {#if $superadmin} @@ -1277,6 +1369,12 @@ >
{/if} + {:else if tab == 'dev_workspace'} + + {:else if tab == 'rulesets'} {:else if tab == 'premium'} - + {#if currentWsForDevTab?.parent_workspace_id} + + This workspace is a fork of {currentWsForDevTab.parent_workspace_id}. It + runs on the parent's plan and its executions count toward the parent's usage and + bill, so there is no separate subscription here. Manage billing, seats, and quotas + from the parent workspace's settings. + + {:else} + + {/if} {:else if tab == 'slack'} { - const ws = $workspaceStore ?? '' - // Land on the parent workspace if this is a fork and the - // parent is still accessible — otherwise fall back to the - // workspace picker. - const parentId = $userWorkspaces.find((w) => w.id === ws)?.parent_workspace_id - const parentStillAccessible = !!( - parentId && $userWorkspaces.find((w) => w.id === parentId) - ) - await WorkspaceService.archiveWorkspace({ workspace: ws }) - sendUserToast(`Archived workspace ${ws}`) - if (parentStillAccessible && parentId) { - // Refresh the list so the just-archived workspace drops out before - // we land on the parent. Guarded: a refresh failure must not block - // the switch (the list reloads on next page load). - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces after archive', e) - } - switchWorkspace(parentId) - await goto('/') - } else { - workspaceStore.set(undefined) - usersWorkspaceStore.set(undefined) - await goto('/user/workspaces') - } - }} + on:click={openArchiveConfirm} > Archive workspace @@ -1581,18 +1661,51 @@ disabled={$workspaceStore === 'admins' || $workspaceStore === 'starter'} size="sm" btnClasses="mt-2" - on:click={async () => { - await WorkspaceService.deleteWorkspace({ workspace: $workspaceStore ?? '' }) - sendUserToast(`Deleted workspace ${$workspaceStore}`) - workspaceStore.set(undefined) - usersWorkspaceStore.set(undefined) - goto('/user/workspaces') - }} + on:click={openDeleteConfirm} > Delete workspace (superadmin) {/if}
+ + { + archiveConfirmOpen = false + await doArchiveWorkspace() + }} + onCanceled={() => (archiveConfirmOpen = false)} + > +
+ + Archiving this workspace also archives its AI sessions{affectedSessionCount > 0 + ? ` (${affectedSessionCount})` + : ''}. Unarchiving the workspace restores them. + +
+
+ + { + deleteConfirmOpen = false + await doDeleteWorkspace() + }} + onCanceled={() => (deleteConfirmOpen = false)} + > +
+ + Permanently deleting this workspace also permanently deletes its AI sessions{affectedSessionCount > + 0 + ? ` (${affectedSessionCount})` + : ''}. This cannot be undone. + +
+
{:else if tab == 'webhook'} void) | undefined + + // Embedder side: validate access (main session cookie or shared JWT) and mint + // a scoped embed token for the opaque iframe (WIN-2006). + async function fetchEmbedToken(): Promise<{ token?: string }> { if (parsedCustomPath.jwt) { - const token = 'jwt_ext_' + parsedCustomPath.jwt - OpenAPI.TOKEN = token - setContext<{ token?: string }>('AuthToken', { token }) - jwtError = false + OpenAPI.TOKEN = 'jwt_ext_' + parsedCustomPath.jwt } + const headers: Record = {} + if (typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + headers['Authorization'] = `Bearer ${OpenAPI.TOKEN}` + } + const res = await fetch( + `${OpenAPI.BASE}/apps_u/embed_token_by_custom_path/${parsedCustomPath.path}`, + { headers } + ) + if (!res.ok) { + const err: any = new Error('Failed to fetch embed token') + err.status = res.status + throw err + } + return await res.json() + } + + // Viewer side: load the app + user using the embed token handed to the iframe. + async function loadApp() { try { app = await AppService.getPublicAppByCustomPath({ customPath: parsedCustomPath.path @@ -62,9 +91,13 @@ workspaceStore.set(app.workspace_id) noPermission = false notExists = false + jwtError = false try { userStore.set(await getUserExt(app.workspace_id)) + // A JWT in the custom path that fails to resolve a user is surfaced as a + // toast (matches the pre-sandbox custom-path viewer) rather than silently + // falling through to anonymous. if (!$userStore && parsedCustomPath.jwt) { jwtError = true sendUserToast('Could not authentify user with jwt token', true) @@ -74,7 +107,8 @@ } } catch (e) { if (e.status == 401) { - noPermission = true + // Embed token missing/expired — ask the embedder for a fresh one. + refresh?.() } else { notExists = true } @@ -83,17 +117,25 @@ if (BROWSER) { setLicense() - loadApp() } - { + { + refresh = requestTokenRefresh loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte new file mode 100644 index 0000000000..d0af10b433 --- /dev/null +++ b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte @@ -0,0 +1,99 @@ + + + { + refresh = requestTokenRefresh + loadApp() + }} +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js deleted file mode 100644 index 42a8b51427..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load({ params }) { - return { - stuff: { title: `Public App` } - } -} diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte deleted file mode 100644 index 38563cfbd9..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte +++ /dev/null @@ -1,11 +0,0 @@ - - - diff --git a/frontend/src/routes/kitchen_sink/+page.svelte b/frontend/src/routes/kitchen_sink/+page.svelte index 0e07158244..4285ee0921 100644 --- a/frontend/src/routes/kitchen_sink/+page.svelte +++ b/frontend/src/routes/kitchen_sink/+page.svelte @@ -4,21 +4,135 @@ import TabContent from '$lib/components/common/tabs/TabContent.svelte' import Tabs from '$lib/components/common/tabs/Tabs.svelte' import DarkModeToggle from '$lib/components/sidebar/DarkModeToggle.svelte' + import GfmMarkdown from '$lib/components/GfmMarkdown.svelte' + import AssistantMessage from '$lib/components/copilot/chat/AssistantMessage.svelte' + import type { DisplayMessage } from '$lib/components/copilot/chat/shared' + import DraggableTabs, { type TabItem } from '$lib/components/common/tabs/DraggableTabs.svelte' import { Globe } from 'lucide-svelte' let tab = $state('button') + + // Enough tabs to overflow a narrow strip so the shared ScrollableX hover + // scrollbar is exercised: drag to reorder, hover to reveal the 4px thumb. + let draggableTabs = $state( + Array.from({ length: 14 }, (_, i) => ({ id: `t${i}`, label: `Preview tab ${i + 1}` })) + ) + let activeDraggableTab = $state('t0') + + const sampleMarkdown = `# Heading 1 + +## Heading 2 + +Body text with **bold**, *italic*, a [link](https://windmill.dev), and \`inline code\` that must stay readable in both themes. + +> A block quote should be legible too. + +- First bullet +- Second bullet with \`code\` + +1. Ordered one +2. Ordered two + +\`\`\`ts +const block = 'code block' +console.log(block) +\`\`\` + +An unlabeled fence should render plain (no forced syntax colors): + +\`\`\` +just some plain text +no language, no coloring +\`\`\` + +Raw sanitized HTML (via rehypeRaw) must keep its content, not render empty: + +
raw pre content stays visible
+ +| Column A | Column B | +| -------- | -------- | +| one | two | +| three | four | + +--- +` let dropdownItems = [ { label: 'Lorem ipsum', onClick: () => {} } ] + + // Mirrors how the AI chat renders assistant answers: markdown flows through + // AssistantMessage, and fenced code blocks go through CodeDisplay → + // HighlightCode. Exercise several languages + prose so the code-block styling + // can be tuned against the real render path, not an approximation. + const chatSampleContent = `Here's how you'd wire up the trigger. First, some prose with \`inline code\`, a [link](https://windmill.dev), and **bold** text so we can see how code sits next to surrounding content. + +\`\`\`python +def main(name: str = "world"): + # a short python snippet + greeting = f"hello, {name}!" + print(greeting) + return {"greeting": greeting} +\`\`\` + +A one-liner in the middle of a sentence like \`SELECT * FROM users\` should stay inline. Now a TypeScript block: + +\`\`\`ts +export async function main(count: number) { + const rows = await Promise.all( + Array.from({ length: count }, (_, i) => fetchRow(i)) + ) + return rows.filter((r) => r.active).map((r) => r.id) +} +\`\`\` + +A block with a very long line to check horizontal overflow behaviour: + +\`\`\`bash +curl -sSL "https://app.windmill.dev/api/w/demo/jobs/run_wait_result/p/u/admin/very_long_script_path?token=abcdef0123456789&include_header=Authorization" | jq '.result.value' +\`\`\` + +Some SQL: + +\`\`\`sql +select id, name, created_at +from users +where active = true +order by created_at desc +limit 10; +\`\`\` + +And a bulleted list where an item carries \`code\`: + +- First item with \`some_var\` +- Second item +- Third item + +\`\`\`rust +fn main() { + let items = vec![1, 2, 3]; + let sum: i32 = items.iter().sum(); + println!("sum = {sum}"); +} +\`\`\` + +That's the full round-trip.` + + const chatMessage: DisplayMessage = { + role: 'assistant', + content: chatSampleContent + } + + + {#snippet content()} @@ -72,5 +186,31 @@
+ + + + +
+ Rendered through the real chat path (AssistantMessage → + CodeDisplayHighlightCode), constrained to the chat panel width. +
+
+ +
+
+ +
+ DraggableTabs (uses the shared ScrollableX, 4px bar): hover to reveal the + thumb, drag to reorder. +
+
+ (activeDraggableTab = id)} + onReorder={(next) => (draggableTabs = next)} + /> +
+
{/snippet} diff --git a/frontend/src/routes/kitchen_sink/deploy_animation/+page.svelte b/frontend/src/routes/kitchen_sink/deploy_animation/+page.svelte new file mode 100644 index 0000000000..6cd7e7197f --- /dev/null +++ b/frontend/src/routes/kitchen_sink/deploy_animation/+page.svelte @@ -0,0 +1,194 @@ + + +
+
+

Deploy animation playground

+ +
+

+ Drives the real WorkspaceDiffDrawer with a mock deploy model. Tune the knobs, open the + drawer, hit Deploy, feel the transition. Reset re-arms the drafts. +

+ +
+ + + + + +
+ + + +
+
+ +
+ + +
+
+ + diff --git a/frontend/src/routes/pipeline_dev/+page.svelte b/frontend/src/routes/pipeline_dev/+page.svelte new file mode 100644 index 0000000000..0f2e3d0607 --- /dev/null +++ b/frontend/src/routes/pipeline_dev/+page.svelte @@ -0,0 +1,7 @@ + + +
+ +
diff --git a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte index c661aa48c6..089b95c97d 100644 --- a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte +++ b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte @@ -4,21 +4,20 @@ import { AppService, OpenAPI, type AppWithLastVersion } from '$lib/gen' import { userStore } from '$lib/stores' - import { setContext } from 'svelte' import { setLicense } from '$lib/enterpriseUtils' import { getUserExt } from '$lib/user' - import { sendUserToast } from '$lib/toast' import { page } from '$app/state' + import { base } from '$lib/base' import PublicApp from '$lib/components/apps/editor/PublicApp.svelte' + import PublicAppFrame from '$lib/components/apps/editor/PublicAppFrame.svelte' let app: (AppWithLastVersion & { value: any }) | undefined = $state(undefined) let notExists = $state(false) let noPermission = $state(false) - let jwtError = $state(false) - function parseSecret(secret: string): { secret: string; jwt: string } { + function parseSecret(secret: string): { secret: string; jwt: string | undefined } { const parts = secret.split('/') return { secret: parts[0], @@ -27,18 +26,59 @@ } const parsedSecret = parseSecret(page.params.secret ?? '') + const workspace = page.params.workspace ?? '' + // URL for the opaque viewer iframe: the share URL WITHOUT the trailing JWT + // segment. The JWT is a viewer credential (broader and longer-lived than the + // scoped embed token) consumed here on the embedder side only — it must never + // appear in the iframe's own location, where app-authored code could read it. + // Captured once (not reactively): the embedder mirrors the app's hash/query + // back onto this page's URL, and re-deriving the src from it would reload the + // app on its every navigation. + const viewerUrl = `${base}/public/${workspace}/${parsedSecret.secret}${page.url.search}${page.url.hash}` + + let refresh: (() => void) | undefined + + // Embedder side: validate access (using the main session cookie or the shared + // JWT) and mint a scoped embed token for the opaque iframe (WIN-2006). + async function fetchEmbedToken(): Promise<{ token?: string }> { + if (parsedSecret.jwt) { + OpenAPI.TOKEN = 'jwt_ext_' + parsedSecret.jwt + } + const headers: Record = {} + if (typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + headers['Authorization'] = `Bearer ${OpenAPI.TOKEN}` + } + const res = await fetch( + `${OpenAPI.BASE}/w/${workspace}/apps_u/embed_token/${parsedSecret.secret}`, + { headers } + ) + if (!res.ok) { + const err: any = new Error('Failed to fetch embed token') + err.status = res.status + throw err + } + return await res.json() + } + + // Viewer side: load the app + user using the embed token handed to the iframe. async function loadApp() { + try { + userStore.set(await getUserExt(workspace)) + } catch (e) { + console.warn('Anonymous user') + } try { app = await AppService.getPublicAppBySecret({ - workspace: page.params.workspace ?? '', + workspace, path: parsedSecret.secret }) noPermission = false notExists = false } catch (e) { if (e.status == 401) { - noPermission = true + // Embed token missing/expired — ask the embedder for a fresh one. + refresh?.() } else { notExists = true } @@ -47,42 +87,25 @@ if (BROWSER) { setLicense() - loadAll() - } - - function loadAll() { - console.log('loadAll') - loadUser().then(() => { - loadApp() - }) - } - - async function loadUser() { - if (parsedSecret.jwt) { - const token = 'jwt_ext_' + parsedSecret.jwt - OpenAPI.TOKEN = token - setContext<{ token?: string }>('AuthToken', { token }) - jwtError = false - } - try { - userStore.set(await getUserExt(page.params.workspace ?? '')) - if (!$userStore && parsedSecret.jwt) { - jwtError = true - sendUserToast('Could not authentify user with jwt token', true) - } - } catch (e) { - console.warn('Anonymous user') - } } - { - loadAll() + { + refresh = requestTokenRefresh + loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/tailwind.config.cjs b/frontend/tailwind.config.cjs index bae0531c17..9bed325d03 100644 --- a/frontend/tailwind.config.cjs +++ b/frontend/tailwind.config.cjs @@ -295,7 +295,8 @@ const config = { 600: '#d97706', 700: '#b45309', 800: '#92400e', - 900: '#78350f' + 900: '#78350f', + 950: '#451a03' }, emerald: { 50: '#ecfdf5', @@ -455,6 +456,49 @@ const config = { ] }, extend: { + // Tailwind Typography hardcodes every `prose` color (body, headings, + // borders, code, ...) to fixed gray shades. Our `.prose` usages mostly + // render without `dark:prose-invert`, so in dark mode those light-mode + // grays stay put and text/borders/code render near-black on a dark + // surface. Point the whole palette at our theme tokens — which already + // flip with the active theme — for BOTH the default and inverted sets, so + // bare `.prose` and `dark:prose-invert` usages both track the theme. + typography: { + DEFAULT: { + css: { + '--tw-prose-body': 'rgb(var(--color-text-secondary))', + '--tw-prose-headings': 'rgb(var(--color-text-primary))', + '--tw-prose-lead': 'rgb(var(--color-text-secondary))', + '--tw-prose-bold': 'rgb(var(--color-text-primary))', + '--tw-prose-counters': 'rgb(var(--color-text-tertiary))', + '--tw-prose-bullets': 'rgb(var(--color-text-tertiary))', + '--tw-prose-hr': 'rgb(var(--color-border-light))', + '--tw-prose-quotes': 'rgb(var(--color-text-secondary))', + '--tw-prose-quote-borders': 'rgb(var(--color-border-light))', + '--tw-prose-captions': 'rgb(var(--color-text-tertiary))', + '--tw-prose-code': 'rgb(var(--color-text-primary))', + '--tw-prose-pre-code': 'rgb(var(--color-text-primary))', + '--tw-prose-pre-bg': 'rgb(var(--color-surface-secondary))', + '--tw-prose-th-borders': 'rgb(var(--color-border-light))', + '--tw-prose-td-borders': 'rgb(var(--color-border-light))', + '--tw-prose-invert-body': 'rgb(var(--color-text-secondary))', + '--tw-prose-invert-headings': 'rgb(var(--color-text-primary))', + '--tw-prose-invert-lead': 'rgb(var(--color-text-secondary))', + '--tw-prose-invert-bold': 'rgb(var(--color-text-primary))', + '--tw-prose-invert-counters': 'rgb(var(--color-text-tertiary))', + '--tw-prose-invert-bullets': 'rgb(var(--color-text-tertiary))', + '--tw-prose-invert-hr': 'rgb(var(--color-border-light))', + '--tw-prose-invert-quotes': 'rgb(var(--color-text-secondary))', + '--tw-prose-invert-quote-borders': 'rgb(var(--color-border-light))', + '--tw-prose-invert-captions': 'rgb(var(--color-text-tertiary))', + '--tw-prose-invert-code': 'rgb(var(--color-text-primary))', + '--tw-prose-invert-pre-code': 'rgb(var(--color-text-primary))', + '--tw-prose-invert-pre-bg': 'rgb(var(--color-surface-secondary))', + '--tw-prose-invert-th-borders': 'rgb(var(--color-border-light))', + '--tw-prose-invert-td-borders': 'rgb(var(--color-border-light))' + } + } + }, border: { color: 'red' }, @@ -641,6 +685,16 @@ const config = { ".dark [type='checkbox']:checked": { backgroundImage: `url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3cpath d='M12.207 4.793a1 1 0 010 1.414l-5 5a1 1 0 01-1.414 0l-2-2a1 1 0 011.414-1.414L6.5 9.086l4.293-4.293a1 1 0 011.414 0z'/%3e%3c/svg%3e")` }, + // @tailwindcss/forms draws the indeterminate dash white-on-currentColor, + // but the checkbox theme above keeps a white/dark background — so restyle + // the dash to match the checkmark's fill in each mode. + "[type='checkbox']:indeterminate": { + backgroundColor: 'transparent', + backgroundImage: `url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='black' xmlns='http://www.w3.org/2000/svg'%3e%3cpath d='M4 8a1 1 0 011-1h6a1 1 0 110 2H5a1 1 0 01-1-1z'/%3e%3c/svg%3e")` + }, + ".dark [type='checkbox']:indeterminate": { + backgroundImage: `url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3cpath d='M4 8a1 1 0 011-1h6a1 1 0 110 2H5a1 1 0 01-1-1z'/%3e%3c/svg%3e")` + }, 'input:not(.windmillapp):not(.no-default-style),input[type="text"]:not(.windmillapp):not(.no-default-style),input[type="email"]:not(.windmillapp):not(.no-default-style),input[type="url"]:not(.windmillapp):not(.no-default-style),input[type="password"]:not(.windmillapp):not(.no-default-style),input[type="number"]:not(.windmillapp):not(.no-default-style),input[type="date"]:not(.windmillapp):not(.no-default-style),input[type="datetime-local"]:not(.windmillapp):not(.no-default-style),input[type="month"]:not(.windmillapp):not(.no-default-style),input[type="search"]:not(.windmillapp):not(.no-default-style),input[type="tel"]:not(.windmillapp):not(.no-default-style),input[type="time"]:not(.windmillapp):not(.no-default-style),input[type="week"]:not(.windmillapp):not(.no-default-style),textarea:not(.windmillapp):not(.no-default-style):not(.monaco-mouse-cursor-text),select:not(.windmillapp):not(.no-default-style)': { display: 'block', diff --git a/frontend/vite.config.js b/frontend/vite.config.js index 989fc5cbe2..ac927801dc 100644 --- a/frontend/vite.config.js +++ b/frontend/vite.config.js @@ -1,5 +1,5 @@ import { sveltekit } from '@sveltejs/kit/vite' -import { readFileSync } from 'fs' +import { existsSync, readFileSync } from 'fs' import { fileURLToPath } from 'url' import mkcert from 'vite-plugin-mkcert' @@ -7,6 +7,15 @@ const file = fileURLToPath(new URL('package.json', import.meta.url)) const json = readFileSync(file, 'utf8') const version = JSON.parse(json) +// The postinstall downloads the pinned UI Builder artifact into static/ui_builder, +// which SvelteKit serves at /ui_builder. Serve that directly; only proxy to a +// live UI Builder dev server on :4000 when the bundle is absent (mirrors the +// backend's static-vs-:4000 fallback). Delete static/ui_builder to develop the +// builder against :4000. +const uiBuilderStaticPresent = existsSync( + fileURLToPath(new URL('static/ui_builder/app-preview.html', import.meta.url)) +) + const remoteUrl = process.env.REMOTE ?? (process.env.BACKEND_PORT @@ -84,15 +93,19 @@ const config = { changeOrigin: true, ws: true }, - '^/ui_builder/.*': { - target: 'http://localhost:4000', - changeOrigin: true, - headers: { - 'Cross-Origin-Opener-Policy': 'same-origin', - 'Cross-Origin-Embedder-Policy': 'require-corp', - 'Cross-Origin-Resource-Policy': 'cross-origin' - } - } + ...(uiBuilderStaticPresent + ? {} + : { + '^/ui_builder/.*': { + target: 'http://localhost:4000', + changeOrigin: true, + headers: { + 'Cross-Origin-Opener-Policy': 'same-origin', + 'Cross-Origin-Embedder-Policy': 'require-corp', + 'Cross-Origin-Resource-Policy': 'cross-origin' + } + } + }) } }, preview: { port: 3001 }, diff --git a/integration_tests/ai_agent_tests/conftest.py b/integration_tests/ai_agent_tests/conftest.py index 681304e614..5e1e2cbf9f 100644 --- a/integration_tests/ai_agent_tests/conftest.py +++ b/integration_tests/ai_agent_tests/conftest.py @@ -18,6 +18,80 @@ load_dotenv(Path(__file__).parent / ".env") TEST_IMAGE_PATH = Path(__file__).parent / "test_image.webp" TEST_IMAGE_S3_KEY = "test_images/test_image.webp" +# Env vars each provider needs before its cases can run. Cases for a provider +# whose keys are absent are skipped instead of failed, so a CI run (or a local +# dev) can exercise only the providers it has credentials for. +PROVIDER_ENV_REQUIREMENTS: dict[str, list[str]] = { + "openai": ["OPENAI_API_KEY"], + "azure_openai": ["AZURE_OPENAI_API_KEY", "AZURE_OPENAI_BASE_URL"], + "anthropic": ["ANTHROPIC_API_KEY"], + "google_ai": ["GOOGLE_AI_API_KEY"], + "openrouter": ["OPENROUTER_API_KEY"], + "bedrock": ["BEDROCK_API_KEY"], + "bedrock_api_key": ["BEDROCK_API_KEY"], + "bedrock_iam": ["BEDROCK_IAM_ACCESS_KEY_ID", "BEDROCK_IAM_SECRET_ACCESS_KEY"], + "bedrock_iam_session": [ + "BEDROCK_SESSION_ACCESS_KEY_ID", + "BEDROCK_SESSION_SECRET_ACCESS_KEY", + "BEDROCK_SESSION_TOKEN", + ], + "bedrock_env": ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"], +} + + +def pytest_configure(config): + config.addinivalue_line( + "markers", + "requires_provider(provider): skip test when the provider's credentials are absent", + ) + + +def _provider_name_from_item(item) -> str | None: + callspec = getattr(item, "callspec", None) + if callspec is None: + marker = item.get_closest_marker("requires_provider") + if marker is None: + return None + provider = marker.args[0] if marker.args else marker.kwargs.get("provider") + else: + provider = callspec.params.get("provider_config") + + if isinstance(provider, str): + return provider + if isinstance(provider, dict) and isinstance(provider.get("name"), str): + return provider["name"] + return None + + +def _provider_skip_reason(provider_name: str) -> str | None: + required = PROVIDER_ENV_REQUIREMENTS.get(provider_name, []) + missing = [name for name in required if not os.environ.get(name)] + if missing: + return f"{provider_name}: missing {', '.join(missing)}" + return None + + +def pytest_collection_modifyitems(config, items): + for item in items: + provider_name = _provider_name_from_item(item) + if provider_name is None: + continue + + reason = _provider_skip_reason(provider_name) + if reason is not None: + item.add_marker(pytest.mark.skip(reason=reason)) + + +@pytest.fixture(autouse=True) +def skip_provider_without_credentials(request): + """Skip dynamic provider cases when that provider's keys are not set.""" + provider_name = _provider_name_from_item(request.node) + if provider_name is None: + return + reason = _provider_skip_reason(provider_name) + if reason is not None: + pytest.skip(reason) + class AIAgentTestClient: """HTTP client for testing AI agents via the preview_flow endpoint.""" diff --git a/integration_tests/ai_agent_tests/test_completion_params.py b/integration_tests/ai_agent_tests/test_completion_params.py index c937c540a6..af53ce4ce0 100644 --- a/integration_tests/ai_agent_tests/test_completion_params.py +++ b/integration_tests/ai_agent_tests/test_completion_params.py @@ -5,7 +5,7 @@ Tests that AI agents correctly handle temperature and max_completion_tokens: - Default parameters (undefined) - Low temperature (0.0 - deterministic) - High temperature (0.9 - more random) -- Low max_completion_tokens (10 - short response) +- Low max_completion_tokens (16 - short response; OpenAI's minimum) - High max_completion_tokens (4096 - longer response allowed) - Combined parameters """ @@ -132,14 +132,15 @@ class TestCompletionParams: provider_config, ): """ - Test with max_completion_tokens=10 (short response). - The response should be truncated or very short. + Test with a low max_completion_tokens (16 — short response). + The response should be truncated or very short. 16 is OpenAI's minimum + for max_output_tokens; lower values (e.g. 10) are rejected with a 400. """ flow_value = create_ai_agent_flow( provider_input_transform=provider_config["input_transform"], system_prompt="You are a helpful assistant.", output_type="text", - max_completion_tokens=10, + max_completion_tokens=16, ) result = client.run_preview_flow( @@ -153,7 +154,7 @@ class TestCompletionParams: result_str = str(result) assert len(result_str) > 0, f"Expected non-empty result: {result}" - print(f"Low max_tokens (10) result from {provider_config['name']}: {result}") + print(f"Low max_tokens (16) result from {provider_config['name']}: {result}") @pytest.mark.parametrize( "provider_config", diff --git a/integration_tests/ai_agent_tests/test_tool_calling.py b/integration_tests/ai_agent_tests/test_tool_calling.py index 7ac559250a..c80158fd3a 100644 --- a/integration_tests/ai_agent_tests/test_tool_calling.py +++ b/integration_tests/ai_agent_tests/test_tool_calling.py @@ -119,6 +119,7 @@ class TestToolCalling: assert "23" in result_str, f"Expected '23' in result: {result}" print(f"Workspace script tool result from {provider_config['name']}: {result}") + @pytest.mark.requires_provider("google_ai") def test_nested_ai_agent_tool_with_gemini_3( self, client: AIAgentTestClient, diff --git a/lsp/Pipfile b/lsp/Pipfile index e3c213dd00..62b4fb3a18 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.733.1" +wmill = ">=1.753.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index bd3250d838..25019ca61f 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.733.1 + version: 1.753.0 title: OpenFlow Spec contact: name: Ruben Fiszel @@ -452,6 +452,7 @@ components: enum: - openai - azure_openai + - azure_foundry - anthropic - mistral - deepseek @@ -474,6 +475,9 @@ components: model: type: string description: Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro') + reasoning_effort: + type: string + description: Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default. required: - kind - resource @@ -1027,7 +1031,7 @@ components: - $ref: '#/components/schemas/InputTransform' description: | Boolean. If true, stream the AI response incrementally. - Streaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result + Streaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result memory: $ref: '#/components/schemas/MemoryTransform' output_schema: diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index 39e214fca6..e76ad1a97f 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.733.1' + ModuleVersion = '1.753.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/powershell-client/WindmillClient/WindmillClient.psm1 b/powershell-client/WindmillClient/WindmillClient.psm1 index 9f0aca4b1a..f2d241ddc7 100644 --- a/powershell-client/WindmillClient/WindmillClient.psm1 +++ b/powershell-client/WindmillClient/WindmillClient.psm1 @@ -258,14 +258,15 @@ function Invoke-WindmillScript { [string] $Hash = $null, [Hashtable] $Arguments = @{}, [boolean] $AssertResultIsNotNull = $true, - [int] $Timeout = $null + [int] $Timeout = $null, + [string] $Tag = $null ) if (-not $script:WindmillConnection) { throw "Windmill connection not established. Run Connect-Windmill first." } - $jobId = Start-WindmillScript -Path $Path -Hash $Hash -Arguments $Arguments + $jobId = Start-WindmillScript -Path $Path -Hash $Hash -Arguments $Arguments -Tag $Tag $until = if ($Timeout) { (Get-Date).AddSeconds($Timeout) } else { [DateTime]::MaxValue } return $script:WindmillConnection.WaitJob($jobId, $until, $AssertResultIsNotNull) } @@ -279,14 +280,15 @@ function Start-WindmillScript { [string] $Path = $null, [string] $Hash = $null, [Hashtable] $Arguments = @{}, - [int] $ScheduledInSecs = $null + [int] $ScheduledInSecs = $null, + [string] $Tag = $null ) if (-not $script:WindmillConnection) { throw "Windmill connection not established. Run Connect-Windmill first." } - return $script:WindmillConnection.RunScriptAsync($Path, $Hash, $Arguments, $ScheduledInSecs) + return $script:WindmillConnection.RunScriptAsync($Path, $Hash, $Arguments, $ScheduledInSecs, $Tag) } <# @@ -297,14 +299,15 @@ function Start-WindmillFlow { param( [string] $Path = $null, [Hashtable] $Arguments = @{}, - [int] $ScheduledInSecs = $null + [int] $ScheduledInSecs = $null, + [string] $Tag = $null ) if (-not $script:WindmillConnection) { throw "Windmill connection not established. Run Connect-Windmill first." } - return $script:WindmillConnection.RunFlowAsync($Path, $Arguments, $ScheduledInSecs) + return $script:WindmillConnection.RunFlowAsync($Path, $Arguments, $ScheduledInSecs, $Tag) } <# @@ -596,7 +599,13 @@ class Windmill { return $result } + # Preserve the original 4-arg arity (PowerShell class dispatch is by exact + # argument count, so existing direct callers would otherwise break). [PSCustomObject] RunScriptAsync([string] $Path, [string] $Hash, [Hashtable] $Arguments, [int] $ScheduledInSecs) { + return $this.RunScriptAsync($Path, $Hash, $Arguments, $ScheduledInSecs, $null) + } + + [PSCustomObject] RunScriptAsync([string] $Path, [string] $Hash, [Hashtable] $Arguments, [int] $ScheduledInSecs, [string] $Tag) { $params = @{} if ($Path -and $Hash) { @@ -607,6 +616,10 @@ class Windmill { $params["scheduled_in_secs"] = $ScheduledInSecs } + if ($Tag) { + $params["tag"] = $Tag + } + if ($env:WM_JOB_ID) { $params["parent_job"] = $env:WM_JOB_ID } @@ -631,13 +644,23 @@ class Windmill { return $this.Post($endpoint, $Arguments, $true).Content } + # Preserve the original 3-arg arity (PowerShell class dispatch is by exact + # argument count, so existing direct callers would otherwise break). [string] RunFlowAsync([string] $Path, [Hashtable] $Arguments, [int] $ScheduledInSecs) { + return $this.RunFlowAsync($Path, $Arguments, $ScheduledInSecs, $null) + } + + [string] RunFlowAsync([string] $Path, [Hashtable] $Arguments, [int] $ScheduledInSecs, [string] $Tag) { $params = @{} if ($ScheduledInSecs -ne $null) { $params["scheduled_in_secs"] = $ScheduledInSecs } + if ($Tag) { + $params["tag"] = $Tag + } + # TODO: Figure out why this fails when we set parent_job (at least for HN Discord Feed) if ($env:WM_JOB_ID) { $params["parent_job"] = $env:WM_JOB_ID diff --git a/python-client/tests/wmill_client_test.py b/python-client/tests/wmill_client_test.py index 3309e1ab5f..6df80919cf 100644 --- a/python-client/tests/wmill_client_test.py +++ b/python-client/tests/wmill_client_test.py @@ -124,5 +124,49 @@ SET s3_secret_access_key='80yMndIMcyXwEujxVNINQbf0tBlIzRaLPyM2m1n4'; wmill.load_s3_file(s3_obj) +class TestParseS3Object(unittest.TestCase): + """Pure-unit tests for parse_s3_object — no network/env needed.""" + + def test_bare_string_raises_with_uri_hint(self): + # A bare key is rejected rather than silently uploading under an + # auto-generated name; the error points at the s3:/// spelling. + with self.assertRaisesRegex(ValueError, "s3:///dir/file.json"): + wmill.parse_s3_object("dir/file.json") + + def test_triple_slash_uri_is_default_storage(self): + self.assertEqual( + wmill.parse_s3_object("s3:///dir/file.json"), + S3Object(s3="dir/file.json", storage=None), + ) + + def test_full_uri_splits_storage_and_key(self): + self.assertEqual( + wmill.parse_s3_object("s3://bucket/dir/f"), + S3Object(s3="dir/f", storage="bucket"), + ) + + def test_malformed_uri_raises(self): + # `s3://x` has no key part — fail loudly instead of silently + # misplacing the object. + with self.assertRaises(ValueError): + wmill.parse_s3_object("s3://broken") + + def test_empty_key_uri_raises(self): + # An empty key is never a valid target: it would fall back to an + # auto-generated key, which is requested by omitting the object. + with self.assertRaises(ValueError): + wmill.parse_s3_object("s3:///") + with self.assertRaises(ValueError): + wmill.parse_s3_object("s3://bucket/") + + def test_empty_string_raises(self): + # Auto-generated keys are requested by omitting the object (None), + # not by an empty string. + with self.assertRaises(ValueError): + wmill.parse_s3_object("") + + def test_s3object_passes_through(self): + self.assertEqual(wmill.parse_s3_object(S3Object(s3="x")), S3Object(s3="x")) + if __name__ == "__main__": unittest.main() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index 5dc2fd6c77..d756855c9f 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.733.1" +version = "1.753.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/python-client/wmill/wmill/client.py b/python-client/wmill/wmill/client.py index 18573e297d..fc4c34737d 100644 --- a/python-client/wmill/wmill/client.py +++ b/python-client/wmill/wmill/client.py @@ -168,16 +168,17 @@ class Windmill: hash_: str = None, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job and return its job id. - + .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. """ logging.warning( "run_script_async is deprecated. Use run_script_by_path_async or run_script_by_hash_async instead.", ) assert not (path and hash_), "path and hash_ are mutually exclusive" - return self._run_script_async_internal(path=path, hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs) + return self._run_script_async_internal(path=path, hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs, tag=tag) def _run_script_async_internal( self, @@ -185,10 +186,13 @@ class Windmill: hash_: str = None, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Internal helper for running scripts asynchronously.""" args = args or {} params = {"scheduled_in_secs": scheduled_in_secs} if scheduled_in_secs else {} + if tag: + params["tag"] = tag if os.environ.get("WM_JOB_ID"): params["parent_job"] = os.environ.get("WM_JOB_ID") if os.environ.get("WM_ROOT_FLOW_JOB_ID"): @@ -208,18 +212,20 @@ class Windmill: path: str, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job by path and return its job id.""" - return self._run_script_async_internal(path=path, args=args, scheduled_in_secs=scheduled_in_secs) + return self._run_script_async_internal(path=path, args=args, scheduled_in_secs=scheduled_in_secs, tag=tag) def run_script_by_hash_async( self, hash_: str, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job by hash and return its job id.""" - return self._run_script_async_internal(hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs) + return self._run_script_async_internal(hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs, tag=tag) def run_flow_async( self, @@ -230,10 +236,13 @@ class Windmill: # as otherwise the child flow and its own child will store their state in the parent job which will # lead to incorrectness and failures do_not_track_in_parent: bool = True, + tag: str = None, ) -> str: """Create a flow job and return its job id.""" args = args or {} params = {"scheduled_in_secs": scheduled_in_secs} if scheduled_in_secs else {} + if tag: + params["tag"] = tag if not do_not_track_in_parent: if os.environ.get("WM_JOB_ID"): params["parent_job"] = os.environ.get("WM_JOB_ID") @@ -254,9 +263,10 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Run script synchronously and return its result. - + .. deprecated:: Use run_script_by_path or run_script_by_hash instead. """ logging.warning( @@ -265,7 +275,7 @@ class Windmill: assert not (path and hash_), "path and hash_ are mutually exclusive" return self._run_script_internal( path=path, hash_=hash_, args=args, timeout=timeout, verbose=verbose, - cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none + cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none, tag=tag ) def _run_script_internal( @@ -277,6 +287,7 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Internal helper for running scripts synchronously.""" args = args or {} @@ -290,7 +301,7 @@ class Windmill: if isinstance(timeout, dt.timedelta): timeout = timeout.total_seconds() - job_id = self._run_script_async_internal(path=path, hash_=hash_, args=args) + job_id = self._run_script_async_internal(path=path, hash_=hash_, args=args, tag=tag) return self.wait_job( job_id, timeout, verbose, cleanup, assert_result_is_not_none ) @@ -303,11 +314,12 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Run script by path synchronously and return its result.""" return self._run_script_internal( path=path, args=args, timeout=timeout, verbose=verbose, - cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none + cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none, tag=tag ) def run_script_by_hash( @@ -318,11 +330,12 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Run script by hash synchronously and return its result.""" return self._run_script_internal( hash_=hash_, args=args, timeout=timeout, verbose=verbose, - cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none + cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none, tag=tag ) def run_inline_script_preview( @@ -1453,6 +1466,7 @@ def run_script_async( hash_or_path: str, args: Dict[str, Any] = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job and return its job ID. @@ -1460,6 +1474,7 @@ def run_script_async( hash_or_path: Script hash or path (determined by presence of '/') args: Script arguments scheduled_in_secs: Delay before execution in seconds + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1472,6 +1487,7 @@ def run_script_async( path=path, args=args, scheduled_in_secs=scheduled_in_secs, + tag=tag, ) @@ -1484,6 +1500,7 @@ def run_flow_async( # as otherwise the child flow and its own child will store their state in the parent job which will # lead to incorrectness and failures do_not_track_in_parent: bool = True, + tag: str = None, ) -> str: """Create a flow job and return its job ID. @@ -1492,6 +1509,7 @@ def run_flow_async( args: Flow arguments scheduled_in_secs: Delay before execution in seconds do_not_track_in_parent: Whether to track in parent job (default: True) + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1501,6 +1519,7 @@ def run_flow_async( args=args, scheduled_in_secs=scheduled_in_secs, do_not_track_in_parent=do_not_track_in_parent, + tag=tag, ) @@ -1512,6 +1531,7 @@ def run_script_sync( assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, + tag: str = None, ) -> Any: """Run a script synchronously by hash and return its result. @@ -1522,6 +1542,7 @@ def run_script_sync( assert_result_is_not_none: Raise exception if result is None cleanup: Register cleanup handler to cancel job on exit timeout: Maximum time to wait + tag: Override the worker tag the job runs on Returns: Script result @@ -1533,6 +1554,7 @@ def run_script_sync( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -1541,6 +1563,7 @@ def run_script_by_path_async( path: str, args: Dict[str, Any] = None, scheduled_in_secs: Union[None, int] = None, + tag: str = None, ) -> str: """Create a script job by path and return its job ID. @@ -1548,6 +1571,7 @@ def run_script_by_path_async( path: Script path args: Script arguments scheduled_in_secs: Delay before execution in seconds + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1556,6 +1580,7 @@ def run_script_by_path_async( path=path, args=args, scheduled_in_secs=scheduled_in_secs, + tag=tag, ) @@ -1564,6 +1589,7 @@ def run_script_by_hash_async( hash_: str, args: Dict[str, Any] = None, scheduled_in_secs: Union[None, int] = None, + tag: str = None, ) -> str: """Create a script job by hash and return its job ID. @@ -1571,6 +1597,7 @@ def run_script_by_hash_async( hash_: Script hash args: Script arguments scheduled_in_secs: Delay before execution in seconds + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1579,6 +1606,7 @@ def run_script_by_hash_async( hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs, + tag=tag, ) @@ -1590,6 +1618,7 @@ def run_script_by_path_sync( assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, + tag: str = None, ) -> Any: """Run a script synchronously by path and return its result. @@ -1600,6 +1629,7 @@ def run_script_by_path_sync( assert_result_is_not_none: Raise exception if result is None cleanup: Register cleanup handler to cancel job on exit timeout: Maximum time to wait + tag: Override the worker tag the job runs on Returns: Script result @@ -1611,6 +1641,7 @@ def run_script_by_path_sync( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -2062,9 +2093,10 @@ def run_script( verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = True, + tag: str = None, ) -> Any: """Run script synchronously and return its result. - + .. deprecated:: Use run_script_by_path or run_script_by_hash instead. """ return _client.run_script( @@ -2075,6 +2107,7 @@ def run_script( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -2086,6 +2119,7 @@ def run_script_by_path( verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = True, + tag: str = None, ) -> Any: """Run script by path synchronously and return its result.""" return _client.run_script_by_path( @@ -2095,6 +2129,7 @@ def run_script_by_path( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -2106,6 +2141,7 @@ def run_script_by_hash( verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = True, + tag: str = None, ) -> Any: """Run script by hash synchronously and return its result.""" return _client.run_script_by_hash( @@ -2115,6 +2151,7 @@ def run_script_by_hash( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @init_global_client @@ -2175,12 +2212,27 @@ def parse_resource_syntax(s: str) -> Optional[str]: return None def parse_s3_object(s3_object: S3Object | str) -> S3Object: - """Parse S3 object from string or S3Object format.""" + """Parse S3 object from a `s3:///` URI string (`s3:///` + for the default storage) or S3Object format. Any other string raises + rather than falling back to an auto-generated key: an auto key is + requested by omitting the object, and a fallback would silently misplace + the upload on any typo. + """ if isinstance(s3_object, str): - match = re.match(r'^s3://([^/]*)/(.*)$', s3_object) + match = re.match(r'^s3://([^/]*)/(.+)$', s3_object) if match: - return S3Object(s3=match.group(2) or "", storage=match.group(1) or None) - return S3Object(s3="") + return S3Object(s3=match.group(2), storage=match.group(1) or None) + if s3_object.startswith("s3://"): + raise ValueError( + f"Invalid s3 object URI {s3_object!r}: expected " + "s3:/// with a non-empty key " + "(s3:/// for the default storage)" + ) + raise ValueError( + f"Invalid s3 object {s3_object!r}: expected an s3:/// " + f"URI (e.g. 's3:///{s3_object}' for key {s3_object!r} in the default " + "storage) or S3Object(s3=)" + ) else: return s3_object @@ -2289,6 +2341,131 @@ class DucklakeClient: ) ) + def _qualified(self, table: str, schema: str = None) -> str: + return f'dl."{schema}"."{table}"' if schema else f"dl.{table}" + + def _materialize_finish(self, sql, table, schema, partition, partition_col): + """Return the materialize query; in a pipeline (WM_PIPELINE) append a + summary read and record materialized_partition state after a successful + run so SDK-materialized slices appear in the grid like `// materialize` + ones. Outside a pipeline it stays a plain query (no recording).""" + bind = {} if partition is None else {"_wm_partition": partition} + if os.environ.get("WM_PIPELINE") != "true": + return self.query(sql, **bind) + t = self._qualified(table, schema) + where = f" WHERE {partition_col} = $_wm_partition" if partition is not None else "" + summary = ( + f"\nSELECT (SELECT count(*) FROM {t}{where}) AS rows, " + f"(SELECT max(snapshot_id) FROM ducklake_snapshots('dl')) AS snapshot_id;" + ) + q = self.query(sql + summary, **bind) + # Asset path mirrors the `// materialize` engine: /. + # for an explicit schema, else /
. Dropping the schema would + # hide the row from the grid and collide distinct schemas under one key. + asset_path = f"{self.name}/{schema}.{table}" if schema else f"{self.name}/{table}" + return _RecordingSqlQuery(q, self.client, asset_path, partition or "") + + def upsert_partition( + self, + table: str, + select_sql: str, + partition: str = None, + unique_key: str = None, + partition_col: str = "_wm_partition", + schema: str = None, + ): + """Idempotently materialize the rows of `select_sql` into ducklake + `table` for one `partition` (or the whole table when `partition` is + None). Client-side equivalent of the `// materialize` engine: with + `unique_key` it upserts within the slice (delete-by-key + insert); + without it, it replaces (whole table → CREATE OR REPLACE; partition → + delete the partition + insert). Re-running the same slice is safe — the + backfill / failure-recovery contract. + + The partition value is bound as a DuckDB arg (never string-interpolated) + so it cannot inject SQL. `select_sql` is trusted (your own query). + """ + t = self._qualified(table, schema) + # Whole-table (no partition): no partition column; replace rebuilds the + # table with CREATE OR REPLACE, merge upserts the whole table by key. + if partition is None: + if unique_key: + sql = ( + f"CREATE TABLE IF NOT EXISTS {t} AS SELECT * FROM ({select_sql}) WHERE false;\n" + f"BEGIN TRANSACTION;\n" + f"DELETE FROM {t} WHERE {unique_key} IN (SELECT {unique_key} FROM ({select_sql}));\n" + f"INSERT INTO {t} SELECT * FROM ({select_sql});\n" + f"COMMIT;" + ) + else: + sql = f"CREATE OR REPLACE TABLE {t} AS SELECT * FROM ({select_sql});" + return self._materialize_finish(sql, table, schema, partition, partition_col) + src = f"SELECT *, $_wm_partition AS {partition_col} FROM ({select_sql})" + if unique_key: + # Upsert via delete-by-key + insert (not MERGE — DuckLake's MERGE + # fails writing the first rows of a fresh partition). + body = ( + f"DELETE FROM {t} WHERE {partition_col} = $_wm_partition " + f"AND {unique_key} IN (SELECT {unique_key} FROM ({select_sql}));\n" + f"INSERT INTO {t} {src};" + ) + else: + body = ( + f"DELETE FROM {t} WHERE {partition_col} = $_wm_partition;\n" + f"INSERT INTO {t} {src};" + ) + sql = ( + f"CREATE TABLE IF NOT EXISTS {t} AS " + f"SELECT *, CAST(NULL AS VARCHAR) AS {partition_col} FROM ({select_sql}) WHERE false;\n" + f"ALTER TABLE {t} SET PARTITIONED BY ({partition_col});\n" + f"BEGIN TRANSACTION;\n{body}\nCOMMIT;" + ) + return self._materialize_finish(sql, table, schema, partition, partition_col) + + def append_partition( + self, + table: str, + select_sql: str, + partition: str = None, + partition_col: str = "_wm_partition", + schema: str = None, + ): + """INSERT-only materialization (no dedup / no replace) for an immutable + event-log table — for one `partition`, or the whole table when + `partition` is None. NOTE: unlike `upsert_partition`, re-running the same + slice duplicates rows — use only for append-only sources.""" + t = self._qualified(table, schema) + # Whole-table (no partition): insert into the bare table, no partition col. + if partition is None: + sql = ( + f"CREATE TABLE IF NOT EXISTS {t} AS SELECT * FROM ({select_sql}) WHERE false;\n" + f"INSERT INTO {t} SELECT * FROM ({select_sql});" + ) + return self._materialize_finish(sql, table, schema, partition, partition_col) + sql = ( + f"CREATE TABLE IF NOT EXISTS {t} AS " + f"SELECT *, CAST(NULL AS VARCHAR) AS {partition_col} FROM ({select_sql}) WHERE false;\n" + f"ALTER TABLE {t} SET PARTITIONED BY ({partition_col});\n" + f"INSERT INTO {t} SELECT *, $_wm_partition AS {partition_col} FROM ({select_sql});" + ) + return self._materialize_finish(sql, table, schema, partition, partition_col) + + def read( + self, + table: str, + partition: str = None, + partition_col: str = "_wm_partition", + schema: str = None, + ): + """Read a materialized ducklake table, optionally a single partition.""" + t = self._qualified(table, schema) + if partition is not None: + return self.query( + f"SELECT * FROM {t} WHERE {partition_col} = $_wm_partition", + _wm_partition=partition, + ) + return self.query(f"SELECT * FROM {t}") + class SqlQuery: """Query result handler for DataTable and DuckLake queries.""" @@ -2337,6 +2514,60 @@ class SqlQuery: """ self.fetch_one() + +class _RecordingSqlQuery: + """Wraps a ducklake materialize query so that, on a successful run, the + trailing summary (row count + snapshot id) is captured and the + materialized_partition state is recorded (best-effort). Only used in pipeline + context — outside it the helpers return a plain SqlQuery. Mirrors SqlQuery's + terminal methods so `.execute()` / `.fetch_one()` behave the same.""" + + def __init__(self, inner, client, asset_path, partition): + self._inner = inner + self._client = client + self._asset_path = asset_path + self._partition = partition + self.sql = inner.sql + + def execute(self): + self._run() + + def fetch_one(self): + return self._run() + + def fetch(self, result_collection=None): + return self._run() + + def _run(self): + try: + row = self._inner.fetch_one() + except Exception as e: + self._record("failed", None, None, str(e)) + raise + snap = row.get("snapshot_id") if isinstance(row, dict) else None + rows = row.get("rows") if isinstance(row, dict) else None + self._record("materialized", snap, rows, None) + return row + + def _record(self, status, snapshot_id, row_count, error): + try: + self._client.post( + f"/w/{self._client.workspace}/assets/record_materialization", + json={ + "asset_kind": "ducklake", + "asset_path": self._asset_path, + "partition": self._partition, + "status": status, + "snapshot_id": snapshot_id, + "row_count": row_count, + "job_id": os.environ.get("WM_JOB_ID"), + "error": error, + }, + ) + except Exception: + pass # best-effort; never fail the user's materialization + + def infer_sql_type(value) -> str: """ DuckDB executor requires explicit argument types at declaration diff --git a/rust-client/src/client.rs b/rust-client/src/client.rs index 4fc9eca192..451dc58d85 100644 --- a/rust-client/src/client.rs +++ b/rust-client/src/client.rs @@ -629,7 +629,43 @@ impl Windmill { ret!(async move { let job_id = self - .run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs) + .run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, None) + .await?; + self.wait_job_inner( + &job_id.to_string(), + timeout_secs, + verbose, + assert_result_is_not_none, + ) + .await + }); + } + + /// Same as [`Windmill::run_script_sync`] but allows overriding the worker `tag` + /// the job runs on. + /// + /// # Parameters + /// In addition to the parameters of [`Windmill::run_script_sync`]: + /// - `tag`: Optional worker tag override (the job is dispatched to workers + /// listening on this tag instead of the script's default tag) + pub fn run_script_sync_with_tag<'a>( + &'a self, + ident: &'a str, + ident_is_hash: bool, + args: Value, + scheduled_in_secs: Option, + timeout_secs: Option, + verbose: bool, + assert_result_is_not_none: bool, + tag: Option<&'a str>, + ) -> MaybeFuture<'a, Result> { + if verbose { + println!("running `{ident}` synchronously with {:?}", &args); + } + + ret!(async move { + let job_id = self + .run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, tag) .await?; self.wait_job_inner( &job_id.to_string(), @@ -690,7 +726,25 @@ impl Windmill { args: Value, scheduled_in_secs: Option, ) -> MaybeFuture<'a, Result> { - ret!(self.run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs)); + ret!(self.run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, None)); + } + + /// Same as [`Windmill::run_script_async`] but allows overriding the worker `tag` + /// the job runs on. + /// + /// # Arguments + /// In addition to the arguments of [`Windmill::run_script_async`]: + /// * `tag` - Optional worker tag override (the job is dispatched to workers + /// listening on this tag instead of the script's default tag) + pub fn run_script_async_with_tag<'a>( + &'a self, + ident: &'a str, + ident_is_hash: bool, + args: Value, + scheduled_in_secs: Option, + tag: Option<&'a str>, + ) -> MaybeFuture<'a, Result> { + ret!(self.run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, tag)); } async fn run_script_async_inner<'a>( @@ -699,6 +753,7 @@ impl Windmill { ident_is_hash: bool, mut args: Value, scheduled_in_secs: Option, + tag: Option<&'a str>, ) -> Result { if let Ok(parent_job) = var("WM_JOB_ID") { args["parent_job"] = json!(parent_job); @@ -712,6 +767,9 @@ impl Windmill { args["scheduled_in_secs"] = json!(scheduled_in_secs); } + // The `None`s below map positionally to the query params of the generated + // job API. The 5th one is the worker `tag` override (after scheduled_for, + // scheduled_in_secs, skip_preprocessor and parent_job). let uuid = if ident_is_hash { job_api::run_script_by_hash( &self.client_config, @@ -722,7 +780,7 @@ impl Windmill { None, None, None, - None, + tag, None, None, None, @@ -746,7 +804,7 @@ impl Windmill { None, None, None, - None, + tag, None, None, None, diff --git a/sandbox-image/Dockerfile.sandbox b/sandbox-image/Dockerfile.sandbox index 7c9ca181a4..7666366720 100644 --- a/sandbox-image/Dockerfile.sandbox +++ b/sandbox-image/Dockerfile.sandbox @@ -1,4 +1,4 @@ -FROM debian:bookworm-slim +FROM debian:trixie-slim # ── Minimal system deps ────────────────────────────────────────────────────── RUN apt-get update && apt-get install -y --no-install-recommends \ diff --git a/system_prompts/auto-generated/cli/cli-commands.md b/system_prompts/auto-generated/cli/cli-commands.md index 2e54a87395..3d5d6429d4 100644 --- a/system_prompts/auto-generated/cli/cli-commands.md +++ b/system_prompts/auto-generated/cli/cli-commands.md @@ -77,6 +77,13 @@ datatable related commands - `datatable run ` - run a SQL query on a datatable - `-n --name ` - Datatable name (default: main) - `-s --silent` - Output only the final result as JSON. Useful for scripting. +- `datatable migrate` - manage datatable migrations + - `datatable migrate new ` - scaffold a new migration (.up.sql / .down.sql files) + - `-d --datatable ` - Target datatable (default: main) + - `datatable migrate up` - apply all pending migrations to the main datatable (or one via --datatable) + - `-d --datatable ` - Target datatable (default: main) + - `datatable migrate down` - roll back the most recent migration on the main datatable (or one via --datatable) + - `-d --datatable ` - Target datatable (default: main) - `datatable create [name:string]` - register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable:// - `--resource ` - Back the datatable with an existing postgresql resource path instead of the instance database - `--force` - Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved) @@ -345,19 +352,18 @@ Manage jobs (list, inspect, cancel) ### jobs -Pull completed and queued jobs from workspace - -**Arguments:** `[workspace:string]` - -**Options:** -- `-c, --completed-output ` - Completed jobs output file (default: completed_jobs.json) -- `-q, --queued-output ` - Queued jobs output file (default: queued_jobs.json) -- `--skip-worker-check` - Skip checking for active workers before export +Manage jobs (import/export) **Subcommands:** -- `jobs pull` -- `jobs push` +- `jobs pull [workspace:string]` - Pull completed and queued jobs from workspace + - `-c, --completed-output ` - Completed jobs output file (default: completed_jobs.json) + - `-q, --queued-output ` - Queued jobs output file (default: queued_jobs.json) + - `--skip-worker-check` - Skip checking for active workers before export +- `jobs push [workspace:string]` - Push completed and queued jobs to workspace + - `-c, --completed-file ` - Completed jobs input file (default: completed_jobs.json) + - `-q, --queued-file ` - Queued jobs input file (default: queued_jobs.json) + - `--skip-worker-check` - Skip checking for active workers before import ### lint @@ -419,6 +425,22 @@ inspect asset-driven pipelines (scripts marked `// pipeline`, wired by `// on ` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal - `--json` - Output the raw asset graph as JSON + - `--local` - Build the graph from local working-tree files (// pipeline scripts) instead of the deployed workspace — no deploy needed. +- `pipeline run ` - run a cascade: from --from (a root OR any mid-DAG model), fan downstream up to the --to end node(s) + - `--from ` - Start script (short name or path). May be any node, including a mid-DAG model — that node plus its transitive downstream runs, upstream is NOT re-run (dbt `--select model+`). Defaults to the folder's sole schedule/manual root. + - `--to ` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream. + - `--dry-run` - Print the topological run plan without executing. + - `--json` - Output the plan as JSON (for piping to jq). + - `--local` - Run the local working-tree scripts via preview (no deploy) instead of the deployed versions; the graph is built from local files. + - `--upload ` - Bind an object to a data_upload/webhook entry point so it runs in the cascade, as SCRIPT[:PARAM]=SOURCE (SOURCE is a local file or an s3://key). Local files are uploaded to the workspace store; the S3Object param is inferred when the script has exactly one. Repeatable. + - `--arg ` - Pass a plain run arg to a script in the cascade, as SCRIPT:PARAM=VALUE (VALUE is parsed as JSON when possible, else taken as a string — e.g. daily_report:partition=2026-07-02). Repeatable. + - `--partition ` - Partition value for `// partitioned` scripts in the run (e.g. 2026-06-30) — use it to backfill a past slice. With --local, time kinds (daily/hourly/weekly/monthly) default to the current UTC period when omitted; `dynamic` always needs it. Deployed runs without it defer to backend run-start resolution. +- `pipeline docs ` - generate PIPELINE.md (+ AGENTS.md pointer) describing a folder's pipeline graph and datatable schemas, for an editor / agentic loop + - `--local` - Build the graph from local working-tree files instead of the deployed workspace. +- `pipeline dev [folder:string]` - Live-preview a data pipeline from local files: watch an `f/` of `// pipeline` scripts, push the working-tree graph to the dev page, and run the cascade via preview (no deploy). + - `--port ` - Port for the dev WebSocket server. + - `--no-open` - Do not open the browser automatically. + - `--frontend ` - Origin serving the /pipeline_dev page (e.g. http://localhost:3000 for a locally-run frontend). Defaults to the workspace remote; use it when the remote's deployed frontend predates the dev page. ### protection-rules @@ -744,13 +766,19 @@ workspace related commands - `--branch ` - Git branch to associate (default: workspace name) - `workspace unbind` - Remove baseUrl and workspaceId from a workspace entry - `--workspace ` - Workspace to unbind -- `workspace fork [workspace_name:string] [workspace_id:string]` - Create a forked workspace +- `workspace fork [workspace_name:string] [workspace_id:string]` - Create a forked workspace from its parent workspace. + +The parent is resolved from your current git branch, not from the active profile: run this from a git repo checked out on the branch mapped to the parent workspace in wmill.yaml's `workspaces:` section (a fork branch of it resolves to the same parent). `wmill workspace switch` does not change which workspace is forked. + +Arguments (omit both to be prompted interactively): + [workspace_name] Friendly display name for the fork, shown in the UI. May contain spaces, so quote it in the shell (e.g. "My Fork"). Max 50 chars. Defaults to "'s fork". + [workspace_id] Id for the fork. Must be a slug (no spaces or special characters) and is automatically prefixed with `wm-fork-`, so pass just the bare slug (e.g. `my-fork` becomes `wm-fork-my-fork`). This id also determines the fork's git branch name. Defaults to a slug derived from the name — or, when you are converting an existing branch into the fork branch, from that branch. - `--create-workspace-name ` - Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id. - `--color ` - Workspace color (hex code, e.g. #ff0000) - `--datatable-behavior ` - How to handle datatables: skip, schema_only, or schema_and_data (default: interactive prompt) - `--from-branch ` - Non-interactive override for the 'turn my current working branch into the fork' workflow: base the fork on (its bound workspace is the parent) and rename the current branch onto wm-fork//. Usually unneeded — from a working branch `wmill workspace fork` offers this interactively; from a base branch it creates a fresh fork branch. - `-y --yes` - Skip interactive prompts (defaults datatable behavior to 'skip'). On a non-base branch, requires --from-branch since the base branch can't be prompted for. -- `workspace delete-fork ` - Delete a forked workspace and git branch +- `workspace delete-fork ` - Delete a forked workspace - `-y --yes` - Skip confirmation prompt - `workspace merge` - Compare and deploy changes between a fork and its parent workspace - `--direction ` - Deploy direction: to-parent or to-fork diff --git a/system_prompts/auto-generated/flow.md b/system_prompts/auto-generated/flow.md index 97d0d5143a..08bcc2d641 100644 --- a/system_prompts/auto-generated/flow.md +++ b/system_prompts/auto-generated/flow.md @@ -139,8 +139,11 @@ input_transforms: ## Approval / Suspend Structure +An approval step is a normal **script** step (`type: rawscript` or `type: script`) that is turned into an approval by adding a module-level `suspend`. Its script calls `wmill.getResumeUrls(approver)` to generate the secret resume/cancel URLs and returns them so they can be sent to the approver(s) (Slack, email, etc.) or approved from the run page. + - `suspend` belongs on the flow module object itself, as a sibling of `id` and `value` - Never put `suspend` inside `value` +- Do NOT use `type: identity` for an approval step. An identity step suspends but never produces the resume URLs, so approvers have no link to act on — it is not a functional approval. Correct shape: @@ -156,10 +159,23 @@ Correct shape: type: string required: [comment] value: - type: identity + type: rawscript + language: bun + input_transforms: + approver: + type: static + value: '' + content: | + import * as wmill from "windmill-client" + + export async function main(approver?: string) { + const urls = await wmill.getResumeUrls(approver) + // send urls.resume / urls.cancel to the approver(s), e.g. via Slack or email + return urls + } ``` -Incorrect shape: +Incorrect shape (suspend misplaced inside `value`): ```yaml - id: request_approval @@ -169,6 +185,16 @@ Incorrect shape: required_events: 1 ``` +Incorrect shape (identity has no resume URLs — not a real approval): + +```yaml +- id: request_approval + suspend: + required_events: 1 + value: + type: identity +``` + ## Branch Result Scope Rules - Inside a branch, you may reference earlier outer steps and earlier steps in the same branch @@ -295,4 +321,4 @@ Reference a specific resource using `$res:` prefix: ## OpenFlow Schema -{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}} \ No newline at end of file +{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}} \ No newline at end of file diff --git a/system_prompts/auto-generated/index.d.ts b/system_prompts/auto-generated/index.d.ts index f7d8d632b4..8333db7bdc 100644 --- a/system_prompts/auto-generated/index.d.ts +++ b/system_prompts/auto-generated/index.d.ts @@ -3,5 +3,6 @@ export declare function getScriptPrompt(language: string): string; export declare function getFlowPrompt(): string; export declare function getResourcePrompt(): string; export declare function getRawAppPrompt(): string; +export declare function getPipelinePrompt(): string; export declare function getDatatableSdkReference(language?: string): string; export declare function getWorkflowAsCodePrompt(language?: string): string; diff --git a/system_prompts/auto-generated/index.ts b/system_prompts/auto-generated/index.ts index 6ee3e20f4a..2e2a88cae6 100644 --- a/system_prompts/auto-generated/index.ts +++ b/system_prompts/auto-generated/index.ts @@ -54,6 +54,11 @@ export function getRawAppPrompt(): string { return prompts.RAW_APP_BASE; } +// Helper for data pipeline authoring (chat consumers) +export function getPipelinePrompt(): string { + return prompts.PIPELINE_BASE; +} + // Helper to get the datatable SQL SDK reference (wmill.datatable()). // Pass a language to get only that SDK; omit it to get both. export function getDatatableSdkReference(language?: string): string { diff --git a/system_prompts/auto-generated/prompts.d.ts b/system_prompts/auto-generated/prompts.d.ts index 32ddcdc4bd..e74555504e 100644 --- a/system_prompts/auto-generated/prompts.d.ts +++ b/system_prompts/auto-generated/prompts.d.ts @@ -1,34 +1,37 @@ -export declare const SCRIPT_BASE = "# Windmill Script Writing Guide\n\n## General Principles\n\n- Scripts must export a main function (do not call it)\n- Libraries are installed automatically - do not show installation instructions\n- Credentials and configuration are stored in resources and passed as parameters\n- The windmill client (`wmill`) provides APIs for interacting with the platform\n\n## Function Naming\n\n- Main function: `main` (or `preprocessor` for preprocessor scripts)\n- Must be async for TypeScript variants\n\n## Return Values\n\n- Scripts can return any JSON-serializable value\n- Return values become available to subsequent flow steps via `results.step_id`\n\n## Preprocessor Scripts\n\nPreprocessor scripts process raw trigger data from various sources (webhook, custom HTTP route, SQS, WebSocket, Kafka, NATS, MQTT, Postgres, or email) before passing it to the flow. This separates the trigger logic from the flow logic and keeps the auto-generated UI clean.\n\nThe returned object determines the parameter values passed to the flow.\ne.g., `{ b: 1, a: 2 }` calls the flow with `a = 2` and `b = 1`, assuming the flow has two inputs called `a` and `b`.\n\nThe preprocessor receives a single parameter called `event`.\n"; -export declare const FLOW_BASE = "# Windmill Flow Building Guide\n\n## OpenFlow Schema\n\nThe OpenFlow schema (openflow.openapi.yaml) is the source of truth for flow structure. Refer to OPENFLOW_SCHEMA for the complete type definitions.\n\n## Reserved Module IDs\n\n- `failure` - Reserved for failure handler module\n- `preprocessor` - Reserved for preprocessor module\n- `Input` - Reserved for flow input reference\n\n## Hard Structural Rules\n\nThese are strict Windmill schema rules. Follow them exactly.\n\n- `value.modules` is only for normal sequential steps\n- `value.preprocessor_module` and `value.failure_module` are special top-level fields inside `value`, not entries in `value.modules`\n- If a flow needs a preprocessor, create `value.preprocessor_module` with `id: preprocessor`\n- If a flow needs a failure handler, create `value.failure_module` with `id: failure`\n- Do NOT create regular modules inside `value.modules` named `preprocessor` or `failure`\n- `preprocessor_module` and `failure_module` only support `script` or `rawscript`\n- `preprocessor_module` runs before normal modules and cannot reference `results.*`\n- `failure_module` can use the `error` object with `error.message`, `error.step_id`, `error.name`, and `error.stack`\n\nCorrect shape:\n\n```yaml\nvalue:\n preprocessor_module:\n id: preprocessor\n value:\n type: rawscript\n ...\n failure_module:\n id: failure\n value:\n type: rawscript\n ...\n modules:\n - id: process_event\n value:\n type: rawscript\n ...\n```\n\nIncorrect shape:\n\n```yaml\nvalue:\n modules:\n - id: preprocessor\n ...\n - id: process_event\n ...\n - id: failure\n ...\n```\n\n## Module ID Rules\n\n- Must be unique across the entire flow\n- Use underscores, not spaces (e.g., `fetch_data` not `fetch data`)\n- Use descriptive names that reflect the step's purpose\n\n## Common Mistakes to Avoid\n\n- Missing `input_transforms` - Rawscript parameters won't receive values without them\n- Referencing future steps - `results.step_id` only works for steps that execute before the current one\n- Duplicate module IDs - Each module ID must be unique in the flow\n\n## Data Flow Between Steps\n\n- `flow_input.property` - Access flow input parameters\n- `results.step_id` - Access output from a previous step only when that step result is in scope\n- `results.step_id.property` - Access specific property from a previous step output only when that step result is in scope\n- `flow_input.iter.value` - Current iteration value when inside a loop (`forloopflow` or `whileloopflow`)\n- `flow_input.iter.index` - Current loop index when inside a loop (`forloopflow` or `whileloopflow`)\n\n## Loop Structure Rules\n\n- For `whileloopflow`, use module-level `stop_after_if` on the loop module itself when the loop should stop after an iteration result\n- Do NOT put `stop_after_if` inside `value` of a `whileloopflow`\n- `stop_after_all_iters_if` is for checks after the whole loop finishes, not the normal per-iteration break condition\n- When a `whileloopflow` carries state forward between iterations, use `flow_input.iter.value` as the current loop value and provide an explicit first-iteration fallback when needed\n- Use `flow_input.iter.index` only when the loop logic is truly based on the iteration index, not as a replacement for the current loop value\n- If the user asks for a final scalar/object after a loop, add a normal step after the loop that extracts the final value from the loop result instead of returning the whole loop result array\n\nCorrect `whileloopflow` shape:\n\n```yaml\n- id: loop_until_done\n stop_after_if:\n expr: result.done === true\n skip_if_stopped: false\n value:\n type: whileloopflow\n skip_failures: false\n modules:\n - id: advance_state\n value:\n type: rawscript\n input_transforms:\n state:\n type: javascript\n expr: flow_input.iter && flow_input.iter.value !== undefined ? flow_input.iter.value : flow_input.initial_state\n- id: return_final_state\n value:\n type: rawscript\n input_transforms:\n final_state:\n type: javascript\n expr: results.loop_until_done[results.loop_until_done.length - 1]\n```\n\nIncorrect `whileloopflow` patterns:\n\n```yaml\n- id: loop_until_done\n value:\n type: whileloopflow\n stop_after_if:\n expr: result.done === true\n```\n\n```yaml\ninput_transforms:\n state:\n type: javascript\n expr: flow_input.iter.index\n```\n\n```yaml\ninput_transforms:\n final_state:\n type: javascript\n expr: results.loop_until_done\n```\n\n## Approval / Suspend Structure\n\n- `suspend` belongs on the flow module object itself, as a sibling of `id` and `value`\n- Never put `suspend` inside `value`\n\nCorrect shape:\n\n```yaml\n- id: request_approval\n suspend:\n required_events: 1\n resume_form:\n schema:\n type: object\n properties:\n comment:\n type: string\n required: [comment]\n value:\n type: identity\n```\n\nIncorrect shape:\n\n```yaml\n- id: request_approval\n value:\n type: rawscript\n suspend:\n required_events: 1\n```\n\n## Branch Result Scope Rules\n\n- Inside a branch, you may reference earlier outer steps and earlier steps in the same branch\n- Outside a `branchone`, do NOT reference ids of steps that only exist inside its branches or default branch. Use `results.` instead\n- Outside a `branchall`, do NOT reference ids of steps inside its branches. Use `results.` instead\n- If downstream steps need a stable shape after a branch, make each branch return the same fields\n- When needed, add a normalization step immediately after the branch and consume `results.` there\n\nCorrect after `branchone`:\n\n```yaml\n- id: route_order\n value:\n type: branchone\n ...\n- id: send_confirmation\n value:\n input_transforms:\n routed:\n type: javascript\n expr: results.route_order\n```\n\nIncorrect after `branchone`:\n\n```yaml\nexpr: results.create_shipment\nexpr: results.create_backorder\n```\n\nCorrect after `branchall`:\n\n```yaml\n- id: enrich_parallel\n value:\n type: branchall\n parallel: true\n ...\n- id: combine_data\n value:\n input_transforms:\n enrichments:\n type: javascript\n expr: results.enrich_parallel\n```\n\n## Input Transforms\n\nEvery rawscript module needs `input_transforms` to map function parameters to values:\n\nStatic transform (fixed value):\n{\"param_name\": {\"type\": \"static\", \"value\": \"fixed_string\"}}\n\nJavaScript transform (dynamic expression):\n{\"param_name\": {\"type\": \"javascript\", \"expr\": \"results.previous_step.data\"}}\n\n## Resource References\n\n- For flow inputs: Use type `\"object\"` with format `\"resource-{type}\"` (e.g., `\"resource-postgresql\"`)\n- For step inputs: Use static value `\"$res:path/to/resource\"`\n\n## Final Structural Self-Check\n\nBefore finalizing a flow, verify:\n\n- any preprocessor is in `value.preprocessor_module`\n- any failure handler is in `value.failure_module`\n- any approval step has module-level `suspend`\n- no downstream step references inner branch step ids from outside the branch\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations.\n\nTo accept an S3 object as flow input:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"file\": {\n \"type\": \"object\",\n \"format\": \"resource-s3_object\",\n \"description\": \"File to process\"\n }\n }\n}\n```\n\n## Using Resources in Flows\n\nOn Windmill, credentials and configuration are stored in resources. Resource types define the format of the resource.\n\n### As Flow Input\n\nIn the flow schema, set the property type to `\"object\"` with format `\"resource-{type}\"`:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"database\": {\n \"type\": \"object\",\n \"format\": \"resource-postgresql\",\n \"description\": \"Database connection\"\n }\n }\n}\n```\n\n### As Step Input (Static Reference)\n\nReference a specific resource using `$res:` prefix:\n\n```json\n{\n \"database\": {\n \"type\": \"static\",\n \"value\": \"$res:f/folder/my_database\"\n }\n}\n```\n"; -export declare const RESOURCES_BASE = "# Windmill Resources\n\nResources store credentials and configuration for external services.\n\n## File Format\n\nResource files use the pattern: `{path}.resource.json`\n\nExample: `f/databases/postgres_prod.resource.json`\n\n## Resource Structure\n\n```json\n{\n \"value\": {\n \"host\": \"db.example.com\",\n \"port\": 5432,\n \"user\": \"admin\",\n \"password\": \"$var:g/all/db_password\",\n \"dbname\": \"production\"\n },\n \"description\": \"Production PostgreSQL database\",\n \"resource_type\": \"postgresql\"\n}\n```\n\n## Required Fields\n\n- `value` - Object containing the resource configuration\n- `resource_type` - Name of the resource type (e.g., \"postgresql\", \"slack\")\n\n## Variable References\n\nReference variables in resource values:\n\n```json\n{\n \"value\": {\n \"api_key\": \"$var:g/all/api_key\",\n \"secret\": \"$var:u/admin/secret\"\n }\n}\n```\n\n**Reference formats:**\n- `$var:g/all/name` - Global variable\n- `$var:u/username/name` - User variable\n- `$var:f/folder/name` - Folder variable\n\n## Resource References\n\nReference other resources:\n\n```json\n{\n \"value\": {\n \"database\": \"$res:f/databases/postgres\"\n }\n}\n```\n\n## Common Resource Types\n\n### PostgreSQL\n```json\n{\n \"resource_type\": \"postgresql\",\n \"value\": {\n \"host\": \"localhost\",\n \"port\": 5432,\n \"user\": \"postgres\",\n \"password\": \"$var:g/all/pg_password\",\n \"dbname\": \"windmill\",\n \"sslmode\": \"prefer\"\n }\n}\n```\n\n### MySQL\n```json\n{\n \"resource_type\": \"mysql\",\n \"value\": {\n \"host\": \"localhost\",\n \"port\": 3306,\n \"user\": \"root\",\n \"password\": \"$var:g/all/mysql_password\",\n \"database\": \"myapp\"\n }\n}\n```\n\n### Slack\n```json\n{\n \"resource_type\": \"slack\",\n \"value\": {\n \"token\": \"$var:g/all/slack_token\"\n }\n}\n```\n\n### AWS S3\n```json\n{\n \"resource_type\": \"s3\",\n \"value\": {\n \"bucket\": \"my-bucket\",\n \"region\": \"us-east-1\",\n \"accessKeyId\": \"$var:g/all/aws_access_key\",\n \"secretAccessKey\": \"$var:g/all/aws_secret_key\"\n }\n}\n```\n\n### HTTP/API\n```json\n{\n \"resource_type\": \"http\",\n \"value\": {\n \"baseUrl\": \"https://api.example.com\",\n \"headers\": {\n \"Authorization\": \"Bearer $var:g/all/api_token\"\n }\n }\n}\n```\n\n### Kafka\n```json\n{\n \"resource_type\": \"kafka\",\n \"value\": {\n \"brokers\": \"broker1:9092,broker2:9092\",\n \"sasl_mechanism\": \"PLAIN\",\n \"security_protocol\": \"SASL_SSL\",\n \"username\": \"$var:g/all/kafka_user\",\n \"password\": \"$var:g/all/kafka_password\"\n }\n}\n```\n\n### NATS\n```json\n{\n \"resource_type\": \"nats\",\n \"value\": {\n \"servers\": [\"nats://localhost:4222\"],\n \"user\": \"$var:g/all/nats_user\",\n \"password\": \"$var:g/all/nats_password\"\n }\n}\n```\n\n### MQTT\n```json\n{\n \"resource_type\": \"mqtt\",\n \"value\": {\n \"host\": \"mqtt.example.com\",\n \"port\": 8883,\n \"username\": \"$var:g/all/mqtt_user\",\n \"password\": \"$var:g/all/mqtt_password\",\n \"tls\": true\n }\n}\n```\n\n## Custom Resource Types\n\nCreate custom resource types with JSON Schema:\n\n```json\n{\n \"name\": \"custom_api\",\n \"schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"base_url\": {\"type\": \"string\", \"format\": \"uri\"},\n \"api_key\": {\"type\": \"string\"},\n \"timeout\": {\"type\": \"integer\", \"default\": 30}\n },\n \"required\": [\"base_url\", \"api_key\"]\n },\n \"description\": \"Custom API connection\"\n}\n```\n\nSave as: `custom_api.resource-type.json`\n\n## OAuth Resources\n\nOAuth resources are managed through the Windmill UI and marked:\n\n```json\n{\n \"is_oauth\": true,\n \"account\": 123\n}\n```\n\nOAuth tokens are automatically refreshed by Windmill.\n\n## Using Resources in Scripts\n\n### TypeScript (Bun/Deno)\n```typescript\nexport async function main(db: RT.Postgresql) {\n // db contains the resource values\n const { host, port, user, password, dbname } = db;\n}\n```\n\n### Python\n```python\nclass postgresql(TypedDict):\n host: str\n port: int\n user: str\n password: str\n dbname: str\n\ndef main(db: postgresql):\n # db contains the resource values\n pass\n```\n\n## CLI Commands\n\n```bash\n# List resources\nwmill resource list\n\n# List resource types with schemas\nwmill resource-type list --schema\n\n# Get specific resource type schema\nwmill resource-type get postgresql\n\n# Push resources (tell the user to run this, do NOT run it yourself)\nwmill sync push\n```\n"; -export declare const RAW_APP_BASE = "# Windmill Raw Apps\n\nRaw apps let you build custom frontends with React, Svelte, or Vue that connect to Windmill backend runnables and datatables.\n\n## App shape\n\nA raw app has three logical parts:\n\n- **Frontend** \u2014 bundled with esbuild from `index.tsx` as the entrypoint. Files include the entrypoint, components (`App.tsx`), styles, etc.\n- **Backend runnables** \u2014 server-side scripts the frontend calls, each addressed by a unique key.\n- **Data** \u2014 optional whitelisted datatables (managed PostgreSQL) that the backend runnables can query. The frontend never queries the database directly; backend runnables are the only bridge.\n\n## Frontend\n\n### Entrypoint\n\n`index.tsx` is the bundling entrypoint. It typically renders a top-level `App` component. The bundler is esbuild.\n\n### Generated bindings (`wmill.d.ts` / `wmill.ts`)\n\nThe frontend imports a generated module that mirrors the backend runnables. **Never write to it directly** \u2014 it gets regenerated whenever backend runnables change. Modifying it by hand will be overwritten.\n\n### Calling backend runnables\n\nImport the generated bindings and call the runnable like a function:\n\n```typescript\nimport { backend } from './wmill';\n\n// Call a backend runnable\nconst user = await backend.get_user({ user_id: '123' });\n```\n\nThe frontend cannot reach datatables, workspace items, or external services on its own \u2014 it goes through `backend.(args)` for everything server-side.\n\n## Backend runnables\n\nEach runnable has a unique key (used to call it from the frontend) and one of four types:\n\n| Type | What it is |\n|---|---|\n| `inline` | Custom code stored on the app itself. Most common for app-specific logic. |\n| `script` | Reference to an existing workspace script by path. |\n| `flow` | Reference to an existing workspace flow by path. |\n| `hubscript` | Reference to a hub script by path. |\n\n### Inline runnables\n\nInline runnables carry their own source code. For file-based raw apps, the runnable language is determined by the backend file extension. The script must expose a `main` function as its entrypoint.\n\n**TypeScript example** (`backend/get_user.ts`):\n\n```typescript\nimport * as wmill from 'windmill-client';\n\nexport async function main(user_id: string) {\n const sql = wmill.datatable();\n const user = await sql`SELECT * FROM users WHERE id = ${user_id}`.fetchOne();\n return user;\n}\n```\n\n**Python example** (`backend/get_user.py`):\n\n```python\nimport wmill\n\ndef main(user_id: str):\n db = wmill.datatable()\n user = db.query('SELECT * FROM users WHERE id = $1', user_id).fetch_one()\n return user\n```\n\n### Path runnables (script / flow / hubscript)\n\nWhen `type` is `script`, `flow`, or `hubscript`, the runnable just stores a `path` to an existing workspace or hub item \u2014 no inline code. The referenced item's input/output schema becomes the runnable's surface.\n\n### Static inputs\n\n`staticInputs` is an optional `Record` for arguments not overridable from the frontend. Useful with path runnables to pre-fill some args while leaving the rest to the frontend caller.\n\n## Data Tables\n\nData tables are PostgreSQL databases managed by Windmill. Backend runnables query them via the `wmill` client; the frontend never queries them directly.\n\n### Critical rules\n\n1. **Whitelisted tables only**: a runnable can only query tables listed in the app's `data.tables` config. Tables not in this list are not accessible.\n2. **Add tables before using**: queries against unlisted tables fail at runtime. When you introduce a new table, register it in `data.tables` first.\n3. **Use the configured datatable/schema**: the app's `data` config sets the default datatable and schema; reference them consistently across runnables.\n\n### Querying in TypeScript (Bun/Deno)\n\n```typescript\nimport * as wmill from 'windmill-client';\n\nexport async function main(user_id: string) {\n const sql = wmill.datatable(); // Or: wmill.datatable('other_datatable')\n\n // Parameterized queries (safe from SQL injection)\n const user = await sql`SELECT * FROM users WHERE id = ${user_id}`.fetchOne();\n const users = await sql`SELECT * FROM users WHERE active = ${true}`.fetch();\n\n // Insert/Update\n await sql`INSERT INTO users (name, email) VALUES (${name}, ${email})`;\n await sql`UPDATE users SET name = ${newName} WHERE id = ${user_id}`;\n\n return user;\n}\n```\n\n### Querying in Python\n\n```python\nimport wmill\n\ndef main(user_id: str):\n db = wmill.datatable() # Or: wmill.datatable('other_datatable')\n\n # Use $1, $2, etc. for parameters\n user = db.query('SELECT * FROM users WHERE id = $1', user_id).fetch_one()\n users = db.query('SELECT * FROM users WHERE active = $1', True).fetch()\n\n # Insert/Update\n db.query('INSERT INTO users (name, email) VALUES ($1, $2)', name, email)\n db.query('UPDATE users SET name = $1 WHERE id = $2', new_name, user_id)\n\n return user\n```\n\n## Best Practices\n\n1. **Check existing tables** before creating new ones \u2014 reuse beats schema growth.\n2. **Use parameterized queries** \u2014 never concatenate user input into SQL.\n3. **Keep runnables focused** \u2014 one function per runnable; small surface area.\n4. **Use descriptive keys** \u2014 `get_user`, not `a`.\n5. **Always whitelist tables** \u2014 adding a runnable that queries a new table requires the table to be in `data.tables` first.\n"; -export declare const WORKFLOW_AS_CODE_BASE = "# Windmill Workflow-as-Code Writing Guide\n\n## Scope\n\nUse this guide when writing or modifying Windmill Workflow-as-Code (WAC) scripts.\nWAC is authored as a Windmill script and deployed with the normal script workflow. It is not an OpenFlow YAML flow.\n\nSupported WAC authoring targets:\n- Bun TypeScript scripts that import from `windmill-client`\n- Python 3 scripts that import from `wmill`\n\n## File Shape\n\nBun TypeScript:\n\n```typescript\nimport {\n task,\n taskScript,\n taskFlow,\n step,\n sleep,\n waitForApproval,\n getResumeUrls,\n parallel,\n workflow,\n} from \"windmill-client\";\n\nconst process = task(async (x: string): Promise => {\n return `processed: ${x}`;\n});\n\nexport const main = workflow(async (x: string) => {\n const result = await process(x);\n return { result };\n});\n```\n\nPython:\n\n```python\nfrom wmill import task, task_script, task_flow, step, sleep, wait_for_approval, get_resume_urls, parallel, workflow\n\n@task()\nasync def process(x: str) -> str:\n return f\"processed: {x}\"\n\n@workflow\nasync def main(x: str):\n result = await process(x)\n return {\"result\": result}\n```\n\nRules:\n- Do not call `main`.\n- Bun TypeScript should export the workflow entrypoint, preferably `export const main = workflow(async (...) => { ... })`.\n- Python must use `@workflow` on an async top-level function, usually `main`.\n- Define task functions and `taskScript`/`task_script` or `taskFlow`/`task_flow` assignments at module top level with stable names.\n- Use the exact SDK names. Do not alias `workflow`, `task`, `taskScript`, `taskFlow`, `step`, `sleep`, `waitForApproval`, `task_script`, `task_flow`, or `wait_for_approval`; the WAC parser recognizes these names directly.\n\n## Checkpoint And Replay Model\n\nThe parent workflow may rerun from the top after any suspension, retry, approval, or child task completion. Completed durable steps are replayed from the checkpoint.\n\nPut every side effect or non-deterministic value behind a durable WAC boundary:\n- Use `task()` / `@task()` for substantial work that should run as its own child job.\n- Use `taskScript()` / `task_script()` for an existing script or a relative module file.\n- Use `taskFlow()` / `task_flow()` for an existing Windmill flow.\n- Use `step(name, fn)` for lightweight inline work whose result must be checkpointed.\n- Use `sleep(seconds)` for server-side sleeps that do not hold a worker.\n- Use `waitForApproval()` / `wait_for_approval()` for external approval suspension.\n\nNever put API calls, database writes, notifications, random values, timestamps, or irreversible changes directly in the top-level workflow body. The workflow body can be rerun. Put those operations in a task or in `step()`.\n\nBranching on task or step results is safe because those results are checkpointed. Branching on current time, random data, environment reads, or external state is unsafe unless the value is first captured with `step()`.\n\n## Tasks\n\nUse `task()` / `@task()` for inline functions that become workflow steps:\n\n```typescript\nconst enrich = task(async (customerId: string) => {\n return await fetchCustomer(customerId);\n});\n```\n\n```python\n@task(timeout=600, tag=\"etl\")\nasync def enrich(customer_id: str):\n return await fetch_customer(customer_id)\n```\n\nIn TypeScript, prefer assigning each task to a named top-level const. In Python, prefer top-level async functions decorated with `@task()` or `@task`.\n\nFor existing scripts:\n\n```typescript\nconst helper = taskScript(\"./helper.ts\");\nconst existing = taskScript(\"f/data/extract\", { timeout: 600 });\nconst value = await helper({ input: x });\n```\n\n```python\nhelper = task_script(\"./helper.py\")\nexisting = task_script(\"f/data/extract\", timeout=600)\nvalue = await helper(input=x)\n```\n\nFor existing flows:\n\n```typescript\nconst pipeline = taskFlow(\"f/etl/pipeline\");\nconst output = await pipeline({ input: data });\n```\n\n```python\npipeline = task_flow(\"f/etl/pipeline\")\noutput = await pipeline(input=data)\n```\n\n## Inline Steps\n\nUse `step()` for lightweight inline values that must not change during replay:\n\n```typescript\nconst urls = await step(\"get_urls\", () => getResumeUrls());\nconst startedAt = await step(\"started_at\", () => new Date().toISOString());\n```\n\n```python\nurls = await step(\"get_urls\", lambda: get_resume_urls())\n```\n\nUse stable, descriptive step names. Do not generate step names dynamically.\n\n## Parallelism\n\nTo run independent work in parallel, start task promises/coroutines before awaiting them together:\n\n```typescript\nconst [a, b] = await Promise.all([process(\"a\"), process(\"b\")]);\nconst many = await parallel(items, process, { concurrency: 5 });\n```\n\n```python\nimport asyncio\n\na, b = await asyncio.gather(process(\"a\"), process(\"b\"))\nmany = await parallel(items, process, concurrency=5)\n```\n\nOnly parallelize independent steps. Do not read the result of a task before it is awaited.\n\n## Approvals\n\nGenerate resume URLs inside `step()` before sending them:\n\n```typescript\nconst urls = await step(\"get_urls\", () => getResumeUrls());\nawait step(\"notify\", () => sendApprovalEmail(urls.approvalPage));\nconst approval = await waitForApproval({ timeout: 3600 });\n```\n\n```python\nurls = await step(\"get_urls\", lambda: get_resume_urls())\nawait step(\"notify\", lambda: send_approval_email(urls[\"approvalPage\"]))\napproval = await wait_for_approval(timeout=3600)\n```\n\n`selfApproval: false` and `self_approval=False` are Enterprise-only approval behavior. Do not use them unless the user asks for that behavior.\n\n## Error Handling\n\nLet task errors fail the workflow unless the user asks for recovery logic.\n\nPython: `except Exception` is safe around WAC calls because internal suspension inherits from `BaseException`. Avoid bare `except:` in workflow code. If the user asks for recovery logic around failed child work, catch `TaskError` from `wmill` for task failures.\n\nTypeScript: avoid broad `try/catch` around WAC SDK calls. The SDK uses an internal suspension error during initial dispatch; catching it can break workflow suspension. If a broad catch is unavoidable, rethrow internal suspension errors before handling business errors.\n"; -export declare const FLOW_CHAT_SPECIAL_MODULES = "## Special Modules\n\n- Use `set_preprocessor_module` to add, replace, or remove the top-level `value.preprocessor_module`\n- Use `set_failure_module` to add, replace, or remove the top-level `value.failure_module`\n- Use `set_flow_json` only when you are replacing the whole flow, including normal modules and optional special modules\n\n**Example - Update only the special modules:**\n```javascript\nset_preprocessor_module({\n module: JSON.stringify({\n id: \"preprocessor\",\n value: {\n type: \"rawscript\",\n language: \"bun\",\n content: \"export async function preprocessor(payload: string) { const trimmed = payload.trim(); if (!trimmed) { throw new Error('payload must not be empty'); } return { payload: trimmed }; }\",\n input_transforms: {\n payload: { type: \"javascript\", expr: \"flow_input.payload\" }\n }\n }\n })\n})\n\nset_failure_module({\n module: JSON.stringify({\n id: \"failure\",\n value: {\n type: \"rawscript\",\n language: \"bun\",\n content: \"export async function main(message: string, name: string, step_id: string) { return { message, name, step_id }; }\",\n input_transforms: {\n message: { type: \"javascript\", expr: \"error.message\" },\n name: { type: \"javascript\", expr: \"error.name\" },\n step_id: { type: \"javascript\", expr: \"error.step_id\" }\n }\n }\n })\n})\n```\n"; -export declare const SDK_TYPESCRIPT = "# TypeScript SDK (windmill-client)\n\nImport: import * as wmill from 'windmill-client'\n\nworkerHasInternalServer(): boolean\n\n/**\n * Initialize the Windmill client with authentication token and base URL\n * @param token - Authentication token (defaults to WM_TOKEN env variable)\n * @param baseUrl - API base URL (defaults to BASE_INTERNAL_URL or BASE_URL env variable)\n */\nsetClient(token?: string, baseUrl?: string): void\n\n/**\n * Create a client configuration from env variables\n * @returns client configuration\n */\ngetWorkspace(): string\n\n/**\n * Get a resource value by path\n * @param path path of the resource, default to internal state path\n * @param undefinedIfEmpty if the resource does not exist, return undefined instead of throwing an error\n * @returns resource value\n */\nasync getResource(path?: string, undefinedIfEmpty?: boolean): Promise\n\n/**\n * Get the true root job id\n * @param jobId job id to get the root job id from (default to current job)\n * @returns root job id\n */\nasync getRootJobId(jobId?: string): Promise\n\n/**\n * @deprecated Use runScriptByPath or runScriptByHash instead\n */\nasync runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Run a script synchronously by its path and wait for the result\n * @param path - Script path in Windmill\n * @param args - Arguments to pass to the script\n * @param verbose - Enable verbose logging\n * @returns Script execution result\n */\nasync runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Run a script synchronously by its hash and wait for the result\n * @param hash_ - Script hash in Windmill\n * @param args - Arguments to pass to the script\n * @param verbose - Enable verbose logging\n * @returns Script execution result\n */\nasync runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Append a text to the result stream\n * @param text text to append to the result stream\n */\nappendToResultStream(text: string): void\n\n/**\n * Stream to the result stream\n * @param stream stream to stream to the result stream\n */\nasync streamResult(stream: AsyncIterable): Promise\n\n/**\n * Run a flow synchronously by its path and wait for the result\n * @param path - Flow path in Windmill\n * @param args - Arguments to pass to the flow\n * @param verbose - Enable verbose logging\n * @returns Flow execution result\n */\nasync runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Wait for a job to complete and return its result\n * @param jobId - ID of the job to wait for\n * @param verbose - Enable verbose logging\n * @returns Job result when completed\n */\nasync waitJob(jobId: string, verbose: boolean = false): Promise\n\n/**\n * Get the result of a completed job\n * @param jobId - ID of the completed job\n * @returns Job result\n */\nasync getResult(jobId: string): Promise\n\n/**\n * Get the result of a job if completed, or its current status\n * @param jobId - ID of the job\n * @returns Object with started, completed, success, and result properties\n */\nasync getResultMaybe(jobId: string): Promise\n\n/**\n * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead\n */\nasync runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise\n\n/**\n * Run a script asynchronously by its path\n * @param path - Script path in Windmill\n * @param args - Arguments to pass to the script\n * @param scheduledInSeconds - Schedule execution for a future time (in seconds)\n * @returns Job ID of the created job\n */\nasync runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise\n\n/**\n * Run a script asynchronously by its hash\n * @param hash_ - Script hash in Windmill\n * @param args - Arguments to pass to the script\n * @param scheduledInSeconds - Schedule execution for a future time (in seconds)\n * @returns Job ID of the created job\n */\nasync runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise\n\n/**\n * Run a flow asynchronously by its path\n * @param path - Flow path in Windmill\n * @param args - Arguments to pass to the flow\n * @param scheduledInSeconds - Schedule execution for a future time (in seconds)\n * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job)\n * @returns Job ID of the created job\n */\nasync runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise\n\n/**\n * Resolve a resource value in case the default value was picked because the input payload was undefined\n * @param obj resource value or path of the resource under the format `$res:path`\n * @returns resource value\n */\nasync resolveDefaultResource(obj: any): Promise\n\n/**\n * Get the state file path from environment variables\n * @returns State path string\n */\ngetStatePath(): string\n\n/**\n * Set a resource value by path\n * @param path path of the resource to set, default to state path\n * @param value new value of the resource to set\n * @param initializeToTypeIfNotExist if the resource does not exist, initialize it with this type\n */\nasync setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise\n\n/**\n * Set the state\n * @param state state to set\n * @deprecated use setState instead\n */\nasync setInternalState(state: any): Promise\n\n/**\n * Set the state\n * @param state state to set\n * @param path Optional state resource path override. Defaults to `getStatePath()`.\n */\nasync setState(state: any, path?: string): Promise\n\n/**\n * Set the progress\n * Progress cannot go back and limited to 0% to 99% range\n * @param percent Progress to set in %\n * @param jobId? Job to set progress for\n */\nasync setProgress(percent: number, jobId?: any): Promise\n\n/**\n * Get the progress\n * @param jobId? Job to get progress from\n * @returns Optional clamped between 0 and 100 progress value\n */\nasync getProgress(jobId?: any): Promise\n\n/**\n * Set a flow user state\n * @param key key of the state\n * @param value value of the state\n */\nasync setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): Promise\n\n/**\n * Get a flow user state\n * @param path path of the variable\n */\nasync getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise\n\n/**\n * Get the internal state\n * @deprecated use getState instead\n */\nasync getInternalState(): Promise\n\n/**\n * Get the state shared across executions\n * @param path Optional state resource path override. Defaults to `getStatePath()`.\n */\nasync getState(path?: string): Promise\n\n/**\n * Get a variable by path\n * @param path path of the variable\n * @returns variable value\n */\nasync getVariable(path: string): Promise\n\n/**\n * Set a variable by path, create if not exist\n * @param path path of the variable\n * @param value value of the variable\n * @param isSecretIfNotExist if the variable does not exist, create it as secret or not (default: false)\n * @param descriptionIfNotExist if the variable does not exist, create it with this description (default: \"\")\n */\nasync setVariable(path: string, value: string, isSecretIfNotExist?: boolean, descriptionIfNotExist?: string): Promise\n\n/**\n * Build a PostgreSQL connection URL from a database resource\n * @param path - Path to the database resource\n * @returns PostgreSQL connection URL string\n */\nasync databaseUrlFromResource(path: string): Promise\n\nasync polarsConnectionSettings(s3_resource_path: string | undefined): Promise\n\nasync duckdbConnectionSettings(s3_resource_path: string | undefined): Promise\n\n/**\n * Get S3 client settings from a resource or workspace default\n * @param s3_resource_path - Path to S3 resource (uses workspace default if undefined)\n * @returns S3 client configuration settings\n */\nasync denoS3LightClientSettings(s3_resource_path: string | undefined): Promise\n\n/**\n * Load the content of a file stored in S3. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.\n * \n * ```typescript\n * let fileContent = await wmill.loadS3FileContent(inputFile)\n * // if the file is a raw text file, it can be decoded and printed directly:\n * const text = new TextDecoder().decode(fileContentStream)\n * console.log(text);\n * ```\n */\nasync loadS3File(s3object: S3Object, s3ResourcePath: string | undefined = undefined): Promise\n\n/**\n * Load the content of a file stored in S3 as a stream. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.\n * \n * ```typescript\n * let fileContentBlob = await wmill.loadS3FileStream(inputFile)\n * // if the content is plain text, the blob can be read directly:\n * console.log(await fileContentBlob.text());\n * ```\n */\nasync loadS3FileStream(s3object: S3Object, s3ResourcePath: string | undefined = undefined): Promise\n\n/**\n * Persist a file to the S3 bucket. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.\n * \n * ```typescript\n * const s3object = await writeS3File(s3Object, \"Hello Windmill!\")\n * const fileContentAsUtf8Str = (await s3object.toArray()).toString('utf-8')\n * console.log(fileContentAsUtf8Str)\n * ```\n */\nasync writeS3File(s3object: S3Object | undefined, fileContent: string | Blob, s3ResourcePath: string | undefined = undefined, contentType: string | undefined = undefined, contentDisposition: string | undefined = undefined): Promise\n\n/**\n * Sign S3 objects to be used by anonymous users in public apps\n * @param s3objects s3 objects to sign\n * @returns signed s3 objects\n */\nasync signS3Objects(s3objects: S3Object[]): Promise\n\n/**\n * Sign S3 object to be used by anonymous users in public apps\n * @param s3object s3 object to sign\n * @returns signed s3 object\n */\nasync signS3Object(s3object: S3Object): Promise\n\n/**\n * Generate a presigned public URL for an array of S3 objects.\n * If an S3 object is not signed yet, it will be signed first.\n * @param s3Objects s3 objects to sign\n * @returns list of signed public URLs\n */\nasync getPresignedS3PublicUrls(s3Objects: S3Object[], { baseUrl }: { baseUrl?: string } = {}): Promise\n\n/**\n * Generate a presigned public URL for an S3 object. If the S3 object is not signed yet, it will be signed first.\n * @param s3Object s3 object to sign\n * @returns signed public URL\n */\nasync getPresignedS3PublicUrl(s3Objects: S3Object, { baseUrl }: { baseUrl?: string } = {}): Promise\n\n/**\n * Get URLs needed for resuming a flow after this step\n * @param approver approver name\n * @param flowLevel if true, generate resume URLs for the parent flow instead of the specific step.\n * This allows pre-approvals that can be consumed by any later suspend step in the same flow.\n * @returns approval page UI URL, resume and cancel API URLs for resuming the flow\n */\nasync getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{\n approvalPage: string;\n resume: string;\n cancel: string;\n}>\n\n/**\n * @deprecated use getResumeUrls instead\n */\ngetResumeEndpoints(approver?: string): Promise<{\n approvalPage: string;\n resume: string;\n cancel: string;\n}>\n\n/**\n * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only)\n * @param audience audience of the token\n * @param expiresIn Optional number of seconds until the token expires\n * @returns jwt token\n */\nasync getIdToken(audience: string, expiresIn?: number): Promise\n\n/**\n * Convert a base64-encoded string to Uint8Array\n * @param data - Base64-encoded string\n * @returns Decoded Uint8Array\n */\nbase64ToUint8Array(data: string): Uint8Array\n\n/**\n * Convert a Uint8Array to base64-encoded string\n * @param arrayBuffer - Uint8Array to encode\n * @returns Base64-encoded string\n */\nuint8ArrayToBase64(arrayBuffer: Uint8Array): string\n\n/**\n * Get email from workspace username\n * This method is particularly useful for apps that require the email address of the viewer.\n * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app.\n * @param username\n * @returns email address\n */\nasync usernameToEmail(username: string): Promise\n\n/**\n * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields.\n * \n * **[Enterprise Edition Only]** To include form fields in the Slack approval request, go to **Advanced -> Suspend -> Form**\n * and define a form. Learn more at [Windmill Documentation](https://www.windmill.dev/docs/flows/flow_approval#form).\n * \n * @param {Object} options - The configuration options for the Slack approval request.\n * @param {string} options.slackResourcePath - The path to the Slack resource in Windmill.\n * @param {string} options.channelId - The Slack channel ID where the approval request will be sent.\n * @param {string} [options.message] - Optional custom message to include in the Slack approval request.\n * @param {string} [options.approver] - Optional user ID or name of the approver for the request.\n * @param {DefaultArgs} [options.defaultArgsJson] - Optional object defining or overriding the default arguments to a form field.\n * @param {Enums} [options.dynamicEnumsJson] - Optional object overriding the enum default values of an enum form field.\n * @param {string} [options.resumeButtonText] - Optional text for the resume button.\n * @param {string} [options.cancelButtonText] - Optional text for the cancel button.\n * \n * @returns {Promise} Resolves when the Slack approval request is successfully sent.\n * \n * @throws {Error} If the function is not called within a flow or flow preview.\n * @throws {Error} If the `JobService.getSlackApprovalPayload` call fails.\n * \n * **Usage Example:**\n * ```typescript\n * await requestInteractiveSlackApproval({\n * slackResourcePath: \"/u/alex/my_slack_resource\",\n * channelId: \"admins-slack-channel\",\n * message: \"Please approve this request\",\n * approver: \"approver123\",\n * defaultArgsJson: { key1: \"value1\", key2: 42 },\n * dynamicEnumsJson: { foo: [\"choice1\", \"choice2\"], bar: [\"optionA\", \"optionB\"] },\n * resumeButtonText: \"Resume\",\n * cancelButtonText: \"Cancel\",\n * });\n * ```\n * \n * **Note:** This function requires execution within a Windmill flow or flow preview.\n */\nasync requestInteractiveSlackApproval({ slackResourcePath, channelId, message, approver, defaultArgsJson, dynamicEnumsJson, resumeButtonText, cancelButtonText, }: SlackApprovalOptions): Promise\n\n/**\n * Sends an interactive approval request via Teams, allowing optional customization of the message, approver, and form fields.\n * \n * **[Enterprise Edition Only]** To include form fields in the Teams approval request, go to **Advanced -> Suspend -> Form**\n * and define a form. Learn more at [Windmill Documentation](https://www.windmill.dev/docs/flows/flow_approval#form).\n * \n * @param {Object} options - The configuration options for the Teams approval request.\n * @param {string} options.teamName - The Teams team name where the approval request will be sent.\n * @param {string} options.channelName - The Teams channel name where the approval request will be sent.\n * @param {string} [options.message] - Optional custom message to include in the Teams approval request.\n * @param {string} [options.approver] - Optional user ID or name of the approver for the request.\n * @param {DefaultArgs} [options.defaultArgsJson] - Optional object defining or overriding the default arguments to a form field.\n * @param {Enums} [options.dynamicEnumsJson] - Optional object overriding the enum default values of an enum form field.\n * \n * @returns {Promise} Resolves when the Teams approval request is successfully sent.\n * \n * @throws {Error} If the function is not called within a flow or flow preview.\n * @throws {Error} If the `JobService.getTeamsApprovalPayload` call fails.\n * \n * **Usage Example:**\n * ```typescript\n * await requestInteractiveTeamsApproval({\n * teamName: \"admins-teams\",\n * channelName: \"admins-teams-channel\",\n * message: \"Please approve this request\",\n * approver: \"approver123\",\n * defaultArgsJson: { key1: \"value1\", key2: 42 },\n * dynamicEnumsJson: { foo: [\"choice1\", \"choice2\"], bar: [\"optionA\", \"optionB\"] },\n * });\n * ```\n * \n * **Note:** This function requires execution within a Windmill flow or flow preview.\n */\nasync requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise\n\n/**\n * Parse an S3 object from URI string or record format\n * @param s3Object - S3 object as URI string (s3://storage/key) or record\n * @returns S3 object record with storage and s3 key\n */\nparseS3Object(s3Object: S3Object): S3ObjectRecord\n\nsetWorkflowCtx(ctx: WorkflowCtx | null): void\n\nasync sleep(seconds: number): Promise\n\nasync step(name: string, fn: () => T | Promise): Promise\n\n/**\n * Create a task that dispatches to a separate Windmill script.\n * \n * @example\n * const extract = taskScript(\"f/data/extract\");\n * // inside workflow: await extract({ url: \"https://...\" })\n */\ntaskScript(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Create a task that dispatches to a separate Windmill flow.\n * \n * @example\n * const pipeline = taskFlow(\"f/etl/pipeline\");\n * // inside workflow: await pipeline({ input: data })\n */\ntaskFlow(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Mark an async function as a workflow-as-code entry point.\n * \n * The function must be **deterministic**: given the same inputs it must call\n * tasks in the same order on every replay. Branching on task results is fine\n * (results are replayed from checkpoint), but branching on external state\n * (current time, random values, external API calls) must use `step()` to\n * checkpoint the value so replays see the same result.\n */\nworkflow(fn: (...args: any[]) => Promise): void\n\n/**\n * Suspend the workflow and wait for an external approval.\n * \n * Use `getResumeUrls()` (wrapped in `step()`) to obtain resume/cancel/approvalPage\n * URLs before calling this function.\n * \n * @example\n * const urls = await step(\"urls\", () => getResumeUrls());\n * await step(\"notify\", () => sendEmail(urls.approvalPage));\n * const { value, approver } = await waitForApproval({ timeout: 3600 });\n */\nwaitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }>\n\n/**\n * Process items in parallel with optional concurrency control.\n * \n * Each item is processed by calling `fn(item)`, which should be a task().\n * Items are dispatched in batches of `concurrency` (default: all at once).\n * \n * @example\n * const process = task(async (item: string) => { ... });\n * const results = await parallel(items, process, { concurrency: 5 });\n */\nasync parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: { concurrency?: number },): Promise\n\n/**\n * Commit Kafka offsets for a trigger with auto_commit disabled.\n * @param triggerPath - Path to the Kafka trigger (from event.wm_trigger.trigger_path)\n * @param topic - Kafka topic name (from event.topic)\n * @param partition - Partition number (from event.partition)\n * @param offset - Message offset to commit (from event.offset)\n */\nasync commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise\n\n/**\n * Create a SQL template function for PostgreSQL/datatable queries\n * @param name - Database/datatable name (default: \"main\")\n * @returns SQL template function for building parameterized queries\n * @example\n * let sql = wmill.datatable()\n * let name = 'Robin'\n * let age = 21\n * await sql`\n * SELECT * FROM friends\n * WHERE name = ${name} AND age = ${age}::int\n * `.fetch()\n */\ndatatable(name: string = \"main\"): DatatableSqlTemplateFunction\n\n/**\n * Create a SQL template function for DuckDB/ducklake queries\n * @param name - DuckDB database name (default: \"main\")\n * @returns SQL template function for building parameterized queries\n * @example\n * let sql = wmill.ducklake()\n * let name = 'Robin'\n * let age = 21\n * await sql`\n * SELECT * FROM friends\n * WHERE name = ${name} AND age = ${age}\n * `.fetch()\n */\nducklake(name: string = \"main\"): SqlTemplateFunction\n"; -export declare const SDK_PYTHON = "# Python SDK (wmill)\n\nImport: import wmill\n\ndef worker_has_internal_server() -> bool\n\ndef get_mocked_api() -> Optional[dict]\n\n# Get the HTTP client instance.\n# \n# Returns:\n# Configured httpx.Client for API requests\ndef get_client() -> httpx.Client\n\n# Make an HTTP GET request to the Windmill API.\n# \n# Args:\n# endpoint: API endpoint path\n# raise_for_status: Whether to raise an exception on HTTP errors\n# **kwargs: Additional arguments passed to httpx.get\n# \n# Returns:\n# HTTP response object\ndef get(endpoint, raise_for_status = True, **kwargs) -> httpx.Response\n\n# Make an HTTP POST request to the Windmill API.\n# \n# Args:\n# endpoint: API endpoint path\n# raise_for_status: Whether to raise an exception on HTTP errors\n# **kwargs: Additional arguments passed to httpx.post\n# \n# Returns:\n# HTTP response object\ndef post(endpoint, raise_for_status = True, **kwargs) -> httpx.Response\n\n# Create a new authentication token.\n# \n# Args:\n# duration: Token validity duration (default: 1 day)\n# \n# Returns:\n# New authentication token string\ndef create_token(duration = dt.timedelta(days=1)) -> str\n\n# Create a script job and return its job id.\n# \n# .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead.\ndef run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str\n\n# Create a script job by path and return its job id.\ndef run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str\n\n# Create a script job by hash and return its job id.\ndef run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str\n\n# Create a flow job and return its job id.\ndef run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str\n\n# Run script synchronously and return its result.\n# \n# .. deprecated:: Use run_script_by_path or run_script_by_hash instead.\ndef run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any\n\n# Run script by path synchronously and return its result.\ndef run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any\n\n# Run script by hash synchronously and return its result.\ndef run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any\n\n# Run a script on the current worker without creating a job.\n# \n# On agent workers (no internal server), falls back to running a normal\n# preview job and waiting for the result.\ndef run_inline_script_preview(content: str, language: str, args: dict = None) -> Any\n\n# Wait for a job to complete and return its result.\n# \n# Args:\n# job_id: ID of the job to wait for\n# timeout: Maximum time to wait (seconds or timedelta)\n# verbose: Enable verbose logging\n# cleanup: Register cleanup handler to cancel job on exit\n# assert_result_is_not_none: Raise exception if result is None\n# \n# Returns:\n# Job result when completed\n# \n# Raises:\n# TimeoutError: If timeout is reached\n# Exception: If job fails\ndef wait_job(job_id, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False)\n\n# Cancel a specific job by ID.\n# \n# Args:\n# job_id: UUID of the job to cancel\n# reason: Optional reason for cancellation\n# \n# Returns:\n# Response message from the cancel endpoint\ndef cancel_job(job_id: str, reason: str = None) -> str\n\n# Cancel currently running executions of the same script.\ndef cancel_running() -> dict\n\n# Get job details by ID.\n# \n# Args:\n# job_id: UUID of the job\n# \n# Returns:\n# Job details dictionary\ndef get_job(job_id: str) -> dict\n\n# Get the root job ID for a flow hierarchy.\n# \n# Args:\n# job_id: Job ID (defaults to current WM_JOB_ID)\n# \n# Returns:\n# Root job ID\ndef get_root_job_id(job_id: str | None = None) -> dict\n\n# Get an OIDC JWT token for authentication to external services.\n# \n# Args:\n# audience: Token audience (e.g., \"vault\", \"aws\")\n# expires_in: Optional expiration time in seconds\n# \n# Returns:\n# JWT token string\ndef get_id_token(audience: str, expires_in: int | None = None) -> str\n\n# Get the status of a job.\n# \n# Args:\n# job_id: UUID of the job\n# \n# Returns:\n# Job status: \"RUNNING\", \"WAITING\", or \"COMPLETED\"\ndef get_job_status(job_id: str) -> JobStatus\n\n# Get the result of a completed job.\n# \n# Args:\n# job_id: UUID of the completed job\n# assert_result_is_not_none: Raise exception if result is None\n# \n# Returns:\n# Job result\ndef get_result(job_id: str, assert_result_is_not_none: bool = True) -> Any\n\n# Get a variable value by path.\n# \n# Args:\n# path: Variable path in Windmill\n# \n# Returns:\n# Variable value as string\ndef get_variable(path: str) -> str\n\n# Set a variable value by path, creating it if it doesn't exist.\n# \n# Args:\n# path: Variable path in Windmill\n# value: Variable value to set\n# is_secret: Whether the variable should be secret (default: False)\ndef set_variable(path: str, value: str, is_secret: bool = False) -> None\n\n# Get a resource value by path.\n# \n# Args:\n# path: Resource path in Windmill\n# none_if_undefined: Return None instead of raising if not found\n# interpolated: if variables and resources are fully unrolled\n# \n# Returns:\n# Resource value dictionary or None\ndef get_resource(path: str, none_if_undefined: bool = False, interpolated: bool = True) -> dict | None\n\n# Set a resource value by path, creating it if it doesn't exist.\n# \n# Args:\n# value: Resource value to set\n# path: Resource path in Windmill\n# resource_type: Resource type for creation\ndef set_resource(value: Any, path: str, resource_type: str)\n\n# List resources from Windmill workspace.\n# \n# Args:\n# resource_type: Optional resource type to filter by (e.g., \"postgresql\", \"mysql\", \"s3\")\n# page: Optional page number for pagination\n# per_page: Optional number of results per page\n# \n# Returns:\n# List of resource dictionaries\ndef list_resources(resource_type: str = None, page: int = None, per_page: int = None) -> list[dict]\n\n# Set the workflow state.\n# \n# Args:\n# value: State value to set\n# path: Optional state resource path override.\ndef set_state(value: Any, path: str | None = None) -> None\n\n# Get the workflow state.\n# \n# Args:\n# path: Optional state resource path override.\n# \n# Returns:\n# State value or None if not set\ndef get_state(path: str | None = None) -> Any\n\n# Set job progress percentage (0-99).\n# \n# Args:\n# value: Progress percentage\n# job_id: Job ID (defaults to current WM_JOB_ID)\ndef set_progress(value: int, job_id: Optional[str] = None)\n\n# Get job progress percentage.\n# \n# Args:\n# job_id: Job ID (defaults to current WM_JOB_ID)\n# \n# Returns:\n# Progress value (0-100) or None if not set\ndef get_progress(job_id: Optional[str] = None) -> Any\n\n# Set the user state of a flow at a given key\ndef set_flow_user_state(key: str, value: Any) -> None\n\n# Get the user state of a flow at a given key\ndef get_flow_user_state(key: str) -> Any\n\n# Get the Windmill server version.\n# \n# Returns:\n# Version string\ndef version()\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from DuckDB\ndef get_duckdb_connection_settings(s3_resource_path: str = '') -> DuckDbConnectionSettings | None\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from Polars\ndef get_polars_connection_settings(s3_resource_path: str = '') -> PolarsConnectionSettings\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection using boto3\ndef get_boto3_connection_settings(s3_resource_path: str = '') -> Boto3ConnectionSettings\n\n# Load a file from the workspace s3 bucket and returns its content as bytes.\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# my_obj_content = client.load_s3_file(s3_obj)\n# file_content = my_obj_content.decode(\"utf-8\")\n# '''\ndef load_s3_file(s3object: S3Object | str, s3_resource_path: str | None) -> bytes\n\n# Load a file from the workspace s3 bucket and returns the bytes stream.\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# with wmill.load_s3_file_reader(s3object, s3_resource_path) as file_reader:\n# print(file_reader.read())\n# '''\ndef load_s3_file_reader(s3object: S3Object | str, s3_resource_path: str | None) -> BufferedReader\n\n# Write a file to the workspace S3 bucket\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# \n# # for an in memory bytes array:\n# file_content = b'Hello Windmill!'\n# client.write_s3_file(s3_obj, file_content)\n# \n# # for a file:\n# with open(\"my_file.txt\", \"rb\") as my_file:\n# client.write_s3_file(s3_obj, my_file)\n# '''\ndef write_s3_file(s3object: S3Object | str | None, file_content: BufferedReader | bytes, s3_resource_path: str | None, content_type: str | None = None, content_disposition: str | None = None) -> S3Object\n\n# Permanently delete a file from the workspace S3 bucket.\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# client.delete_s3_object(s3_obj)\n# '''\ndef delete_s3_object(s3object: S3Object | str, s3_resource_path: str | None = None) -> None\n\n# Sign S3 objects for use by anonymous users in public apps.\n# \n# Args:\n# s3_objects: List of S3 objects to sign\n# \n# Returns:\n# List of signed S3 objects\ndef sign_s3_objects(s3_objects: list[S3Object | str]) -> list[S3Object]\n\n# Sign a single S3 object for use by anonymous users in public apps.\n# \n# Args:\n# s3_object: S3 object to sign\n# \n# Returns:\n# Signed S3 object\ndef sign_s3_object(s3_object: S3Object | str) -> S3Object\n\n# Generate presigned public URLs for an array of S3 objects.\n# If an S3 object is not signed yet, it will be signed first.\n# \n# Args:\n# s3_objects: List of S3 objects to sign\n# base_url: Optional base URL for the presigned URLs (defaults to WM_BASE_URL)\n# \n# Returns:\n# List of signed public URLs\n# \n# Example:\n# >>> s3_objs = [S3Object(s3=\"/path/to/file1.txt\"), S3Object(s3=\"/path/to/file2.txt\")]\n# >>> urls = client.get_presigned_s3_public_urls(s3_objs)\ndef get_presigned_s3_public_urls(s3_objects: list[S3Object | str], base_url: str | None = None) -> list[str]\n\n# Generate a presigned public URL for an S3 object.\n# If the S3 object is not signed yet, it will be signed first.\n# \n# Args:\n# s3_object: S3 object to sign\n# base_url: Optional base URL for the presigned URL (defaults to WM_BASE_URL)\n# \n# Returns:\n# Signed public URL\n# \n# Example:\n# >>> s3_obj = S3Object(s3=\"/path/to/file.txt\")\n# >>> url = client.get_presigned_s3_public_url(s3_obj)\ndef get_presigned_s3_public_url(s3_object: S3Object | str, base_url: str | None = None) -> str\n\n# Get the current user information.\n# \n# Returns:\n# User details dictionary\ndef whoami() -> dict\n\n# Get the current user information (alias for whoami).\n# \n# Returns:\n# User details dictionary\ndef user() -> dict\n\n# Get the state resource path from environment.\n# \n# Returns:\n# State path string\ndef state_path() -> str\n\n# Get the workflow state.\n# \n# Returns:\n# State value or None if not set\ndef state() -> Any\n\n# Set the state in the shared folder using pickle\ndef set_shared_state_pickle(value: Any, path: str = 'state.pickle') -> None\n\n# Get the state in the shared folder using pickle\ndef get_shared_state_pickle(path: str = 'state.pickle') -> Any\n\n# Set the state in the shared folder using pickle\ndef set_shared_state(value: Any, path: str = 'state.json') -> None\n\n# Get the state in the shared folder using pickle\ndef get_shared_state(path: str = 'state.json') -> None\n\n# Get URLs needed for resuming a flow after suspension.\n# \n# Args:\n# approver: Optional approver name\n# flow_level: If True, generate resume URLs for the parent flow instead of the\n# specific step. This allows pre-approvals that can be consumed by any later\n# suspend step in the same flow.\n# \n# Returns:\n# Dictionary with approvalPage, resume, and cancel URLs\ndef get_resume_urls(approver: str = None, flow_level: bool = None) -> dict\n\n# Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields.\n# \n# **[Enterprise Edition Only]** To include form fields in the Slack approval request, use the \"Advanced -> Suspend -> Form\" functionality.\n# Learn more at: https://www.windmill.dev/docs/flows/flow_approval#form\n# \n# :param slack_resource_path: The path to the Slack resource in Windmill.\n# :type slack_resource_path: str\n# :param channel_id: The Slack channel ID where the approval request will be sent.\n# :type channel_id: str\n# :param message: Optional custom message to include in the Slack approval request.\n# :type message: str, optional\n# :param approver: Optional user ID or name of the approver for the request.\n# :type approver: str, optional\n# :param default_args_json: Optional dictionary defining or overriding the default arguments for form fields.\n# :type default_args_json: dict, optional\n# :param dynamic_enums_json: Optional dictionary overriding the enum default values of enum form fields.\n# :type dynamic_enums_json: dict, optional\n# \n# :raises Exception: If the function is not called within a flow or flow preview.\n# :raises Exception: If the required flow job or flow step environment variables are not set.\n# \n# :return: None\n# \n# **Usage Example:**\n# >>> client.request_interactive_slack_approval(\n# ... slack_resource_path=\"/u/alex/my_slack_resource\",\n# ... channel_id=\"admins-slack-channel\",\n# ... message=\"Please approve this request\",\n# ... approver=\"approver123\",\n# ... default_args_json={\"key1\": \"value1\", \"key2\": 42},\n# ... dynamic_enums_json={\"foo\": [\"choice1\", \"choice2\"], \"bar\": [\"optionA\", \"optionB\"]},\n# ... )\n# \n# **Notes:**\n# - This function must be executed within a Windmill flow or flow preview.\n# - The function checks for required environment variables (`WM_FLOW_JOB_ID`, `WM_FLOW_STEP_ID`) to ensure it is run in the appropriate context.\ndef request_interactive_slack_approval(slack_resource_path: str, channel_id: str, message: str = None, approver: str = None, default_args_json: dict = None, dynamic_enums_json: dict = None) -> None\n\n# Get email from workspace username\n# This method is particularly useful for apps that require the email address of the viewer.\n# Indeed, in the viewer context WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app.\ndef username_to_email(username: str) -> str\n\n# Send a message to a Microsoft Teams conversation with conversation_id, where success is used to style the message\ndef send_teams_message(conversation_id: str, text: str, success: bool = True, card_block: dict = None)\n\n# Get a DataTable client for SQL queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# \n# Returns:\n# DataTableClient instance\ndef datatable(name: str = 'main')\n\n# Get a DuckLake client for DuckDB queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# \n# Returns:\n# DucklakeClient instance\ndef ducklake(name: str = 'main')\n\ndef init_global_client(f)\n\ndef deprecate(in_favor_of: str)\n\n# Get the current workspace ID.\n# \n# Returns:\n# Workspace ID string\ndef get_workspace() -> str\n\ndef get_version() -> str\n\n# Run a script synchronously by hash and return its result.\n# \n# Args:\n# hash: Script hash\n# args: Script arguments\n# verbose: Enable verbose logging\n# assert_result_is_not_none: Raise exception if result is None\n# cleanup: Register cleanup handler to cancel job on exit\n# timeout: Maximum time to wait\n# \n# Returns:\n# Script result\ndef run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any\n\n# Run a script synchronously by path and return its result.\n# \n# Args:\n# path: Script path\n# args: Script arguments\n# verbose: Enable verbose logging\n# assert_result_is_not_none: Raise exception if result is None\n# cleanup: Register cleanup handler to cancel job on exit\n# timeout: Maximum time to wait\n# \n# Returns:\n# Script result\ndef run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from DuckDB\ndef duckdb_connection_settings(s3_resource_path: str = '') -> DuckDbConnectionSettings\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from Polars\ndef polars_connection_settings(s3_resource_path: str = '') -> PolarsConnectionSettings\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection using boto3\ndef boto3_connection_settings(s3_resource_path: str = '') -> Boto3ConnectionSettings\n\n# Get the state resource path from environment.\n# \n# Returns:\n# State path string\ndef get_state_path() -> str\n\n# Parse resource syntax from string.\ndef parse_resource_syntax(s: str) -> Optional[str]\n\n# Parse S3 object from string or S3Object format.\ndef parse_s3_object(s3_object: S3Object | str) -> S3Object\n\n# Parse variable syntax from string.\ndef parse_variable_syntax(s: str) -> Optional[str]\n\n# Append a text to the result stream.\n# \n# Args:\n# text: text to append to the result stream\ndef append_to_result_stream(text: str) -> None\n\n# Stream to the result stream.\n# \n# Args:\n# stream: stream to stream to the result stream\ndef stream_result(stream) -> None\n\n# Execute a SQL query against the DataTable.\n# \n# Args:\n# sql: SQL query string with $1, $2, etc. placeholders\n# *args: Positional arguments to bind to query placeholders\n# \n# Returns:\n# SqlQuery instance for fetching results\ndef query(sql: str, *args) -> SqlQuery\n\n# Execute query and fetch results.\n# \n# Args:\n# result_collection: Optional result collection mode\n# \n# Returns:\n# Query results\ndef fetch(result_collection: str | None = None)\n\n# Execute query and fetch first row of results.\n# \n# Returns:\n# First row of query results\ndef fetch_one()\n\n# Execute query and fetch first row of results. Return result as a scalar value.\n# \n# Returns:\n# First row of query result as a scalar value\ndef fetch_one_scalar()\n\n# Execute query and don't return any results.\n# \ndef execute()\n\n# DuckDB executor requires explicit argument types at declaration\n# These types exist in both DuckDB and Postgres\n# Check that the types exist if you plan to extend this function for other SQL engines.\ndef infer_sql_type(value) -> str\n\ndef parse_sql_client_name(name: str) -> tuple[str, Optional[str]]\n\n# Decorator that marks a function as a workflow task.\n# \n# Works in both WAC v1 (sync, HTTP-based dispatch) and WAC v2\n# (async, checkpoint/replay) modes:\n# \n# - **v2 (inside @workflow)**: dispatches as a checkpoint step.\n# - **v1 (WM_JOB_ID set, no @workflow)**: dispatches via HTTP API.\n# - **Standalone**: executes the function body directly.\n# \n# Usage::\n# \n# @task\n# async def extract_data(url: str): ...\n# \n# @task(path=\"f/external_script\", timeout=600, tag=\"gpu\")\n# async def run_external(x: int): ...\ndef task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill script.\n# \n# Usage::\n# \n# extract = task_script(\"f/data/extract\", timeout=600)\n# \n# @workflow\n# async def main():\n# data = await extract(url=\"https://...\")\ndef task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill flow.\n# \n# Usage::\n# \n# pipeline = task_flow(\"f/etl/pipeline\", priority=10)\n# \n# @workflow\n# async def main():\n# result = await pipeline(input=data)\ndef task_flow(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Decorator marking an async function as a workflow-as-code entry point.\n# \n# The function must be **deterministic**: given the same inputs it must call\n# tasks in the same order on every replay. Branching on task results is fine\n# (results are replayed from checkpoint), but branching on external state\n# (current time, random values, external API calls) must use ``step()`` to\n# checkpoint the value so replays see the same result.\ndef workflow(func)\n\n# Execute ``fn`` inline and checkpoint the result.\n# \n# On replay the cached value is returned without re-executing ``fn``.\n# Use for lightweight deterministic operations (timestamps, random IDs,\n# config reads) that should not incur the overhead of a child job.\nasync def step(name: str, fn)\n\n# Server-side sleep \u2014 suspend the workflow for the given duration without holding a worker.\n# \n# Inside a @workflow, the parent job suspends and auto-resumes after ``seconds``.\n# Outside a workflow, falls back to ``asyncio.sleep``.\nasync def sleep(seconds: int)\n\n# Suspend the workflow and wait for an external approval.\n# \n# Use ``get_resume_urls()`` (wrapped in ``step()``) to obtain\n# resume/cancel/approval URLs before calling this function.\n# \n# Returns a dict with ``value`` (form data), ``approver``, and ``approved``.\n# \n# Args:\n# timeout: Approval timeout in seconds (default 1800).\n# form: Optional form schema for the approval page.\n# self_approval: Whether the user who triggered the flow can approve it (default True).\n# \n# Example::\n# \n# urls = await step(\"urls\", lambda: get_resume_urls())\n# await step(\"notify\", lambda: send_email(urls[\"approvalPage\"]))\n# result = await wait_for_approval(timeout=3600)\nasync def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_approval: bool = True) -> dict\n\n# Process items in parallel with optional concurrency control.\n# \n# Each item is processed by calling ``fn(item)``, which should be a @task.\n# Items are dispatched in batches of ``concurrency`` (default: all at once).\n# \n# Example::\n# \n# @task\n# async def process(item: str):\n# ...\n# \n# results = await parallel(items, process, concurrency=5)\nasync def parallel(items, fn, concurrency: Optional[int] = None)\n\n# Commit Kafka offsets for a trigger with auto_commit disabled.\n# \n# Args:\n# trigger_path: Path to the Kafka trigger (from event['wm_trigger']['trigger_path'])\n# topic: Kafka topic name (from event['topic'])\n# partition: Partition number (from event['partition'])\n# offset: Message offset to commit (from event['offset'])\ndef commit_kafka_offsets(trigger_path: str, topic: str, partition: int, offset: int) -> None\n\n"; -export declare const WAC_SDK_TYPESCRIPT = "## TypeScript Workflow-as-Code API (windmill-client)\n\nImport: `import { workflow, task, taskScript, taskFlow, step, sleep, waitForApproval, getResumeUrls, parallel } from \"windmill-client\"`\n\n```typescript\nexport interface TaskOptions {\n timeout?: number;\n tag?: string;\n cache_ttl?: number;\n priority?: number;\n concurrency_limit?: number;\n concurrency_key?: string;\n concurrency_time_window_s?: number;\n}\n\n/**\n * Get URLs needed for resuming a flow after this step\n * @param approver approver name\n * @param flowLevel if true, generate resume URLs for the parent flow instead of the specific step.\n * This allows pre-approvals that can be consumed by any later suspend step in the same flow.\n * @returns approval page UI URL, resume and cancel API URLs for resuming the flow\n */\nexport async function getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ approvalPage: string; resume: string; cancel: string; }>\n\n/**\n * Wrap an async function as a workflow task.\n *\n * @example\n * const extract_data = task(async (url: string) => { ... });\n * const run_external = task(\"f/external_script\", async (x: number) => { ... });\n *\n * Inside a `workflow()`, calling a task dispatches it as a step.\n * Outside a workflow, the function body executes directly.\n */\nexport function task Promise>(fnOrPath: T | string, maybeFnOrOptions?: T | TaskOptions, maybeOptions?: TaskOptions,): T\n\n/**\n * Create a task that dispatches to a separate Windmill script.\n *\n * @example\n * const extract = taskScript(\"f/data/extract\");\n * // inside workflow: await extract({ url: \"https://...\" })\n */\nexport function taskScript(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Create a task that dispatches to a separate Windmill flow.\n *\n * @example\n * const pipeline = taskFlow(\"f/etl/pipeline\");\n * // inside workflow: await pipeline({ input: data })\n */\nexport function taskFlow(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Mark an async function as a workflow-as-code entry point.\n *\n * The function must be **deterministic**: given the same inputs it must call\n * tasks in the same order on every replay. Branching on task results is fine\n * (results are replayed from checkpoint), but branching on external state\n * (current time, random values, external API calls) must use `step()` to\n * checkpoint the value so replays see the same result.\n */\nexport function workflow(fn: (...args: any[]) => Promise)\n\nexport async function step(name: string, fn: () => T | Promise): Promise\n\nexport async function sleep(seconds: number): Promise\n\n/**\n * Suspend the workflow and wait for an external approval.\n *\n * Use `getResumeUrls()` (wrapped in `step()`) to obtain resume/cancel/approvalPage\n * URLs before calling this function.\n *\n * @example\n * const urls = await step(\"urls\", () => getResumeUrls());\n * await step(\"notify\", () => sendEmail(urls.approvalPage));\n * const { value, approver } = await waitForApproval({ timeout: 3600 });\n */\nexport function waitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }>\n\n/**\n * Process items in parallel with optional concurrency control.\n *\n * Each item is processed by calling `fn(item)`, which should be a task().\n * Items are dispatched in batches of `concurrency` (default: all at once).\n *\n * @example\n * const process = task(async (item: string) => { ... });\n * const results = await parallel(items, process, { concurrency: 5 });\n */\nexport async function parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: { concurrency?: number },): Promise\n```\n"; -export declare const WAC_SDK_PYTHON = "## Python Workflow-as-Code API (wmill)\n\nImport: `from wmill import workflow, task, task_script, task_flow, step, sleep, wait_for_approval, get_resume_urls, parallel, TaskError`\n\n```python\n# Raised when a WAC task step failed.\n#\n# Attributes:\n# step_key: The checkpoint key of the failed step.\n# child_job_id: The UUID of the failed child job.\n# result: The error result from the child job.\nclass TaskError(Exception):\n def __init__(self, message: str, *, step_key: str = '', child_job_id: str = '', result = None)\n\n# Get URLs needed for resuming a flow after suspension.\n#\n# Args:\n# approver: Optional approver name\n# flow_level: If True, generate resume URLs for the parent flow instead of the\n# specific step. This allows pre-approvals that can be consumed by any later\n# suspend step in the same flow.\n#\n# Returns:\n# Dictionary with approvalPage, resume, and cancel URLs\ndef get_resume_urls(approver: str = None, flow_level: bool = None) -> dict\n\n# Decorator that marks a function as a workflow task.\n#\n# Works in both WAC v1 (sync, HTTP-based dispatch) and WAC v2\n# (async, checkpoint/replay) modes:\n#\n# - **v2 (inside @workflow)**: dispatches as a checkpoint step.\n# - **v1 (WM_JOB_ID set, no @workflow)**: dispatches via HTTP API.\n# - **Standalone**: executes the function body directly.\n#\n# Usage::\n#\n# @task\n# async def extract_data(url: str): ...\n#\n# @task(path=\"f/external_script\", timeout=600, tag=\"gpu\")\n# async def run_external(x: int): ...\ndef task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill script.\n#\n# Usage::\n#\n# extract = task_script(\"f/data/extract\", timeout=600)\n#\n# @workflow\n# async def main():\n# data = await extract(url=\"https://...\")\ndef task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill flow.\n#\n# Usage::\n#\n# pipeline = task_flow(\"f/etl/pipeline\", priority=10)\n#\n# @workflow\n# async def main():\n# result = await pipeline(input=data)\ndef task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Decorator marking an async function as a workflow-as-code entry point.\n#\n# The function must be **deterministic**: given the same inputs it must call\n# tasks in the same order on every replay. Branching on task results is fine\n# (results are replayed from checkpoint), but branching on external state\n# (current time, random values, external API calls) must use ``step()`` to\n# checkpoint the value so replays see the same result.\ndef workflow(func)\n\n# Execute ``fn`` inline and checkpoint the result.\n#\n# On replay the cached value is returned without re-executing ``fn``.\n# Use for lightweight deterministic operations (timestamps, random IDs,\n# config reads) that should not incur the overhead of a child job.\nasync def step(name: str, fn)\n\n# Server-side sleep \u2014 suspend the workflow for the given duration without holding a worker.\n#\n# Inside a @workflow, the parent job suspends and auto-resumes after ``seconds``.\n# Outside a workflow, falls back to ``asyncio.sleep``.\nasync def sleep(seconds: int)\n\n# Suspend the workflow and wait for an external approval.\n#\n# Use ``get_resume_urls()`` (wrapped in ``step()``) to obtain\n# resume/cancel/approval URLs before calling this function.\n#\n# Returns a dict with ``value`` (form data), ``approver``, and ``approved``.\n#\n# Args:\n# timeout: Approval timeout in seconds (default 1800).\n# form: Optional form schema for the approval page.\n# self_approval: Whether the user who triggered the flow can approve it (default True).\n#\n# Example::\n#\n# urls = await step(\"urls\", lambda: get_resume_urls())\n# await step(\"notify\", lambda: send_email(urls[\"approvalPage\"]))\n# result = await wait_for_approval(timeout=3600)\nasync def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_approval: bool = True) -> dict\n\n# Process items in parallel with optional concurrency control.\n#\n# Each item is processed by calling ``fn(item)``, which should be a @task.\n# Items are dispatched in batches of ``concurrency`` (default: all at once).\n#\n# Example::\n#\n# @task\n# async def process(item: str):\n# ...\n#\n# results = await parallel(items, process, concurrency=5)\nasync def parallel(items, fn, *, concurrency: Optional[int] = None)\n```\n"; -export declare const DATATABLE_SDK_TYPESCRIPT = "## TypeScript Datatable API (windmill-client)\n\nImport: `import * as wmill from 'windmill-client'`\n\nSQL statement object with query content, arguments, and execution methods\n```typescript\ntype SqlStatement = {\n /** Raw SQL content with formatted arguments */\n content: string;\n\n /** Argument values keyed by parameter name */\n args: Record;\n\n /**\n * Execute the SQL query and return results\n * @param params - Optional parameters including result collection mode\n * @returns Query results based on the result collection mode\n */\n fetch(\n params?: FetchParams // The union is for auto-completion\n ): Promise>;\n\n /**\n * Execute the SQL query and return only the first row\n * @param params - Optional parameters\n * @returns First row of the query result\n */\n fetchOne(\n params?: Omit, \"resultCollection\">\n ): Promise>;\n\n /**\n * Execute the SQL query and return only the first row as a scalar value\n * @param params - Optional parameters\n * @returns First row of the query result\n */\n fetchOneScalar(\n params?: Omit<\n FetchParams<\"last_statement_first_row_scalar\">,\n \"resultCollection\"\n >\n ): Promise>;\n\n /**\n * Execute the SQL query without fetching rows\n * @param params - Optional parameters\n */\n execute(\n params?: Omit, \"resultCollection\">\n ): Promise;\n};\n```\n\n```typescript\n// Template tag function: sql`SELECT * FROM table WHERE id = ${id}`.fetch()\ninterface DatatableSqlTemplateFunction {\n // Tagged template usage:\n (strings: TemplateStringsArray, ...values: any[]): SqlStatement;\n query(sql: string, ...params: any[]): SqlStatement;\n};\n```\n\nCreate a SQL template function for PostgreSQL/datatable queries\n@param name - Database/datatable name (default: \"main\")\n@returns SQL template function for building parameterized queries\n@example\nlet sql = wmill.datatable()\nlet name = 'Robin'\nlet age = 21\nawait sql`\n SELECT * FROM friends\n WHERE name = ${name} AND age = ${age}::int\n`.fetch()\n```typescript\nfunction datatable(name: string = \"main\"): DatatableSqlTemplateFunction\n```\n"; -export declare const DATATABLE_SDK_PYTHON = "## Python Datatable API (wmill)\n\nImport: `import wmill`\n\n# Get a DataTable client for SQL queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# \n# Returns:\n# DataTableClient instance\ndef datatable(name: str = 'main') -> DataTableClient\n\n# Client for executing SQL queries against Windmill DataTables.\nclass DataTableClient:\n # Initialize DataTableClient.\n # \n # Args:\n # client: Windmill client instance\n # name: DataTable name\n def __init__(client: Windmill, name: str)\n\n # Execute a SQL query against the DataTable.\n # \n # Args:\n # sql: SQL query string with $1, $2, etc. placeholders\n # *args: Positional arguments to bind to query placeholders\n # \n # Returns:\n # SqlQuery instance for fetching results\n def query(sql: str, *args) -> SqlQuery\n\n\n# Query result handler for DataTable and DuckLake queries.\nclass SqlQuery:\n # Initialize SqlQuery.\n # \n # Args:\n # sql: SQL query string\n # fetch_fn: Function to execute the query\n def __init__(sql: str, fetch_fn)\n\n # Execute query and fetch results.\n # \n # Args:\n # result_collection: Optional result collection mode\n # \n # Returns:\n # Query results\n def fetch(result_collection: str | None = None)\n\n # Execute query and fetch first row of results.\n # \n # Returns:\n # First row of query results\n def fetch_one()\n\n # Execute query and fetch first row of results. Return result as a scalar value.\n # \n # Returns:\n # First row of query result as a scalar value\n def fetch_one_scalar()\n\n # Execute query and don't return any results.\n # \n def execute()\n\n\n"; -export declare const OPENFLOW_SCHEMA = "## OpenFlow Schema\n\n{\"OpenFlow\":{\"type\":\"object\",\"description\":\"Top-level flow definition containing metadata, configuration, and the flow structure\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Short description of what this flow does\"},\"description\":{\"type\":\"string\",\"description\":\"Detailed documentation for this flow\"},\"value\":{\"$ref\":\"#/components/schemas/FlowValue\"},\"schema\":{\"type\":\"object\",\"description\":\"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')\"},\"on_behalf_of_email\":{\"type\":\"string\",\"description\":\"The flow will be run with the permissions of the user with this email.\"}},\"required\":[\"summary\",\"value\"]},\"FlowValue\":{\"type\":\"object\",\"description\":\"The flow structure containing modules and optional preprocessor/failure handlers\",\"properties\":{\"modules\":{\"type\":\"array\",\"description\":\"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"failure_module\":{\"description\":\"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types\",\"$ref\":\"#/components/schemas/FlowModule\"},\"preprocessor_module\":{\"description\":\"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results\",\"$ref\":\"#/components/schemas/FlowModule\"},\"same_worker\":{\"type\":\"boolean\",\"description\":\"If true, all steps run on the same worker for better performance\"},\"concurrent_limit\":{\"type\":\"number\",\"description\":\"Maximum number of concurrent executions of this flow\"},\"concurrency_key\":{\"type\":\"string\",\"description\":\"Expression to group concurrent executions (e.g., by user ID)\"},\"concurrency_time_window_s\":{\"type\":\"number\",\"description\":\"Time window in seconds for concurrent_limit\"},\"debounce_delay_s\":{\"type\":\"integer\",\"description\":\"Delay in seconds to debounce flow executions\"},\"debounce_key\":{\"type\":\"string\",\"description\":\"Expression to group debounced executions\"},\"debounce_args_to_accumulate\":{\"type\":\"array\",\"description\":\"Arguments to accumulate across debounced executions\",\"items\":{\"type\":\"string\"}},\"max_total_debouncing_time\":{\"type\":\"integer\",\"description\":\"Maximum total time in seconds that a job can be debounced\"},\"max_total_debounces_amount\":{\"type\":\"integer\",\"description\":\"Maximum number of times a job can be debounced\"},\"skip_expr\":{\"type\":\"string\",\"description\":\"JavaScript expression to conditionally skip the entire flow\"},\"cache_ttl\":{\"type\":\"number\",\"description\":\"Cache duration in seconds for flow results\"},\"cache_ignore_s3_path\":{\"type\":\"boolean\"},\"delete_after_secs\":{\"type\":\"integer\",\"description\":\"If set, delete the flow job's args, result and logs after this many seconds following job completion\"},\"flow_env\":{\"type\":\"object\",\"description\":\"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).\",\"additionalProperties\":{}},\"priority\":{\"type\":\"number\",\"description\":\"Execution priority (higher numbers run first)\"},\"early_return\":{\"type\":\"string\",\"description\":\"JavaScript expression to return early from the flow\"},\"chat_input_enabled\":{\"type\":\"boolean\",\"description\":\"Whether this flow accepts chat-style input\"},\"notes\":{\"type\":\"array\",\"description\":\"Sticky notes attached to the flow\",\"items\":{\"$ref\":\"#/components/schemas/FlowNote\"}},\"groups\":{\"type\":\"array\",\"description\":\"Semantic groups of modules for organizational purposes\",\"items\":{\"$ref\":\"#/components/schemas/FlowGroup\"}}},\"required\":[\"modules\"]},\"Retry\":{\"type\":\"object\",\"description\":\"Retry configuration for failed module executions\",\"properties\":{\"constant\":{\"type\":\"object\",\"description\":\"Retry with constant delay between attempts\",\"properties\":{\"attempts\":{\"type\":\"integer\",\"description\":\"Number of retry attempts\"},\"seconds\":{\"type\":\"integer\",\"description\":\"Seconds to wait between retries\"}}},\"exponential\":{\"type\":\"object\",\"description\":\"Retry with exponential backoff (delay doubles each time)\",\"properties\":{\"attempts\":{\"type\":\"integer\",\"description\":\"Number of retry attempts\"},\"multiplier\":{\"type\":\"integer\",\"description\":\"Multiplier for exponential backoff\"},\"seconds\":{\"type\":\"integer\",\"minimum\":1,\"description\":\"Initial delay in seconds\"},\"random_factor\":{\"type\":\"integer\",\"minimum\":0,\"maximum\":100,\"description\":\"Random jitter percentage (0-100) to avoid thundering herd\"}}},\"retry_if\":{\"$ref\":\"#/components/schemas/RetryIf\"}}},\"FlowNote\":{\"type\":\"object\",\"description\":\"A sticky note attached to a flow for documentation and annotation\",\"properties\":{\"id\":{\"type\":\"string\",\"description\":\"Unique identifier for the note\"},\"text\":{\"type\":\"string\",\"description\":\"Content of the note\"},\"position\":{\"type\":\"object\",\"description\":\"Position of the note in the flow editor\",\"properties\":{\"x\":{\"type\":\"number\",\"description\":\"X coordinate\"},\"y\":{\"type\":\"number\",\"description\":\"Y coordinate\"}},\"required\":[\"x\",\"y\"]},\"size\":{\"type\":\"object\",\"description\":\"Size of the note in the flow editor\",\"properties\":{\"width\":{\"type\":\"number\",\"description\":\"Width in pixels\"},\"height\":{\"type\":\"number\",\"description\":\"Height in pixels\"}},\"required\":[\"width\",\"height\"]},\"color\":{\"type\":\"string\",\"description\":\"Color of the note (e.g., \\\"yellow\\\", \\\"#ffff00\\\")\"},\"type\":{\"type\":\"string\",\"enum\":[\"free\",\"group\"],\"description\":\"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes\"},\"locked\":{\"type\":\"boolean\",\"default\":false,\"description\":\"Whether the note is locked and cannot be edited or moved\"},\"contained_node_ids\":{\"type\":\"array\",\"items\":{\"type\":\"string\"},\"description\":\"For group notes, the IDs of nodes contained within this group\"}},\"required\":[\"id\",\"text\",\"color\",\"type\"]},\"FlowGroup\":{\"type\":\"object\",\"description\":\"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Display name for this group\"},\"note\":{\"type\":\"string\",\"description\":\"Markdown note shown below the group header\"},\"autocollapse\":{\"type\":\"boolean\",\"default\":false,\"description\":\"If true, this group is collapsed by default in the flow editor. UI hint only.\"},\"start_id\":{\"type\":\"string\",\"description\":\"ID of the first flow module in this group (topological entry point)\"},\"end_id\":{\"type\":\"string\",\"description\":\"ID of the last flow module in this group (topological exit point)\"},\"color\":{\"type\":\"string\",\"description\":\"Color for the group in the flow editor\"}},\"required\":[\"start_id\",\"end_id\"]},\"RetryIf\":{\"type\":\"object\",\"description\":\"Conditional retry based on error or result\",\"properties\":{\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables\"}},\"required\":[\"expr\"]},\"StopAfterIf\":{\"type\":\"object\",\"description\":\"Early termination condition for a module\",\"properties\":{\"skip_if_stopped\":{\"type\":\"boolean\",\"description\":\"If true, following steps are skipped when this condition triggers\"},\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop\"},\"error_message\":{\"type\":\"string\",\"nullable\":true,\"description\":\"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.\"}},\"required\":[\"expr\"]},\"FlowModule\":{\"type\":\"object\",\"description\":\"A single step in a flow. Can be a script, subflow, loop, or branch\",\"properties\":{\"id\":{\"type\":\"string\",\"description\":\"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)\"},\"value\":{\"$ref\":\"#/components/schemas/FlowModuleValue\"},\"stop_after_if\":{\"description\":\"Early termination condition evaluated after this step completes\",\"$ref\":\"#/components/schemas/StopAfterIf\"},\"stop_after_all_iters_if\":{\"description\":\"For loops only - early termination condition evaluated after all iterations complete\",\"$ref\":\"#/components/schemas/StopAfterIf\"},\"skip_if\":{\"type\":\"object\",\"description\":\"Conditionally skip this step based on previous results or flow inputs\",\"properties\":{\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'\"}},\"required\":[\"expr\"]},\"sleep\":{\"description\":\"Delay before executing this step (in seconds or as expression)\",\"$ref\":\"#/components/schemas/InputTransform\"},\"cache_ttl\":{\"type\":\"number\",\"description\":\"Cache duration in seconds for this step's results\"},\"cache_ignore_s3_path\":{\"type\":\"boolean\"},\"timeout\":{\"description\":\"Maximum execution time in seconds (static value or expression)\",\"$ref\":\"#/components/schemas/InputTransform\"},\"delete_after_secs\":{\"type\":\"integer\",\"description\":\"If set, delete the step's args, result and logs after this many seconds following job completion\"},\"summary\":{\"type\":\"string\",\"description\":\"Short description of what this step does\"},\"mock\":{\"type\":\"object\",\"description\":\"Mock configuration for testing without executing the actual step\",\"properties\":{\"enabled\":{\"type\":\"boolean\",\"description\":\"If true, return mock value instead of executing\"},\"return_value\":{\"description\":\"Value to return when mocked\"}}},\"suspend\":{\"type\":\"object\",\"description\":\"Configuration for approval/resume steps that wait for user input\",\"properties\":{\"required_events\":{\"type\":\"integer\",\"description\":\"Number of approvals required before continuing\"},\"timeout\":{\"type\":\"integer\",\"description\":\"Timeout in seconds before auto-continuing or canceling\"},\"resume_form\":{\"type\":\"object\",\"description\":\"Form schema for collecting input when resuming\",\"properties\":{\"schema\":{\"type\":\"object\",\"description\":\"JSON Schema for the resume form\"}}},\"user_auth_required\":{\"type\":\"boolean\",\"description\":\"If true, only authenticated users can approve\"},\"user_groups_required\":{\"description\":\"Expression or list of groups that can approve\",\"$ref\":\"#/components/schemas/InputTransform\"},\"self_approval_disabled\":{\"type\":\"boolean\",\"description\":\"If true, the user who started the flow cannot approve\"},\"hide_cancel\":{\"type\":\"boolean\",\"description\":\"If true, hide the cancel button on the approval form\"},\"continue_on_disapprove_timeout\":{\"type\":\"boolean\",\"description\":\"If true, continue flow on timeout instead of canceling\"}}},\"priority\":{\"type\":\"number\",\"description\":\"Execution priority for this step (higher numbers run first)\"},\"continue_on_error\":{\"type\":\"boolean\",\"description\":\"If true, flow continues even if this step fails\"},\"retry\":{\"description\":\"Retry configuration if this step fails\",\"$ref\":\"#/components/schemas/Retry\"},\"debouncing\":{\"description\":\"Debounce configuration for this step (EE only)\",\"type\":\"object\",\"properties\":{\"debounce_delay_s\":{\"type\":\"integer\",\"description\":\"Delay in seconds to debounce this step's executions across flow runs\"},\"debounce_key\":{\"type\":\"string\",\"description\":\"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-\"},\"debounce_args_to_accumulate\":{\"type\":\"array\",\"description\":\"Array-type arguments to accumulate across debounced executions\",\"items\":{\"type\":\"string\"}},\"max_total_debouncing_time\":{\"type\":\"integer\",\"description\":\"Maximum total time in seconds before forced execution\"},\"max_total_debounces_amount\":{\"type\":\"integer\",\"description\":\"Maximum number of debounces before forced execution\"}}}},\"required\":[\"value\",\"id\"]},\"InputTransform\":{\"description\":\"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs\",\"oneOf\":[{\"$ref\":\"#/components/schemas/StaticTransform\"},{\"$ref\":\"#/components/schemas/JavascriptTransform\"},{\"$ref\":\"#/components/schemas/AiTransform\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"static\":\"#/components/schemas/StaticTransform\",\"javascript\":\"#/components/schemas/JavascriptTransform\",\"ai\":\"#/components/schemas/AiTransform\"}}},\"StaticTransform\":{\"type\":\"object\",\"description\":\"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'\",\"properties\":{\"value\":{\"description\":\"The static value. For resources, use format '$res:path/to/resource'\"},\"type\":{\"type\":\"string\",\"enum\":[\"static\"]}},\"required\":[\"type\"]},\"JavascriptTransform\":{\"type\":\"object\",\"description\":\"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value\",\"properties\":{\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)\"},\"type\":{\"type\":\"string\",\"enum\":[\"javascript\"]}},\"required\":[\"expr\",\"type\"]},\"AiTransform\":{\"type\":\"object\",\"description\":\"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"ai\"]}},\"required\":[\"type\"]},\"AIProviderKind\":{\"type\":\"string\",\"description\":\"Supported AI provider types\",\"enum\":[\"openai\",\"azure_openai\",\"anthropic\",\"mistral\",\"deepseek\",\"googleai\",\"groq\",\"openrouter\",\"togetherai\",\"customai\",\"aws_bedrock\"]},\"ProviderConfig\":{\"type\":\"object\",\"description\":\"Complete AI provider configuration with resource reference and model selection\",\"properties\":{\"kind\":{\"$ref\":\"#/components/schemas/AIProviderKind\"},\"resource\":{\"type\":\"string\",\"description\":\"Resource reference in format '$res:{resource_path}' pointing to provider credentials\"},\"model\":{\"type\":\"string\",\"description\":\"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')\"}},\"required\":[\"kind\",\"resource\",\"model\"]},\"StaticProviderTransform\":{\"type\":\"object\",\"description\":\"Static provider configuration passed directly to the AI agent\",\"properties\":{\"value\":{\"$ref\":\"#/components/schemas/ProviderConfig\"},\"type\":{\"type\":\"string\",\"enum\":[\"static\"]}},\"required\":[\"type\",\"value\"]},\"ProviderTransform\":{\"description\":\"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined\",\"oneOf\":[{\"$ref\":\"#/components/schemas/StaticProviderTransform\"},{\"$ref\":\"#/components/schemas/JavascriptTransform\"},{\"$ref\":\"#/components/schemas/AiTransform\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"static\":\"#/components/schemas/StaticProviderTransform\",\"javascript\":\"#/components/schemas/JavascriptTransform\",\"ai\":\"#/components/schemas/AiTransform\"}}},\"MemoryOff\":{\"type\":\"object\",\"description\":\"No conversation memory/context\",\"properties\":{\"kind\":{\"type\":\"string\",\"enum\":[\"off\"]}},\"required\":[\"kind\"]},\"MemoryAuto\":{\"type\":\"object\",\"description\":\"Automatic context management\",\"properties\":{\"kind\":{\"type\":\"string\",\"enum\":[\"auto\"]},\"context_length\":{\"type\":\"integer\",\"description\":\"Maximum number of messages to retain in context\"},\"memory_id\":{\"type\":\"string\",\"description\":\"Identifier for persistent memory across agent invocations\"}},\"required\":[\"kind\"]},\"MemoryMessage\":{\"type\":\"object\",\"description\":\"A single message in conversation history\",\"properties\":{\"role\":{\"type\":\"string\",\"enum\":[\"user\",\"assistant\",\"system\"]},\"content\":{\"type\":\"string\"}},\"required\":[\"role\",\"content\"]},\"MemoryManual\":{\"type\":\"object\",\"description\":\"Explicit message history\",\"properties\":{\"kind\":{\"type\":\"string\",\"enum\":[\"manual\"]},\"messages\":{\"type\":\"array\",\"items\":{\"$ref\":\"#/components/schemas/MemoryMessage\"}}},\"required\":[\"kind\",\"messages\"]},\"MemoryConfig\":{\"description\":\"Conversation memory configuration\",\"oneOf\":[{\"$ref\":\"#/components/schemas/MemoryOff\"},{\"$ref\":\"#/components/schemas/MemoryAuto\"},{\"$ref\":\"#/components/schemas/MemoryManual\"}],\"discriminator\":{\"propertyName\":\"kind\",\"mapping\":{\"off\":\"#/components/schemas/MemoryOff\",\"auto\":\"#/components/schemas/MemoryAuto\",\"manual\":\"#/components/schemas/MemoryManual\"}}},\"StaticMemoryTransform\":{\"type\":\"object\",\"description\":\"Static memory configuration passed directly to the AI agent\",\"properties\":{\"value\":{\"$ref\":\"#/components/schemas/MemoryConfig\"},\"type\":{\"type\":\"string\",\"enum\":[\"static\"]}},\"required\":[\"type\",\"value\"]},\"MemoryTransform\":{\"description\":\"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined\",\"oneOf\":[{\"$ref\":\"#/components/schemas/StaticMemoryTransform\"},{\"$ref\":\"#/components/schemas/JavascriptTransform\"},{\"$ref\":\"#/components/schemas/AiTransform\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"static\":\"#/components/schemas/StaticMemoryTransform\",\"javascript\":\"#/components/schemas/JavascriptTransform\",\"ai\":\"#/components/schemas/AiTransform\"}}},\"FlowModuleValue\":{\"description\":\"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type\",\"oneOf\":[{\"$ref\":\"#/components/schemas/RawScript\"},{\"$ref\":\"#/components/schemas/PathScript\"},{\"$ref\":\"#/components/schemas/PathFlow\"},{\"$ref\":\"#/components/schemas/ForloopFlow\"},{\"$ref\":\"#/components/schemas/WhileloopFlow\"},{\"$ref\":\"#/components/schemas/BranchOne\"},{\"$ref\":\"#/components/schemas/BranchAll\"},{\"$ref\":\"#/components/schemas/Identity\"},{\"$ref\":\"#/components/schemas/AiAgent\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"rawscript\":\"#/components/schemas/RawScript\",\"script\":\"#/components/schemas/PathScript\",\"flow\":\"#/components/schemas/PathFlow\",\"forloopflow\":\"#/components/schemas/ForloopFlow\",\"whileloopflow\":\"#/components/schemas/WhileloopFlow\",\"branchone\":\"#/components/schemas/BranchOne\",\"branchall\":\"#/components/schemas/BranchAll\",\"identity\":\"#/components/schemas/Identity\",\"aiagent\":\"#/components/schemas/AiAgent\"}}},\"RawScript\":{\"type\":\"object\",\"description\":\"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments\",\"additionalProperties\":{\"$ref\":\"#/components/schemas/InputTransform\"}},\"content\":{\"type\":\"string\",\"description\":\"The script source code. Should export a 'main' function\"},\"language\":{\"type\":\"string\",\"description\":\"Programming language for this script\",\"enum\":[\"deno\",\"bun\",\"python3\",\"go\",\"bash\",\"powershell\",\"postgresql\",\"mysql\",\"bigquery\",\"snowflake\",\"mssql\",\"oracledb\",\"graphql\",\"nativets\",\"php\",\"rust\",\"ansible\",\"csharp\",\"nu\",\"java\",\"ruby\",\"rlang\",\"duckdb\"]},\"path\":{\"type\":\"string\",\"description\":\"Optional path for saving this script\"},\"lock\":{\"type\":\"string\",\"description\":\"Lock file content for dependencies\"},\"type\":{\"type\":\"string\",\"enum\":[\"rawscript\"]},\"tag\":{\"type\":\"string\",\"description\":\"Worker group tag for execution routing\"},\"concurrent_limit\":{\"type\":\"number\",\"description\":\"Maximum concurrent executions of this script\"},\"concurrency_time_window_s\":{\"type\":\"number\",\"description\":\"Time window for concurrent_limit\"},\"custom_concurrency_key\":{\"type\":\"string\",\"description\":\"Custom key for grouping concurrent executions\"},\"is_trigger\":{\"type\":\"boolean\",\"description\":\"If true, this script is a trigger that can start the flow\"},\"assets\":{\"type\":\"array\",\"description\":\"External resources this script accesses (S3 objects, resources, etc.)\",\"items\":{\"type\":\"object\",\"required\":[\"path\",\"kind\"],\"properties\":{\"path\":{\"type\":\"string\",\"description\":\"Path to the asset\"},\"kind\":{\"type\":\"string\",\"description\":\"Type of asset\",\"enum\":[\"s3object\",\"resource\",\"ducklake\",\"datatable\",\"volume\"]},\"access_type\":{\"type\":\"string\",\"nullable\":true,\"description\":\"Access level for this asset\",\"enum\":[\"r\",\"w\",\"rw\"]},\"alt_access_type\":{\"type\":\"string\",\"nullable\":true,\"description\":\"Alternative access level\",\"enum\":[\"r\",\"w\",\"rw\"]}}}}},\"required\":[\"type\",\"content\",\"language\",\"input_transforms\"]},\"PathScript\":{\"type\":\"object\",\"description\":\"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments\",\"additionalProperties\":{\"$ref\":\"#/components/schemas/InputTransform\"}},\"path\":{\"type\":\"string\",\"description\":\"Path to the script in the workspace (e.g., 'f/scripts/send_email')\"},\"hash\":{\"type\":\"string\",\"description\":\"Optional specific version hash of the script to use\"},\"type\":{\"type\":\"string\",\"enum\":[\"script\"]},\"tag_override\":{\"type\":\"string\",\"description\":\"Override the script's default worker group tag\"},\"is_trigger\":{\"type\":\"boolean\",\"description\":\"If true, this script is a trigger that can start the flow\"}},\"required\":[\"type\",\"path\",\"input_transforms\"]},\"PathFlow\":{\"type\":\"object\",\"description\":\"Reference to an existing flow by path. Use this to call another flow as a subflow\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments\",\"additionalProperties\":{\"$ref\":\"#/components/schemas/InputTransform\"}},\"path\":{\"type\":\"string\",\"description\":\"Path to the flow in the workspace (e.g., 'f/flows/process_user')\"},\"type\":{\"type\":\"string\",\"enum\":[\"flow\"]}},\"required\":[\"type\",\"path\",\"input_transforms\"]},\"ForloopFlow\":{\"type\":\"object\",\"description\":\"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations\",\"properties\":{\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"iterator\":{\"description\":\"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'\",\"$ref\":\"#/components/schemas/InputTransform\"},\"skip_failures\":{\"type\":\"boolean\",\"description\":\"If true, iteration failures don't stop the loop. Failed iterations return null\"},\"type\":{\"type\":\"string\",\"enum\":[\"forloopflow\"]},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency\"},\"parallelism\":{\"description\":\"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression\",\"$ref\":\"#/components/schemas/InputTransform\"},\"squash\":{\"type\":\"boolean\"}},\"required\":[\"modules\",\"iterator\",\"skip_failures\",\"type\"]},\"WhileloopFlow\":{\"type\":\"object\",\"description\":\"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination\",\"properties\":{\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute in each iteration. Use stop_after_if to control when the loop ends\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"skip_failures\":{\"type\":\"boolean\",\"description\":\"If true, iteration failures don't stop the loop. Failed iterations return null\"},\"type\":{\"type\":\"string\",\"enum\":[\"whileloopflow\"]},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, iterations run concurrently (use with caution in while loops)\"},\"parallelism\":{\"description\":\"Maximum number of concurrent iterations when parallel=true\",\"$ref\":\"#/components/schemas/InputTransform\"},\"squash\":{\"type\":\"boolean\"}},\"required\":[\"modules\",\"skip_failures\",\"type\"]},\"BranchOne\":{\"type\":\"object\",\"description\":\"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes\",\"properties\":{\"branches\":{\"type\":\"array\",\"description\":\"Array of branches to evaluate in order. The first branch with expr evaluating to true executes\",\"items\":{\"type\":\"object\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Short description of this branch condition\"},\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins\"},\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute if this branch's expr is true\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}}},\"required\":[\"modules\",\"expr\"]}},\"default\":{\"type\":\"array\",\"description\":\"Steps to execute if no branch expressions match\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"type\":{\"type\":\"string\",\"enum\":[\"branchone\"]}},\"required\":[\"branches\",\"default\",\"type\"]},\"BranchAll\":{\"type\":\"object\",\"description\":\"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently\",\"properties\":{\"branches\":{\"type\":\"array\",\"description\":\"Array of branches that all execute (either in parallel or sequentially)\",\"items\":{\"type\":\"object\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Short description of this branch's purpose\"},\"skip_failure\":{\"type\":\"boolean\",\"description\":\"If true, failure in this branch doesn't fail the entire flow\"},\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute in this branch\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}}},\"required\":[\"modules\"]}},\"type\":{\"type\":\"string\",\"enum\":[\"branchall\"]},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, all branches execute concurrently. If false, they execute sequentially\"}},\"required\":[\"branches\",\"type\"]},\"AgentTool\":{\"type\":\"object\",\"description\":\"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool\",\"properties\":{\"id\":{\"type\":\"string\",\"description\":\"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')\"},\"summary\":{\"type\":\"string\",\"description\":\"Short description of what this tool does (shown to the AI)\"},\"value\":{\"$ref\":\"#/components/schemas/ToolValue\"}},\"required\":[\"id\",\"value\"]},\"ToolValue\":{\"description\":\"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference\",\"oneOf\":[{\"$ref\":\"#/components/schemas/FlowModuleTool\"},{\"$ref\":\"#/components/schemas/McpToolValue\"},{\"$ref\":\"#/components/schemas/WebsearchToolValue\"}],\"discriminator\":{\"propertyName\":\"tool_type\",\"mapping\":{\"flowmodule\":\"#/components/schemas/FlowModuleTool\",\"mcp\":\"#/components/schemas/McpToolValue\",\"websearch\":\"#/components/schemas/WebsearchToolValue\"}}},\"FlowModuleTool\":{\"description\":\"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module\",\"allOf\":[{\"type\":\"object\",\"properties\":{\"tool_type\":{\"type\":\"string\",\"enum\":[\"flowmodule\"]}},\"required\":[\"tool_type\"]},{\"$ref\":\"#/components/schemas/FlowModuleValue\"}]},\"WebsearchToolValue\":{\"type\":\"object\",\"description\":\"A tool implemented as a websearch tool. The AI can call this like any other websearch tool\",\"properties\":{\"tool_type\":{\"type\":\"string\",\"enum\":[\"websearch\"]}},\"required\":[\"tool_type\"]},\"McpToolValue\":{\"type\":\"object\",\"description\":\"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers\",\"properties\":{\"tool_type\":{\"type\":\"string\",\"enum\":[\"mcp\"]},\"resource_path\":{\"type\":\"string\",\"description\":\"Path to the MCP resource/server configuration\"},\"include_tools\":{\"type\":\"array\",\"description\":\"Whitelist of specific tools to include from this MCP server\",\"items\":{\"type\":\"string\"}},\"exclude_tools\":{\"type\":\"array\",\"description\":\"Blacklist of tools to exclude from this MCP server\",\"items\":{\"type\":\"string\"}}},\"required\":[\"tool_type\",\"resource_path\"]},\"AiAgent\":{\"type\":\"object\",\"description\":\"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Input parameters for the AI agent mapped to their values\",\"properties\":{\"provider\":{\"$ref\":\"#/components/schemas/ProviderTransform\"},\"output_type\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n\"},\"user_message\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax.\"},\"system_prompt\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"System instructions that guide the AI's behavior, persona, and response style. Optional.\"},\"streaming\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n\"},\"memory\":{\"$ref\":\"#/components/schemas/MemoryTransform\"},\"output_schema\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n\"},\"user_attachments\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n\"},\"max_completion_tokens\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n\"},\"temperature\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n\"},\"max_iterations\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n\"}},\"required\":[\"provider\",\"user_message\",\"output_type\"]},\"tools\":{\"type\":\"array\",\"description\":\"Array of tools the agent can use. The agent decides which tools to call based on the task\",\"items\":{\"$ref\":\"#/components/schemas/AgentTool\"}},\"type\":{\"type\":\"string\",\"enum\":[\"aiagent\"]},\"omit_output_from_conversation\":{\"type\":\"boolean\",\"default\":false,\"description\":\"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled.\"},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, the agent can execute multiple tool calls in parallel\"}},\"required\":[\"tools\",\"type\",\"input_transforms\"]},\"Identity\":{\"type\":\"object\",\"description\":\"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"identity\"]},\"flow\":{\"type\":\"boolean\",\"description\":\"If true, marks this as a flow identity (special handling)\"}},\"required\":[\"type\"]},\"FlowStatus\":{\"type\":\"object\",\"properties\":{\"step\":{\"type\":\"integer\"},\"modules\":{\"type\":\"array\",\"items\":{\"$ref\":\"#/components/schemas/FlowStatusModule\"}},\"user_states\":{\"additionalProperties\":true},\"preprocessor_module\":{\"allOf\":[{\"$ref\":\"#/components/schemas/FlowStatusModule\"}]},\"failure_module\":{\"allOf\":[{\"$ref\":\"#/components/schemas/FlowStatusModule\"},{\"type\":\"object\",\"properties\":{\"parent_module\":{\"type\":\"string\"}}}]},\"retry\":{\"type\":\"object\",\"properties\":{\"fail_count\":{\"type\":\"integer\"},\"failed_jobs\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"format\":\"uuid\"}}}}},\"required\":[\"step\",\"modules\",\"failure_module\"]},\"FlowStatusModule\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"WaitingForPriorSteps\",\"WaitingForEvents\",\"WaitingForExecutor\",\"InProgress\",\"Success\",\"Failure\"]},\"id\":{\"type\":\"string\"},\"job\":{\"type\":\"string\",\"format\":\"uuid\"},\"count\":{\"type\":\"integer\"},\"progress\":{\"type\":\"integer\"},\"iterator\":{\"type\":\"object\",\"properties\":{\"index\":{\"type\":\"integer\"},\"itered\":{\"type\":\"array\",\"items\":{}},\"itered_len\":{\"type\":\"integer\"},\"args\":{}}},\"flow_jobs\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"flow_jobs_success\":{\"type\":\"array\",\"items\":{\"type\":\"boolean\"}},\"flow_jobs_duration\":{\"type\":\"object\",\"properties\":{\"started_at\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"duration_ms\":{\"type\":\"array\",\"items\":{\"type\":\"integer\"}}}},\"branch_chosen\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"branch\",\"default\"]},\"branch\":{\"type\":\"integer\"}},\"required\":[\"type\"]},\"branchall\":{\"type\":\"object\",\"properties\":{\"branch\":{\"type\":\"integer\"},\"len\":{\"type\":\"integer\"}},\"required\":[\"branch\",\"len\"]},\"approvers\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"resume_id\":{\"type\":\"integer\"},\"approver\":{\"type\":\"string\"}},\"required\":[\"resume_id\",\"approver\"]}},\"failed_retries\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"format\":\"uuid\"}},\"skipped\":{\"type\":\"boolean\"},\"agent_actions\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"oneOf\":[{\"type\":\"object\",\"properties\":{\"job_id\":{\"type\":\"string\",\"format\":\"uuid\"},\"function_name\":{\"type\":\"string\"},\"type\":{\"type\":\"string\",\"enum\":[\"tool_call\"]},\"module_id\":{\"type\":\"string\"}},\"required\":[\"job_id\",\"function_name\",\"type\",\"module_id\"]},{\"type\":\"object\",\"properties\":{\"call_id\":{\"type\":\"string\",\"format\":\"uuid\"},\"function_name\":{\"type\":\"string\"},\"resource_path\":{\"type\":\"string\"},\"type\":{\"type\":\"string\",\"enum\":[\"mcp_tool_call\"]},\"arguments\":{\"type\":\"object\"}},\"required\":[\"call_id\",\"function_name\",\"resource_path\",\"type\"]},{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"web_search\"]}},\"required\":[\"type\"]},{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"message\"]}},\"required\":[\"content\",\"type\"]}]}},\"agent_actions_success\":{\"type\":\"array\",\"items\":{\"type\":\"boolean\"}}},\"required\":[\"type\"]}}"; -export declare const CLI_COMMANDS = "# Windmill CLI Commands\n\nThe Windmill CLI (`wmill`) provides commands for managing scripts, flows, apps, and other resources.\n\n## Global Options\n\n- `--workspace ` - Specify the target workspace. This overrides the default workspace.\n- `--debug --verbose` - Show debug/verbose logs\n- `--show-diffs` - Show diff informations when syncing (may show sensitive informations)\n- `--token ` - Specify an API token. This will override any stored token.\n- `--base-url ` - Specify the base URL of the API. If used, --token and --workspace are required and no local remote/workspace already set will be used.\n- `--config-dir ` - Specify a custom config directory. Overrides WMILL_CONFIG_DIR environment variable and default ~/.config location.\n\n## Commands\n\n### app\n\napp related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `app list` - list all apps\n - `--json` - Output as JSON (for piping to jq)\n- `app get ` - get an app's details\n - `--json` - Output as JSON (for piping to jq)\n- `app push [file_path:string] [remote_path:string]` - push a local app. With no args, infers the app from the current directory and the remote path from its location relative to wmill.yaml.\n- `app dev [app_folder:string]` - Start a development server for building apps with live reload and hot module replacement\n - `--port ` - Port to run the dev server on (will find next available port if occupied)\n - `--host ` - Host to bind the dev server to\n - `--entry ` - Entry point file (default: index.ts for Svelte/Vue, index.tsx otherwise)\n - `--no-open` - Don't automatically open the browser\n- `app lint [app_folder:string]` - Lint a raw app folder to validate structure and buildability\n - `--fix` - Attempt to fix common issues (not implemented yet)\n- `app new` - create a new raw app from a template\n - `--summary ` - App summary (short description). Skips the prompt when provided. Triggers non-interactive mode.\n - `--path ` - App path (e.g., f/folder/my_app or u/username/my_app). Skips the prompt when provided. Triggers non-interactive mode.\n - `--framework ` - Framework template: react19 | react18 | svelte5 | vue. Skips the prompt when provided. Triggers non-interactive mode.\n - `--datatable ` - Datatable to wire up. Without this flag in non-interactive mode, no datatable is configured.\n - `--schema ` - Schema to use with --datatable. Created (CREATE SCHEMA IF NOT EXISTS) if it doesn't already exist.\n - `--overwrite` - Overwrite the target directory if it already exists, without prompting.\n - `--no-open-in-desktop` - Do not prompt to open the new app in Claude Desktop.\n- `app generate-agents [app_folder:string]` - regenerate AGENTS.md and DATATABLES.md from remote workspace\n- `app set-permissioned-as ` - Set the on_behalf_of_email for an app (requires admin or wm_deployers group)\n\n### audit\n\nView audit logs (requires admin)\n\n**Subcommands:**\n\n- `audit list` - List audit log entries\n- `audit get ` - Get a specific audit log entry\n - `--json` - Output as JSON (for piping to jq)\n\n### config\n\nShow all available wmill.yaml configuration options\n\n**Options:**\n- `--json` - Output as JSON for programmatic consumption\n\n**Subcommands:**\n\n- `config migrate` - Migrate wmill.yaml from gitBranches/environments to workspaces format\n\n### datatable\n\ndatatable related commands\n\n**Subcommands:**\n\n- `datatable list` - list all datatables in the workspace\n - `--json` - Output as JSON (for piping to jq)\n- `datatable run ` - run a SQL query on a datatable\n - `-n --name ` - Datatable name (default: main)\n - `-s --silent` - Output only the final result as JSON. Useful for scripting.\n- `datatable serve` - Serve all datatables as a Postgres-wire endpoint (psql, DBeaver, pgAdmin); the client picks the datatable via the database name in its connection string\n - `--port ` - Port to listen on (default: first free port in 5433-5500)\n - `--host ` - Bind address (default: 127.0.0.1)\n - `--password ` - Password for Postgres clients (default: generate a random password at startup)\n- `datatable psql` - Start a serve listener and launch psql connected to it\n - `-n --name ` - Datatable to connect psql to (default: main)\n - `--port ` - Port the proxy listens on (default: first free port in 5433-5500)\n - `--host ` - Bind address for the proxy (default: 127.0.0.1)\n - `--password ` - Password for the temporary Postgres proxy (default: generate a random password at startup)\n\n### dependencies\n\nworkspace dependencies related commands\n\n**Alias:** `deps`\n\n**Subcommands:**\n\n- `dependencies push ` - Push workspace dependencies from a local file\n\n### dev\n\nWatch local file changes and live-reload the dev page for preview. Does NOT deploy to the remote workspace \u2014 use wmill sync push for that.\n\n**Options:**\n- `--includes ` - Filter paths given a glob pattern or path\n- `--proxy-port ` - Port for a localhost reverse proxy to the remote Windmill server\n- `--path ` - Watch a specific windmill path (e.g., u/admin/my_script or f/my_flow)\n- `--no-open` - Do not open the browser automatically\n\n### docs\n\nSearch Windmill documentation.\n\n**Arguments:** ``\n\n**Options:**\n- `--json` - Output results as JSON.\n\n### ducklake\n\nducklake related commands\n\n**Subcommands:**\n\n- `ducklake list` - list all ducklakes in the workspace\n - `--json` - Output as JSON (for piping to jq)\n- `ducklake run ` - run a SQL query on a ducklake\n - `-n --name ` - Ducklake name (default: main)\n - `-s --silent` - Output only the final result as JSON. Useful for scripting.\n\n### flow\n\nflow related commands\n\n**Options:**\n- `--show-archived` - Enable archived flows in output\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `flow list` - list all flows\n - `--show-archived` - Enable archived flows in output\n - `--json` - Output as JSON (for piping to jq)\n- `flow get ` - get a flow's details\n - `--json` - Output as JSON (for piping to jq)\n- `flow push ` - push a local flow spec. This overrides any remote versions.\n - `--message ` - Deployment message\n- `flow run ` - run a flow by path.\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not ouput anything other then the final output. Useful for scripting.\n- `flow preview ` - preview a local flow without deploying it. Runs the flow definition from local files and uses local PathScripts by default.\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not output anything other then the final output. Useful for scripting.\n - `--remote` - Use deployed workspace scripts for PathScript steps instead of local files.\n- `flow new ` - create a new empty flow\n - `--summary ` - flow summary\n - `--description ` - flow description\n- `flow bootstrap ` - create a new empty flow (alias for new)\n - `--summary ` - flow summary\n - `--description ` - flow description\n- `flow history ` - Show version history for a flow\n - `--json` - Output as JSON (for piping to jq)\n- `flow show-version ` - Show a specific version of a flow\n - `--json` - Output as JSON (for piping to jq)\n- `flow set-permissioned-as ` - Set the on_behalf_of_email for a flow (requires admin or wm_deployers group)\n\n### folder\n\nfolder related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `folder list` - list all folders\n - `--json` - Output as JSON (for piping to jq)\n- `folder get ` - get a folder's details\n - `--json` - Output as JSON (for piping to jq)\n- `folder new ` - create a new folder locally\n - `--summary ` - folder summary\n- `folder push ` - push a local folder to the remote by name. This overrides any remote versions.\n- `folder add-missing` - create default folder.meta.yaml for all subdirectories of f/ that are missing one\n - `-y, --yes` - skip confirmation prompt\n- `folder show-rules ` - Show default_permissioned_as rules for a folder. Use --test-path to see which rule matches a given item path.\n - `--test-path ` - Test which rule matches this item path (e.g. f/prod/jobs/my_script)\n - `--json` - Output as JSON\n\n### generate-metadata\n\nGenerate metadata (locks, schemas) for all scripts, flows, and apps\n\n**Arguments:** `[folder:string]`\n\n**Options:**\n- `--yes` - Skip confirmation prompt\n- `--dry-run` - Show what would be updated without making changes\n- `--lock-only` - Re-generate only the lock files\n- `--schema-only` - Re-generate only script schemas (skips flows and apps)\n- `--skip-scripts` - Skip processing scripts\n- `--skip-flows` - Skip processing flows\n- `--skip-apps` - Skip processing apps\n- `--strict-folder-boundaries` - Only update items inside the specified folder (requires folder argument)\n- `--parallel ` - Number of items to process in parallel\n- `-i --includes ` - Comma separated patterns to specify which files to include\n- `-e --excludes ` - Comma separated patterns to specify which files to exclude\n\n**Subcommands:**\n\n- `generate-metadata rehash [folder:string]`\n - `--skip-scripts` - Skip processing scripts\n - `--skip-flows` - Skip processing flows\n - `--skip-apps` - Skip processing apps\n - `--parallel ` - Number of items to process in parallel\n - `-i --includes ` - Comma separated patterns to specify which files to include\n - `-e --excludes ` - Comma separated patterns to specify which files to exclude\n\n### gitsync-settings\n\nManage git-sync settings between local wmill.yaml and Windmill backend\n\n**Subcommands:**\n\n- `gitsync-settings pull` - Pull git-sync settings from Windmill backend to local wmill.yaml\n - `--repository ` - Specify repository path (e.g., u/user/repo)\n - `--default` - Write settings to top-level defaults instead of overrides\n - `--replace` - Replace existing settings (non-interactive mode)\n - `--override` - Add branch-specific override (non-interactive mode)\n - `--diff` - Show differences without applying changes\n - `--json-output` - Output in JSON format\n - `--with-backend-settings ` - Use provided JSON settings instead of querying backend (for testing)\n - `--yes` - Skip interactive prompts and use default behavior\n - `--promotion ` - Use promotionOverrides from the specified branch instead of regular overrides\n- `gitsync-settings push` - Push git-sync settings from local wmill.yaml to Windmill backend\n - `--repository ` - Specify repository path (e.g., u/user/repo)\n - `--diff` - Show what would be pushed without applying changes\n - `--json-output` - Output in JSON format\n - `--with-backend-settings ` - Use provided JSON settings instead of querying backend (for testing)\n - `--yes` - Skip interactive prompts and use default behavior\n - `--promotion ` - Use promotionOverrides from the specified branch instead of regular overrides\n\n### group\n\nManage workspace groups\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `group list` - List all groups in the workspace\n - `--json` - Output as JSON (for piping to jq)\n- `group get ` - Get group details and members\n - `--json` - Output as JSON (for piping to jq)\n- `group create ` - Create a new group\n - `--summary ` - Group summary/description\n- `group delete ` - Delete a group\n- `group add-user ` - Add a user to a group\n- `group remove-user ` - Remove a user from a group\n\n### hub\n\nHub related commands. EXPERIMENTAL. INTERNAL USE ONLY.\n\n**Subcommands:**\n\n- `hub pull` - pull any supported definitions. EXPERIMENTAL.\n\n### init\n\nBootstrap a windmill project with a wmill.yaml file\n\n**Options:**\n- `--use-default` - Use default settings without checking backend\n- `--use-backend` - Use backend git-sync settings if available\n- `--repository ` - Specify repository path (e.g., u/user/repo) when using backend settings\n- `--bind-profile` - Automatically bind active workspace profile to current Git branch\n- `--no-bind-profile` - Skip workspace profile binding prompt\n\n### instance\n\nsync local with a remote instance or the opposite (push or pull)\n\n**Subcommands:**\n\n- `instance add [instance_name:string] [remote:string] [token:string]` - Add a new instance\n- `instance remove ` - Remove an instance\n- `instance switch ` - Switch the current instance\n- `instance pull` - Pull instance settings, users, configs, instance groups and overwrite local\n - `--yes` - Pull without needing confirmation\n - `--dry-run` - Perform a dry run without making changes\n - `--skip-users` - Skip pulling users\n - `--skip-settings` - Skip pulling settings\n - `--skip-configs` - Skip pulling configs (worker groups)\n - `--skip-groups` - Skip pulling instance groups\n - `--include-workspaces` - Also pull workspaces\n - `--folder-per-instance` - Create a folder per instance\n - `--instance ` - Name of the instance to pull from, override the active instance\n - `--prefix ` - Prefix of the local workspaces to pull, used to create the folders when using --include-workspaces\n - `--prefix-settings` - Store instance yamls inside prefixed folders when using --prefix and --folder-per-instance\n- `instance push` - Push instance settings, users, configs, group and overwrite remote\n - `--yes` - Push without needing confirmation\n - `--dry-run` - Perform a dry run without making changes\n - `--skip-users` - Skip pushing users\n - `--skip-settings` - Skip pushing settings\n - `--skip-configs` - Skip pushing configs (worker groups)\n - `--skip-groups` - Skip pushing instance groups\n - `--include-workspaces` - Also push workspaces\n - `--folder-per-instance` - Create a folder per instance\n - `--instance ` - Name of the instance to push to, override the active instance\n - `--prefix ` - Prefix of the local workspaces folders to push\n - `--prefix-settings` - Store instance yamls inside prefixed folders when using --prefix and --folder-per-instance\n- `instance whoami` - Display information about the currently logged-in user\n- `instance get-config` - Dump the current instance config (global settings + worker configs) as YAML\n - `-o, --output-file ` - Write YAML to a file instead of stdout\n - `--show-secrets` - Include sensitive fields (license key, JWT secret) without prompting\n - `--instance ` - Name of the instance, override the active instance\n- `instance connect-slack`\n - `--bot-token ` - Slack bot token (xoxb-...)\n - `--team-id ` - Slack team id\n - `--team-name ` - Slack team name\n - `--instance ` - Instance profile to connect against (defaults to the active instance)\n\n### job\n\nManage jobs (list, inspect, cancel)\n\n**Subcommands:**\n\n- `job list` - List recent jobs\n- `job get ` - Get job details. For flows: shows step tree with sub-job IDs\n - `--json` - Output as JSON (for piping to jq)\n- `job result ` - Get the result of a completed job (machine-friendly)\n- `job logs ` - Get job logs. For flows: aggregates all step logs\n- `job cancel ` - Cancel a running or queued job\n - `--reason ` - Reason for cancellation\n- `job rerun ` - Re-run a completed job with the same args. Prints the new job UUID on stdout.\n- `job restart ` - Restart a completed flow at a given top-level step. Prints the new flow job UUID on stdout.\n - `--step ` - Top-level step id to restart the flow from\n - `--iteration ` - For a top-level branchall or for-loop step, the iteration to restart at\n\n### jobs\n\nPull completed and queued jobs from workspace\n\n**Arguments:** `[workspace:string]`\n\n**Options:**\n- `-c, --completed-output ` - Completed jobs output file (default: completed_jobs.json)\n- `-q, --queued-output ` - Queued jobs output file (default: queued_jobs.json)\n- `--skip-worker-check` - Skip checking for active workers before export\n\n**Subcommands:**\n\n- `jobs pull`\n- `jobs push`\n\n### lint\n\nValidate Windmill flow, schedule, and trigger YAML files in a directory\n\n**Arguments:** `[directory:string]`\n\n**Options:**\n- `--json` - Output results in JSON format\n- `--fail-on-warn` - Exit with code 1 when warnings are emitted\n- `--locks-required` - Fail if scripts or flow inline scripts that need locks have no locks\n- `-w, --watch` - Watch for file changes and re-lint automatically\n\n### protection-rules\n\n**Subcommands:**\n\n- `protection-rules pull [workspace:string]` - Pull protection rules from Windmill into protection-rules.yaml for a workspace\n - `--all` - Pull every workspace defined in wmill.yaml\n - `--dry-run` - Show what would change without writing the file\n - `--json-output` - Output in JSON format\n- `protection-rules push [workspace:string]` - Push protection rules from protection-rules.yaml to Windmill for a workspace (full reconcile: creates, updates, and deletes)\n - `--all` - Push every workspace defined in protection-rules.yaml\n - `--dry-run` - Show what would change without applying\n - `--json-output` - Output in JSON format\n - `--yes` - Skip the confirmation prompt (including deletions)\n\n### queues\n\nList all queues with their metrics\n\n**Arguments:** `[workspace:string] the optional workspace to filter by (default to all workspaces)`\n\n**Options:**\n- `--instance [instance]` - Name of the instance to push to, override the active instance\n- `--base-url [baseUrl]` - If used with --token, will be used as the base url for the instance\n\n### refresh\n\nRefresh wmill-managed project files (AGENTS.cli.md and skills)\n\n**Subcommands:**\n\n- `refresh prompts` - Refresh AGENTS.cli.md and managed skills. User-owned AGENTS.md and CLAUDE.md are never overwritten unless you opt in.\n - `--yes` - Non-interactive: skip the migration prompt for existing AGENTS.md / CLAUDE.md without the expected include; defaults to appending the include.\n\n### resource\n\nresource related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `resource list` - list all resources\n - `--json` - Output as JSON (for piping to jq)\n- `resource get ` - get a resource's details\n - `--json` - Output as JSON (for piping to jq)\n- `resource new ` - create a new resource locally\n- `resource push ` - push a local resource spec. This overrides any remote versions.\n\n### resource-type\n\nresource type related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `resource-type list` - list all resource types\n - `--schema` - Show schema in the output\n - `--json` - Output as JSON (for piping to jq)\n- `resource-type get ` - get a resource type's details\n - `--json` - Output as JSON (for piping to jq)\n- `resource-type new ` - create a new resource type locally\n- `resource-type push ` - push a local resource spec. This overrides any remote versions.\n- `resource-type generate-namespace` - Create a TypeScript definition file with the RT namespace generated from the resource types\n\n### schedule\n\nschedule related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `schedule list` - list all schedules\n - `--json` - Output as JSON (for piping to jq)\n- `schedule get ` - get a schedule's details\n - `--json` - Output as JSON (for piping to jq)\n- `schedule new ` - create a new schedule locally\n- `schedule push ` - push a local schedule spec. This overrides any remote versions.\n- `schedule enable ` - Enable a schedule\n - `--force` - Bypass the fork-conflict warning when the parent workspace has the same schedule (acknowledges that both crons will fire)\n- `schedule disable ` - Disable a schedule\n- `schedule set-permissioned-as ` - Set the email (run-as user) for a schedule (requires admin or wm_deployers group)\n\n### script\n\nscript related commands\n\n**Options:**\n- `--show-archived` - Show archived scripts instead of active ones\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `script list` - list all scripts\n - `--show-archived` - Show archived scripts instead of active ones\n - `--json` - Output as JSON (for piping to jq)\n- `script push ` - push a local script spec. This overrides any remote versions. Use the script file (.ts, .js, .py, .sh)\n - `--message ` - Deployment message\n- `script get ` - get a script's details\n - `--json` - Output as JSON (for piping to jq)\n- `script show ` - show a script's content (alias for get)\n- `script run ` - run a script by path\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not output anything other then the final output. Useful for scripting.\n- `script preview ` - preview a local script without deploying it. Supports both regular and codebase scripts.\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not output anything other than the final output. Useful for scripting.\n- `script new ` - create a new script\n - `--summary ` - script summary\n - `--description ` - script description\n- `script bootstrap ` - create a new script (alias for new)\n - `--summary ` - script summary\n - `--description ` - script description\n- `script set-permissioned-as ` - Set the on_behalf_of_email for a script (requires admin or wm_deployers group)\n- `script history ` - show version history for a script\n - `--json` - Output as JSON (for piping to jq)\n\n### sync\n\nsync local with a remote workspaces or the opposite (push or pull)\n\n**Subcommands:**\n\n- `sync pull` - Pull any remote changes and apply them locally.\n - `--yes` - Pull without needing confirmation\n - `--dry-run` - Show changes that would be pulled without actually pushing\n - `--plain-secrets` - Pull secrets as plain text\n - `--json` - Use JSON instead of YAML\n - `--skip-variables` - Skip syncing variables (including secrets)\n - `--skip-secrets` - Skip syncing only secrets variables\n - `--include-secrets` - Include secrets in sync (overrides skipSecrets in wmill.yaml)\n - `--skip-resources` - Skip syncing resources\n - `--skip-resource-types` - Skip syncing resource types\n - `--skip-scripts` - Skip syncing scripts\n - `--skip-flows` - Skip syncing flows\n - `--skip-apps` - Skip syncing apps\n - `--skip-folders` - Skip syncing folders\n - `--skip-workspace-dependencies` - Skip syncing workspace dependencies\n - `--skip-scripts-metadata` - Skip syncing scripts metadata, focus solely on logic\n - `--include-schedules` - Include syncing schedules\n - `--include-triggers` - Include syncing triggers\n - `--include-users` - Include syncing users\n - `--include-groups` - Include syncing groups\n - `--include-settings` - Include syncing workspace settings\n - `--include-key` - Include workspace encryption key\n - `--skip-branch-validation` - Skip git branch validation and prompts\n - `--json-output` - Output results in JSON format\n - `-i --includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string). Overrides wmill.yaml includes\n - `-e --excludes ` - Comma separated patterns to specify which file to NOT take into account. Overrides wmill.yaml excludes\n - `--extra-includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string). Useful to still take wmill.yaml into account and act as a second pattern to satisfy\n - `--repository ` - Specify repository path (e.g., u/user/repo) when multiple repositories exist\n - `--promotion ` - Use promotionOverrides from the specified branch instead of regular overrides\n - `--branch, --env ` - [Deprecated: use --workspace] Override the current git branch/environment\n- `sync push` - Push any local changes and apply them remotely.\n - `--yes` - Push without needing confirmation\n - `--dry-run` - Show changes that would be pushed without actually pushing\n - `--plain-secrets` - Push secrets as plain text\n - `--json` - Use JSON instead of YAML\n - `--skip-variables` - Skip syncing variables (including secrets)\n - `--skip-secrets` - Skip syncing only secrets variables\n - `--include-secrets` - Include secrets in sync (overrides skipSecrets in wmill.yaml)\n - `--skip-resources` - Skip syncing resources\n - `--skip-resource-types` - Skip syncing resource types\n - `--skip-scripts` - Skip syncing scripts\n - `--skip-flows` - Skip syncing flows\n - `--skip-apps` - Skip syncing apps\n - `--skip-folders` - Skip syncing folders\n - `--skip-workspace-dependencies` - Skip syncing workspace dependencies\n - `--skip-scripts-metadata` - Skip syncing scripts metadata, focus solely on logic\n - `--include-schedules` - Include syncing schedules\n - `--include-triggers` - Include syncing triggers\n - `--include-users` - Include syncing users\n - `--include-groups` - Include syncing groups\n - `--include-settings` - Include syncing workspace settings\n - `--include-key` - Include workspace encryption key\n - `--skip-branch-validation` - Skip git branch validation and prompts\n - `--json-output` - Output results in JSON format\n - `-i --includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string)\n - `-e --excludes ` - Comma separated patterns to specify which file to NOT take into account.\n - `--extra-includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string). Useful to still take wmill.yaml into account and act as a second pattern to satisfy\n - `--message ` - Include a message that will be added to all scripts/flows/apps updated during this push\n - `--parallel ` - Number of changes to process in parallel\n - `--repository ` - Specify repository path (e.g., u/user/repo) when multiple repositories exist\n - `--branch, --env ` - [Deprecated: use --workspace] Override the current git branch/environment\n - `--lint` - Run lint validation before pushing\n - `--locks-required` - Fail if scripts or flow inline scripts that need locks have no locks\n - `--auto-metadata` - Automatically regenerate stale metadata (locks and schemas) before pushing\n - `--accept-overriding-permissioned-as-with-self` - Accept that items with a different permissioned_as will be updated with your own user\n\n### token\n\nManage API tokens\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `token list` - List API tokens\n - `--json` - Output as JSON (for piping to jq)\n- `token create` - Create a new API token\n - `--label ` - Token label\n - `--expiration ` - Token expiration (ISO 8601 timestamp)\n- `token delete ` - Delete a token by its prefix\n\n### trigger\n\ntrigger related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `trigger list` - list all triggers\n - `--json` - Output as JSON (for piping to jq)\n- `trigger get ` - get a trigger's details\n - `--json` - Output as JSON (for piping to jq)\n - `--kind ` - Trigger kind (http, websocket, kafka, nats, postgres, mqtt, sqs, gcp, azure, email). Recommended for faster lookup\n- `trigger new ` - create a new trigger locally\n - `--kind ` - Trigger kind (required: http, websocket, kafka, nats, postgres, mqtt, sqs, gcp, azure, email)\n- `trigger push ` - push a local trigger spec. This overrides any remote versions.\n- `trigger set-permissioned-as ` - Set the email (run-as user) for a trigger (requires admin or wm_deployers group)\n - `--kind ` - Trigger kind (required: http, websocket, kafka, nats, postgres, mqtt, sqs, gcp, azure, email)\n\n### user\n\nuser related commands\n\n**Subcommands:**\n\n- `user add [password:string]` - Create a user\n - `--superadmin` - Specify to make the new user superadmin.\n - `--company ` - Specify to set the company of the new user.\n - `--name ` - Specify to set the name of the new user.\n- `user remove ` - Delete a user\n- `user create-token` - Create a new API token for the authenticated user\n - `--email ` - Specify credentials to use for authentication. This will not be stored. It will only be used to exchange for a token with the API server, which will not be stored either.\n - `--password ` - Specify credentials to use for authentication. This will not be stored. It will only be used to exchange for a token with the API server, which will not be stored either.\n\n### variable\n\nvariable related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `variable list` - list all variables\n - `--json` - Output as JSON (for piping to jq)\n- `variable get ` - get a variable's details\n - `--json` - Output as JSON (for piping to jq)\n- `variable new ` - create a new variable locally\n- `variable push ` - Push a local variable spec. This overrides any remote versions.\n - `--plain-secrets` - Push secrets as plain text\n- `variable add ` - Create a new variable on the remote. This will update the variable if it already exists.\n - `--plain-secrets` - Push secrets as plain text\n - `--public` - Legacy option, use --plain-secrets instead\n\n### version\n\nShow version information\n\n### worker-groups\n\ndisplay worker groups, pull and push worker groups configs\n\n**Subcommands:**\n\n- `worker-groups pull` - Pull worker groups (similar to `wmill instance pull --skip-users --skip-settings --skip-groups`)\n - `--instance` - Name of the instance to push to, override the active instance\n - `--base-url` - Base url to be passed to the instance settings instead of the local one\n - `--yes` - Pull without needing confirmation\n- `worker-groups push` - Push worker groups (similar to `wmill instance push --skip-users --skip-settings --skip-groups`)\n - `--instance [instance]` - Name of the instance to push to, override the active instance\n - `--base-url [baseUrl]` - If used with --token, will be used as the base url for the instance\n - `--yes` - Push without needing confirmation\n\n### workers\n\nList all workers grouped by worker groups\n\n**Options:**\n- `--instance [instance]` - Name of the instance to push to, override the active instance\n- `--base-url [baseUrl]` - If used with --token, will be used as the base url for the instance\n\n### workspace\n\nworkspace related commands\n\n**Alias:** `profile`\n\n**Subcommands:**\n\n- `workspace switch ` - Switch to another workspace\n- `workspace add [workspace_name:string] [workspace_id:string] [remote:string]` - Add a workspace\n - `-c --create` - Create the workspace if it does not exist\n - `--create-workspace-name ` - Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id.\n - `--create-username ` - Specify your own username in the newly created workspace. Ignored if --create is not specified, the workspace already exists or automatic username creation is enabled on the instance.\n- `workspace remove ` - Remove a workspace\n- `workspace whoami` - Show the currently active user\n- `workspace list` - List local workspace profiles\n- `workspace list-remote` - List workspaces on the remote server that you have access to\n - `--as-superadmin` - List ALL workspaces on the instance (requires the token to belong to a superadmin/devops user)\n- `workspace list-forks` - List forked workspaces on the remote server\n- `workspace bind` - Create or update a workspace entry in wmill.yaml from the active profile\n - `--workspace ` - Workspace name (default: current branch or workspaceId)\n - `--branch ` - Git branch to associate (default: workspace name)\n- `workspace unbind` - Remove baseUrl and workspaceId from a workspace entry\n - `--workspace ` - Workspace to unbind\n- `workspace fork [workspace_name:string] [workspace_id:string]` - Create a forked workspace\n - `--create-workspace-name ` - Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id.\n - `--color ` - Workspace color (hex code, e.g. #ff0000)\n - `--datatable-behavior ` - How to handle datatables: skip, schema_only, or schema_and_data (default: interactive prompt)\n - `-y --yes` - Skip interactive prompts (defaults datatable behavior to 'skip')\n- `workspace delete-fork ` - Delete a forked workspace and git branch\n - `-y --yes` - Skip confirmation prompt\n- `workspace merge` - Compare and deploy changes between a fork and its parent workspace\n - `--direction ` - Deploy direction: to-parent or to-fork\n - `--all` - Deploy all changed items including conflicts\n - `--skip-conflicts` - Skip items modified in both workspaces\n - `--include ` - Comma-separated kind:path items to include (e.g. script:f/test/main,flow:f/my/flow)\n - `--exclude ` - Comma-separated kind:path items to exclude\n - `--preserve-on-behalf-of` - Preserve original on_behalf_of/permissioned_as values\n - `-y --yes` - Non-interactive mode (deploy without prompts)\n- `workspace connect-slack` - Non-interactively connect Slack to the active workspace using a pre-minted bot token (xoxb-...). Produces the same artifacts as the UI OAuth flow: workspace_settings fields, g/slack group, f/slack_bot folder, and the encrypted bot token variable + resource at f/slack_bot/bot_token.\n - `--bot-token ` - Slack bot token (xoxb-...)\n - `--team-id ` - Slack team id\n - `--team-name ` - Slack team name\n- `workspace disconnect-slack`\n\n"; -export declare const LANG_BASH = "# Bash\n\n## Structure\n\nDo not include `#!/bin/bash`. Arguments are obtained as positional parameters:\n\n```bash\n# Get arguments\nvar1=\"$1\"\nvar2=\"$2\"\n\necho \"Processing $var1 and $var2\"\n\n# Return JSON by echoing to stdout\necho \"{\\\"result\\\": \\\"$var1\\\", \\\"count\\\": $var2}\"\n```\n\n**Important:**\n- Do not include shebang (`#!/bin/bash`)\n- Arguments are always strings\n- Access with `$1`, `$2`, etc.\n\n## Output\n\nThe script output is captured as the result. For structured data, output valid JSON:\n\n```bash\nname=\"$1\"\ncount=\"$2\"\n\n# Output JSON result\ncat << EOF\n{\n \"name\": \"$name\",\n \"count\": $count,\n \"timestamp\": \"$(date -Iseconds)\"\n}\nEOF\n```\n\n## Environment Variables\n\nEnvironment variables set in Windmill are available:\n\n```bash\n# Access environment variable\necho \"Workspace: $WM_WORKSPACE\"\necho \"Job ID: $WM_JOB_ID\"\n```\n"; -export declare const LANG_BIGQUERY = "# BigQuery\n\nArguments use `@name` syntax.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- @name1 (string)\n-- @name2 (int64) = 0\nSELECT * FROM users WHERE name = @name1 AND age > @name2;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as a `STRING` JSON parameter \u2014 Parquet/CSV\nfiles are decoded server-side into a JSON array of records, JSON/JSONL pass\nthrough. Consume with `JSON_EXTRACT_ARRAY` / `JSON_VALUE`:\n\n```sql\n-- @file (s3object)\nSELECT\n CAST(JSON_VALUE(row, '$.id') AS INT64) AS id,\n JSON_VALUE(row, '$.name') AS name\nFROM UNNEST(JSON_EXTRACT_ARRAY(@file)) AS row;\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered, bypassing the 10000-row return cap.\n"; -export declare const LANG_BUN = "# TypeScript (Bun)\n\nBun runtime with full npm ecosystem and fastest execution.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function. Libraries are installed automatically.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n```typescript\nimport Stripe from \"stripe\";\nimport { someFunction } from \"some-package\";\n```\n\n## Windmill Client\n\nImport the windmill client for platform interactions:\n\n```typescript\nimport * as wmill from \"windmill-client\";\n```\n\nSee the SDK documentation for available methods.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id,\n };\n}\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form \u2014 always use it instead of redefining your own.\n\n### Receiving an S3Object as a script parameter\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\nexport async function main(file: wmill.S3Object) {\n const content = await wmill.loadS3File(file);\n // ...\n}\n```\n\n### S3 operations\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\n// Load file content from S3\nconst content: Uint8Array = await wmill.loadS3File(s3object);\n\n// Load file as stream\nconst blob: Blob = await wmill.loadS3FileStream(s3object);\n\n// Write file to S3\nconst result: wmill.S3Object = await wmill.writeS3File(\n s3object, // Target path (or undefined to auto-generate)\n fileContent, // string or Blob\n s3ResourcePath // Optional: specific S3 resource to use\n);\n```\n"; -export declare const LANG_BUNNATIVE = "# TypeScript (Bun Native)\n\nNative TypeScript execution with fetch only - no external imports allowed.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n**No imports allowed.** Use the globally available `fetch` function:\n\n```typescript\nexport async function main(url: string) {\n const response = await fetch(url);\n return await response.json();\n}\n```\n\n## Windmill Client\n\nThe windmill client is not available in native TypeScript mode. Use fetch to call APIs directly.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id,\n };\n}\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form \u2014 always use it instead of redefining your own.\n\n### Receiving an S3Object as a script parameter\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\nexport async function main(file: wmill.S3Object) {\n const content = await wmill.loadS3File(file);\n // ...\n}\n```\n\n### S3 operations\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\n// Load file content from S3\nconst content: Uint8Array = await wmill.loadS3File(s3object);\n\n// Load file as stream\nconst blob: Blob = await wmill.loadS3FileStream(s3object);\n\n// Write file to S3\nconst result: wmill.S3Object = await wmill.writeS3File(\n s3object, // Target path (or undefined to auto-generate)\n fileContent, // string or Blob\n s3ResourcePath // Optional: specific S3 resource to use\n);\n```\n"; -export declare const LANG_CSHARP = "# C#\n\nThe script must contain a public static `Main` method inside a class:\n\n```csharp\npublic class Script\n{\n public static object Main(string name, int count)\n {\n return new { Name = name, Count = count };\n }\n}\n```\n\n**Important:**\n- Class name is irrelevant\n- Method must be `public static`\n- Return type can be `object` or specific type\n\n## NuGet Packages\n\nAdd packages using the `#r` directive at the top:\n\n```csharp\n#r \"nuget: Newtonsoft.Json, 13.0.3\"\n#r \"nuget: RestSharp, 110.2.0\"\n\nusing Newtonsoft.Json;\nusing RestSharp;\n\npublic class Script\n{\n public static object Main(string url)\n {\n var client = new RestClient(url);\n var request = new RestRequest();\n var response = client.Get(request);\n return JsonConvert.DeserializeObject(response.Content);\n }\n}\n```\n"; -export declare const LANG_DENO = "# TypeScript (Deno)\n\nDeno runtime with npm support via `npm:` prefix and native Deno libraries.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function. Libraries are installed automatically.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n```typescript\n// npm packages use npm: prefix\nimport Stripe from \"npm:stripe\";\nimport { someFunction } from \"npm:some-package\";\n\n// Deno standard library\nimport { serve } from \"https://deno.land/std/http/server.ts\";\n```\n\n## Windmill Client\n\nImport the windmill client for platform interactions:\n\n```typescript\nimport * as wmill from \"windmill-client\";\n```\n\nSee the SDK documentation for available methods.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id,\n };\n}\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form \u2014 always use it instead of redefining your own.\n\n### Receiving an S3Object as a script parameter\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\nexport async function main(file: wmill.S3Object) {\n const content = await wmill.loadS3File(file);\n // ...\n}\n```\n\n### S3 operations\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\n// Load file content from S3\nconst content: Uint8Array = await wmill.loadS3File(s3object);\n\n// Load file as stream\nconst blob: Blob = await wmill.loadS3FileStream(s3object);\n\n// Write file to S3\nconst result: wmill.S3Object = await wmill.writeS3File(\n s3object, // Target path (or undefined to auto-generate)\n fileContent, // string or Blob\n s3ResourcePath // Optional: specific S3 resource to use\n);\n```\n"; -export declare const LANG_DUCKDB = "# DuckDB\n\nArguments are defined with comments and used with `$name` syntax:\n\n```sql\n-- $name (text) = default\n-- $age (integer)\nSELECT * FROM users WHERE name = $name AND age > $age;\n```\n\n## Ducklake Integration\n\nAttach Ducklake for data lake operations:\n\n```sql\n-- Main ducklake\nATTACH 'ducklake' AS dl;\n\n-- Named ducklake\nATTACH 'ducklake://my_lake' AS dl;\n\n-- Then query\nSELECT * FROM dl.schema.table;\n```\n\n## External Database Connections\n\nConnect to external databases using resources:\n\n```sql\nATTACH '$res:path/to/resource' AS db (TYPE postgres);\nSELECT * FROM db.schema.table;\n```\n\n## S3 File Operations\n\nRead files from S3 storage:\n\n```sql\n-- Default storage\nSELECT * FROM read_csv('s3:///path/to/file.csv');\n\n-- Named storage\nSELECT * FROM read_csv('s3://storage_name/path/to/file.csv');\n\n-- Parquet files\nSELECT * FROM read_parquet('s3:///path/to/file.parquet');\n\n-- JSON files\nSELECT * FROM read_json('s3:///path/to/file.json');\n```\n\n### Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for it\nand binds the arg as the bare `s3://storage/key` URI, which DuckDB's reader\nfunctions consume directly:\n\n```sql\n-- $file (s3object)\nSELECT * FROM read_parquet($file);\n```\n\nWorks with any DuckDB reader: `read_csv($file)`, `read_json($file)`, etc.\n\n### Writing query results to S3\n\nDuckDB writes to S3 natively via `COPY ... TO`:\n\n```sql\nCOPY (SELECT * FROM users) TO 's3:///exports/users.parquet' (FORMAT PARQUET);\n```\n\nUse this instead of the `-- s3` streaming directive supported by the other SQL\ndialects \u2014 that directive is not available in DuckDB.\n"; -export declare const LANG_GO = "# Go\n\n## Structure\n\nThe file package must be `inner` and export a function called `main`:\n\n```go\npackage inner\n\nfunc main(param1 string, param2 int) (map[string]interface{}, error) {\n return map[string]interface{}{\n \"result\": param1,\n \"count\": param2,\n }, nil\n}\n```\n\n**Important:**\n- Package must be `inner`\n- Return type must be `({return_type}, error)`\n- Function name is `main` (lowercase)\n\n## Return Types\n\nThe return type can be any Go type that can be serialized to JSON:\n\n```go\npackage inner\n\ntype Result struct {\n Name string `json:\"name\"`\n Count int `json:\"count\"`\n}\n\nfunc main(name string, count int) (Result, error) {\n return Result{\n Name: name,\n Count: count,\n }, nil\n}\n```\n\n## Error Handling\n\nReturn errors as the second return value:\n\n```go\npackage inner\n\nimport \"errors\"\n\nfunc main(value int) (string, error) {\n if value < 0 {\n return \"\", errors.New(\"value must be positive\")\n }\n return \"success\", nil\n}\n```\n"; -export declare const LANG_GRAPHQL = "# GraphQL\n\n## Structure\n\nWrite GraphQL queries or mutations. Arguments can be added as query parameters:\n\n```graphql\nquery GetUser($id: ID!) {\n user(id: $id) {\n id\n name\n email\n }\n}\n```\n\n## Variables\n\nVariables are passed as script arguments and automatically bound to the query:\n\n```graphql\nquery SearchProducts($query: String!, $limit: Int = 10) {\n products(search: $query, first: $limit) {\n edges {\n node {\n id\n name\n price\n }\n }\n }\n}\n```\n\n## Mutations\n\n```graphql\nmutation CreateUser($input: CreateUserInput!) {\n createUser(input: $input) {\n id\n name\n createdAt\n }\n}\n```\n"; -export declare const LANG_JAVA = "# Java\n\nThe script must contain a Main public class with a `public static main()` method:\n\n```java\npublic class Main {\n public static Object main(String name, int count) {\n java.util.Map result = new java.util.HashMap<>();\n result.put(\"name\", name);\n result.put(\"count\", count);\n return result;\n }\n}\n```\n\n**Important:**\n- Class must be named `Main`\n- Method must be `public static Object main(...)`\n- Return type is `Object` or `void`\n\n## Maven Dependencies\n\nAdd dependencies using comments at the top:\n\n```java\n//requirements:\n//com.google.code.gson:gson:2.10.1\n//org.apache.httpcomponents:httpclient:4.5.14\n\nimport com.google.gson.Gson;\n\npublic class Main {\n public static Object main(String input) {\n Gson gson = new Gson();\n return gson.fromJson(input, Object.class);\n }\n}\n```\n"; -export declare const LANG_MSSQL = "# Microsoft SQL Server (MSSQL)\n\nArguments use `@P1`, `@P2`, etc.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- @P1 name1 (varchar)\n-- @P2 name2 (int) = 0\nSELECT * FROM users WHERE name = @P1 AND age > @P2;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as `nvarchar(max)` JSON text \u2014 Parquet/CSV\nfiles are decoded server-side into a JSON array of records, JSON/JSONL pass\nthrough. Consume with `OPENJSON`:\n\n```sql\n-- @P1 file (s3object)\nSELECT id, name\nFROM OPENJSON(@P1)\nWITH (id INT, name NVARCHAR(200));\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered as the script return value.\n"; -export declare const LANG_MYSQL = "# MySQL\n\nArguments use `?` placeholders.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- ? name1 (text)\n-- ? name2 (int) = 0\nSELECT * FROM users WHERE name = ? AND age > ?;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as JSON text \u2014 Parquet/CSV files are\ndecoded server-side into a JSON array of records, JSON/JSONL pass through.\nConsume with `JSON_TABLE`:\n\n```sql\n-- ? file (s3object)\nSELECT id, name\nFROM JSON_TABLE(?, '$[*]'\n COLUMNS (id INT PATH '$.id', name VARCHAR(200) PATH '$.name')\n) AS r;\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered as the script return value.\n"; -export declare const LANG_NATIVETS = "# TypeScript (Native)\n\nNative TypeScript execution with fetch only - no external imports allowed.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n**No imports allowed.** Use the globally available `fetch` function:\n\n```typescript\nexport async function main(url: string) {\n const response = await fetch(url);\n return await response.json();\n}\n```\n\n## Windmill Client\n\nThe windmill client is not available in native TypeScript mode. Use fetch to call APIs directly.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id\n };\n}\n```\n"; -export declare const LANG_PHP = "# PHP\n\n## Structure\n\nThe script must start with ` $param1, \"count\" => $param2];\n}\n```\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nYou need to **redefine** the type of the resources that are needed before the main function. Always check if the class already exists using `class_exists`:\n\n```php\n $2::INT;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as a `jsonb` parameter \u2014 Parquet/CSV files\nare decoded server-side into a JSON array of records, JSON/JSONL pass through.\nConsume with `jsonb_to_recordset` (or any `jsonb` API):\n\n```sql\n-- $1 file (s3object)\nSELECT *\nFROM jsonb_to_recordset($1::jsonb) AS r(id INT, name TEXT);\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered as the script return value.\n"; -export declare const LANG_POWERSHELL = "# PowerShell\n\n## Structure\n\nArguments are obtained by calling the `param` function on the first line:\n\n```powershell\nparam($Name, $Count = 0, [int]$Age)\n\n# Your code here\nWrite-Output \"Processing $Name, count: $Count, age: $Age\"\n\n# Return object\n@{\n name = $Name\n count = $Count\n age = $Age\n}\n```\n\n## Parameter Types\n\nYou can specify types for parameters:\n\n```powershell\nparam(\n [string]$Name,\n [int]$Count = 0,\n [bool]$Enabled = $true,\n [array]$Items\n)\n\n@{\n name = $Name\n count = $Count\n enabled = $Enabled\n items = $Items\n}\n```\n\n## Return Values\n\nReturn values by outputting them at the end of the script:\n\n```powershell\nparam($Input)\n\n$result = @{\n processed = $true\n data = $Input\n timestamp = Get-Date -Format \"o\"\n}\n\n$result\n```\n"; -export declare const LANG_PYTHON3 = "# Python\n\n## Structure\n\nThe script must contain at least one function called `main`:\n\n```python\ndef main(param1: str, param2: int):\n # Your code here\n return {\"result\": param1, \"count\": param2}\n```\n\nDo not call the main function. Libraries are installed automatically.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nYou need to **redefine** the type of the resources that are needed before the main function as TypedDict:\n\n```python\nfrom typing import TypedDict\n\nclass postgresql(TypedDict):\n host: str\n port: int\n user: str\n password: str\n dbname: str\n\ndef main(db: postgresql):\n # db contains the database connection details\n pass\n```\n\n**Important rules:**\n\n- The resource type name must be **IN LOWERCASE**\n- Only include resource types if they are actually needed\n- If an import conflicts with a resource type name, **rename the imported object, not the type name**\n- Make sure to import TypedDict from typing **if you're using it**\n\n## Imports\n\nLibraries are installed automatically. Do not show installation instructions.\n\n```python\nimport requests\nimport pandas as pd\nfrom datetime import datetime\n```\n\nIf an import name conflicts with a resource type:\n\n```python\n# Wrong - don't rename the type\nimport stripe as stripe_lib\nclass stripe_type(TypedDict): ...\n\n# Correct - rename the import\nimport stripe as stripe_sdk\nclass stripe(TypedDict):\n api_key: str\n```\n\n## Windmill Client\n\nImport the windmill client for platform interactions:\n\n```python\nimport wmill\n```\n\nSee the SDK documentation for available methods.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```python\nfrom typing import TypedDict, Literal, Any\n\nclass Event(TypedDict):\n kind: Literal[\"webhook\", \"http\", \"websocket\", \"kafka\", \"email\", \"nats\", \"postgres\", \"sqs\", \"mqtt\", \"gcp\"]\n body: Any\n headers: dict[str, str]\n query: dict[str, str]\n\ndef preprocessor(event: Event):\n # Transform the event into flow input parameters\n return {\n \"param1\": event[\"body\"][\"field1\"],\n \"param2\": event[\"query\"][\"id\"]\n }\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations.\n\n### Receiving an S3Object as a script parameter\n\nTo accept a file from S3 as input to a script, type the parameter with `S3Object` (imported from `wmill`):\n\n```python\nimport wmill\nfrom wmill import S3Object\n\ndef main(file: S3Object):\n content = wmill.load_s3_file(file)\n # ...\n```\n\n### S3 operations\n\n```python\nimport wmill\n\n# Load file content from S3\ncontent: bytes = wmill.load_s3_file(s3object)\n\n# Load file as stream reader\nreader: BufferedReader = wmill.load_s3_file_reader(s3object)\n\n# Write file to S3\nresult: S3Object = wmill.write_s3_file(\n s3object, # Target path (or None to auto-generate)\n file_content, # bytes or BufferedReader\n s3_resource_path, # Optional: specific S3 resource\n content_type, # Optional: MIME type\n content_disposition # Optional: Content-Disposition header\n)\n```\n"; -export declare const LANG_RLANG = "# R\n\n## Structure\n\nDefine a `main` function using `<-` or `=` assignment. Parameters become the script inputs:\n\n```r\nlibrary(dplyr)\nlibrary(jsonlite)\n\nmain <- function(x, name = \"default\", flag = TRUE) {\n df <- tibble(x = x, name = name)\n result <- df %>% mutate(greeting = paste(\"Hello\", name))\n return(toJSON(result, auto_unbox = TRUE))\n}\n```\n\n**Important:**\n- The `main` function is required\n- Use `library()` to load packages \u2014 they are resolved and installed automatically\n- `jsonlite` is always available (used internally for argument parsing)\n- Return values must be JSON-serializable\n\n## Parameters\n\nR types map to Windmill types:\n- `numeric` \u2192 float/int\n- `character` \u2192 string\n- `logical` \u2192 bool (use `TRUE`/`FALSE`)\n- `list` \u2192 object/dict\n- `NULL` \u2192 null\n\nDefault values are inferred from the function signature:\n\n```r\nmain <- function(\n name, # required string\n count = 10, # optional int, default 10\n verbose = FALSE # optional bool, default FALSE\n) {\n # ...\n}\n```\n\n## Resources and Variables\n\nUse the built-in Windmill helpers (no import needed):\n\n```r\nmain <- function() {\n # Get a variable\n api_key <- get_variable(\"f/my_folder/api_key\")\n\n # Get a resource (returns a list)\n db <- get_resource(\"f/my_folder/postgres_config\")\n host <- db$host\n port <- db$port\n\n return(list(host = host, port = port))\n}\n```\n\n## Output\n\nReturn any JSON-serializable value from `main`. The return value becomes the step result:\n\n```r\nmain <- function(x) {\n # Return a scalar\n return(x + 1)\n\n # Or a list (becomes JSON object)\n return(list(result = x + 1, status = \"ok\"))\n}\n```\n\n## Annotations\n\nControl execution behavior with comment annotations:\n\n```r\n#renv_verbose = true # Show verbose renv output during resolution\n#renv_install_verbose = true # Show verbose output during package installation\n#sandbox = true # Run in nsjail sandbox (requires nsjail)\n```\n"; -export declare const LANG_RUST = "# Rust\n\n## Structure\n\nThe script must contain a function called `main` with proper return type:\n\n```rust\nuse anyhow::anyhow;\nuse serde::Serialize;\n\n#[derive(Serialize, Debug)]\nstruct ReturnType {\n result: String,\n count: i32,\n}\n\nfn main(param1: String, param2: i32) -> anyhow::Result {\n Ok(ReturnType {\n result: param1,\n count: param2,\n })\n}\n```\n\n**Important:**\n- Arguments should be owned types\n- Return type must be serializable (`#[derive(Serialize)]`)\n- Return type is `anyhow::Result`\n\n## Dependencies\n\nPackages must be specified with a partial cargo.toml at the beginning of the script:\n\n```rust\n//! ```cargo\n//! [dependencies]\n//! anyhow = \"1.0.86\"\n//! reqwest = { version = \"0.11\", features = [\"json\"] }\n//! tokio = { version = \"1\", features = [\"full\"] }\n//! ```\n\nuse anyhow::anyhow;\n// ... rest of the code\n```\n\n**Note:** Serde is already included, no need to add it again.\n\n## Async Functions\n\nIf you need to handle async functions (e.g., using tokio), keep the main function sync and create the runtime inside:\n\n```rust\n//! ```cargo\n//! [dependencies]\n//! anyhow = \"1.0.86\"\n//! tokio = { version = \"1\", features = [\"full\"] }\n//! reqwest = { version = \"0.11\", features = [\"json\"] }\n//! ```\n\nuse anyhow::anyhow;\nuse serde::Serialize;\n\n#[derive(Serialize, Debug)]\nstruct Response {\n data: String,\n}\n\nfn main(url: String) -> anyhow::Result {\n let rt = tokio::runtime::Runtime::new()?;\n rt.block_on(async {\n let resp = reqwest::get(&url).await?.text().await?;\n Ok(Response { data: resp })\n })\n}\n```\n"; -export declare const LANG_SNOWFLAKE = "# Snowflake\n\nArguments use `?` placeholders.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- ? name1 (text)\n-- ? name2 (number) = 0\nSELECT * FROM users WHERE name = ? AND age > ?;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as JSON text \u2014 Parquet/CSV files are\ndecoded server-side into a JSON array of records, JSON/JSONL pass through.\nWrap the bind with `PARSE_JSON(?)` and walk it with `LATERAL FLATTEN`:\n\n```sql\n-- ? file (s3object)\nSELECT\n v.value:id::NUMBER AS id,\n v.value:name::STRING AS name\nFROM LATERAL FLATTEN(input => PARSE_JSON(?)) v;\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered, bypassing the 10000-row return cap.\n"; +// Auto-generated by generate.py - DO NOT EDIT + +export declare const SCRIPT_BASE: string; +export declare const FLOW_BASE: string; +export declare const RESOURCES_BASE: string; +export declare const RAW_APP_BASE: string; +export declare const PIPELINE_BASE: string; +export declare const WORKFLOW_AS_CODE_BASE: string; +export declare const FLOW_CHAT_SPECIAL_MODULES: string; +export declare const SDK_TYPESCRIPT: string; +export declare const SDK_PYTHON: string; +export declare const WAC_SDK_TYPESCRIPT: string; +export declare const WAC_SDK_PYTHON: string; +export declare const DATATABLE_SDK_TYPESCRIPT: string; +export declare const DATATABLE_SDK_PYTHON: string; +export declare const OPENFLOW_SCHEMA: string; +export declare const CLI_COMMANDS: string; +export declare const LANG_ANSIBLE: string; +export declare const LANG_BASH: string; +export declare const LANG_BIGQUERY: string; +export declare const LANG_BUN: string; +export declare const LANG_BUNNATIVE: string; +export declare const LANG_CSHARP: string; +export declare const LANG_DENO: string; +export declare const LANG_DUCKDB: string; +export declare const LANG_GO: string; +export declare const LANG_GRAPHQL: string; +export declare const LANG_JAVA: string; +export declare const LANG_MSSQL: string; +export declare const LANG_MYSQL: string; +export declare const LANG_PHP: string; +export declare const LANG_POSTGRESQL: string; +export declare const LANG_POWERSHELL: string; +export declare const LANG_PYTHON3: string; +export declare const LANG_RLANG: string; +export declare const LANG_RUST: string; +export declare const LANG_SNOWFLAKE: string; diff --git a/system_prompts/auto-generated/prompts.ts b/system_prompts/auto-generated/prompts.ts index b9659d4bbe..43b09cfe14 100644 --- a/system_prompts/auto-generated/prompts.ts +++ b/system_prompts/auto-generated/prompts.ts @@ -170,8 +170,11 @@ input_transforms: ## Approval / Suspend Structure +An approval step is a normal **script** step (\`type: rawscript\` or \`type: script\`) that is turned into an approval by adding a module-level \`suspend\`. Its script calls \`wmill.getResumeUrls(approver)\` to generate the secret resume/cancel URLs and returns them so they can be sent to the approver(s) (Slack, email, etc.) or approved from the run page. + - \`suspend\` belongs on the flow module object itself, as a sibling of \`id\` and \`value\` - Never put \`suspend\` inside \`value\` +- Do NOT use \`type: identity\` for an approval step. An identity step suspends but never produces the resume URLs, so approvers have no link to act on — it is not a functional approval. Correct shape: @@ -187,10 +190,23 @@ Correct shape: type: string required: [comment] value: - type: identity + type: rawscript + language: bun + input_transforms: + approver: + type: static + value: '' + content: | + import * as wmill from "windmill-client" + + export async function main(approver?: string) { + const urls = await wmill.getResumeUrls(approver) + // send urls.resume / urls.cancel to the approver(s), e.g. via Slack or email + return urls + } \`\`\` -Incorrect shape: +Incorrect shape (suspend misplaced inside \`value\`): \`\`\`yaml - id: request_approval @@ -200,6 +216,16 @@ Incorrect shape: required_events: 1 \`\`\` +Incorrect shape (identity has no resume URLs — not a real approval): + +\`\`\`yaml +- id: request_approval + suspend: + required_events: 1 + value: + type: identity +\`\`\` + ## Branch Result Scope Rules - Inside a branch, you may reference earlier outer steps and earlier steps in the same branch @@ -587,6 +613,8 @@ A raw app has three logical parts: \`index.tsx\` is the bundling entrypoint. It typically renders a top-level \`App\` component. The bundler is esbuild. +**Always begin every React file (\`.tsx\`/\`.jsx\`) that uses JSX with \`import React from 'react'\`.** esbuild uses the classic JSX transform, so \`React\` must be in scope wherever JSX appears — a missing import compiles fine but throws \`React is not defined\` at runtime, leaving a blank screen. + ### Generated bindings (\`wmill.d.ts\` / \`wmill.ts\`) The frontend imports a generated module that mirrors the backend runnables. **Never write to it directly** — it gets regenerated whenever backend runnables change. Modifying it by hand will be overwritten. @@ -708,6 +736,68 @@ def main(user_id: str): 5. **Always whitelist tables** — adding a runnable that queries a new table requires the table to be in \`data.tables\` first. `; +export const PIPELINE_BASE = `# Data pipeline authoring + +A **data pipeline** is NOT a flow. A flow is one runnable that orchestrates steps internally. A data pipeline is a set of **independent scripts**, each deployed on its own, that form a DAG by reading and writing shared **storage assets** (DuckLake tables, data tables, S3 objects, volumes, resources) and by declaring execution **triggers**. The pipeline is visualized and edited at \`/pipeline/\`; every node is a normal workspace script that happens to carry pipeline annotations. When the user asks for a "data pipeline" (or to "ingest / transform / materialize" data across steps), build pipeline-annotated scripts — do NOT build a flow. + +## What makes a script a pipeline node + +A script joins the pipeline when its source begins with the \`pipeline\` annotation as a top-of-file comment, **written in the script's own comment syntax** — \`//\` for TS/JS (bun), \`--\` for SQL (DuckDB/Postgres), \`#\` for Python/Bash. So it's \`-- pipeline\` in a DuckDB node, \`# pipeline\` in a Python node, \`// pipeline\` in a bun node. Every annotation below uses that same prefix (the \`//\` shown is the TS form). All other wiring is expressed as annotation comments near the top of the file: + +- \`// on \` — declares an execution-DAG **input** (what triggers/feeds this node). \`\` is either: + - an **asset URI** (the node runs when that asset is produced upstream): \`ducklake://main/orders\`, \`datatable://main/users\`, \`s3://\`, \`$res:f/folder/my_resource\`, \`volume://name/path\`. + - a **native trigger kind**: \`schedule\`, \`webhook\`, \`email\`, \`kafka\`, \`mqtt\`, \`nats\`, \`postgres\`, \`sqs\`, \`gcp\`, or \`data_upload\` (a user-uploaded S3 file). For these the actual trigger row (cron, topic, …) is created separately; the annotation only declares the binding. +- **Outputs** are inferred from what the body writes — a \`CREATE TABLE\`, a \`wmill.writeS3File(...)\`, a DuckLake/datatable write. To declare a managed output explicitly, use \`// materialize \`. +- Optional badges: \`// partitioned \`, \`// freshness \` (e.g. \`1h\`), \`// tag \`, \`// retry [delay]\`, \`// data_test ...\`. + +## Materialize (the managed output) + +> **\`// materialize\` is DuckDB-only**, and its target must be a DuckLake table (\`ducklake:///
\`). Deploy **rejects** \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`) or a non-DuckLake target. For a non-DuckDB node, do **not** use \`// materialize\` — write the output via the SDK (\`wmill.writeS3File(...)\`, a postgresql \`CREATE TABLE\`, ducklake helpers, …) and let it be inferred. Use \`duckdb\` when a node should materialize a DuckLake table. + +\`// materialize \` tells the runtime to write the node's output table **for you**: write the body as a single \`SELECT\` and the runtime wraps it in the create/replace — do **not** also write your own \`CREATE TABLE\` / \`INSERT\`. Write strategy: + +- no option → **replace** the whole table each run (full refresh; the only mode whose output columns may change); +- \`// materialize append\` → INSERT-append rows (incremental); +- \`// materialize key=\` → merge/upsert on \`\`. + +\`// materialize manual \` opts **out** of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. + +\`materialize\` pairs with partitioning for incremental pipelines: a \`// partitioned \` node runs **once per partition** (append/merge into a fixed-schema table), and the \`{partition}\` token inside any asset URI is substituted with the current partition value at run time. + +\`materialize\` is an output **declaration** on a node — not a command. There is no "materialize run". + +## How to build one in chat + +1. Put every node in the **same folder**: \`f//\`. The folder is the pipeline. +2. Author each node as a **script draft** with \`write_script\` (or \`edit_script\`), language chosen for the work: \`duckdb\` or \`postgresql\` for SQL-shaped data work, \`bun\`/\`python3\` for general transforms. SQL-heavy lakehouse steps usually use \`duckdb\`. +3. Start each body with \`// pipeline\`, then the \`// on\` input declarations, then the transform that writes the output. +4. **Chain nodes by asset URI**: read an upstream node's output asset, then \`// on \` in the downstream node so the edge forms. Reuse exact asset paths from existing nodes rather than inventing parallel ones. +5. Leave nodes as drafts unless the user asks to deploy. A pipeline only "runs" once its scripts are deployed and their triggers exist. + +When the user already has the \`/pipeline/\` editor open, prefer the dedicated \`build_pipeline_node\` / \`edit_pipeline_node\` tools (they stage reviewable, canvas-highlighted proposals). Outside the editor, use the standard script-draft tools with the annotations above. + +## Example (DuckDB → DuckLake, scheduled ingest + downstream transform) + +Node \`f/sales/orders_ingest\` (runs on a schedule, materializes a DuckLake table): + +\`\`\`sql +-- pipeline +-- on schedule +-- materialize ducklake://main/orders +SELECT * FROM read_csv('s3://raw/orders/*.csv') +\`\`\` + +Node \`f/sales/orders_daily\` (runs when \`orders\` is produced, writes a rollup): + +\`\`\`sql +-- pipeline +-- on ducklake://main/orders +-- materialize ducklake://main/orders_daily +SELECT date_trunc('day', ts) AS day, count(*) AS n +FROM ducklake.main.orders GROUP BY 1 +\`\`\` +`; + export const WORKFLOW_AS_CODE_BASE = `# Windmill Workflow-as-Code Writing Guide ## Scope @@ -965,25 +1055,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -1002,9 +1094,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -1031,25 +1124,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -1057,9 +1152,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -1379,13 +1475,6 @@ async requestInteractiveSlackApproval({ slackResourcePath, channelId, message, a */ async requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise -/** - * Parse an S3 object from URI string or record format - * @param s3Object - S3 object as URI string (s3://storage/key) or record - * @returns S3 object record with storage and s3 key - */ -parseS3Object(s3Object: S3Object): S3ObjectRecord - setWorkflowCtx(ctx: WorkflowCtx | null): void async sleep(seconds: number): Promise @@ -1455,6 +1544,17 @@ async parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: */ async commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise +/** + * Parse an S3 object from URI string or record format + * @param s3Object - S3 object as URI string (\`s3://storage/key\`, \`s3:///key\` + * for the default storage) or record. Any other string throws rather than + * falling back to an auto-generated key: an auto key is requested by + * omitting the object, and a fallback would silently misplace the upload + * on any typo. + * @returns S3 object record with storage and s3 key + */ +parseS3Object(s3Object: S3Object): S3ObjectRecord + /** * Create a SQL template function for PostgreSQL/datatable queries * @param name - Database/datatable name (default: "main") @@ -1487,6 +1587,29 @@ datatable(name: string = "main"): DatatableSqlTemplateFunction * let sql = wmill.ducklake("my_lake:analytics") */ ducklake(name: string = "main"): SqlTemplateFunction + +/** + * Idempotently materialize \`selectSql\` into a ducklake table for one + * partition (or the whole table when \`partition\` is omitted) — the client-side + * equivalent of the \`// materialize\` engine. + * With \`uniqueKey\` it upserts the slice (delete-by-key + insert); otherwise it + * replaces it (whole table → \`CREATE OR REPLACE\`; partition → delete + insert). + * Safe to re-run for the same partition (backfill / failure-recovery). + * + * Returns a lazy statement — call \`.execute()\` to run it: + * \`await wmill.upsertPartition({ table, selectSql, partition }).execute()\`. + */ +upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement + +/** + * INSERT-only materialization (no dedup/replace) for append-only tables. + * Re-running the same partition duplicates rows — use only for immutable + * event-log sources. + * + * Returns a lazy statement — call \`.execute()\` to run it: + * \`await wmill.appendPartition({ table, selectSql, partition }).execute()\`. + */ +appendPartition(opts: Omit,): SqlStatement `; export const SDK_PYTHON = `# Python SDK (wmill) @@ -1537,27 +1660,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -1975,10 +2098,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -1989,10 +2113,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB @@ -2015,7 +2140,11 @@ def get_state_path() -> str # Parse resource syntax from string. def parse_resource_syntax(s: str) -> Optional[str] -# Parse S3 object from string or S3Object format. +# Parse S3 object from a \`s3:///\` URI string (\`s3:///\` +# for the default storage) or S3Object format. Any other string raises +# rather than falling back to an auto-generated key: an auto key is +# requested by omitting the object, and a fallback would silently misplace +# the upload on any typo. def parse_s3_object(s3_object: S3Object | str) -> S3Object # Parse variable syntax from string. @@ -2043,6 +2172,27 @@ def stream_result(stream) -> None # SqlQuery instance for fetching results def query(sql: str, *args) -> SqlQuery +# Idempotently materialize the rows of \`select_sql\` into ducklake +# \`table\` for one \`partition\` (or the whole table when \`partition\` is +# None). Client-side equivalent of the \`// materialize\` engine: with +# \`unique_key\` it upserts within the slice (delete-by-key + insert); +# without it, it replaces (whole table → CREATE OR REPLACE; partition → +# delete the partition + insert). Re-running the same slice is safe — the +# backfill / failure-recovery contract. +# +# The partition value is bound as a DuckDB arg (never string-interpolated) +# so it cannot inject SQL. \`select_sql\` is trusted (your own query). +def upsert_partition(table: str, select_sql: str, partition: str = None, unique_key: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# INSERT-only materialization (no dedup / no replace) for an immutable +# event-log table — for one \`partition\`, or the whole table when +# \`partition\` is None. NOTE: unlike \`upsert_partition\`, re-running the same +# slice duplicates rows — use only for append-only sources. +def append_partition(table: str, select_sql: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# Read a materialized ducklake table, optionally a single partition. +def read(table: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + # Execute query and fetch results. # # Args: @@ -2550,7 +2700,7 @@ class SqlQuery: export const OPENFLOW_SCHEMA = `## OpenFlow Schema -{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`; +{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`; export const CLI_COMMANDS = `# Windmill CLI Commands @@ -2631,6 +2781,13 @@ datatable related commands - \`datatable run \` - run a SQL query on a datatable - \`-n --name \` - Datatable name (default: main) - \`-s --silent\` - Output only the final result as JSON. Useful for scripting. +- \`datatable migrate\` - manage datatable migrations + - \`datatable migrate new \` - scaffold a new migration (.up.sql / .down.sql files) + - \`-d --datatable \` - Target datatable (default: main) + - \`datatable migrate up\` - apply all pending migrations to the main datatable (or one via --datatable) + - \`-d --datatable \` - Target datatable (default: main) + - \`datatable migrate down\` - roll back the most recent migration on the main datatable (or one via --datatable) + - \`-d --datatable \` - Target datatable (default: main) - \`datatable create [name:string]\` - register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable:// - \`--resource \` - Back the datatable with an existing postgresql resource path instead of the instance database - \`--force\` - Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved) @@ -2899,19 +3056,18 @@ Manage jobs (list, inspect, cancel) ### jobs -Pull completed and queued jobs from workspace - -**Arguments:** \`[workspace:string]\` - -**Options:** -- \`-c, --completed-output \` - Completed jobs output file (default: completed_jobs.json) -- \`-q, --queued-output \` - Queued jobs output file (default: queued_jobs.json) -- \`--skip-worker-check\` - Skip checking for active workers before export +Manage jobs (import/export) **Subcommands:** -- \`jobs pull\` -- \`jobs push\` +- \`jobs pull [workspace:string]\` - Pull completed and queued jobs from workspace + - \`-c, --completed-output \` - Completed jobs output file (default: completed_jobs.json) + - \`-q, --queued-output \` - Queued jobs output file (default: queued_jobs.json) + - \`--skip-worker-check\` - Skip checking for active workers before export +- \`jobs push [workspace:string]\` - Push completed and queued jobs to workspace + - \`-c, --completed-file \` - Completed jobs input file (default: completed_jobs.json) + - \`-q, --queued-file \` - Queued jobs input file (default: queued_jobs.json) + - \`--skip-worker-check\` - Skip checking for active workers before import ### lint @@ -2973,6 +3129,22 @@ inspect asset-driven pipelines (scripts marked \`// pipeline\`, wired by \`// on - \`--json\` - Output as JSON (for piping to jq) - \`pipeline show \` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal - \`--json\` - Output the raw asset graph as JSON + - \`--local\` - Build the graph from local working-tree files (// pipeline scripts) instead of the deployed workspace — no deploy needed. +- \`pipeline run \` - run a cascade: from --from (a root OR any mid-DAG model), fan downstream up to the --to end node(s) + - \`--from \` - Start script (short name or path). May be any node, including a mid-DAG model — that node plus its transitive downstream runs, upstream is NOT re-run (dbt \`--select model+\`). Defaults to the folder's sole schedule/manual root. + - \`--to \` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream. + - \`--dry-run\` - Print the topological run plan without executing. + - \`--json\` - Output the plan as JSON (for piping to jq). + - \`--local\` - Run the local working-tree scripts via preview (no deploy) instead of the deployed versions; the graph is built from local files. + - \`--upload \` - Bind an object to a data_upload/webhook entry point so it runs in the cascade, as SCRIPT[:PARAM]=SOURCE (SOURCE is a local file or an s3://key). Local files are uploaded to the workspace store; the S3Object param is inferred when the script has exactly one. Repeatable. + - \`--arg \` - Pass a plain run arg to a script in the cascade, as SCRIPT:PARAM=VALUE (VALUE is parsed as JSON when possible, else taken as a string — e.g. daily_report:partition=2026-07-02). Repeatable. + - \`--partition \` - Partition value for \`// partitioned\` scripts in the run (e.g. 2026-06-30) — use it to backfill a past slice. With --local, time kinds (daily/hourly/weekly/monthly) default to the current UTC period when omitted; \`dynamic\` always needs it. Deployed runs without it defer to backend run-start resolution. +- \`pipeline docs \` - generate PIPELINE.md (+ AGENTS.md pointer) describing a folder's pipeline graph and datatable schemas, for an editor / agentic loop + - \`--local\` - Build the graph from local working-tree files instead of the deployed workspace. +- \`pipeline dev [folder:string]\` - Live-preview a data pipeline from local files: watch an \`f/\` of \`// pipeline\` scripts, push the working-tree graph to the dev page, and run the cascade via preview (no deploy). + - \`--port \` - Port for the dev WebSocket server. + - \`--no-open\` - Do not open the browser automatically. + - \`--frontend \` - Origin serving the /pipeline_dev page (e.g. http://localhost:3000 for a locally-run frontend). Defaults to the workspace remote; use it when the remote's deployed frontend predates the dev page. ### protection-rules @@ -3298,13 +3470,19 @@ workspace related commands - \`--branch \` - Git branch to associate (default: workspace name) - \`workspace unbind\` - Remove baseUrl and workspaceId from a workspace entry - \`--workspace \` - Workspace to unbind -- \`workspace fork [workspace_name:string] [workspace_id:string]\` - Create a forked workspace +- \`workspace fork [workspace_name:string] [workspace_id:string]\` - Create a forked workspace from its parent workspace. + +The parent is resolved from your current git branch, not from the active profile: run this from a git repo checked out on the branch mapped to the parent workspace in wmill.yaml's \`workspaces:\` section (a fork branch of it resolves to the same parent). \`wmill workspace switch\` does not change which workspace is forked. + +Arguments (omit both to be prompted interactively): + [workspace_name] Friendly display name for the fork, shown in the UI. May contain spaces, so quote it in the shell (e.g. "My Fork"). Max 50 chars. Defaults to "'s fork". + [workspace_id] Id for the fork. Must be a slug (no spaces or special characters) and is automatically prefixed with \`wm-fork-\`, so pass just the bare slug (e.g. \`my-fork\` becomes \`wm-fork-my-fork\`). This id also determines the fork's git branch name. Defaults to a slug derived from the name — or, when you are converting an existing branch into the fork branch, from that branch. - \`--create-workspace-name \` - Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id. - \`--color \` - Workspace color (hex code, e.g. #ff0000) - \`--datatable-behavior \` - How to handle datatables: skip, schema_only, or schema_and_data (default: interactive prompt) - \`--from-branch \` - Non-interactive override for the 'turn my current working branch into the fork' workflow: base the fork on (its bound workspace is the parent) and rename the current branch onto wm-fork//. Usually unneeded — from a working branch \`wmill workspace fork\` offers this interactively; from a base branch it creates a fresh fork branch. - \`-y --yes\` - Skip interactive prompts (defaults datatable behavior to 'skip'). On a non-base branch, requires --from-branch since the base branch can't be prompted for. -- \`workspace delete-fork \` - Delete a forked workspace and git branch +- \`workspace delete-fork \` - Delete a forked workspace - \`-y --yes\` - Skip confirmation prompt - \`workspace merge\` - Compare and deploy changes between a fork and its parent workspace - \`--direction \` - Deploy direction: to-parent or to-fork diff --git a/system_prompts/auto-generated/script.md b/system_prompts/auto-generated/script.md index 89d8e62cfa..408f1354b7 100644 --- a/system_prompts/auto-generated/script.md +++ b/system_prompts/auto-generated/script.md @@ -1468,25 +1468,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -1505,9 +1507,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -1534,25 +1537,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -1560,9 +1565,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -1882,13 +1888,6 @@ async requestInteractiveSlackApproval({ slackResourcePath, channelId, message, a */ async requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise -/** - * Parse an S3 object from URI string or record format - * @param s3Object - S3 object as URI string (s3://storage/key) or record - * @returns S3 object record with storage and s3 key - */ -parseS3Object(s3Object: S3Object): S3ObjectRecord - setWorkflowCtx(ctx: WorkflowCtx | null): void async sleep(seconds: number): Promise @@ -1958,6 +1957,17 @@ async parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: */ async commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise +/** + * Parse an S3 object from URI string or record format + * @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key` + * for the default storage) or record. Any other string throws rather than + * falling back to an auto-generated key: an auto key is requested by + * omitting the object, and a fallback would silently misplace the upload + * on any typo. + * @returns S3 object record with storage and s3 key + */ +parseS3Object(s3Object: S3Object): S3ObjectRecord + /** * Create a SQL template function for PostgreSQL/datatable queries * @param name - Database/datatable name (default: "main") @@ -1991,6 +2001,29 @@ datatable(name: string = "main"): DatatableSqlTemplateFunction */ ducklake(name: string = "main"): SqlTemplateFunction +/** + * Idempotently materialize `selectSql` into a ducklake table for one + * partition (or the whole table when `partition` is omitted) — the client-side + * equivalent of the `// materialize` engine. + * With `uniqueKey` it upserts the slice (delete-by-key + insert); otherwise it + * replaces it (whole table → `CREATE OR REPLACE`; partition → delete + insert). + * Safe to re-run for the same partition (backfill / failure-recovery). + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.upsertPartition({ table, selectSql, partition }).execute()`. + */ +upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement + +/** + * INSERT-only materialization (no dedup/replace) for append-only tables. + * Re-running the same partition duplicates rows — use only for immutable + * event-log sources. + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.appendPartition({ table, selectSql, partition }).execute()`. + */ +appendPartition(opts: Omit,): SqlStatement + # Python SDK (wmill) @@ -2040,27 +2073,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -2478,10 +2511,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -2492,10 +2526,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB @@ -2518,7 +2553,11 @@ def get_state_path() -> str # Parse resource syntax from string. def parse_resource_syntax(s: str) -> Optional[str] -# Parse S3 object from string or S3Object format. +# Parse S3 object from a `s3:///` URI string (`s3:///` +# for the default storage) or S3Object format. Any other string raises +# rather than falling back to an auto-generated key: an auto key is +# requested by omitting the object, and a fallback would silently misplace +# the upload on any typo. def parse_s3_object(s3_object: S3Object | str) -> S3Object # Parse variable syntax from string. @@ -2546,6 +2585,27 @@ def stream_result(stream) -> None # SqlQuery instance for fetching results def query(sql: str, *args) -> SqlQuery +# Idempotently materialize the rows of `select_sql` into ducklake +# `table` for one `partition` (or the whole table when `partition` is +# None). Client-side equivalent of the `// materialize` engine: with +# `unique_key` it upserts within the slice (delete-by-key + insert); +# without it, it replaces (whole table → CREATE OR REPLACE; partition → +# delete the partition + insert). Re-running the same slice is safe — the +# backfill / failure-recovery contract. +# +# The partition value is bound as a DuckDB arg (never string-interpolated) +# so it cannot inject SQL. `select_sql` is trusted (your own query). +def upsert_partition(table: str, select_sql: str, partition: str = None, unique_key: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# INSERT-only materialization (no dedup / no replace) for an immutable +# event-log table — for one `partition`, or the whole table when +# `partition` is None. NOTE: unlike `upsert_partition`, re-running the same +# slice duplicates rows — use only for append-only sources. +def append_partition(table: str, select_sql: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# Read a materialized ducklake table, optionally a single partition. +def read(table: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + # Execute query and fetch results. # # Args: diff --git a/system_prompts/auto-generated/sdks/python.md b/system_prompts/auto-generated/sdks/python.md index e0b1bd3dfd..c750f03bcf 100644 --- a/system_prompts/auto-generated/sdks/python.md +++ b/system_prompts/auto-generated/sdks/python.md @@ -46,27 +46,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -484,10 +484,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -498,10 +499,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB @@ -524,7 +526,11 @@ def get_state_path() -> str # Parse resource syntax from string. def parse_resource_syntax(s: str) -> Optional[str] -# Parse S3 object from string or S3Object format. +# Parse S3 object from a `s3:///` URI string (`s3:///` +# for the default storage) or S3Object format. Any other string raises +# rather than falling back to an auto-generated key: an auto key is +# requested by omitting the object, and a fallback would silently misplace +# the upload on any typo. def parse_s3_object(s3_object: S3Object | str) -> S3Object # Parse variable syntax from string. @@ -552,6 +558,27 @@ def stream_result(stream) -> None # SqlQuery instance for fetching results def query(sql: str, *args) -> SqlQuery +# Idempotently materialize the rows of `select_sql` into ducklake +# `table` for one `partition` (or the whole table when `partition` is +# None). Client-side equivalent of the `// materialize` engine: with +# `unique_key` it upserts within the slice (delete-by-key + insert); +# without it, it replaces (whole table → CREATE OR REPLACE; partition → +# delete the partition + insert). Re-running the same slice is safe — the +# backfill / failure-recovery contract. +# +# The partition value is bound as a DuckDB arg (never string-interpolated) +# so it cannot inject SQL. `select_sql` is trusted (your own query). +def upsert_partition(table: str, select_sql: str, partition: str = None, unique_key: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# INSERT-only materialization (no dedup / no replace) for an immutable +# event-log table — for one `partition`, or the whole table when +# `partition` is None. NOTE: unlike `upsert_partition`, re-running the same +# slice duplicates rows — use only for append-only sources. +def append_partition(table: str, select_sql: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# Read a materialized ducklake table, optionally a single partition. +def read(table: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + # Execute query and fetch results. # # Args: diff --git a/system_prompts/auto-generated/sdks/typescript.md b/system_prompts/auto-generated/sdks/typescript.md index ff57632eef..8100929776 100644 --- a/system_prompts/auto-generated/sdks/typescript.md +++ b/system_prompts/auto-generated/sdks/typescript.md @@ -35,25 +35,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -72,9 +74,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -101,25 +104,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -127,9 +132,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -449,13 +455,6 @@ async requestInteractiveSlackApproval({ slackResourcePath, channelId, message, a */ async requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise -/** - * Parse an S3 object from URI string or record format - * @param s3Object - S3 object as URI string (s3://storage/key) or record - * @returns S3 object record with storage and s3 key - */ -parseS3Object(s3Object: S3Object): S3ObjectRecord - setWorkflowCtx(ctx: WorkflowCtx | null): void async sleep(seconds: number): Promise @@ -525,6 +524,17 @@ async parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: */ async commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise +/** + * Parse an S3 object from URI string or record format + * @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key` + * for the default storage) or record. Any other string throws rather than + * falling back to an auto-generated key: an auto key is requested by + * omitting the object, and a fallback would silently misplace the upload + * on any typo. + * @returns S3 object record with storage and s3 key + */ +parseS3Object(s3Object: S3Object): S3ObjectRecord + /** * Create a SQL template function for PostgreSQL/datatable queries * @param name - Database/datatable name (default: "main") @@ -557,3 +567,26 @@ datatable(name: string = "main"): DatatableSqlTemplateFunction * let sql = wmill.ducklake("my_lake:analytics") */ ducklake(name: string = "main"): SqlTemplateFunction + +/** + * Idempotently materialize `selectSql` into a ducklake table for one + * partition (or the whole table when `partition` is omitted) — the client-side + * equivalent of the `// materialize` engine. + * With `uniqueKey` it upserts the slice (delete-by-key + insert); otherwise it + * replaces it (whole table → `CREATE OR REPLACE`; partition → delete + insert). + * Safe to re-run for the same partition (backfill / failure-recovery). + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.upsertPartition({ table, selectSql, partition }).execute()`. + */ +upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement + +/** + * INSERT-only materialization (no dedup/replace) for append-only tables. + * Re-running the same partition duplicates rows — use only for immutable + * event-log sources. + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.appendPartition({ table, selectSql, partition }).execute()`. + */ +appendPartition(opts: Omit,): SqlStatement diff --git a/system_prompts/auto-generated/skills/cli-commands/SKILL.md b/system_prompts/auto-generated/skills/cli-commands/SKILL.md index 0afcda0f5a..3a61a67c12 100644 --- a/system_prompts/auto-generated/skills/cli-commands/SKILL.md +++ b/system_prompts/auto-generated/skills/cli-commands/SKILL.md @@ -82,6 +82,13 @@ datatable related commands - `datatable run ` - run a SQL query on a datatable - `-n --name ` - Datatable name (default: main) - `-s --silent` - Output only the final result as JSON. Useful for scripting. +- `datatable migrate` - manage datatable migrations + - `datatable migrate new ` - scaffold a new migration (.up.sql / .down.sql files) + - `-d --datatable ` - Target datatable (default: main) + - `datatable migrate up` - apply all pending migrations to the main datatable (or one via --datatable) + - `-d --datatable ` - Target datatable (default: main) + - `datatable migrate down` - roll back the most recent migration on the main datatable (or one via --datatable) + - `-d --datatable ` - Target datatable (default: main) - `datatable create [name:string]` - register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable:// - `--resource ` - Back the datatable with an existing postgresql resource path instead of the instance database - `--force` - Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved) @@ -350,19 +357,18 @@ Manage jobs (list, inspect, cancel) ### jobs -Pull completed and queued jobs from workspace - -**Arguments:** `[workspace:string]` - -**Options:** -- `-c, --completed-output ` - Completed jobs output file (default: completed_jobs.json) -- `-q, --queued-output ` - Queued jobs output file (default: queued_jobs.json) -- `--skip-worker-check` - Skip checking for active workers before export +Manage jobs (import/export) **Subcommands:** -- `jobs pull` -- `jobs push` +- `jobs pull [workspace:string]` - Pull completed and queued jobs from workspace + - `-c, --completed-output ` - Completed jobs output file (default: completed_jobs.json) + - `-q, --queued-output ` - Queued jobs output file (default: queued_jobs.json) + - `--skip-worker-check` - Skip checking for active workers before export +- `jobs push [workspace:string]` - Push completed and queued jobs to workspace + - `-c, --completed-file ` - Completed jobs input file (default: completed_jobs.json) + - `-q, --queued-file ` - Queued jobs input file (default: queued_jobs.json) + - `--skip-worker-check` - Skip checking for active workers before import ### lint @@ -424,6 +430,22 @@ inspect asset-driven pipelines (scripts marked `// pipeline`, wired by `// on ` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal - `--json` - Output the raw asset graph as JSON + - `--local` - Build the graph from local working-tree files (// pipeline scripts) instead of the deployed workspace — no deploy needed. +- `pipeline run ` - run a cascade: from --from (a root OR any mid-DAG model), fan downstream up to the --to end node(s) + - `--from ` - Start script (short name or path). May be any node, including a mid-DAG model — that node plus its transitive downstream runs, upstream is NOT re-run (dbt `--select model+`). Defaults to the folder's sole schedule/manual root. + - `--to ` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream. + - `--dry-run` - Print the topological run plan without executing. + - `--json` - Output the plan as JSON (for piping to jq). + - `--local` - Run the local working-tree scripts via preview (no deploy) instead of the deployed versions; the graph is built from local files. + - `--upload ` - Bind an object to a data_upload/webhook entry point so it runs in the cascade, as SCRIPT[:PARAM]=SOURCE (SOURCE is a local file or an s3://key). Local files are uploaded to the workspace store; the S3Object param is inferred when the script has exactly one. Repeatable. + - `--arg ` - Pass a plain run arg to a script in the cascade, as SCRIPT:PARAM=VALUE (VALUE is parsed as JSON when possible, else taken as a string — e.g. daily_report:partition=2026-07-02). Repeatable. + - `--partition ` - Partition value for `// partitioned` scripts in the run (e.g. 2026-06-30) — use it to backfill a past slice. With --local, time kinds (daily/hourly/weekly/monthly) default to the current UTC period when omitted; `dynamic` always needs it. Deployed runs without it defer to backend run-start resolution. +- `pipeline docs ` - generate PIPELINE.md (+ AGENTS.md pointer) describing a folder's pipeline graph and datatable schemas, for an editor / agentic loop + - `--local` - Build the graph from local working-tree files instead of the deployed workspace. +- `pipeline dev [folder:string]` - Live-preview a data pipeline from local files: watch an `f/` of `// pipeline` scripts, push the working-tree graph to the dev page, and run the cascade via preview (no deploy). + - `--port ` - Port for the dev WebSocket server. + - `--no-open` - Do not open the browser automatically. + - `--frontend ` - Origin serving the /pipeline_dev page (e.g. http://localhost:3000 for a locally-run frontend). Defaults to the workspace remote; use it when the remote's deployed frontend predates the dev page. ### protection-rules @@ -749,13 +771,19 @@ workspace related commands - `--branch ` - Git branch to associate (default: workspace name) - `workspace unbind` - Remove baseUrl and workspaceId from a workspace entry - `--workspace ` - Workspace to unbind -- `workspace fork [workspace_name:string] [workspace_id:string]` - Create a forked workspace +- `workspace fork [workspace_name:string] [workspace_id:string]` - Create a forked workspace from its parent workspace. + +The parent is resolved from your current git branch, not from the active profile: run this from a git repo checked out on the branch mapped to the parent workspace in wmill.yaml's `workspaces:` section (a fork branch of it resolves to the same parent). `wmill workspace switch` does not change which workspace is forked. + +Arguments (omit both to be prompted interactively): + [workspace_name] Friendly display name for the fork, shown in the UI. May contain spaces, so quote it in the shell (e.g. "My Fork"). Max 50 chars. Defaults to "'s fork". + [workspace_id] Id for the fork. Must be a slug (no spaces or special characters) and is automatically prefixed with `wm-fork-`, so pass just the bare slug (e.g. `my-fork` becomes `wm-fork-my-fork`). This id also determines the fork's git branch name. Defaults to a slug derived from the name — or, when you are converting an existing branch into the fork branch, from that branch. - `--create-workspace-name ` - Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id. - `--color ` - Workspace color (hex code, e.g. #ff0000) - `--datatable-behavior ` - How to handle datatables: skip, schema_only, or schema_and_data (default: interactive prompt) - `--from-branch ` - Non-interactive override for the 'turn my current working branch into the fork' workflow: base the fork on (its bound workspace is the parent) and rename the current branch onto wm-fork//. Usually unneeded — from a working branch `wmill workspace fork` offers this interactively; from a base branch it creates a fresh fork branch. - `-y --yes` - Skip interactive prompts (defaults datatable behavior to 'skip'). On a non-base branch, requires --from-branch since the base branch can't be prompted for. -- `workspace delete-fork ` - Delete a forked workspace and git branch +- `workspace delete-fork ` - Delete a forked workspace - `-y --yes` - Skip confirmation prompt - `workspace merge` - Compare and deploy changes between a fork and its parent workspace - `--direction ` - Deploy direction: to-parent or to-fork diff --git a/system_prompts/auto-generated/skills/raw-app/SKILL.md b/system_prompts/auto-generated/skills/raw-app/SKILL.md index 292f93a7fd..624fdc97ac 100644 --- a/system_prompts/auto-generated/skills/raw-app/SKILL.md +++ b/system_prompts/auto-generated/skills/raw-app/SKILL.md @@ -251,6 +251,8 @@ A raw app has three logical parts: `index.tsx` is the bundling entrypoint. It typically renders a top-level `App` component. The bundler is esbuild. +**Always begin every React file (`.tsx`/`.jsx`) that uses JSX with `import React from 'react'`.** esbuild uses the classic JSX transform, so `React` must be in scope wherever JSX appears — a missing import compiles fine but throws `React is not defined` at runtime, leaving a blank screen. + ### Generated bindings (`wmill.d.ts` / `wmill.ts`) The frontend imports a generated module that mirrors the backend runnables. **Never write to it directly** — it gets regenerated whenever backend runnables change. Modifying it by hand will be overwritten. diff --git a/system_prompts/auto-generated/skills/write-flow/SKILL.md b/system_prompts/auto-generated/skills/write-flow/SKILL.md index 8c8057d48f..360b0d1ec9 100644 --- a/system_prompts/auto-generated/skills/write-flow/SKILL.md +++ b/system_prompts/auto-generated/skills/write-flow/SKILL.md @@ -225,8 +225,11 @@ input_transforms: ## Approval / Suspend Structure +An approval step is a normal **script** step (`type: rawscript` or `type: script`) that is turned into an approval by adding a module-level `suspend`. Its script calls `wmill.getResumeUrls(approver)` to generate the secret resume/cancel URLs and returns them so they can be sent to the approver(s) (Slack, email, etc.) or approved from the run page. + - `suspend` belongs on the flow module object itself, as a sibling of `id` and `value` - Never put `suspend` inside `value` +- Do NOT use `type: identity` for an approval step. An identity step suspends but never produces the resume URLs, so approvers have no link to act on — it is not a functional approval. Correct shape: @@ -242,10 +245,23 @@ Correct shape: type: string required: [comment] value: - type: identity + type: rawscript + language: bun + input_transforms: + approver: + type: static + value: '' + content: | + import * as wmill from "windmill-client" + + export async function main(approver?: string) { + const urls = await wmill.getResumeUrls(approver) + // send urls.resume / urls.cancel to the approver(s), e.g. via Slack or email + return urls + } ``` -Incorrect shape: +Incorrect shape (suspend misplaced inside `value`): ```yaml - id: request_approval @@ -255,6 +271,16 @@ Incorrect shape: required_events: 1 ``` +Incorrect shape (identity has no resume URLs — not a real approval): + +```yaml +- id: request_approval + suspend: + required_events: 1 + value: + type: identity +``` + ## Branch Result Scope Rules - Inside a branch, you may reference earlier outer steps and earlier steps in the same branch @@ -381,4 +407,4 @@ Reference a specific resource using `$res:` prefix: ## OpenFlow Schema -{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}} \ No newline at end of file +{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}} \ No newline at end of file diff --git a/system_prompts/auto-generated/skills/write-script-bun/SKILL.md b/system_prompts/auto-generated/skills/write-script-bun/SKILL.md index 0b656c1f75..7dcaec30bc 100644 --- a/system_prompts/auto-generated/skills/write-script-bun/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-bun/SKILL.md @@ -206,25 +206,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -243,9 +245,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -272,25 +275,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -298,9 +303,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -620,13 +626,6 @@ async requestInteractiveSlackApproval({ slackResourcePath, channelId, message, a */ async requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise -/** - * Parse an S3 object from URI string or record format - * @param s3Object - S3 object as URI string (s3://storage/key) or record - * @returns S3 object record with storage and s3 key - */ -parseS3Object(s3Object: S3Object): S3ObjectRecord - setWorkflowCtx(ctx: WorkflowCtx | null): void async sleep(seconds: number): Promise @@ -696,6 +695,17 @@ async parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: */ async commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise +/** + * Parse an S3 object from URI string or record format + * @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key` + * for the default storage) or record. Any other string throws rather than + * falling back to an auto-generated key: an auto key is requested by + * omitting the object, and a fallback would silently misplace the upload + * on any typo. + * @returns S3 object record with storage and s3 key + */ +parseS3Object(s3Object: S3Object): S3ObjectRecord + /** * Create a SQL template function for PostgreSQL/datatable queries * @param name - Database/datatable name (default: "main") @@ -728,3 +738,26 @@ datatable(name: string = "main"): DatatableSqlTemplateFunction * let sql = wmill.ducklake("my_lake:analytics") */ ducklake(name: string = "main"): SqlTemplateFunction + +/** + * Idempotently materialize `selectSql` into a ducklake table for one + * partition (or the whole table when `partition` is omitted) — the client-side + * equivalent of the `// materialize` engine. + * With `uniqueKey` it upserts the slice (delete-by-key + insert); otherwise it + * replaces it (whole table → `CREATE OR REPLACE`; partition → delete + insert). + * Safe to re-run for the same partition (backfill / failure-recovery). + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.upsertPartition({ table, selectSql, partition }).execute()`. + */ +upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement + +/** + * INSERT-only materialization (no dedup/replace) for append-only tables. + * Re-running the same partition duplicates rows — use only for immutable + * event-log sources. + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.appendPartition({ table, selectSql, partition }).execute()`. + */ +appendPartition(opts: Omit,): SqlStatement diff --git a/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md b/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md index 4759590606..86c3bfd010 100644 --- a/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md @@ -206,25 +206,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -243,9 +245,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -272,25 +275,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -298,9 +303,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -620,13 +626,6 @@ async requestInteractiveSlackApproval({ slackResourcePath, channelId, message, a */ async requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise -/** - * Parse an S3 object from URI string or record format - * @param s3Object - S3 object as URI string (s3://storage/key) or record - * @returns S3 object record with storage and s3 key - */ -parseS3Object(s3Object: S3Object): S3ObjectRecord - setWorkflowCtx(ctx: WorkflowCtx | null): void async sleep(seconds: number): Promise @@ -696,6 +695,17 @@ async parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: */ async commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise +/** + * Parse an S3 object from URI string or record format + * @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key` + * for the default storage) or record. Any other string throws rather than + * falling back to an auto-generated key: an auto key is requested by + * omitting the object, and a fallback would silently misplace the upload + * on any typo. + * @returns S3 object record with storage and s3 key + */ +parseS3Object(s3Object: S3Object): S3ObjectRecord + /** * Create a SQL template function for PostgreSQL/datatable queries * @param name - Database/datatable name (default: "main") @@ -728,3 +738,26 @@ datatable(name: string = "main"): DatatableSqlTemplateFunction * let sql = wmill.ducklake("my_lake:analytics") */ ducklake(name: string = "main"): SqlTemplateFunction + +/** + * Idempotently materialize `selectSql` into a ducklake table for one + * partition (or the whole table when `partition` is omitted) — the client-side + * equivalent of the `// materialize` engine. + * With `uniqueKey` it upserts the slice (delete-by-key + insert); otherwise it + * replaces it (whole table → `CREATE OR REPLACE`; partition → delete + insert). + * Safe to re-run for the same partition (backfill / failure-recovery). + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.upsertPartition({ table, selectSql, partition }).execute()`. + */ +upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement + +/** + * INSERT-only materialization (no dedup/replace) for append-only tables. + * Re-running the same partition duplicates rows — use only for immutable + * event-log sources. + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.appendPartition({ table, selectSql, partition }).execute()`. + */ +appendPartition(opts: Omit,): SqlStatement diff --git a/system_prompts/auto-generated/skills/write-script-deno/SKILL.md b/system_prompts/auto-generated/skills/write-script-deno/SKILL.md index bab600af98..f7f3a517db 100644 --- a/system_prompts/auto-generated/skills/write-script-deno/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-deno/SKILL.md @@ -206,25 +206,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -243,9 +245,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -272,25 +275,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -298,9 +303,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -620,13 +626,6 @@ async requestInteractiveSlackApproval({ slackResourcePath, channelId, message, a */ async requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise -/** - * Parse an S3 object from URI string or record format - * @param s3Object - S3 object as URI string (s3://storage/key) or record - * @returns S3 object record with storage and s3 key - */ -parseS3Object(s3Object: S3Object): S3ObjectRecord - setWorkflowCtx(ctx: WorkflowCtx | null): void async sleep(seconds: number): Promise @@ -696,6 +695,17 @@ async parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: */ async commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise +/** + * Parse an S3 object from URI string or record format + * @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key` + * for the default storage) or record. Any other string throws rather than + * falling back to an auto-generated key: an auto key is requested by + * omitting the object, and a fallback would silently misplace the upload + * on any typo. + * @returns S3 object record with storage and s3 key + */ +parseS3Object(s3Object: S3Object): S3ObjectRecord + /** * Create a SQL template function for PostgreSQL/datatable queries * @param name - Database/datatable name (default: "main") @@ -728,3 +738,26 @@ datatable(name: string = "main"): DatatableSqlTemplateFunction * let sql = wmill.ducklake("my_lake:analytics") */ ducklake(name: string = "main"): SqlTemplateFunction + +/** + * Idempotently materialize `selectSql` into a ducklake table for one + * partition (or the whole table when `partition` is omitted) — the client-side + * equivalent of the `// materialize` engine. + * With `uniqueKey` it upserts the slice (delete-by-key + insert); otherwise it + * replaces it (whole table → `CREATE OR REPLACE`; partition → delete + insert). + * Safe to re-run for the same partition (backfill / failure-recovery). + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.upsertPartition({ table, selectSql, partition }).execute()`. + */ +upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement + +/** + * INSERT-only materialization (no dedup/replace) for append-only tables. + * Re-running the same partition duplicates rows — use only for immutable + * event-log sources. + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.appendPartition({ table, selectSql, partition }).execute()`. + */ +appendPartition(opts: Omit,): SqlStatement diff --git a/system_prompts/auto-generated/skills/write-script-python3/SKILL.md b/system_prompts/auto-generated/skills/write-script-python3/SKILL.md index b07f3c0d59..49c0ab057f 100644 --- a/system_prompts/auto-generated/skills/write-script-python3/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-python3/SKILL.md @@ -231,27 +231,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -669,10 +669,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -683,10 +684,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB @@ -709,7 +711,11 @@ def get_state_path() -> str # Parse resource syntax from string. def parse_resource_syntax(s: str) -> Optional[str] -# Parse S3 object from string or S3Object format. +# Parse S3 object from a `s3:///` URI string (`s3:///` +# for the default storage) or S3Object format. Any other string raises +# rather than falling back to an auto-generated key: an auto key is +# requested by omitting the object, and a fallback would silently misplace +# the upload on any typo. def parse_s3_object(s3_object: S3Object | str) -> S3Object # Parse variable syntax from string. @@ -737,6 +743,27 @@ def stream_result(stream) -> None # SqlQuery instance for fetching results def query(sql: str, *args) -> SqlQuery +# Idempotently materialize the rows of `select_sql` into ducklake +# `table` for one `partition` (or the whole table when `partition` is +# None). Client-side equivalent of the `// materialize` engine: with +# `unique_key` it upserts within the slice (delete-by-key + insert); +# without it, it replaces (whole table → CREATE OR REPLACE; partition → +# delete the partition + insert). Re-running the same slice is safe — the +# backfill / failure-recovery contract. +# +# The partition value is bound as a DuckDB arg (never string-interpolated) +# so it cannot inject SQL. `select_sql` is trusted (your own query). +def upsert_partition(table: str, select_sql: str, partition: str = None, unique_key: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# INSERT-only materialization (no dedup / no replace) for an immutable +# event-log table — for one `partition`, or the whole table when +# `partition` is None. NOTE: unlike `upsert_partition`, re-running the same +# slice duplicates rows — use only for append-only sources. +def append_partition(table: str, select_sql: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + +# Read a materialized ducklake table, optionally a single partition. +def read(table: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None) + # Execute query and fetch results. # # Args: diff --git a/system_prompts/base/flow-base.md b/system_prompts/base/flow-base.md index 17f46333f3..def01a3535 100644 --- a/system_prompts/base/flow-base.md +++ b/system_prompts/base/flow-base.md @@ -139,8 +139,11 @@ input_transforms: ## Approval / Suspend Structure +An approval step is a normal **script** step (`type: rawscript` or `type: script`) that is turned into an approval by adding a module-level `suspend`. Its script calls `wmill.getResumeUrls(approver)` to generate the secret resume/cancel URLs and returns them so they can be sent to the approver(s) (Slack, email, etc.) or approved from the run page. + - `suspend` belongs on the flow module object itself, as a sibling of `id` and `value` - Never put `suspend` inside `value` +- Do NOT use `type: identity` for an approval step. An identity step suspends but never produces the resume URLs, so approvers have no link to act on — it is not a functional approval. Correct shape: @@ -156,10 +159,23 @@ Correct shape: type: string required: [comment] value: - type: identity + type: rawscript + language: bun + input_transforms: + approver: + type: static + value: '' + content: | + import * as wmill from "windmill-client" + + export async function main(approver?: string) { + const urls = await wmill.getResumeUrls(approver) + // send urls.resume / urls.cancel to the approver(s), e.g. via Slack or email + return urls + } ``` -Incorrect shape: +Incorrect shape (suspend misplaced inside `value`): ```yaml - id: request_approval @@ -169,6 +185,16 @@ Incorrect shape: required_events: 1 ``` +Incorrect shape (identity has no resume URLs — not a real approval): + +```yaml +- id: request_approval + suspend: + required_events: 1 + value: + type: identity +``` + ## Branch Result Scope Rules - Inside a branch, you may reference earlier outer steps and earlier steps in the same branch diff --git a/system_prompts/base/pipeline-base.md b/system_prompts/base/pipeline-base.md new file mode 100644 index 0000000000..a0f2e2492a --- /dev/null +++ b/system_prompts/base/pipeline-base.md @@ -0,0 +1,60 @@ +# Data pipeline authoring + +A **data pipeline** is NOT a flow. A flow is one runnable that orchestrates steps internally. A data pipeline is a set of **independent scripts**, each deployed on its own, that form a DAG by reading and writing shared **storage assets** (DuckLake tables, data tables, S3 objects, volumes, resources) and by declaring execution **triggers**. The pipeline is visualized and edited at `/pipeline/`; every node is a normal workspace script that happens to carry pipeline annotations. When the user asks for a "data pipeline" (or to "ingest / transform / materialize" data across steps), build pipeline-annotated scripts — do NOT build a flow. + +## What makes a script a pipeline node + +A script joins the pipeline when its source begins with the `pipeline` annotation as a top-of-file comment, **written in the script's own comment syntax** — `//` for TS/JS (bun), `--` for SQL (DuckDB/Postgres), `#` for Python/Bash. So it's `-- pipeline` in a DuckDB node, `# pipeline` in a Python node, `// pipeline` in a bun node. Every annotation below uses that same prefix (the `//` shown is the TS form). All other wiring is expressed as annotation comments near the top of the file: + +- `// on ` — declares an execution-DAG **input** (what triggers/feeds this node). `` is either: + - an **asset URI** (the node runs when that asset is produced upstream): `ducklake://main/orders`, `datatable://main/users`, `s3://`, `$res:f/folder/my_resource`, `volume://name/path`. + - a **native trigger kind**: `schedule`, `webhook`, `email`, `kafka`, `mqtt`, `nats`, `postgres`, `sqs`, `gcp`, or `data_upload` (a user-uploaded S3 file). For these the actual trigger row (cron, topic, …) is created separately; the annotation only declares the binding. +- **Outputs** are inferred from what the body writes — a `CREATE TABLE`, a `wmill.writeS3File(...)`, a DuckLake/datatable write. To declare a managed output explicitly, use `// materialize `. +- Optional badges: `// partitioned `, `// freshness ` (e.g. `1h`), `// tag `, `// retry [delay]`, `// data_test ...`. + +## Materialize (the managed output) + +> **`// materialize` is DuckDB-only**, and its target must be a DuckLake table (`ducklake:///
`). Deploy **rejects** `// materialize` on any other language (`python3`, `bun`, `postgresql`) or a non-DuckLake target. For a non-DuckDB node, do **not** use `// materialize` — write the output via the SDK (`wmill.writeS3File(...)`, a postgresql `CREATE TABLE`, ducklake helpers, …) and let it be inferred. Use `duckdb` when a node should materialize a DuckLake table. + +`// materialize ` tells the runtime to write the node's output table **for you**: write the body as a single `SELECT` and the runtime wraps it in the create/replace — do **not** also write your own `CREATE TABLE` / `INSERT`. Write strategy: + +- no option → **replace** the whole table each run (full refresh; the only mode whose output columns may change); +- `// materialize append` → INSERT-append rows (incremental); +- `// materialize key=` → merge/upsert on ``. + +`// materialize manual ` opts **out** of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. + +`materialize` pairs with partitioning for incremental pipelines: a `// partitioned ` node runs **once per partition** (append/merge into a fixed-schema table), and the `{partition}` token inside any asset URI is substituted with the current partition value at run time. + +`materialize` is an output **declaration** on a node — not a command. There is no "materialize run". + +## How to build one in chat + +1. Put every node in the **same folder**: `f//`. The folder is the pipeline. +2. Author each node as a **script draft** with `write_script` (or `edit_script`), language chosen for the work: `duckdb` or `postgresql` for SQL-shaped data work, `bun`/`python3` for general transforms. SQL-heavy lakehouse steps usually use `duckdb`. +3. Start each body with `// pipeline`, then the `// on` input declarations, then the transform that writes the output. +4. **Chain nodes by asset URI**: read an upstream node's output asset, then `// on ` in the downstream node so the edge forms. Reuse exact asset paths from existing nodes rather than inventing parallel ones. +5. Leave nodes as drafts unless the user asks to deploy. A pipeline only "runs" once its scripts are deployed and their triggers exist. + +When the user already has the `/pipeline/` editor open, prefer the dedicated `build_pipeline_node` / `edit_pipeline_node` tools (they stage reviewable, canvas-highlighted proposals). Outside the editor, use the standard script-draft tools with the annotations above. + +## Example (DuckDB → DuckLake, scheduled ingest + downstream transform) + +Node `f/sales/orders_ingest` (runs on a schedule, materializes a DuckLake table): + +```sql +-- pipeline +-- on schedule +-- materialize ducklake://main/orders +SELECT * FROM read_csv('s3://raw/orders/*.csv') +``` + +Node `f/sales/orders_daily` (runs when `orders` is produced, writes a rollup): + +```sql +-- pipeline +-- on ducklake://main/orders +-- materialize ducklake://main/orders_daily +SELECT date_trunc('day', ts) AS day, count(*) AS n +FROM ducklake.main.orders GROUP BY 1 +``` diff --git a/system_prompts/base/raw-app.md b/system_prompts/base/raw-app.md index 04103e434b..004302cc74 100644 --- a/system_prompts/base/raw-app.md +++ b/system_prompts/base/raw-app.md @@ -16,6 +16,8 @@ A raw app has three logical parts: `index.tsx` is the bundling entrypoint. It typically renders a top-level `App` component. The bundler is esbuild. +**Always begin every React file (`.tsx`/`.jsx`) that uses JSX with `import React from 'react'`.** esbuild uses the classic JSX transform, so `React` must be in scope wherever JSX appears — a missing import compiles fine but throws `React is not defined` at runtime, leaving a blank screen. + ### Generated bindings (`wmill.d.ts` / `wmill.ts`) The frontend imports a generated module that mirrors the backend runnables. **Never write to it directly** — it gets regenerated whenever backend runnables change. Modifying it by hand will be overwritten. diff --git a/system_prompts/generate.py b/system_prompts/generate.py index ecae4b3bb0..dac36ad97a 100644 --- a/system_prompts/generate.py +++ b/system_prompts/generate.py @@ -340,13 +340,54 @@ def extract_description(section: str) -> str | None: return ''.join(_unquote_js_string(p) for p in parts).strip() or None -def parse_command_block(content: str, file_path: Path | None = None) -> dict: +def extract_named_command_block(content: str, var_name: str) -> str | None: + """Return the chained-call body of `const = new Command() ...`, + from just after `new Command()` up to the next top-level statement. + + Returns None when the var isn't a *direct* `new Command()` (e.g. it's wrapped + in a helper call like `auditListOptions(new Command()...)`), so callers can + fall back to a looser match. + """ + m = re.search( + r'const\s+' + re.escape(var_name) + r'\s*=\s*new\s+Command\(\)' + r'([\s\S]*?)(?=\n(?:const|let|var|async|function|export)\b)', + content, + ) + return m.group(1) if m else None + + +def extract_exported_command_block(content: str) -> str | None: + """Return the chained-call body of the command that is `export default`ed. + + A command file may define helper `new Command()` groups (assigned to local + consts and mounted as nested subcommands via `.command("x", localCmd)`) + *before* the exported command. Anchoring on the first `new Command()` in the + file would merge those helpers into the top-level command, so resolve the + exported variable first and only then fall back to the first `new Command()` + (which covers inline/wrapped exports). + """ + export_match = re.search(r'export\s+default\s+(\w+)\s*;', content) + if export_match: + block = extract_named_command_block(content, export_match.group(1)) + if block is not None: + return block + command_match = re.search( + r'(?:const\s+command\s*=\s*)?new\s+Command\(\)([\s\S]*?)(?=export\s+default)', + content, + ) + return command_match.group(1) if command_match else None + + +def parse_command_block( + content: str, file_path: Path | None = None, block: str | None = None +) -> dict: """ Parse a Cliffy Command() definition block and extract metadata. Returns a dict with: description, options, subcommands, arguments, alias If file_path is provided, imported subcommands will be resolved by parsing - the imported files. + the imported files. `block` may be passed to parse a specific pre-extracted + command body (used to recurse into locally-defined nested command groups). """ result = { 'description': '', @@ -357,15 +398,11 @@ def parse_command_block(content: str, file_path: Path | None = None) -> dict: } # Find the command block - command_match = re.search( - r'(?:const\s+command\s*=\s*)?new\s+Command\(\)([\s\S]*?)(?=export\s+default)', - content - ) - if not command_match: + if block is None: + block = extract_exported_command_block(content) + if block is None: return result - block = command_match.group(1) - # Find where subcommands start first_subcommand_pos = block.find('.command(') if first_subcommand_pos == -1: @@ -451,12 +488,31 @@ def parse_command_block(content: str, file_path: Path | None = None) -> dict: 'name': cmd_name, 'description': imported_cmd.get('description', ''), 'arguments': imported_cmd.get('arguments', ''), - 'options': imported_cmd.get('options', []) + 'options': imported_cmd.get('options', []), + 'subcommands': imported_cmd.get('subcommands', []), }) continue except Exception as e: print(f" Warning: Could not parse imported command {second_arg}: {e}") cmd_desc = '' + elif second_arg and re.search( + r'const\s+' + re.escape(second_arg) + r'\s*=\s*new\s+Command\(\)', content + ): + # Locally-defined command group mounted as a subcommand + # (e.g. `.command("migrate", migrateCommand)`): recurse into its + # definition so its own subcommands/options are captured. + nested_block = extract_named_command_block(content, second_arg) + if nested_block is not None: + nested = parse_command_block(content, file_path, block=nested_block) + result['subcommands'].append({ + 'name': cmd_name, + 'description': nested.get('description', ''), + 'arguments': nested.get('arguments', ''), + 'options': nested.get('options', []), + 'subcommands': nested.get('subcommands', []), + }) + continue + cmd_desc = '' else: cmd_desc = '' @@ -628,6 +684,16 @@ def generate_cli_commands_markdown(cli_data: dict) -> str: for opt in sub['options']: md += f" - `{opt['flag']}` - {opt['description']}\n" + # Nested sub-subcommands (e.g. `datatable migrate new`) + for subsub in sub.get('subcommands', []): + ss_args = f" {subsub['arguments']}" if subsub.get('arguments') else "" + md += f" - `{cmd['name']} {sub_name} {subsub['name']}{ss_args}`" + if subsub.get('description'): + md += f" - {subsub['description']}" + md += "\n" + for opt in subsub.get('options', []): + md += f" - `{opt['flag']}` - {opt['description']}\n" + md += "\n" return md @@ -689,6 +755,21 @@ def generate_ts_exports(prompts: dict[str, str]) -> str: return ts +def generate_ts_declarations(prompts: dict[str, str]) -> str: + """Generate the .d.ts for prompts.ts. + + Each export is declared as a plain `string` rather than a string-literal + type so the declaration file does not embed (and drift against) the prompt + contents — those live only in prompts.ts. + """ + dts = "// Auto-generated by generate.py - DO NOT EDIT\n\n" + + for name in prompts.keys(): + dts += f"export declare const {name}: string;\n" + + return dts + + # ============================================================================= # Schema File Generation # ============================================================================= @@ -2354,6 +2435,7 @@ def main(): flow_base = read_markdown_file(base_dir / "flow-base.md") resources_base = read_markdown_file(base_dir / "resources.md") raw_app_base = read_markdown_file(base_dir / "raw-app.md") + pipeline_base = read_markdown_file(base_dir / "pipeline-base.md") workflow_as_code_base = read_markdown_file(base_dir / "workflow-as-code.md") flow_cli = read_markdown_file(base_dir / "flow-cli.md") flow_chat_special_modules = read_markdown_file(base_dir / "flow-chat-special-modules.md") @@ -2420,6 +2502,7 @@ def main(): 'FLOW_BASE': flow_base, 'RESOURCES_BASE': resources_base, 'RAW_APP_BASE': raw_app_base, + 'PIPELINE_BASE': pipeline_base, 'WORKFLOW_AS_CODE_BASE': workflow_as_code_base, 'FLOW_CHAT_SPECIAL_MODULES': flow_chat_special_modules, @@ -2447,6 +2530,7 @@ def main(): # Generate TypeScript exports ts_exports = generate_ts_exports(prompts) (OUTPUT_GENERATED_DIR / "prompts.ts").write_text(ts_exports) + (OUTPUT_GENERATED_DIR / "prompts.d.ts").write_text(generate_ts_declarations(prompts)) # Generate complete script.md (all languages combined) script_md_parts = [script_base] @@ -2518,6 +2602,11 @@ export function getRawAppPrompt(): string { return prompts.RAW_APP_BASE; } +// Helper for data pipeline authoring (chat consumers) +export function getPipelinePrompt(): string { + return prompts.PIPELINE_BASE; +} + // Helper to get the datatable SQL SDK reference (wmill.datatable()). // Pass a language to get only that SDK; omit it to get both. export function getDatatableSdkReference(language?: string): string { @@ -2570,6 +2659,7 @@ export declare function getScriptPrompt(language: string): string; export declare function getFlowPrompt(): string; export declare function getResourcePrompt(): string; export declare function getRawAppPrompt(): string; +export declare function getPipelinePrompt(): string; export declare function getDatatableSdkReference(language?: string): string; export declare function getWorkflowAsCodePrompt(language?: string): string; """ @@ -2615,6 +2705,7 @@ export declare function getWorkflowAsCodePrompt(language?: string): string; print(f" - auto-generated/sdks/wac-python.md") print(f" - auto-generated/cli/cli-commands.md (auto-generated from CLI source)") print(f" - auto-generated/prompts.ts") + print(f" - auto-generated/prompts.d.ts") print(f" - auto-generated/index.ts") print(f" - auto-generated/script.md") print(f" - auto-generated/flow.md") diff --git a/typescript-client/build.jsr.sh b/typescript-client/build.jsr.sh index 6dd8968b8f..a8e3cdc390 100755 --- a/typescript-client/build.jsr.sh +++ b/typescript-client/build.jsr.sh @@ -15,6 +15,6 @@ cp "${script_dirpath}/sqlUtils.ts" "${script_dirpath}/src/" echo "" >> "${script_dirpath}/src/index.ts" echo 'export type { DenoS3LightClientSettings } from "./s3Types";' >> "${script_dirpath}/src/index.ts" echo "" >> "${script_dirpath}/src/index.ts" -echo 'export { type Base64, setClient, getVariable, setVariable, getResource, setResource, getResumeUrls, setState, setProgress, getProgress, getState, getIdToken, denoS3LightClientSettings, loadS3FileStream, loadS3File, writeS3File, deleteS3File, signS3Objects, signS3Object, getPresignedS3PublicUrls, getPresignedS3PublicUrl, task, runScript, runScriptAsync, runScriptByPath, runScriptByHash, runScriptByPathAsync, runScriptByHashAsync, runFlow, runFlowAsync, waitJob, getRootJobId, setFlowUserState, getFlowUserState, usernameToEmail, requestInteractiveSlackApproval, Sql, requestInteractiveTeamsApproval, appendToResultStream, streamResult, datatable, ducklake, type DatatableSqlTemplateFunction, type SqlTemplateFunction, type S3Object, type S3ObjectRecord, type S3ObjectURI } from "./client";' >> "${script_dirpath}/src/index.ts" +echo 'export { type Base64, setClient, getVariable, setVariable, getResource, setResource, getResumeUrls, setState, setProgress, getProgress, getState, getIdToken, denoS3LightClientSettings, loadS3FileStream, loadS3File, writeS3File, deleteS3File, signS3Objects, signS3Object, getPresignedS3PublicUrls, getPresignedS3PublicUrl, task, runScript, runScriptAsync, runScriptByPath, runScriptByHash, runScriptByPathAsync, runScriptByHashAsync, runFlow, runFlowAsync, waitJob, getRootJobId, setFlowUserState, getFlowUserState, usernameToEmail, requestInteractiveSlackApproval, Sql, requestInteractiveTeamsApproval, appendToResultStream, streamResult, datatable, ducklake, upsertPartition, appendPartition, type DucklakeMaterializeOptions, type SqlStatement, type DatatableSqlTemplateFunction, type SqlTemplateFunction, type S3Object, type S3ObjectRecord, type S3ObjectURI } from "./client";' >> "${script_dirpath}/src/index.ts" diff --git a/typescript-client/build.sh b/typescript-client/build.sh index beb44f448c..b3bd5b5c1c 100755 --- a/typescript-client/build.sh +++ b/typescript-client/build.sh @@ -40,7 +40,7 @@ cp "${script_dirpath}/sqlUtils.ts" "${script_dirpath}/src/" echo "" >> "${script_dirpath}/src/index.ts" echo 'export type { DenoS3LightClientSettings } from "./s3Types";' >> "${script_dirpath}/src/index.ts" echo "" >> "${script_dirpath}/src/index.ts" -echo 'export { type Base64, setClient, getVariable, setVariable, getResource, setResource, getResumeUrls, setState, setProgress, getProgress, getState, getIdToken, denoS3LightClientSettings, loadS3FileStream, loadS3File, writeS3File, deleteS3File, signS3Objects, signS3Object, getPresignedS3PublicUrls, getPresignedS3PublicUrl, task, taskScript, taskFlow, workflow, step, sleep, parallel, waitForApproval, type TaskOptions, WorkflowCtx, _workflowCtx, setWorkflowCtx, StepSuspend, runScript, runScriptAsync, runScriptByPath, runScriptByHash, runScriptByPathAsync, runScriptByHashAsync, runFlow, runFlowAsync, waitJob, getRootJobId, setFlowUserState, getFlowUserState, usernameToEmail, requestInteractiveSlackApproval, type Sql, requestInteractiveTeamsApproval, appendToResultStream, streamResult, datatable, ducklake, type DatatableSqlTemplateFunction, type SqlTemplateFunction, type S3Object, type S3ObjectRecord, type S3ObjectURI, commitKafkaOffsets } from "./client";' >> "${script_dirpath}/src/index.ts" +echo 'export { type Base64, setClient, getVariable, setVariable, getResource, setResource, getResumeUrls, setState, setProgress, getProgress, getState, getIdToken, denoS3LightClientSettings, loadS3FileStream, loadS3File, writeS3File, deleteS3File, signS3Objects, signS3Object, getPresignedS3PublicUrls, getPresignedS3PublicUrl, task, taskScript, taskFlow, workflow, step, sleep, parallel, waitForApproval, type TaskOptions, WorkflowCtx, _workflowCtx, setWorkflowCtx, StepSuspend, runScript, runScriptAsync, runScriptByPath, runScriptByHash, runScriptByPathAsync, runScriptByHashAsync, runFlow, runFlowAsync, waitJob, getRootJobId, setFlowUserState, getFlowUserState, usernameToEmail, requestInteractiveSlackApproval, type Sql, requestInteractiveTeamsApproval, appendToResultStream, streamResult, datatable, ducklake, upsertPartition, appendPartition, type DucklakeMaterializeOptions, type SqlStatement, type DatatableSqlTemplateFunction, type SqlTemplateFunction, type S3Object, type S3ObjectRecord, type S3ObjectURI, commitKafkaOffsets } from "./client";' >> "${script_dirpath}/src/index.ts" # Build default export by combining client utilities + services # This preserves backward compatibility for `import wmill from "windmill-client"` @@ -99,6 +99,8 @@ import { streamResult, datatable, ducklake, + upsertPartition, + appendPartition, SHARED_FOLDER, getWorkspace, getStatePath, @@ -185,6 +187,8 @@ const wmill = { streamResult, datatable, ducklake, + upsertPartition, + appendPartition, SHARED_FOLDER, getWorkspace, getStatePath, diff --git a/typescript-client/client.d.ts b/typescript-client/client.d.ts index e789353e27..1fa1ed1ca6 100644 --- a/typescript-client/client.d.ts +++ b/typescript-client/client.d.ts @@ -22,6 +22,10 @@ export { export { datatable, ducklake, + upsertPartition, + appendPartition, + type DucklakeMaterializeOptions, + type SqlStatement, type SqlTemplateFunction, type DatatableSqlTemplateFunction, } from "./sqlUtils"; @@ -56,7 +60,8 @@ export declare function runScript( path?: string | null, hash_?: string | null, args?: Record | null, - verbose?: boolean + verbose?: boolean, + tag?: string | null ): Promise; export declare function waitJob(jobId: string, verbose?: boolean): Promise; export declare function getResult(jobId: string): Promise; @@ -66,7 +71,8 @@ export declare function runScriptAsync( path: string | null, hash_: string | null, args: Record | null, - scheduledInSeconds?: number | null + scheduledInSeconds?: number | null, + tag?: string | null ): Promise; /** * Resolve a resource value in case the default value was picked because the input payload was undefined diff --git a/typescript-client/client.ts b/typescript-client/client.ts index f388ff6457..5d2342471b 100644 --- a/typescript-client/client.ts +++ b/typescript-client/client.ts @@ -16,10 +16,12 @@ import { OpenAPI } from "./core/OpenAPI"; import { DenoS3LightClientSettings, S3ObjectRecord, + parseS3Object, type S3Object, } from "./s3Types"; export { + parseS3Object, type S3Object, type S3ObjectRecord, type S3ObjectURI, @@ -27,8 +29,12 @@ export { export { datatable, ducklake, + upsertPartition, + appendPartition, type SqlTemplateFunction, type DatatableSqlTemplateFunction, + type DucklakeMaterializeOptions, + type SqlStatement, } from "./sqlUtils"; // Services are NOT re-exported here to enable tree-shaking @@ -71,7 +77,7 @@ function getPublicBaseUrl(): string { return getEnv("WM_BASE_URL") ?? "http://localhost:3000"; } -const getEnv = (key: string) => { +export const getEnv = (key: string) => { if (typeof window === "undefined") { // node return process?.env?.[key]; @@ -149,7 +155,8 @@ export async function runScript( path: string | null = null, hash_: string | null = null, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { console.warn( "runScript is deprecated. Use runScriptByPath or runScriptByHash instead." @@ -157,14 +164,15 @@ export async function runScript( if (path && hash_) { throw new Error("path and hash_ are mutually exclusive"); } - return _runScriptInternal(path, hash_, args, verbose); + return _runScriptInternal(path, hash_, args, verbose, tag); } async function _runScriptInternal( path: string | null = null, hash_: string | null = null, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { args = args || {}; @@ -179,7 +187,7 @@ async function _runScriptInternal( } } - const jobId = await _runScriptAsyncInternal(path, hash_, args); + const jobId = await _runScriptAsyncInternal(path, hash_, args, null, tag); return await waitJob(jobId, verbose); } @@ -188,14 +196,16 @@ async function _runScriptInternal( * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ export async function runScriptByPath( path: string, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { - return _runScriptInternal(path, null, args, verbose); + return _runScriptInternal(path, null, args, verbose, tag); } /** @@ -203,14 +213,16 @@ export async function runScriptByPath( * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ export async function runScriptByHash( hash_: string, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { - return _runScriptInternal(null, hash_, args, verbose); + return _runScriptInternal(null, hash_, args, verbose, tag); } /** @@ -236,12 +248,14 @@ export async function streamResult(stream: AsyncIterable) { * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ export async function runFlow( path: string | null = null, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { args = args || {}; @@ -249,7 +263,7 @@ export async function runFlow( console.info(`running \`${path}\` synchronously with args:`, args); } - const jobId = await runFlowAsync(path, args, null, false); + const jobId = await runFlowAsync(path, args, null, false, tag); return await waitJob(jobId, verbose); } @@ -364,7 +378,8 @@ export async function runScriptAsync( path: string | null, hash_: string | null, args: Record | null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { console.warn( "runScriptAsync is deprecated. Use runScriptByPathAsync or runScriptByHashAsync instead." @@ -373,14 +388,15 @@ export async function runScriptAsync( if (path && hash_) { throw new Error("path and hash_ are mutually exclusive"); } - return _runScriptAsyncInternal(path, hash_, args, scheduledInSeconds); + return _runScriptAsyncInternal(path, hash_, args, scheduledInSeconds, tag); } async function _runScriptAsyncInternal( path: string | null = null, hash_: string | null = null, args: Record | null = null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { // Create a script job and return its job id. args = args || {}; @@ -390,6 +406,10 @@ async function _runScriptAsyncInternal( params["scheduled_in_secs"] = scheduledInSeconds; } + if (tag) { + params["tag"] = tag; + } + let parentJobId = getEnv("WM_JOB_ID"); if (parentJobId !== undefined) { params["parent_job"] = parentJobId; @@ -427,14 +447,16 @@ async function _runScriptAsyncInternal( * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ export async function runScriptByPathAsync( path: string, args: Record | null = null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { - return _runScriptAsyncInternal(path, null, args, scheduledInSeconds); + return _runScriptAsyncInternal(path, null, args, scheduledInSeconds, tag); } /** @@ -442,14 +464,16 @@ export async function runScriptByPathAsync( * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ export async function runScriptByHashAsync( hash_: string, args: Record | null = null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { - return _runScriptAsyncInternal(null, hash_, args, scheduledInSeconds); + return _runScriptAsyncInternal(null, hash_, args, scheduledInSeconds, tag); } /** @@ -458,6 +482,7 @@ export async function runScriptByHashAsync( * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ export async function runFlowAsync( @@ -467,7 +492,8 @@ export async function runFlowAsync( // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures - doNotTrackInParent: boolean = true + doNotTrackInParent: boolean = true, + tag: string | null = null ): Promise { // Create a script job and return its job id. @@ -478,6 +504,10 @@ export async function runFlowAsync( params["scheduled_in_secs"] = scheduledInSeconds; } + if (tag) { + params["tag"] = tag; + } + if (!doNotTrackInParent) { let parentJobId = getEnv("WM_JOB_ID"); if (parentJobId !== undefined) { @@ -1477,17 +1507,6 @@ function parseResourceSyntax(s: string | undefined) { if (s?.startsWith("res://")) return s.substring(6); } -/** - * Parse an S3 object from URI string or record format - * @param s3Object - S3 object as URI string (s3://storage/key) or record - * @returns S3 object record with storage and s3 key - */ -export function parseS3Object(s3Object: S3Object): S3ObjectRecord { - if (typeof s3Object === "object") return s3Object; - const match = s3Object.match(/^s3:\/\/([^/]*)\/(.*)$/); - return { storage: match?.[1] || undefined, s3: match?.[2] ?? "" }; -} - function parseVariableSyntax(s: string) { if (s.startsWith("var://")) return s.substring(6); } diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 3f5ebe702d..280be5e53f 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.733.1", + "version": "1.753.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index a9fe521bb2..46bbda8c03 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.733.1", + "version": "1.753.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/typescript-client/s3Types.d.ts b/typescript-client/s3Types.d.ts index 4903edc8ff..57a4af2222 100644 --- a/typescript-client/s3Types.d.ts +++ b/typescript-client/s3Types.d.ts @@ -14,3 +14,4 @@ export type DenoS3LightClientSettings = { secretKey?: string; pathStyle?: boolean; }; +export declare function parseS3Object(s3Object: S3Object): S3ObjectRecord; diff --git a/typescript-client/s3Types.ts b/typescript-client/s3Types.ts index e9af9b2d30..d2049570f8 100644 --- a/typescript-client/s3Types.ts +++ b/typescript-client/s3Types.ts @@ -4,7 +4,8 @@ export type S3Object = S3ObjectURI | S3ObjectRecord; /** - * S3 object URI in the format `s3://storage/key` + * S3 object URI in the format `s3://storage/key` (`s3:///key` targets the + * workspace default storage) */ export type S3ObjectURI = `s3://${string}/${string}`; @@ -39,3 +40,26 @@ export type DenoS3LightClientSettings = { /** Use path-style URLs instead of virtual-hosted style */ pathStyle?: boolean; }; + +/** + * Parse an S3 object from URI string or record format + * @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key` + * for the default storage) or record. Any other string throws rather than + * falling back to an auto-generated key: an auto key is requested by + * omitting the object, and a fallback would silently misplace the upload + * on any typo. + * @returns S3 object record with storage and s3 key + */ +export function parseS3Object(s3Object: S3Object): S3ObjectRecord { + if (typeof s3Object === "object") return s3Object; + const match = s3Object.match(/^s3:\/\/([^/]*)\/(.+)$/); + if (match) return { storage: match[1] || undefined, s3: match[2] }; + if (s3Object.startsWith("s3://")) { + throw new Error( + `Invalid s3 object URI ${JSON.stringify(s3Object)}: expected s3:/// with a non-empty key (s3:/// for the default storage)` + ); + } + throw new Error( + `Invalid s3 object ${JSON.stringify(s3Object)}: expected an s3:/// URI (e.g. "s3:///${s3Object}" for key "${s3Object}" in the default storage) or { s3: }` + ); +} diff --git a/typescript-client/sqlUtils.d.ts b/typescript-client/sqlUtils.d.ts index 7ff4041792..c791c8833d 100644 --- a/typescript-client/sqlUtils.d.ts +++ b/typescript-client/sqlUtils.d.ts @@ -87,3 +87,18 @@ export interface DatatableSqlTemplateFunction extends SqlTemplateFunction { export declare function datatable(name: string): DatatableSqlTemplateFunction; export declare function ducklake(name: string): SqlTemplateFunction; + +export interface DucklakeMaterializeOptions { + ducklake?: string; + table: string; + selectSql: string; + partition?: string; + uniqueKey?: string; + partitionCol?: string; +} +export declare function upsertPartition( + opts: DucklakeMaterializeOptions, +): SqlStatement; +export declare function appendPartition( + opts: Omit, +): SqlStatement; diff --git a/typescript-client/sqlUtils.ts b/typescript-client/sqlUtils.ts index abe596ce03..85c563095b 100644 --- a/typescript-client/sqlUtils.ts +++ b/typescript-client/sqlUtils.ts @@ -1,4 +1,5 @@ -import { getWorkspace, workerHasInternalServer } from "./client"; +import { getEnv, getWorkspace, workerHasInternalServer } from "./client"; +import { OpenAPI } from "./core/OpenAPI"; import { JobService } from "./services.gen"; type ResultCollection = @@ -385,6 +386,185 @@ export function ducklake(name: string = "main"): SqlTemplateFunction { return buildSqlTemplateFunction(ducklakeProvider(n, schema)); } +/** Options for the ducklake materialization helpers. `partition` is bound as a + * DuckDB arg (never interpolated); `selectSql`, `table`, `schema`, `uniqueKey` + * are trusted structural SQL inlined via `raw`. */ +export interface DucklakeMaterializeOptions { + /** ducklake name (default "main"), optionally "name:schema". */ + ducklake?: string; + /** target table within the ducklake. */ + table: string; + /** the SELECT producing the rows for this slice. */ + selectSql: string; + /** the partition value (bound). Omit for a whole-table materialization — no + * partition column, and replace becomes a `CREATE OR REPLACE TABLE`. */ + partition?: string; + /** dedup key → upsert in slice (delete-by-key + insert); omit → replace (delete partition + insert). */ + uniqueKey?: string; + /** physical partition column (default "_wm_partition"). */ + partitionCol?: string; +} + +/** Idempotently materialize `selectSql` into a ducklake table for one + * partition (or the whole table when `partition` is omitted) — the client-side + * equivalent of the `// materialize` engine. + * With `uniqueKey` it upserts the slice (delete-by-key + insert); otherwise it + * replaces it (whole table → `CREATE OR REPLACE`; partition → delete + insert). + * Safe to re-run for the same partition (backfill / failure-recovery). + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.upsertPartition({ table, selectSql, partition }).execute()`. */ +export function upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement { + return finishMaterialize(buildUpsertStatement(opts), opts); +} +function buildUpsertStatement(opts: DucklakeMaterializeOptions): SqlStatement { + let { name: n, schema } = parseName(opts.ducklake ?? "main"); + let sql = buildSqlTemplateFunction(ducklakeProvider(n, schema)); + let pcol = sql.raw(opts.partitionCol ?? "_wm_partition"); + let t = sql.raw(`dl.${opts.table}`); + let body = sql.raw(opts.selectSql); + // Whole-table (no partition): no partition column. Replace rebuilds the table + // with CREATE OR REPLACE (handles schema changes); merge upserts by key. + if (opts.partition === undefined) { + if (opts.uniqueKey) { + let uk = sql.raw(opts.uniqueKey); + return sql`CREATE TABLE IF NOT EXISTS ${t} AS SELECT * FROM (${body}) WHERE false; +BEGIN TRANSACTION; +DELETE FROM ${t} WHERE ${uk} IN (SELECT ${uk} FROM (${body})); +INSERT INTO ${t} SELECT * FROM (${body}); +COMMIT;`; + } + return sql`CREATE OR REPLACE TABLE ${t} AS SELECT * FROM (${body});`; + } + if (opts.uniqueKey) { + let uk = sql.raw(opts.uniqueKey); + // Upsert via delete-by-key + insert (not MERGE — DuckLake's MERGE fails + // writing the first rows of a fresh partition). + return sql`CREATE TABLE IF NOT EXISTS ${t} AS SELECT *, CAST(NULL AS VARCHAR) AS ${pcol} FROM (${body}) WHERE false; +ALTER TABLE ${t} SET PARTITIONED BY (${pcol}); +BEGIN TRANSACTION; +DELETE FROM ${t} WHERE ${pcol} = ${opts.partition} AND ${uk} IN (SELECT ${uk} FROM (${body})); +INSERT INTO ${t} SELECT *, ${opts.partition} AS ${pcol} FROM (${body}); +COMMIT;`; + } + return sql`CREATE TABLE IF NOT EXISTS ${t} AS SELECT *, CAST(NULL AS VARCHAR) AS ${pcol} FROM (${body}) WHERE false; +ALTER TABLE ${t} SET PARTITIONED BY (${pcol}); +BEGIN TRANSACTION; +DELETE FROM ${t} WHERE ${pcol} = ${opts.partition}; +INSERT INTO ${t} SELECT *, ${opts.partition} AS ${pcol} FROM (${body}); +COMMIT;`; +} + +/** INSERT-only materialization (no dedup/replace) for append-only tables. + * Re-running the same partition duplicates rows — use only for immutable + * event-log sources. + * + * Returns a lazy statement — call `.execute()` to run it: + * `await wmill.appendPartition({ table, selectSql, partition }).execute()`. */ +export function appendPartition( + opts: Omit, +): SqlStatement { + return finishMaterialize(buildAppendStatement(opts), opts); +} +function buildAppendStatement( + opts: Omit, +): SqlStatement { + let { name: n, schema } = parseName(opts.ducklake ?? "main"); + let sql = buildSqlTemplateFunction(ducklakeProvider(n, schema)); + let pcol = sql.raw(opts.partitionCol ?? "_wm_partition"); + let t = sql.raw(`dl.${opts.table}`); + let body = sql.raw(opts.selectSql); + // Whole-table (no partition): insert into the bare table, no partition column. + if (opts.partition === undefined) { + return sql`CREATE TABLE IF NOT EXISTS ${t} AS SELECT * FROM (${body}) WHERE false; +INSERT INTO ${t} SELECT * FROM (${body});`; + } + return sql`CREATE TABLE IF NOT EXISTS ${t} AS SELECT *, CAST(NULL AS VARCHAR) AS ${pcol} FROM (${body}) WHERE false; +ALTER TABLE ${t} SET PARTITIONED BY (${pcol}); +INSERT INTO ${t} SELECT *, ${opts.partition} AS ${pcol} FROM (${body});`; +} + +// In pipeline context (WM_PIPELINE), wrap a materialize statement so a +// successful run captures the slice's row count + snapshot and records +// materialized_partition state — making SDK materializations appear in the grid +// like `// materialize` ones. Outside a pipeline it's a passthrough (no record). +function finishMaterialize( + stmt: SqlStatement, + opts: Pick, +): SqlStatement { + if (getEnv("WM_PIPELINE") !== "true") return stmt; + let { name: n, schema } = parseName(opts.ducklake ?? "main"); + let sql = buildSqlTemplateFunction(ducklakeProvider(n, schema)); + let t = sql.raw(`dl.${opts.table}`); + let pcol = opts.partitionCol ?? "_wm_partition"; + let where = + opts.partition !== undefined + ? sql.raw(`WHERE ${pcol} = '${String(opts.partition).replace(/'/g, "''")}'`) + : sql.raw(""); + let summary = sql`SELECT (SELECT count(*) FROM ${t} ${where}) AS rows, (SELECT max(snapshot_id) FROM ducklake_snapshots('dl')) AS snapshot_id`; + // Asset path mirrors the `// materialize` engine: /.
for + // an explicit schema, else /
— so the grid lookup matches and + // distinct schemas don't collide under one state key. + let assetPath = schema ? `${n}/${schema}.${opts.table}` : `${n}/${opts.table}`; + let partition = opts.partition ?? ""; + let run = async () => { + try { + await stmt.execute(); + } catch (e) { + await recordMaterialization(assetPath, partition, "failed", null, null, String(e)); + throw e; + } + let snapshot_id: number | null = null; + let row_count: number | null = null; + try { + let s: any = await summary.fetchOne(); + snapshot_id = s?.snapshot_id ?? null; + row_count = s?.rows ?? null; + } catch { + /* summary read is best-effort */ + } + await recordMaterialization(assetPath, partition, "materialized", snapshot_id, row_count, null); + }; + return { + ...stmt, + execute: (() => run()) as any, + fetch: (() => run()) as any, + fetchOne: (() => run()) as any, + fetchOneScalar: (() => run()) as any, + }; +} + +async function recordMaterialization( + assetPath: string, + partition: string, + status: string, + snapshot_id: number | null, + row_count: number | null, + error: string | null, +): Promise { + try { + await fetch(`${OpenAPI.BASE}/w/${getWorkspace()}/assets/record_materialization`, { + method: "POST", + headers: { + "content-type": "application/json", + authorization: `Bearer ${OpenAPI.TOKEN as string}`, + }, + body: JSON.stringify({ + asset_kind: "ducklake", + asset_path: assetPath, + partition, + status, + snapshot_id, + row_count, + job_id: getEnv("WM_JOB_ID") ?? null, + error, + }), + }); + } catch { + // best-effort; never fail the user's materialization + } +} + // --------------------------------------------------------------------------- // Utilities // --------------------------------------------------------------------------- diff --git a/typescript-client/tests/s3Types.test.ts b/typescript-client/tests/s3Types.test.ts new file mode 100644 index 0000000000..6f67a6493c --- /dev/null +++ b/typescript-client/tests/s3Types.test.ts @@ -0,0 +1,52 @@ +import { describe, expect, test } from "bun:test"; +import { parseS3Object } from "../s3Types"; + +describe("parseS3Object", () => { + test("bare string throws with the s3:/// hint", () => { + // A bare key is rejected rather than silently uploading under an + // auto-generated name; the error points at the s3:/// spelling. + expect(() => parseS3Object("dir/file.json" as any)).toThrow( + /s3:\/\/\/dir\/file\.json/ + ); + }); + + test("triple-slash URI targets the default storage", () => { + expect(parseS3Object("s3:///dir/file.json")).toEqual({ + storage: undefined, + s3: "dir/file.json", + }); + }); + + test("full URI splits storage and key", () => { + expect(parseS3Object("s3://bucket/dir/f")).toEqual({ + storage: "bucket", + s3: "dir/f", + }); + }); + + test("malformed s3:// URI throws", () => { + // `s3://x` has no key part — fail loudly instead of silently misplacing + // the object. + expect(() => parseS3Object("s3://broken" as any)).toThrow( + /Invalid s3 object/ + ); + }); + + test("empty-key URIs throw", () => { + // An empty key is never a valid target: it would fall back to an + // auto-generated key, which is requested by omitting the object. + expect(() => parseS3Object("s3:///" as any)).toThrow(/Invalid s3 object/); + expect(() => parseS3Object("s3://bucket/")).toThrow(/Invalid s3 object/); + }); + + test("empty string throws (omit the object for an auto-generated key)", () => { + expect(() => parseS3Object("" as any)).toThrow(/Invalid s3 object/); + }); + + test("record form passes through", () => { + expect(parseS3Object({ s3: "x", storage: "b" })).toEqual({ + s3: "x", + storage: "b", + }); + }); +}); diff --git a/version.txt b/version.txt index 71cc29b78b..ab8ba721b4 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.733.1 +1.753.0